RSE Classes

Client

class rucio.client.rseclient.RSEClient(rucio_host=None, auth_host=None, account=None, ca_cert=None, auth_type=None, creds=None, timeout=600, user_agent='rucio-clients', vo=None)

Bases: rucio.client.baseclient.BaseClient

RSE client class for working with rucio RSEs

RSE_BASEURL = 'rses'
add_distance(source, destination, parameters)

Add a src-dest distance.

Parameters

  • source – The source.

  • destination – The destination.

  • parameters – A dictionnary with property.

add_protocol(rse, params)

Sends the request to create a new protocol for the given RSE.

Parameters

  • rse – the name of the rse.

  • scheme – identifier of this protocol

  • params – Attributes of the protocol. Supported are: hostname: hostname for this protocol (default = localhost) port: port for this protocol (default = 0) prefix: string used as a prfeix for this protocol when generating the PFN (default = None) impl: qualified name of the implementation class for this protocol (mandatory) read: integer representing the priority of this procotol for read operations (default = -1) write: integer representing the priority of this procotol for write operations (default = -1) delete: integer representing the priority of this procotol for delete operations (default = -1) extended_attributes: miscellaneous protocol specific information e.g. spacetoken for SRM (default = None)

Returns

True if protocol was created successfully else False.

Raises

  • Duplicate – if protocol with same hostname, port and protocol identifier already exists for the given RSE.

  • RSENotFound – if the RSE doesn’t exist.

  • KeyNotFound – if params is missing manadtory attributes to create the protocol.

  • AccessDenied – if not authorized.

add_qos_policy(rse, qos_policy)

Add a QoS policy from an RSE.

Parameters

  • rse_id – The id of the RSE.

  • qos_policy – The QoS policy to add.

  • session – The database session in use.

Raises

Duplicate – If the QoS policy already exists.

Returns

True if successful, except otherwise.

add_rse(rse, **kwargs)

Sends the request to create a new RSE.

Parameters

  • rse – the name of the rse.

  • deterministic – Boolean to know if the pfn is generated deterministically.

  • volatile – Boolean for RSE cache.

  • city – City for the RSE.

  • region_code – The region code for the RSE.

  • country_name – The country.

  • continent – The continent.

  • time_zone – Timezone.

  • staging_area – Staging area.

  • ISP – Internet service provider.

  • rse_type – RSE type.

  • latitude – Latitude coordinate of RSE.

  • longitude – Longitude coordinate of RSE.

  • ASN – Access service network.

  • availability – Availability.

Returns

True if location was created successfully else False.

Raises

Duplicate – if rse already exists.

add_rse_attribute(rse, key, value)

Sends the request to add a RSE attribute.

Parameters

  • rse – the name of the rse.

  • key – the attribute key.

  • value – the attribute value.

Returns

True if RSE attribute was created successfully else False.

Raises

Duplicate – if RSE attribute already exists.

delete_protocols(rse, scheme, hostname=None, port=None)

Deletes matching protocols from RSE. Protocols using the same identifier can be distinguished by hostname and port.

Parameters

  • rse – the RSE name.

  • scheme – identifier of the protocol.

  • hostname – hostname of the protocol.

  • port – port of the protocol.

Returns

True if success.

Raises
delete_qos_policy(rse, qos_policy)

Delete a QoS policy from an RSE.

Parameters

  • rse_id – The id of the RSE.

  • qos_policy – The QoS policy to delete.

  • session – The database session in use.

Returns

True if successful, silent failure if QoS policy does not exist.

delete_rse(rse)

Sends the request to delete a rse.

Parameters

rse – the name of the rse.

Returns

True if location was created successfully else False.

delete_rse_attribute(rse, key)

Sends the request to delete a RSE attribute.

Parameters

  • rse – the RSE name.

  • key – the attribute key.

Returns

True if RSE attribute was deleted successfully else False.

get_distance(source, destination)

Get distances between rses.

Parameters

  • source – The source RSE.

  • destination – The destination RSE.

Returns distance

List of dictionaries.

get_protocols(rse, protocol_domain='ALL', operation=None, default=False, scheme=None)

Returns protocol information. Parameter comibantions are: (operation OR default) XOR protocol.

Parameters

  • rse – the RSE name.

  • protocol_domain – The scope of the protocol. Supported are ‘LAN’, ‘WAN’, and ‘ALL’ (as default).

  • operation – The name of the requested operation (read, write, or delete). If None, all operations are queried.

  • default – Indicates if only the default operations should be returned.

  • scheme – The identifier of the requested protocol.

Returns

A list with details about each matching protocol.

Raises
get_rse(rse)

Returns details about the referred RSE.

Parameters

rse – Name of the referred RSE

Returns

A dict containing all attributes of the referred RSE

Raises

RSENotFound – if the referred RSE was not found in the database

get_rse_limits(rse)

Get RSE limits.

Parameters

rse – The RSE name.

Returns

True if successful, otherwise false.

get_rse_usage(rse, filters=None)

Get RSE usage information.

Parameters

  • rse – the RSE name.

  • filters – dictionary of attributes by which the results should be filtered

Returns

True if successful, otherwise false.

lfns2pfns(rse, lfns, protocol_domain='ALL', operation=None, scheme=None)

Returns PFNs that should be used at a RSE, corresponding to requested LFNs. The PFNs are generated for the RSE regardless of whether a replica exists for the LFN.

Parameters

  • rse – the RSE name

  • lfns – A list of LFN strings to translate to PFNs.

  • protocol_domain – The scope of the protocol. Supported are ‘LAN’, ‘WAN’, and ‘ALL’ (as default).

  • operation – The name of the requested operation (read, write, or delete). If None, all operations are queried.

  • scheme – The identifier of the requested protocol (gsiftp, https, davs, etc).

Returns

A dictionary of LFN / PFN pairs.

Raises
list_qos_policies(rse)

List all QoS policies of an RSE.

Parameters

  • rse_id – The id of the RSE.

  • session – The database session in use.

Returns

List containing all QoS policies.

list_rse_attributes(rse)

Sends the request to get RSE attributes.

Parameters

rse – The RSE name.

Returns

A dict with the RSE attribute name/value pairs.

list_rse_usage_history(rse, filters=None)

List RSE usage history information.

Parameters

  • rse – The RSE name.

  • filters – dictionary of attributes by which the results should be filtered.

Returns

list of dictionnaries.

list_rses(rse_expression=None)

Sends the request to list all rucio locations(RSEs).

Rse_expression

RSE Expression to use as filter.

Returns

a list containing the names of all rucio locations.

set_rse_limits(rse, name, value)

Set RSE limit information.

Parameters

  • rse – The RSE name.

  • name – The name of the limit.

  • value – The feature value. Set to -1 to remove the limit.

Returns

True if successful, otherwise false.

set_rse_usage(rse, source, used, free)

Set RSE usage information.

Parameters

  • rse – the RSE name.

  • source – the information source, e.g. srm.

  • used – the used space in bytes.

  • free – the free in bytes.

Returns

True if successful, otherwise false.

swap_protocols(rse, domain, operation, scheme_a, scheme_b)

Swaps the priorities of the provided operation.

Parameters

  • rse – the RSE name.

  • domain – the domain in which priorities should be swapped i.e. wan or lan.

  • operation – the operation that should be swapped i.e. read, write, or delete.

  • scheme_a – the scheme of one of the two protocols to be swapped, e.g. srm.

  • scheme_b – the scheme of the other of the two protocols to be swapped, e.g. http.

Returns

True if success.

Raises
update_distance(source, destination, parameters)

Update distances with the given RSE ids.

Parameters

  • source – The source.

  • destination – The destination.

  • parameters – A dictionnary with property.

update_protocols(rse, scheme, data, hostname=None, port=None)

Updates matching protocols from RSE. Protocol using the same identifier can be distinguished by hostname and port.

Parameters

  • rse – the RSE name.

  • scheme – identifier of the protocol.

  • data – A dict providing the new values of the protocol attibutes. Keys must match column names in database.

  • hostname – hostname of the protocol.

  • port – port of the protocol.

Returns

True if success.

Raises
update_rse(rse, parameters)

Update RSE properties like availability or name.

Parameters

  • rse – the name of the new rse.

  • parameters – A dictionnary with property (name, read, write, delete as keys).

Manager

rucio.rse.rsemanager.create_protocol(rse_settings, operation, scheme=None, domain='wan', auth_token=None, logger=<Logger rucio.rse.rsemanager (WARNING)>)

Instanciates the protocol defined for the given operation.

Parameters

  • rse_settings – RSE attributes

  • operation – Intended operation for this protocol

  • scheme – Optional filter if no specific protocol is defined in rse_setting for the provided operation

  • domain – Optional specification of the domain

  • auth_token – Optionally passing JSON Web Token (OIDC) string for authentication

Returns

An instance of the requested protocol

rucio.rse.rsemanager.delete(rse_settings, lfns, domain='wan', auth_token=None)

Delete a file from the connected storage. Providing a list indicates the bulk mode.

Rse_settings

RSE attributes

Parameters

  • lfns – a single dict or a list with dicts containing ‘scope’ and ‘name’. E.g. [{‘name’: ‘1_rse_remote_delete.raw’, ‘scope’: ‘user.jdoe’}, {‘name’: ‘2_rse_remote_delete.raw’, ‘scope’: ‘user.jdoe’}]

  • domain – The network domain, either ‘wan’ (default) or ‘lan’

  • auth_token – Optionally passing JSON Web Token (OIDC) string for authentication

Returns

True/False for a single file or a dict object with ‘scope:name’ as keys and True or the exception as value for each file in bulk mode

Raises
rucio.rse.rsemanager.exists(rse_settings, files, domain='wan', auth_token=None, logger=<Logger rucio.rse.rsemanager (WARNING)>)

Checks if a file is present at the connected storage. Providing a list indicates the bulk mode.

Rse_settings

RSE attributes

Parameters

  • files – a single dict or a list with dicts containing ‘scope’ and ‘name’ if LFNs are used and only ‘name’ if PFNs are used. E.g. {‘name’: ‘2_rse_remote_get.raw’, ‘scope’: ‘user.jdoe’}, {‘name’: ‘user/jdoe/5a/98/3_rse_remote_get.raw’}

  • domain – The network domain, either ‘wan’ (default) or ‘lan’

  • auth_token – Optionally passing JSON Web Token (OIDC) string for authentication

  • logger – An optional logging.Logger object

Returns

True/False for a single file or a dict object with ‘scope:name’ for LFNs or ‘name’ for PFNs as keys and True or the exception as value for each file in bulk mode

Raises

RSENotConnected – no connection to a specific storage has been established

rucio.rse.rsemanager.find_matching_scheme(rse_settings_dest, rse_settings_src, operation_src, operation_dest, domain='wan', scheme=None)

Find the best matching scheme between two RSEs

Parameters

  • rse_settings_dest – RSE settings for the destination RSE.

  • rse_settings_src – RSE settings for the src RSE.

  • operation_src – Source Operation such as read, write.

  • operation_dest – Dest Operation such as read, write.

  • domain – Domain such as lan, wan.

  • scheme – List of supported schemes.

Returns

Tuple of matching schemes (dest_scheme, src_scheme, dest_scheme_priority, src_scheme_priority).

rucio.rse.rsemanager.get_protocols_ordered(rse_settings, operation, scheme=None, domain='wan')
rucio.rse.rsemanager.get_rse_info(rse=None, vo='def', rse_id=None, session=None)

Returns all protocol related RSE attributes. Call with either rse and vo, or (in server mode) rse_id

Parameters

  • rse – Name of the requested RSE

  • vo – The VO for the RSE.

  • rse_id – The id of the rse (use in server mode to avoid db calls)

  • session – The eventual database session.

Returns

a dict object with the following attributes: id … an internal identifier rse … the name of the RSE as string type … the storage type odf the RSE e.g. DISK volatile … boolean indictaing if the RSE is volatile verify_checksum … boolean indicating whether RSE supports requests for checksums deteministic … boolean indicating of the nameing of the files follows the defined determinism domain … indictaing the domain that should be assumed for transfers. Values are ‘ALL’, ‘LAN’, or ‘WAN’ protocols … all supported protocol in form of a list of dict objects with the followig structure - scheme … protocol scheme e.g. http, srm, … - hostname … hostname of the site - prefix … path to the folder where the files are stored - port … port used for this protocol - impl … naming the python class of the protocol implementation - extended_attributes … additional information for the protocol - domains … a dict naming each domain and the priority of the protocol for each operation (lower is better, zero is not upported)

Raises

RSENotFound – if the provided RSE coud not be found in the database.

rucio.rse.rsemanager.get_space_usage(rse_settings, scheme=None, domain='wan', auth_token=None)

Get RSE space usage information.

Rse_settings

RSE attributes

Parameters

  • scheme – optional filter to select which protocol to be used.

  • domain – The network domain, either ‘wan’ (default) or ‘lan’

  • auth_token – Optionally passing JSON Web Token (OIDC) string for authentication

Returns

a list with dict containing ‘totalsize’ and ‘unusedsize’

Raises

ServiceUnavailable – if some generic error occured in the library.

rucio.rse.rsemanager.lfns2pfns(rse_settings, lfns, operation='write', scheme=None, domain='wan', auth_token=None)

Convert the lfn to a pfn

Rse_settings

RSE attributes

Parameters

  • lfns – logical file names as a dict containing ‘scope’ and ‘name’ as keys. For bulk a list of dicts can be provided

  • operation – Intended operation for this protocol

  • scheme – Optional filter if no specific protocol is defined in rse_setting for the provided operation

  • domain – Optional specification of the domain

  • auth_token – Optionally passing JSON Web Token (OIDC) string for authentication

Returns

a dict with scope:name as key and the PFN as value

rucio.rse.rsemanager.parse_pfns(rse_settings, pfns, operation='read', domain='wan', auth_token=None)

Checks if a PFN is feasible for a given RSE. If so it splits the pfn in its various components.

Rse_settings

RSE attributes

Parameters

  • pfns – list of PFNs

  • operation – Intended operation for this protocol

  • domain – Optional specification of the domain

  • auth_token – Optionally passing JSON Web Token (OIDC) string for authentication

Returns

A dict with the parts known by the selected protocol e.g. scheme, hostname, prefix, path, name

Raises
rucio.rse.rsemanager.rename(rse_settings, files, domain='wan', auth_token=None)

Rename files stored on the connected storage. Providing a list indicates the bulk mode.

Rse_settings

RSE attributes

Parameters

  • files – a single dict or a list with dicts containing ‘scope’, ‘name’, ‘new_scope’ and ‘new_name’ if LFNs are used or only ‘name’ and ‘new_name’ if PFNs are used. If ‘new_scope’ or ‘new_name’ are not provided, the current one is used. Examples: [ {‘name’: ‘3_rse_remote_rename.raw’, ‘scope’: ‘user.jdoe’, ‘new_name’: ‘3_rse_new.raw’, ‘new_scope’: ‘user.jdoe’}, {‘name’: ‘user/jdoe/d9/cb/9_rse_remote_rename.raw’, ‘new_name’: ‘user/jdoe/c6/4a/9_rse_new.raw’} ]

  • domain – The network domain, either ‘wan’ (default) or ‘lan’

  • auth_token – Optionally passing JSON Web Token (OIDC) string for authentication

Returns

True/False for a single file or a dict object with LFN (key) and True/False (value) in bulk mode

Raises
rucio.rse.rsemanager.select_protocol(rse_settings, operation, scheme=None, domain='wan')
rucio.rse.rsemanager.upload(rse_settings, lfns, domain='wan', source_dir=None, force_pfn=None, force_scheme=None, transfer_timeout=None, delete_existing=False, sign_service=None, auth_token=None, logger=<Logger rucio.rse.rsemanager (WARNING)>)

Uploads a file to the connected storage. Providing a list indicates the bulk mode.

Rse_settings

RSE attributes

Parameters

  • lfns – a single dict or a list with dicts containing ‘scope’ and ‘name’. Examples: [ {‘name’: ‘1_rse_local_put.raw’, ‘scope’: ‘user.jdoe’, ‘filesize’: 42, ‘adler32’: ‘87HS3J968JSNWID’}, {‘name’: ‘2_rse_local_put.raw’, ‘scope’: ‘user.jdoe’, ‘filesize’: 4711, ‘adler32’: ‘RSSMICETHMISBA837464F’} ] If the ‘filename’ key is present, it will be used by Rucio as the actual name of the file on disk (separate from the Rucio ‘name’).

  • domain – The network domain, either ‘wan’ (default) or ‘lan’

  • source_dir – path to the local directory including the source files

  • force_pfn – use the given PFN – can lead to dark data, use sparingly

  • force_scheme – use the given protocol scheme, overriding the protocol priority in the RSE description

  • transfer_timeout – set this timeout (in seconds) for the transfers, for protocols that support it

  • sign_service – use the given service (e.g. gcs, s3, swift) to sign the URL

  • auth_token – Optionally passing JSON Web Token (OIDC) string for authentication

  • logger – An optional logging.Logger object

Returns

True/False for a single file or a dict object with ‘scope:name’ as keys and True or the exception as value for each file in bulk mode

Raises

Storage

class rucio.rse.protocols.protocol.RSEProtocol(protocol_attr, rse_settings, logger=None)

Bases: object

This class is virtual and acts as a base to inherit new protocols from. It further provides some common functionality which applies for the amjority of the protocols.

close()

Closes the connection to RSE.

connect()

Establishes the actual connection to the referred RSE.

Raises

RSEAccessDenied – if no connection could be established.

delete(path)

Deletes a file from the connected RSE.

Parameters

path – path to the to be deleted file

Raises
exists(path)

Checks if the requested file is known by the referred RSE.

Parameters

path – Physical file name

Returns

True if the file exists, False if it doesn’t

Raises

SourceNotFound – if the source file was not found on the referred storage.

get(path, dest, transfer_timeout=None)

Provides access to files stored inside connected the RSE.

Parameters

  • path – Physical file name of requested file

  • dest – Name and path of the files when stored at the client

  • transfer_timeout – Transfer timeout (in seconds)

Raises
get_space_usage()

Get RSE space usage information.

Returns

a list with dict containing ‘totalsize’ and ‘unusedsize’

Raises

ServiceUnavailable – if some generic error occured in the library.

lfns2pfns(lfns)

Retruns a fully qualified PFN for the file referred by path.

Parameters

path – The path to the file.

Returns

Fully qualified PFN.

parse_pfns(pfns)

Splits the given PFN into the parts known by the protocol. It is also checked if the provided protocol supportes the given PFNs.

Parameters

pfns – a list of a fully qualified PFNs

Returns

dic with PFN as key and a dict with path and name as value

Raises

RSEFileNameNotSupported – if the provided PFN doesn’t match with the protocol settings

put(source, target, source_dir, transfer_timeout=None)

Allows to store files inside the referred RSE.

Parameters

  • source – path to the source file on the client file system

  • target – path to the destination file on the storage

  • source_dir – Path where the to be transferred files are stored in the local file system

  • transfer_timeout – Transfer timeout (in seconds)

Raises
rename(path, new_path)

Allows to rename a file stored inside the connected RSE.

Parameters

  • path – path to the current file on the storage

  • new_path – path to the new file on the storage

Raises
stat(path)

Returns the stats of a file.

Parameters

path – path to file

Raises
Returns

a dict with two keys, filesize and adler32 of the file provided in path.