Download Methods¶
-
class
rucio.client.downloadclient.
BaseExtractionTool
(program_name, useability_check_args, extract_args, logger)¶ Bases:
object
-
is_useable
()¶ Checks if the extraction tool is installed and usable
- Returns
True if it is usable otherwise False
-
try_extraction
(archive_file_path, file_to_extract, dest_dir_path)¶ Calls the extraction program to extract a file from an archive
- Parameters
archive_file_path – path to the archive
file_to_extract – file name to extract from the archive
dest_dir_path – destination directory where the extracted file will be stored
- Returns
True on success otherwise False
-
-
class
rucio.client.downloadclient.
DownloadClient
(client=None, logger=None, tracing=True, check_admin=False, check_pcache=False)¶ Bases:
object
-
download_aria2c
(items, trace_custom_fields={}, filters={})¶ Uses aria2c to download the items with given DIDs. This function can also download datasets and wildcarded DIDs. It only can download files that are available via https/davs. Aria2c needs to be installed and X509_USER_PROXY needs to be set!
- Parameters
items – List of dictionaries. Each dictionary describing an item to download. Keys: did - DID string of this file (e.g. ‘scope:file.name’). Wildcards are not allowed rse - Optional: rse name (e.g. ‘CERN-PROD_DATADISK’) or rse expression from where to download base_dir - Optional: base directory where the downloaded files will be stored. (Default: ‘.’) no_subdir - Optional: If true, files are written directly into base_dir and existing files are overwritten. (Default: False) nrandom - Optional: if the DID addresses a dataset, nrandom files will be randomly choosen for download from the dataset ignore_checksum - Optional: If true, skips the checksum validation between the downloaded file and the rucio catalouge. (Default: False)
trace_custom_fields – Custom key value pairs to send with the traces
filters – dictionary containing filter options
- Returns
a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState
- Raises
InputValidationError – if one of the input items is in the wrong format
NoFilesDownloaded – if no files could be downloaded
NotAllFilesDownloaded – if not all files could be downloaded
RucioException – if something went wrong during the download (e.g. aria2c could not be started)
-
download_dids
(items, num_threads=2, trace_custom_fields={}, traces_copy_out=None)¶ Download items with given DIDs. This function can also download datasets and wildcarded DIDs.
- Parameters
items – List of dictionaries. Each dictionary describing an item to download. Keys: did - DID string of this file (e.g. ‘scope:file.name’) filters - Filter to select DIDs for download. Optional if DID is given rse - Optional: rse name (e.g. ‘CERN-PROD_DATADISK’) or rse expression from where to download no_resolve_archives - Optional: bool indicating whether archives should not be considered for download (Default: False) resolve_archives - Deprecated: Use no_resolve_archives instead force_scheme - Optional: force a specific scheme to download this item. (Default: None) base_dir - Optional: base directory where the downloaded files will be stored. (Default: ‘.’) no_subdir - Optional: If true, files are written directly into base_dir and existing files are overwritten. (Default: False) nrandom - Optional: if the DID addresses a dataset, nrandom files will be randomly choosen for download from the dataset ignore_checksum - Optional: If true, skips the checksum validation between the downloaded file and the rucio catalouge. (Default: False) transfer_timeout - Optional: Timeout time for the download protocols. (Default: None)
num_threads – Suggestion of number of threads to use for the download. It will be lowered if it’s too high.
trace_custom_fields – Custom key value pairs to send with the traces.
traces_copy_out – reference to an external list, where the traces should be uploaded
- Returns
a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState
- Raises
InputValidationError – if one of the input items is in the wrong format
NoFilesDownloaded – if no files could be downloaded
NotAllFilesDownloaded – if not all files could be downloaded
RucioException – if something unexpected went wrong during the download
-
download_from_metalink_file
(item, metalink_file_path, num_threads=2, trace_custom_fields={}, traces_copy_out=None)¶ Download items using a given metalink file.
- Parameters
item – dictionary describing an item to download. Keys: base_dir - Optional: base directory where the downloaded files will be stored. (Default: ‘.’) no_subdir - Optional: If true, files are written directly into base_dir and existing files are overwritten. (Default: False) ignore_checksum - Optional: If true, skips the checksum validation between the downloaded file and the rucio catalouge. (Default: False) transfer_timeout - Optional: Timeout time for the download protocols. (Default: None)
num_threads – Suggestion of number of threads to use for the download. It will be lowered if it’s too high.
trace_custom_fields – Custom key value pairs to send with the traces.
traces_copy_out – reference to an external list, where the traces should be uploaded
- Returns
a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState
- Raises
InputValidationError – if one of the input items is in the wrong format
NoFilesDownloaded – if no files could be downloaded
NotAllFilesDownloaded – if not all files could be downloaded
RucioException – if something unexpected went wrong during the download
-
download_pfns
(items, num_threads=2, trace_custom_fields={}, traces_copy_out=None)¶ Download items with a given PFN. This function can only download files, no datasets.
- Parameters
items – List of dictionaries. Each dictionary describing a file to download. Keys: pfn - PFN string of this file did - DID string of this file (e.g. ‘scope:file.name’). Wildcards are not allowed rse - rse name (e.g. ‘CERN-PROD_DATADISK’). RSE Expressions are not allowed base_dir - Optional: Base directory where the downloaded files will be stored. (Default: ‘.’) no_subdir - Optional: If true, files are written directly into base_dir and existing files are overwritten. (Default: False) adler32 - Optional: The adler32 checmsum to compare the downloaded files adler32 checksum with md5 - Optional: The md5 checksum to compare the downloaded files md5 checksum with transfer_timeout - Optional: Timeout time for the download protocols. (Default: None)
num_threads – Suggestion of number of threads to use for the download. It will be lowered if it’s too high.
trace_custom_fields – Custom key value pairs to send with the traces
traces_copy_out – reference to an external list, where the traces should be uploaded
- Returns
a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState clientState can be one of the following: ALREADY_DONE, DONE, FILE_NOT_FOUND, FAIL_VALIDATE, FAILED
- Raises
InputValidationError – if one of the input items is in the wrong format
NoFilesDownloaded – if no files could be downloaded
NotAllFilesDownloaded – if not all files could be downloaded
RucioException – if something unexpected went wrong during the download
-