Daemon rucio-conveyor-submitter

The Conveyor-Submitter daemon is responsible for managing non-tape file transfers. It prepares transfer jobs and submits them to the transfertool.

usage: rucio-conveyor-submitter [-h] [--run-once] [--total-threads TOTAL_THREADS] [--bulk BULK] [--group-bulk GROUP_BULK] [--group-policy GROUP_POLICY]
                                [--mock] [--source-strategy SOURCE_STRATEGY] [--exclude-rses EXCLUDE_RSES] [--include-rses INCLUDE_RSES]
                                [--rses RSES [RSES ...]] [--vos VOS [VOS ...]] [--activities ACTIVITIES [ACTIVITIES ...]]
                                [--exclude-activities EXCLUDE_ACTIVITIES [EXCLUDE_ACTIVITIES ...]] [--sleep-time SLEEP_TIME] [--max-sources MAX_SOURCES]
                                [--retry-other-fts]

Named Arguments

--run-once

One iteration only

Default: False

--total-threads

Concurrency control: total number of threads per process

Default: 1

--bulk

Bulk control: number of requests

Default: 100

--group-bulk

Group control: number of requests per group

Default: 1

--group-policy

Group control: policy used to group. enum{rule, dest, src_dest, rule_src_dest, activity_dest, activity_src_dest}

Default: “rule”

--mock

Fake source replicas for requests

Default: False

--source-strategy

Source strategy. Overload the strategy defined in config DB.

--exclude-rses

RSE expression to exclude

--include-rses

RSE expression to include

--rses

Explicit list of RSEs to include

--vos

Optional list of VOs to consider. Only used in multi-VO mode.

--activities

Explicit list of activities to include

--exclude-activities

Explicit list of activities to exclude

--sleep-time

Seconds to sleep if few requests

Default: 600

--max-sources

Maximum source replicas per FTS job

Default: 4

--retry-other-fts

retry on a different FTS

Default: False

Upload a file and create a replication rule:

$ rucio upload --scope mock --rse MOCK --name file filename.txt
$ rucio add-rule mock:file 1 MOCK2
$ rucio-admin rse add-distance MOCK2 MOCK --distance 1 --ranking 1

The rule should replicate the file from RSE MOCK to RSE MOCK2. Therefor a distance between these RSEs is needed.

Check transfer requests for the DID:

$ python
from rucio.db.sqla import session,models
session.get_session().query(models.Request).filter_by(scope='mock', name='file').first()
# {'request_type': TRANSFER, 'state': QUEUED', ...}

A queued request was created which can be picked uped by the Conveyor-Submiter daemon.

Run the daemon:

$ rucio-conveyor-submitter --run-once

Check again the transfer requests for the DID:

$ python
from rucio.db.sqla import session,models
session.get_session().query(models.Request).filter_by(scope='mock', name='file').first()
# {'request_type': TRANSFER, 'state': SUBMITTED', ...}

A tranfer request got created by executing the transfer. Depending on the transfer submission, the request state can be different. In this example the transfer got submitted successfully.

When run in multi-VO mode, by default the daemon will run on RSEs from all VOs:

$ rucio-conveyor-submitter --run-once
2020-07-29 13:51:09,436 5784    INFO    This instance will work on VOs: def, abc, xyz, 123
2020-07-29 13:51:13,315 5784    INFO    RSE selection: automatic for relevant VOs
2020-07-29 13:51:13,316 5784    INFO    starting submitter threads

By using the --vos argument only the VO or VOs specified will be affected:

$ rucio-conveyor-submitter --run-once --vos abc xyz
2020-07-29 13:51:09,436 5784    INFO    This instance will work on VOs: abc, xyz
2020-07-29 13:51:13,315 5784    INFO    RSE selection: automatic for relevant VOs
2020-07-29 13:51:13,316 5784    INFO    starting submitter threads

Note that attempting the use the --vos argument when in single-VO mode will have no affect:

$ rucio-conveyor-submitter --run-once --vos abc xyz
2020-07-29 13:39:37,263 5752    INFO    RSE selection: automatic
2020-07-29 13:39:37,264 5752    INFO    starting submitter threads