Rucio Power User Guide

In this guide you'll learn how to configure a Rucio instance (version 38LTS) provided as SaaS. This can be done from a Rucio account with admin privileges, i.e. a Power User.

RSE Creation, distances and multi-hop

The official Rucio RSE definition states: "A Rucio Storage Element (RSE) is the logical abstraction of a storage system for physical files."

Managing data with Rucio means defining replica rules on RSE, as well as their behaviour, in a declarative fashion.

RSE creation requires admin privileges, thus this operation must be performed from a root account or from an account with the admin:True attribute.

Creating an RSE is as simple as typing:

$ rucio rse add <RSE NAME>
Added new deterministic RSE: <RSE NAME>

A newly created RSE will display the following information when asked:

[user@root-client-6fb648c88d-g8z4g ~]$ rucio rse show <RSE NAME>
Settings:
=========
  availability: 7
  availability_delete: True
  availability_read: True
  availability_write: True
  credentials: None
  delete_protocol: 1
  deterministic: True
  domain: ['lan', 'wan']
  id: 44f0394ee83246c4a1a1629c09183598
  lfn2pfn_algorithm: identity
  qos_class: None
  read_protocol: 1
  rse: <RSE NAME>
  rse_type: DISK
  sign_url: None
  staging_area: False
  third_party_copy_read_protocol: 1
  third_party_copy_write_protocol: 1
  verify_checksum: True
  volatile: False
  write_protocol: 1
Attributes:
===========
  <RSE NAME>: True
Protocols:
==========
Usage:
======
  rucio
    files: 0
    free: None
    rse: <RSE NAME>
    rse_id: 44f0394ee83246c4a1a1629c09183598
    source: rucio
    total: 0
    updated_at: 2026-05-22 21:34:00
    used: 0
RSE limits:
===========

Attributes and Protocols depend specifically on the storage. We'll see next the values for the typical INFN storage devices.

• Attributes

  define what kind of physical storage we're abstracting and which FTS endpoint to use to transfer file on it. Each attribute can be managed with the following commands:

# Add attribute
rucio rse attribute add --key <KEY> --value <VALUE> <RSE NAME>

# Delete attribute
rucio rse attribute remove --attribute <KEY> <RSE NAME>

• Protocols

  define the protocol to use to communicate with the storage. Each protocol must be set as:

# Add protocol
rucio rse protocol add --host-name <HOSTNAME> --scheme <SCHEME> --prefix <PREFIX> [OPTIONAL ARGUMENTS] <RSE NAME>

# Delete protocol
rucio rse protocol remove --scheme <SCHEME> <RSE NAME>

Finally, accounts allowed to interact with an RSE can have a quota on it.

The power user cannot create accounts, but can modify the attributes of the existing ones. The accounts are created by the users themselves by registering with the Rucio registration service at https://<vo>-registration.rucio.cloud.infn.it. By default, they are assigned a quota of 100 GB on each of the RSEs of the data lake.

The power user can set a new quota via the proper command:

# Give 1000GB quota
$ rucio account limit add --rse <RSE NAME> --bytes 1000GB <ACCOUNT NAME>
Set account limit for account <ACCOUNT_NAME> on RSE <RSE NAME>: 1000.000 GB

To see an account quota on a specific RSE:

$ rucio account limit list --rse <RSE NAME> <ACCOUNT NAME>
Quota on <RSE NAME> for <ACCOUNT NAME> : 1000.000 GB

INFN supports a number of storage "flavours": S3 disk storage (with CEPH backend), WebDAV disk storage (with STORM and dCache backends), WebDAV tape storage (with STORM backed)

S3 disk storage (CEPH backend)

The official documentation for S3 RSE can be found here.

• Attributes

  The key-value pairs of interest are:

--key QoS --value DISK
--key backend --value CEPH
--key fts --value <FTS URL endpoint>
--key s3_url_style --value path
--key sign_url --value s3
--key site --value <storage location>
--key skip_upload_stat --value True
--key strict_copy --value True
--key verify_checksum --value False

• Protocols

  Set up the https protocol as:

rucio rse protocol add \
--host-name <storage hostname> \
--scheme https \
--port <storage port> \
--prefix <dedicated path inside the storage> \
--impl rucio.rse.protocols.gfal.NoRename \
--domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' \
 <RSE NAME>

The RSE credentials must be passed to Rucio in order to access the S3 storage. This step requires pushing these credentials in the OpenBao server used by Rucio to store all its relevant secrets.

First of all, from any Linux server (not necessarily from the server or the container where you are using the Rucio client), create a JSON file rse-account.cfg with the S3 credentials:

{
  "<RSE ID>": {
    "access_key": "<S3 storage access key>",
    "secret_key": "<S3 storage secret key>",
    "signature_version": "s3v4",
    "region": "eu-south-1" # adapt as necessary
  }
}

<RSE ID> is the alphanumeric string identifying the RSE. You can obtain it with:

$ rucio rse show <RSE_NAME> | grep rse_id
rse_id: 44f0394ee83246c4a1a1629c09183598

In the case of multiple S3 storage devices, add the credentials of each storage to the JSON file, i.e.

{
  "<RSE ID 1>": {
    "access_key": "<S3 storage 1 access key>",
    "secret_key": "<S3 storage 1 secret key>",
    "signature_version": "s3v4",
    "region": "eu-south-1" # adapt as necessary
  },
  "<RSE ID 2>": {
    "access_key": "<S3 storage 2 access key>",
    "secret_key": "<S3 storage 2 secret key>",
    "signature_version": "s3v4",
    "region": "eu-south-1" # adapt as necessary
  },
  ...
}

Install OpenBao in the Linux server (e.g.sudo dnf install openbao -y in AlmaLinux) and ask the RUCIO operations team for the environment variables to set (e.g. BAO_ADDR=https://openbao-dev.cloud.infn.it, BAO_TOKEN=<vo-specific-token> and BAO_NAMESPACE=rucio/dev).

Rucio server and daemons requires the credentials to be able to connect to the S3 storage. Thus, we'll mount rse-account.cfg to the pods as a secret. We need therefore load the secret to OpenBao in the $VO path:

$ bao kv put --mount=kv-rucio-it rucio/$VO/rse-accounts rse-accounts.cfg=$(base64 -w0 -i rse-account.cfg)

Finally, we need to set the account attribute sign-gcs:True to any Rucio account allowed to perform operation on S3 storage:

# Give sign-gcs attribute
$ rucio account attribute add --key sign-gcs --value true <ACCOUNT NAME>

To see account attributes, run:

$ rucio account attribute list <ACCOUNT NAME>
+----------+---------+
|    Key   |  Value  |
|----------+---------|
| sign-gcs |  True   |
+----------+---------+

WebDAV tape or disk storage (STORM backend)

• Attributes: The key-value pairs of interest are:

--key QoS --value TAPE (or DISK)
--key backend --value STORM
--key fts --value <FTS URL endpoint>
--key site --value <storage location>

• Protocols: Set up the davs protocol as:

$ rucio rse protocol add \
--host-name <HOST URL> \
--scheme davs \
--port <STORAGE PORT> \
--prefix <PATH IN THE STORAGE> \
--impl rucio.rse.protocols.gfal.NoRename (or rucio.rse.protocols.gfal.Default in case of disk RSE) \
--domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' \
<Tape or Disk WebDAV RSE>

• Settings

  By default, any newly created RSE has rse_type:DISK among its settings. To change it:

$ rucio rse update --key rse_type --value TAPE <TAPE RSE NAME>

WebDAV disk storage (dCache backend)

• Attributes: The key-value pairs of interest are:

--rse <WebDAV RSE> --key QoS --value DISK
--rse <WebDAV RSE> --key backend --value DCACHE
--rse <WebDAV RSE> --key fts --value <FTS URL endpoint>
--rse <WebDAV RSE> --key site --value <storage location>

• Protocols Set up the davs protocol as:

$ rucio rse protocol add \
--host-name <HOST URL> \
--scheme davs \
--port <STORAGE PORT> \
--prefix <PATH IN THE STORAGE> \
--impl rucio.rse.protocols.gfal.Default \
--domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' \
<WebDAV RSE>

• User scopes

  In some cases, scopes are required to authorize users to perform actions on the WebDAV storage. Scopes are embedded in the JWT token retrieved from the IAM user client that has been set up by the RUCIO operations team. Rucio will ask for scopes defined inside the client rucio.cfg configuration file.

  • IAM client must know which scopes Rucio can ask for, e.g. storage.read, storage.modify, etc... The power user or the site manager hosting the RSE must tell to the RUCIO operations team which scopes, IAM groups or WLCG groups are authorized to read/write on the RSE.

  • Rucio client must know which scopes to ask for. These scopes must be added to the key oidc_scope in the [client] section of the Rucio client configuration file rucio.cfg, in order to have the proper authorizations enabled.

Distances

Rucio interacts with a federation of storage areas, abstracted as RSEs. When a replication rule is created, Rucio will trigger the transfer of files from a random RSE, without giving priority to any of them.

We can define in Rucio rules describing how the RSEs interact among them. Here comes the concept of distances.

Distance between two RSEs can be seen as the "priority" level between them, the lower the distance the higher the priority. If a data is replicated onto two storage RSE1 and RSE2 and we ask Rucio to create a third replica on RSE3, Rucio will trigger the TPC from the RSE with the lowest distance to RSE3.

Furthermore, by setting null distance between two RSEs, we can forbid the TPC among them. Thus, setting distances is crucial to make the datalake behave as planned.

To set a distance between two RSEs admin privileges are required:

$ rucio rse distance add --distance 1 <RSE-1> <RSE-2>
Set distance from <RSE-1> to <RSE-2> to 1
$ rucio rse distance add --distance 1 <RSE-2> <RSE-1>
Set distance from <RSE-2> to <RSE-1> to 1

Distance is defined only one-way, thus you'll have to set distance both from RSE-1 to RSE-2 and from RSE-2 to RSE-1. Distances can differ in the two "directions", allowing the creation of specific "routes" that the replicas must follow.

Multi-hop represents the best example of infrastructure with asymmetric RSE distances.

Multi-hop

Multi-hop allows to create a replica from an RSE to another by using a third RSE in between. This is useful when there are issues making two storage talk to each other, for example a STORM storage towards a S3 storage.

In fact, the transfer from S3 to STORM is simply done by FTS, the vice-versa doesn't work and we need a third RSE to mediate the transfer. Since a TPC can be performed from a STORM to dCache/WebDAV storage, we can use a dCache/WebDAV RSE to mediate the transfer. Distances need to be defined properly, i.e.:

# S3 to STORM direct transfer
$ rucio rse distance add --distance 1 <S3 RSE> <STORM RSE>

# STORM to S3 multi-hop transfer
$ rucio rse distance add --distance 1 <STORM RSE> <dCache/WebDAV RSE>
$ rucio rse distance add --distance 1 <dCache/WebDAV RSE> <S3 RSE>

# No direct trasnfer from STORM to S3
$ rucio rse distance remove <STORM RSE> <S3 RSE>

To enable multi-hop, the RSE that acts as mediator must have the available_for_multihop attribute set to True, i.e.:

rucio rse attribute add --key available_for_multihop --value True <Multi-hop RSE>

Rule approval

Rucio implements a rule approval feature, preventing the creation of new rules that fulfill a certain requirement without explicit approval of a specific user.

As an example, let's say we want that every rule on a specific RSE must be approved by the user with a given account. On this RSE, the power user can add the attribute:

rucio rse attribute add --key rule_approvers --value <ACCOUNT NAME> <RSE NAME>

Then ask the RUCIO operations team to give the account the permission to approve this kind of rules. More than one rule approver can be defined. With this attribute set, end users can add rules anywhere, except on RSE <RSE NAME> where approval (in the form of --ask-approval flag after the add-rule command) is required.

After a rule is added with the flag --ask-approval, the approver will receive an e-mail with instruction on how approve or reject the rule.