Access the INFN Cloud Object Storage from a scientific environment

Introduction

INFN Cloud Object Storage data can be accessed and managed via command line or mounted directly as a folder or synchronised on your computer or on a cloud environment. This can be achieved by using a customised version of Rclone and the oidc-agent software.

Important

this solution is not suitable for storing or reading data from highly I/O demanding applications, such as analysis or montecarlo programs that continuously read or write to file, data base management systems, etc… Every operation on data would be immediately committed to the remote storage system, generating network traffic that may harm the centralised object storage service itself. Use this solution with caution.

Installing Rclone

Go to the releases page and download the most recent version for your platform.

Please find below the instructions to download and test Rclone for linux:

#download rclone

wget https://repo.cloud.cnaf.infn.it/repository/rclone/rclone-linux/2.0.0/rclone-linux-2.0.0 -O rclone_linux

#make it executable
chmod +x rclone_linux

#test if it is executable
./rclone_linux --help

Installing oidc-agent

Follow the instructions listed in the official installation guide.

Configuration

Important

Please note that with oidc-agent >= 5 aud mode must be specified in the configuration. It can be configured with by creating the file $HOME/.config/oidc-agent/issuer.config.d/infn-cloud with the following content.
{
"issuer": "https://iam.cloud.infn.it/",
"register": "https://iam.cloud.infn.it/manage/dev/dynreg",
"legacy_aud_mode": true
}

oidc client

First of all, it is necessary to configure an OpenID Connect (oidc) client on each machine you want to access the object storage from using Rclone.

You can follow the steps below to register a client named “oidc_client”:

# the following command has to be executed once in every shell, can be put
# in the ~/.bashrc file
eval `oidc-agent-service use`

# the following command allows to register a new client named "oidc_client";
# the client name can be freely changed, just make sure to change its name
# accordingly in next steps; you need to run this command only once
oidc-gen -w device --iss=https://iam.cloud.infn.it/ \
--scope="openid profile offline_access" oidc_client

# the following command has to be executed at every restart of the
# oidc-agent; spurious executions are not an error
oidc-add oidc_client

The oidc-gen command will ask you to log into the INFN-Cloud IAM (https://iam.cloud.infn.it/) and authorise the client creation and the use of scopes.

Rclone

Rclone can be configured creating the file $HOME/.config/rclone/rclone.conf with the following content:

[rgw-backbone]
type = s3
provider = INFN Cloud
account = <oidc_client_name>
oidc_agent = True
endpoint = https://rgw.cloud.infn.it/
role_name = IAMaccess
audience = object
env_auth = false

Usage

Interact with the buckets

Rclone supports a number of storage management operations. Refer to the output of the ./rclone_linux --help command for a detailed list of commands.

A non-comprehensive list of examples is reported below:

./rclone_linux ls rgw-backbone:/<object>     # list the specified object, bucket
                                    # file, or folder
./rclone_linux cat rgw-backbone:/<object>    # dump the content of the specified
                                    # object to stdout
./rclone_linux copy <source> <dest>   # copy the "source" into destination
                                    # "dest". One of the two can be a
                                    # "rgw-backbone:/<object>"
./rclone_linux delete rgw-backbone:/<object> # delete the specified object

Mount as a filesystem

You can use the mount rclone command to mount INFN-Cloud buckets as remote filesystems.

mkdir -p ~/infncloud
./rclone_linux mount rgw-backbone:/cpellegr ~/infncloud/ --daemon

To un-mount the filesystem use the fusermount command:

fusermount -u ~/infncloud