Best practices for securely backup user data on INFN Cloud

Important

You and the system administrator of the VM are the sole responsible for the security of the data stored in the VM. The following best practices are recommendations to help you in this process.

Table of contents

Best practices for securely backup user data on INFN Cloud
- Introduction
  - Use the provided Docker solution
  - Manual setup

Introduction

This document provides a set of best practices to securely backup your VM data. The backup process is a critical part of the data management process and it is important to ensure that it is secure and reliable. In this guide, we will provide instructions to set up a custom solution for backing up any directory in a VM. The proposed solutions are primarily intended for single user environments, but can be easily adapted for multi-user environments. You can either deploy this solution directly using the provided Docker images or manually set it up by following the step-by-step instructions included.

While there are numerous tools and services available for backing up data, for the aims of this guide we will use the INFN Cloud object storage as a backend for the backup, accessed through a patched version of Rclone that supports authentication via oidc-agent, in combination with either restic or duplicity.

If you want to use those tools to backup your data, we strongly encourage you to read more in their user guides at the following links:

Use the provided Docker solution

To simplify the process of setting up the backup solution, we provide a Docker solution that includes all the necessary dependencies to run the patched rclone using the INFN Cloud object storage as a backend and spawn a REST server that can be used as repository by restic. You can check out this repository which includes both the Dockerfile, with the necessary scripts and binaries, and an example docker compose setup.

This solution will setup a REST server that can be used by restic as a repository. You can then use the included restic client to backup your data or use an external solution either via another docker service or directly from the host machine. In the following we will show how to configure and use it with both the integrated restic client and an external one.

Please note that this setup is primarily intended for on demand VMs deployed by the user, which should be system administrator. Moreover, please be sure that you have docker installed on your VM and that you have the necessary permissions to run docker commands.

If you have a different setup or you want to customize the solution to suit your needs, you can the source code available in the repository as a starting point.

Set-up the REST server

Start by downloading the docker-compose.yaml and .env files from the repository mentioned above in your VM or simply copy their content, as you will need to edit both.

First, in the docker-compose file you will see two services. The first one is 'cloud-backup' which is the integrated solution that we provide.

cloud-backup:
image: harbor.cloud.infn.it/library/cloud-user-backup:latest
container_name: cloud-backup
hostname: my-VM
working_dir: /backup
env_file:
- .env
volumes:
  - cloud-backup-data:/home/user # This volume is necessary to allow persistence of the setup
  - /local/directory/to/backup/:/backup/dir1
  - /another/local/directory/to/backup/:/backup/dir2
command: start-server
networks:
  default:
    aliases:
      - cloud-backup
# expose: # Uncomment this line if you want to expose the port to other services in the same 'backup-net' network
#   - 8080
# ports: # Uncomment this line if you want to expose the port to the host machine
#   - "8080:8080"

The image will be downloaded from harbor when launching the service, if that is not the case you can download it manually using the following command:

$ docker pull harbor.cloud.infn.it/library/cloud-user-backup:latest

Next, if you want to use the integrated restic client, you will need to change the volumes field to mount in the container the host directories you want to backup.

We suggest you to mount all under the same [/backup]{.title-ref} directory and then set this directory as the [working_dir]{.title-ref}. This is to avoid that restic sees a change in the metadata of the root backup directory each time yuo restart the container, which will be then considered as a change that needs to be backed up, even if you set the [--skip-if-unchanged]{.title-ref} flag in restic.

Also, it is important that you set the hostname of the container to a name that identify your instance, as it will be then used by restic to identify the backup. If you don't set it, restic will then use the container id as hostname which is random generated. This will cause a duplication of the snapshots each time the instance is restarted, that will not be cleaned when using the [forget]{.title-ref} command, as it will be considered as it was coming from a totally different host. The hostname can be either set in the docker-compose file for the service, using the hostname field, or only when running the backup, setting the RESTIC_HOST environment variable when scheduling the job, as it will be later explained.

If you plan to also use this container to restore the data, you should also mount the directories where you want to restore the data.

If you want to expose the REST server to external restic client on the machine, you can either use the expose field, to open the port to other services in the same docker network, or the ports field to map the container port to one on the host machine. For this use case you don't need to expose the port, so you can leave the fields commented.

Finally, you need to edit the .env file to include the necessary environment variables.

# Mandatory variables
AAI_USER=
OIDC_PASSWORD=
BACKUP_BUCKET=
RESTIC_PASSWORD=

# Optional variables

# To secure the restic server with authentication
RESTIC_REST_PASSWORD=
RESTIC_REST_USERNAME=

# To change the restic server port (default: 8080)
RESTIC_REST_PORT=

In detail, you have to define

AAI_USER: your INFN AAI username
OIDC_PASSWORD: a custom password that will be used to secure the oidc-agent profile
BACKUP_BUCKET: the path, without leading /, of the directory in your INFN Cloud personal bucket;
for example if you want to use the directory '/backup/VM1', here you need to put just 'backup/VM1';
if the directory does not exist, it will be created automatically;
RESTIC_PASSWORD: a custom password that will be used to encrypt the data in the restic repository;

Moreover, if you need to expose the REST server, you can also define the following variables:

RESTIC_REST_USERNAME and RESTIC_REST_PASSWORD: a custom user/password credentials pair that will be used to access the REST server;
RESTIC_REST_PORT: the internal container port on which the REST server will listen (default: 8080).

Once you have edited the files, you can setup both the oidc-agent and the patched rclone by running the following command:

$ docker compose run --rm cloud-backup config

Then follow the instructions on the screen to complete the setup of the service. At this point you could already start the REST server, that will be used as repository by restic. However, the OIDC token will expire after 30 days, so you need to regularly monitor the token expiration and renew it when necessary. In the provided docker image an helper script has been included to help you with this task, as it will described in the dedicated section.

Initialize the repository

If you will use the integrated helper scripts to backup your data, as described in the next section, the repository will be automatically initialized when running the backup for the first time. Otherwise you have to initialize the repository by running the following command: .. code-block:: bash

\$ docker compose run --rm cloud-backup restic init

Schedule jobs with Ofelia

To schedule and automate the necessary process, such as the monitoring of the OIDC refresh token or the backup process itself, a second service is used in the compose file: ofelia, a cron-like job scheduler for Docker containers.

ofelia:
    image: mcuadros/ofelia:latest
    container_name: ofelia
    depends_on:
    - cloud-backup
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - ./ofelia/logs/:/var/log/ofelia
    - ./ofelia-config.ini:/etc/ofelia/config.ini

This service must be configured to depend on the cloud-backup service, so that it can run jobs on it. You will also need to mount:

the docker socket, so that ofelia can run jobs on the cloud-backup container;
the ofelia configuration file, which defines the jobs to run;
a directory to store the logs of the jobs, if needed.

You can find an example configuration file in the repository, but you can also create your own by following the instructions in the ofelia documentation.

In this example four different jobs are scheduled:

check if the OIDC token is still valid, every day at 1:00 AM;
backup of the desired directories, every day at 2:00 AM;
clean up of older backups from the repository, every day at 3:00 AM;
removal of ofelia logs older than 1 month, every day at 4:00 AM.

The jobs are spread over different times to avoid potential conflicts, especially between the backup and the clean up jobs.

[global]
save-folder = var/log/ofelia

[job-exec "check-refresh-token"]
schedule = 0 0 1 * * *
container = cloud-backup
user= user
environment = OIDC_TOKEN_CHECK_URL=https://hc.cloud.infn.it/ping/<uuid>
command = check-refresh-token

[job-exec "backup"]
container= cloud-backup
schedule= 0 0 2 * * *
user= user
environment = BACKUP_CHECK_URL=https://hc.cloud.infn.it/ping/<uuid>
environment = RESTIC_HOST=my-VM
command= backup --tag example /backup/dir1

[job-exec "forget"]
container= cloud-backup
schedule= @every 1d 03:00
user= user
environment = FORGET_CHECK_URL=https://hc.cloud.infn.it/ping/<uuid>
environment = RESTIC_HOST=my-VM
command= forget --keep-within 1m --prune

[job-local "clean-logs"]
schedule = 0 0 4 * * *
command = find /var/log/ofelia/ -type f -mtime +30 -print -delete

For each execution of the jobs, Ofelia will create three log files: one containing the instructions to run the job, one containing the stderr output and one containing the stdout output. However, this requires the user to constantly check on them or use an automated system to parse the log files and check for errors.

For this reason, in the docker image are also included various helper scripts that will help you to monitor the execution of the jobs using the INFN Cloud Healthchecks service. A guide on how to use this service can be found in the dedicated page of the user guide. We suggest you to create a new check for each job you want to monitor, to immediately see which job has failed and receive an instant alert via the preferred notification channel. You will then need to substitute the <uuid> in the environment variable with the UUID of your check.

This is critical not only for the backup procedure but also to regularly check if the OIDC token is still valid, as it has to be manually renewed every 30 days.

The relevant commands that are included in the docker image are:

check-refresh-token: checks if the OIDC token is still valid and how long it will be valid for, then sends a request to the provided check URL, attaching the output of the command as the body of the request;
backup: is a wrapper for the restic backup command and it will pass any additional argument after the 'backup' command to it;
it will then assert if the backup or successful or not and send a request to the provided check URL, attaching the result of the command as the body of the request;
if the backup does not succeed because the repository is not initialized, it will initialize the repository and retry the backup;
if the backup fails for any other reason, the backup will stop and the specific error message will be sent to Healthchecks;
forget: is a wrapper for the restic forget command and it will pass any additional argument after the 'forget' command to it;
as the backup command, it will report the result of the execution to Healthchecks.

All these commands will also print the full output in the Ofelia logs, so you can check those if more details are needed.

If you need or want to use another service to monitor the jobs, you can simply modify these scripts and rebuild a custom version of the docker image. Otherwise, if you do not need or want to use any external monitor service at all, you can substitute the commands in the ofelia configuration file with the relevant restic commands.

We strongly suggest you to read the restic documentation to understand how to use it properly to securely and efficiently backup your data.

Start the solution

Now that everything is properly configured, you can start the solution by running the following command:

$ docker compose up -d

This will start both the cloud-backup and ofelia services in detached mode. If everything is working correctly, the scheduled jobs will start running at the specified times and you will receive notifications from Healthchecks.

Renew the OIDC refresh token

As mentioned before, the OIDC token will expire after 30 days, so you need to manually renew it regularly. When the expiration date is one week away, the check-refresh-token job will set the corresponding check to "down" and send in the body of the request the remaining days.

If the refresh token will expire all the scheduled jobs will stop working. Before that happens, you will need to renew it by running the following command:

$ docker compose run --rm cloud-backup reauth

Then follow the instructions on the screen to complete the authentication.

External restic client

If you don't want to use the integrated restic client or you need a more flexible setup, you can use an external restic client to backup your data. In this case you have multiple choices, as there are a plethora of clients available, all based on restic but different goals and features. For example, you can use:

official restic client installed in the host machine;
official restic docker image;
resticprofile, a configuration profiles manager for restic, which allows more complex configurations;
rustic, an alternative restic client written in Rust, available also as a docker image;
backrest, a simple restic wrapper with a user-friendly web interface.

These are just a few examples, we suggest you to read their respective documentation if you want to use them. Depending on the client you choose, you will simply need to edit the cloud-backup service in the docker-compose.yml file to expose the REST server port either to the host machine or to other services in the same docker network.

Then you should edit ofelia configuration file accordingly, removing unnecessary jobs. If you plan to use a client provided via a docker image, you can still rely on Ofelia to schedule the jobs. In any case, if you do not have an alternative in place, you should still keep Ofelia and at least the check-refresh-token job as this is the only way to monitor the OIDC token expiration without manually checking the logs.

Moreover, if you plan to use a client provided via a docker image, please note that the directories that need to be backed up should be mounted in this container, not in the cloud-backup container. When using the official restic docker image, you should also run the service with [working_dir]{.title-ref} set to the root [/backup]{.title-ref} directory, where you mounted the directories to backup, as explained above. The same issue should not be present when using rustic or backrest, but we suggest you to check the documentation of the client you choose to see if there are any specific requirements.

Manual setup

If you prefer to manually set up everything or you would like to use a different backup tool than restic, you can follow the instructions below.

:::: important ::: title Important ::: ::::

In this section we will assume that you are a system administrator with root access in the VM where you have to run the backup software, as some packages needs to be installed system-wide. If you are not the system administrator, you can ask for help to the system administrator of the VM.

Configuring oidc-agent

Using Rclone as a backend for this guide allows to abstract the backup process from the specific storage solution chosen, making it easier to switch between different providers. In this document we will use as an example the INFN Cloud object storage (which has a quota of 200 GB for each user), but the same procedure can be adapted to any other storage provider supported by Rclone. If you already have a storage provider for your experiment data, you can skip this section and configure Rclone accordingly your use case.

In order to use Rclone with the INFN Cloud object storage, we must use a patched version of Rclone that supports authentication via oidc-agent, a set of tools designed to manage OpenID Connect tokens and make them easily usable from the command line.

To install and configure oidc-agent you can follow the instructions in the oidc-agent documentation. If your VM is running AlmaLinux9, you can already find the updated oidc-agent in the EPEL repositories and install it with the following command:

$ sudo dnf install oidc-agent-cli

If you are using Debian or Ubuntu, as described in the official documentation, you need to configure an additional repository that can be found here https://repo.data.kit.edu/

After installing oidc-agent, you need to export and load the environment variables by executing the following commands:

$ eval `oidc-agent-service use

To make it persistent, you can add the command to your .bashrc file, running the following while in your home directory:

$ echo 'eval `oidc-agent-service use`' >> .bashrc

With oidc-agent version >= 5, the aud mode must be explicitly enable by creating the file [\~/.oidc-agent/issuer.config.d/infn-cloud]{.title-ref} with the following content:

{
"issuer": "https://iam.cloud.infn.it/",
"register": "https://iam.cloud.infn.it/manage/dev/dynreg",
"legacy_aud_mode": true
}

We can now configure oidc-agent to work with the INFN Cloud IAM in order to issue the necessary tokens to access the object storage. To do this, we need to create a new profile in oidc-agent by executing the following commands:

$ export OIDC_PASSWORD=changeme
$ oidc-gen --flow="device" --dae https://iam.cloud.infn.it/devicecode --pw-env=OIDC_PASSWORD --issuer="https://iam.cloud.infn.it" --scope="openid profile offline_access email" infncloud

The OIDC_PASSWORD environment variable is used to secure the profile, you can choose it as you prefer.

:::: important ::: title Important ::: ::::

Please remember that executing the above command interactively in a terminal will store your password in the bash history, to avoid this you should define the environment variable in a separate file and source it or use a password file with the --pw-file option.

To complete the process the software will ask you to open a link to the INFN Cloud IAM in your browser, login with your account (if needed) and enter the code provided (in the following is redacted with XXXXX for security reasons) to approve the registered client. If your terminal view is too short it may be hidden by the QR code appended at the end of the output, you can simply scroll up to see the code.

Registering Client ...
Generating account configuration ...
accepted

Using a browser on any device, visit:
https://iam.cloud.infn.it/device

And enter the code: XXXXXX

The configuration of oidc-agent should be now complete, you can check the profile by executing the following command:

$ oidc-token infncloud

If the command returns a token, the configuration is correct and you can proceed to configure Rclone. Instead, if the command returns an error, please check the previous steps and try again.

Configuring Rclone with INFN Cloud object storage

To use Rclone with the INFN Cloud object storage, we need to download a patched version of Rclone that supports authentication via the token issued by oidc-token.

You can find the source code in the following github: https://github.com/DODAS-TS/rclone/ or you can download the compiled binary in your home directory, using the following commands:

$ wget https://repo.cloud.cnaf.infn.it/repository/rclone/rclone-linux/2.0.0/rclone-linux-2.0.0 -O /home/$USER/rclone
$ chmod +x /home/$USER/rclone

You will now need to define the Rclone configuration file, you can do this by creating the configuration file [/home/\$USER/.config/rclone/rclone.config]{.title-ref} with the following content:

[infncloud]
type = s3
provider = INFN Cloud
account = infncloud
oidc_agent = True
endpoint = https://rgw.cloud.infn.it/
role_name = IAMaccess
audience = object
env_auth = false

You can now test the configuration by executing the following command, where <username> should be replaced with your INFN AAI username, which is also the name of your bucket in the object storage:

$ ./rclone ls infncloud:/<username>

If everything is working correctly, the command should print your bucket content.

Both restic and duplicity needs the rclone binary to be in the PATH. If you don't have the official version of Rclone installed, and don't plan to use it, you can simply copy the binary in a directory already in PATH, usually [/user/bin]{.title-ref}.

$ sudo cp /home/$USER/rclone /usr/bin/rclone

Otherwise, in order to use the patched version of Rclone only with restic/duplicity and still keep the compatibility with the official Rclone client, you need to define an alias before running restic/duplicity by issuing the following command:

$ alias rclone=/home/$USER/rclone

Please remember to unset the alias when done. This can be easily achieved by inserting all the commands in a bash script.

Configure and run Restic

Restic is a fast, secure, and efficient backup program that supports deduplication, encryption, and data integrity verification. It is designed to be easy to use and to work well with cloud storage providers. In this section we will show how to configure Restic to use Rclone as a backend and INFN Cloud as storage provider. Thus, we will assume that you have already successfully configured Rclone as described in the previous section or with any other storage provider of your choice. If you are using a different storage provider, you will need to modify the Rclone path accordingly in the following commands.

We will also assume that

To install Restic, we suggest you to use the package manager of your distribution of choice. For Ubuntu, you can use the following commands to download and install Restic:

$ sudo apt update
$ sudo apt install restic

For other distributions, you can find more detailed installation instructions at https://restic.readthedocs.io/en/stable/020_installation.html.

We will now assume that you can successfully run the 'restic' command in your terminal.

To initialize a new repository with Restic, you can use the following command:

$ restic -r rclone:infncloud:/<username>/<bucket-name> init

Where <username> is your INFN AAI username and <bucket-name> is the name of the bucket in the object storage. This will create a new repository in the specified bucket. You will be prompted to enter a password for the repository. Make sure to remember this password, as you will need it to access the repository in the future.

To create a backup with Restic, you can use the following command:

$ restic -r rclone:infncloud:/<username>/<bucket-name> backup /path/to/backup

Where [/path/to/backup]{.title-ref} is the local directory you want to backup. You will be prompted to enter the repository password and then Restic will create a backup of the specified directory in the repository.

To list the snapshots in the repository, you can use the following command:

$ restic -r rclone:infncloud:/<username>/<bucket-name> snapshots

To restore a backup, you can use the following command:

$ restic -r rclone:infncloud:/<username> restore <snapshot_id> --target /path/where/to/restore

Where [<snapshot_id>]{.title-ref} is the ID of the snapshot you want to restore, and [/path/where/to/restore]{.title-ref} is the directory where you want to restore the backup. Restic will then restore the specified snapshot to the specified directory.

For more information on the available commands and options, you can consult the Restic documentation at https://restic.readthedocs.io/.

Configure and run Duplicity

Duplicity is a software suite that provides encrypted, digitally signed, versioned, local or remote backup of files. On the first run it performs a complete (full) backup, then any subsequent (incremental) backups only add differences from the latest version of the backup. Duplicity can use GPG to encrypt the backups and supports a wide range of backends, including local filesystem, SSH, FTP, Amazon S3, Google Drive, Microsoft OneDrive and Rclone. The latter is a command line program that manages files hosted on cloud storages such as Amazon S3, Google Drive, Dropbox, and many others.

In this section we will show how to configure Duplicity to use Rclone as a backend and INFN Cloud as storage provider. Thus, we will assume that you have already successfully configured Rclone as described in the previous section or with any other storage provider of your choice. If you are using a different storage provider, you will need to modify the Rclone path accordingly in the following commands.

There are multiple ways to install Duplicity, depending on the distribution you are using. For Ubuntu, you can use the official PPA at https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git and install it with the following commands:

$ sudo add-apt-repository ppa:duplicity-team/duplicity-release-git
$ sudo apt update
$ sudo apt install duplicity

For other distributions, you can find more detailed installation instructions at https://duplicity.gitlab.io/ in the "Download" section. In summary, you can either download the official release from github and install it manually, or use the pip3 package manager:

$ pip3 install duplicity

We will now assume that you can successfully run the 'duplicity' command in your terminal.

To execute a backup with Duplicity, using Rclone as a backend you can simply run the following command:

$ duplicity /path/to/backup rclone://infncloud:/<username>

Where /path/to/backup is the path of the directory you want to put under backup, <username> is your INFN AAI username (or the name of the bucket in the object storage). The command will create a full backup of the directory on the first run, and then only the differences on subsequent runs. Duplicity has also the ability to encrypted and sign the backup with a GPG key, you just need to provide the id of public key. You can create a new GPG key by running the following command:

$ gpg --gen-key

and follow the on screen prompts. After the key is generated, you can list the keys with the following command:

$ gpg --list-keys --keyid-format LONG

The output will look like this:

/home/$USER/.gnupg/pubring.kbx
----------------------------------
pub   rsa3072/PUBLIC_KEY 2024-03-15 [SC] [expires: 2026-03-15]
uid                 [ultimate] Name <email address>
ub   rsa3072/PRIVATE_KEY 2024-03-15 [E] [expires: 2026-03-15]

The id we need is the one reported here as PUBLIC_KEY. You can now use this key to encrypt and sign the backup by defining adding the proper option to the duplicity command:

$ PASSPHRASE=<your GPG key passphrase> duplicity --encrypt-sign-key=PUBLIC_KEY /path/to/backup rclone://infncloud:/<username>

Notice that the PASSPHRASE environment variable is used to provide the passphrase of the GPG key used to encrypt and sign the backup. If you don't want to provide the passphrase in the command line, you can define it in a separate file and source it before running the command.

If you want to restore the backup, you can use the following command:

$ PASSPHRASE=<your GPG key passphrase> duplicity rclone://infncloud:/<username>

This will restore the latest version of the backup to the current directory. If you want to restore a specific version of the backup, you can add the --time option followed by the date of the backup you want to restore:

$ PASSPHRASE=<your GPG key passphrase> duplicity --time 1D rclone://infncloud:/<username>

This will restore the backup as it was 1 day ago. You can also use other time formats, such as 1W for 1 week ago, 1M for 1 month ago, etc. You can also use the --force option to force the restore, even if the current directory is not empty. For more information on the available options, you can consult the Duplicity documentation at https://duplicity.gitlab.io/.

Bash script to automate the backup process

To automate the backup process, you can create a bash script that runs the Duplicity command and then schedule it with cron. Create a new file, we will call it backup.sh in this example, in your home directory with the following content:

#!/bin/bash

export OIDC_PASSWORD=changeme
eval `oidc-agent-service use
oidc-add --pw-env=OIDC_PASSWORD infncloud

alias rclone='/home/$USER/rclone_linux'

DIR_TO_BACKUP=/home/$USER/
DEST_BUCKET=infncloud:/username/

GPG_KEY="YOUR_PUBLIC_KEY"
PASSPHRASE="YOUR_GPG_PASSPHRASE"

duplicity --verbosity info --encrypt-sign-key=$GPG_KEY --log-file /home/$USER/.duplicity/info.log --full-if-older-than=1M ${DIR_TO_BACKUP} rclone://${DEST_BUCKET}
duplicity --encrypt-sign-key=$GPG_KEY --force remove-all-but-n-full 3 rclone://${DEST_BUCKET}

unset OIDC_PASSWORD
unalias rclone

:::: important ::: title Important :::

Please remember to edit the script, using your own values for the defined variables, before running it. ::::

In this script we define the directory to backup, the destination bucket, the GPG key to use for encryption and signing, and the passphrase of the GPG key. The script will run the Duplicity command to create a full backup of the directory on first run and every time the last backup is older than one month and then remove all but the last 3 full backups.

You can make the script executable by running the following command:

$ chmod +x /home/$USER/backup.sh

To schedule the backup, you can use the cron daemon. You can edit the crontab file by running the following command:

$ crontab -e

This will open the crontab file in your default text editor. You can then add a new line with the following content:

0 4 * * * /home/$USER/backup.sh

This will run the backup script every day at 4:00 AM. You can customize the schedule by changing the values in the first five fields. For more information on the crontab syntax, you can consult the cron documentation.