Best practices for securely backing up user data on INFN Cloud

Introduction

The backup process is a critical part of the data management process and it is important to ensure that it is secure and reliable.

While there are numerous tools and services available for backing up data, in this guide, we will focus on how to set up a custom solution for backing up any directory in your VM to the INFN Cloud object storage. To this aim, please be sure to have already created your personal bucket in the object storage, as described in the INFN Cloud object storage service user guide.

The instructions provided in this guide can be adapted to other cloud storage providers supported by the backup tools that we will use.

The custom solution we propose is composed of multiple components that work together to securely access the INFN Cloud object storage and perform the backup process:

• oidc-agent, a tool to obtain OIDC tokens from the INFN Cloud IAM;
• a helper script, provided by INFN Cloud, that uses the OIDC token to obtain temporary S3 credentials to access the object storage;
• a backup tool compatible with S3 object storage.

You can either deploy this solution directly on your VM or use the containerized version.

Docker-based Setup

To simplify the process of setting up the backup solution, we provide a Dockerized solution that includes all the necessary dependencies to run the backup software. An example of a docker-compose.yaml file that sets up the necessary services to perform the backup using Restic (more details on the tool will be provided in the manual setup section) is provided in this repository.

Configuration steps

To set up the Docker solution, follow these preliminary steps:

• Create a directory to store all the necessary files for the setup, e.g. backup-setup, and navigate into it.
• Download the docker-compose.yaml file from the provided repository;
• Create a file named oidc-pass containing a secure password that will be used to protect the oidc-agent profile;
• Create an empty file named access-token that will be used to share the generated access token between services;
• Create a file named restic_password containing a secure password that will be used to encrypt the Restic repository;
• Create a directory named oidc-data that will be used to store the oidc-agent configuration and tokens;
• Create a file named aws_cred that will be used to store the AWS credentials. This file will be automatically populated by the cloud-s3-cred service;
• Create a directory named ofelia with a sub-directory logs to store the logs of the scheduled jobs.

mkdir backup-setup && cd backup-setup
curl -fsSL -o docker-compose.yaml \
    https://baltig.infn.it/infn-cloud/user-support-scripts/-/raw/main/backup/docker-compose.yaml

# Create password files (you will be prompted to enter passwords)
read -rsp "Please enter a secure password for the OIDC agent profile: " oidc_pass && \
    echo && printf '%s\n' "$oidc_pass" > oidc-pass && unset oidc_pass
read -rsp "Please enter a secure password for the Restic repository encryption: " restic_pass && \
    echo && printf '%s\n' "$restic_pass" > restic_password && unset restic_pass

# Create empty files
: > access-token
: > aws_cred

# Create directories
mkdir -p oidc-data ofelia/logs

# Restrict sensitive file permissions
chmod 600 oidc-pass restic_password aws_cred access-token
chmod 700 oidc-data

The name of the directories and files can be changed as needed, but make sure to update the docker-compose.yaml file accordingly. Make sure to set the proper permissions to avoid other users reading its content.

Next you should edit the docker-compose.yaml file to reflect your specific setup. In the following we will break down each service and explain the necessary configuration.

oidc-agent

The oidc-agent service is responsible for managing OpenID Connect (OIDC) authentication with the INFN Cloud IAM. This service obtains and maintains access tokens that are used by cloud-s3-cred service to authenticate with the cloud infrastructure.

What it does:

• Handles OIDC token generation and renewal from the INFN Cloud IAM
• Stores the OIDC profile configuration securely
• Makes tokens available to other services through a shared file

Volumes:

• ./oidc-data/:/root/.oidc-agent - Persists OIDC agent configuration and profile data
• ./oidc-pass:/root/.oidc-password - Contains the password to unlock the OIDC profile
• ./access-token:/root/access-token - Shares the generated access token with other services

Environment Variables:

• OIDC_PASSWORD_FILE=/root/.oidc-password - Path to the file containing the OIDC profile password
• OIDC_OUTPUT_FILE=/root/access-token - Path where the access token will be written

cloud-s3-cred

The cloud-s3-cred service exchanges OIDC tokens for temporary S3 credentials. It communicates with the INFN Cloud STS (Security Token Service) to generate AWS-compatible credentials that allow access to the object storage.

What it does:

• Reads the OIDC access token generated by oidc-agent
• Calls the INFN Cloud STS endpoint to exchange the token for temporary AWS credentials
• Writes the S3 credentials to a file that other services can use

Volumes:

• ./aws_cred:/root/.aws/credentials:rw - Stores the temporary S3 credentials
• ./access-token:/root/access-token:ro - Reads the access token from oidc-agent

Environment Variables:

• OIDC_TOKEN_FILE=/root/access-token - Path to the access token file to read

backup-dir0

The backup-dir0 service performs the actual backup of your data using Restic. You can create multiple instances of this service (backup-dir1, backup-dir2, etc...) to backup different directories.

What it does:

• Reads data from the mounted source directory
• Uses S3 credentials from cloud-s3-cred to authenticate with object storage
• Creates encrypted backups in the INFN Cloud object storage
• Supports incremental backups with deduplication

Volumes:

• ./aws_cred:/root/.aws/credentials:ro - Reads the S3 credentials (generated by cloud-s3-cred)
• ./restic_password:/root/restic_password:ro - Reads the Restic repository encryption password
• ./source/to/backup:/backup0:ro - The directory you want to backup (mounted read-only inside the container)

Environment Variables:

• RESTIC_REPOSITORY=s3:https://rgw.cloud.infn.it/<bucket_name>/<bucket_path>
• The S3 path where backups will be stored
• RESTIC_PASSWORD_FILE=/root/restic_password - Path to the file containing the Restic repository password
• RESTIC_HOST=<custom_name> - A unique identifier for this backup host (used to organize snapshots)

Parameters to Change:

• ./source/to/backup - Change to the actual path on your host machine that you want to backup
• /backup0 - Can be renamed if you prefer a different mount point inside the container
• <bucket_name> - Your INFN AAI username (your personal bucket name)
• <bucket_path> - The path within your bucket where backups should be stored (e.g., backup/vm1)
• <custom_name> - A descriptive name for this backup instance (e.g., the VM hostname)

ofelia

The ofelia service is a scheduler that automates the execution of backup and maintenance tasks on a defined schedule. It monitors Docker containers and runs commands at specified times.

What it does:

• Schedules periodic execution of backup jobs
• Refreshes S3 credentials and OIDC tokens before they expire (default expiry is 1 hour)
• Cleans up old backup snapshots to save storage space
• Maintains logs of all job executions
• Allows monitoring through webhook (INFN Cloud Healthchecks service can be used)

Initialize the setup

Once you have configured the docker-compose.yaml file and created the necessary files and directories, you should create the oidc-agent profile by running the following command:

docker compose run --rm oidc-agent config

This command will prompt you to authenticate with the INFN Cloud IAM, create the necessary OIDC profile and generate the initial access token.

Generate the S3 credentials by running:

docker compose run --rm cloud-s3-cred

At this point you should be ready to initialize the Restic repository. You can do this by running the following command:

docker compose run --rm backup-dir0 init

This command will create a new Restic repository in the specified S3 path, encrypted with the password you provided in the restic_password file.

You are now ready to start the automated backup process with the command:

docker compose up -d

Restore a backup

To restore data from a Restic backup you need:

• access to the repository: the repository path (RESTIC_REPOSITORY) and valid S3 credentials. The credentials are refreshed automatically by the scheduled jobs; if they are not running, generate them with docker compose run --rm cloud-s3-cred;
• the repository password stored in the restic_password file. Without it the backup cannot be decrypted;
• a local directory where the restored files will be written.

If the original VM is lost, set up the Docker solution on a new VM as described above, using the same restic_password, RESTIC_REPOSITORY and RESTIC_HOST values of the original setup. Run the oidc-agent config and cloud-s3-cred commands, but do not run init, since the repository already exists.

The restore is then done in four steps.

1. Find the snapshot you want to restore. Each backup run creates a snapshot, identified by an ID and the time it was taken:

docker compose run --rm backup-dir0 snapshots

Snapshots are filtered by the RESTIC_HOST value, which is why it must match the one used to create them.

2. Look inside the snapshot (optional) to find the files you need:

docker compose run --rm backup-dir0 ls <snapshot_id>

3. Restore the snapshot into a local directory, mounted inside the container. Use the snapshot ID, or latest for the most recent one:

mkdir -p restore
docker compose run --rm -v ./restore:/restore backup-dir0 restore <snapshot_id> --target /restore

To restore only some files or directories, add one or more --include options with their path inside the snapshot, e.g. --include /backup0/path/to/file.

4. Check the restored data and copy it back. The files are restored under the same path they had inside the container, e.g. restore/backup0/.... Once you have verified them, copy them to their original location, for example with:

rsync -a restore/backup0/ /source/to/backup/

If you are using the manual setup, the steps are the same: run the commands with restic instead of docker compose run --rm backup-dir0, after exporting the RESTIC_REPOSITORY and RESTIC_PASSWORD_FILE variables, and pass any local directory to --target.

For all the available options, e.g. restoring single files to standard output or overwriting files in place, see the Restic documentation.

Other Restic commands

Any Restic command can be run through the backup service by appending it to docker compose run --rm backup-dir0. For example, to run a backup immediately or to check the integrity of the repository:

docker compose run --rm backup-dir0 backup /backup0
docker compose run --rm backup-dir0 check

Renew the OIDC refresh token

The access tokens and S3 credentials used by the backup jobs are short-lived and are refreshed automatically by the scheduled jobs. To do so, oidc-agent uses an OIDC refresh token, which expires after 30 days and must be renewed manually.

To help you remember, the check-refresh-token job scheduled by ofelia monitors the refresh token expiration. When it is one week away, the job reports the corresponding Healthchecks check as "down", including the number of remaining days in the body of the request.

If the refresh token expires, all the scheduled jobs will stop working. Before that happens, renew it by running the following command:

docker compose run --rm oidc-agent reauth

Then follow the instructions on the screen to complete the authentication.

Manual setup

If you are interested in manually setting up the backup solution without using the provided Docker setup, you can follow the instructions in this section. The manual setup is more flexible and allows you to customize the solution to better suit your needs, but it requires more technical knowledge and effort to set up and maintain. You should only follow this section if you are comfortable with the command line and have experience with the tools involved in the backup process.

Install and configure oidc-agent

To install and configure oidc-agent you can follow the instructions in the oidc-agent documentation. If your VM is running AlmaLinux9, you can already find the updated oidc-agent in the EPEL repositories and install it with the following command:

sudo dnf install oidc-agent-cli

If you are using Debian or Ubuntu, as described in the official documentation, you need to configure an additional repository that can be found here

After installing oidc-agent, you need to export and load the environment variables by executing the following commands:

eval `oidc-agent-service use`

To make it persistent, you can add the command to your .bashrc file, running the following while in your home directory:

echo 'eval `oidc-agent-service use`' >> .bashrc

With oidc-agent version >= 5, the legacy audience mode must be explicitly enabled by creating the file ~/.oidc-agent/issuer.config.d/infn-cloud with the following content:

{
  "issuer": "https://iam.cloud.infn.it/",
  "register": "https://iam.cloud.infn.it/manage/dev/dynreg",
  "legacy_aud_mode": true
}

We can now configure oidc-agent to work with the INFN Cloud IAM in order to issue the necessary tokens to access the object storage. To do this, we need to create a new profile in oidc-agent by executing the following commands:

oidc-gen -w device --iss=https://iam.cloud.infn.it/ \
--scope="openid profile offline_access" --aud=object infn-cloud

Follow the instructions on the screen to complete the process.

You will be asked to open a link to the INFN Cloud IAM in your browser, login with your account (if needed) and enter the code provided (in the following is redacted with XXXXXX for security reasons) to approve the registered client. If your terminal view is too short it may be hidden by the QR code appended at the end of the output, you can simply scroll up to see the code.

Registering Client ...
Generating account configuration ...
accepted

Using a browser on any device, visit:
https://iam.cloud.infn.it/device

And enter the code: XXXXXX

Once you complete the authentication via the browser, the profile will be created and you will be asked to set a password to secure it.

If you do so, you will need to provide it each time you want to obtain a token for this profile. The best way of doing this is by either setting an environment variable or add the password to a text file and pass them as an argument of the oidc command line with the --pw-env or --pw-file option, respectively.

The configuration of oidc-agent should be now complete, you can check the profile by executing the following command:

oidc-token infn-cloud

If the command returns a token, the configuration is correct and you can proceed, if the command returns an error instead, please check the previous steps and try again.

Configure S3 credentials

To interact with the INFN Cloud object storage, we need to obtain temporary S3 credentials using the OIDC token obtained with oidc-agent. To simplify this process, INFN Cloud provides a helper script that will take care of this for you.

Make sure you have the jq and xq tools installed, as they are required by the script for parsing JSON and XML responses. Then download the script and save it as $HOME/.local/bin/s3-cred-oidc.sh:

mkdir -p $HOME/.local/bin
curl -fsSL -o $HOME/.local/bin/s3-cred-oidc.sh \
    https://baltig.infn.it/infn-cloud/user-support-scripts/-/raw/main/obj-storage/s3-cred-oidc.sh

Finally, make the script executable:

chmod +x $HOME/.local/bin/s3-cred-oidc.sh

You can read more about the script usage and parameters running the script with the --help flag:

$HOME/.local/bin/s3-cred-oidc.sh --help

To configure any S3 compatible tool to use the helper script to obtain the temporary credentials, you need to create or modify the $HOME/.aws/config file with the following content, replacing <user> with your username (the path must be absolute, as environment variables such as $HOME are not expanded by all tools):

[default]
credential_process = /home/<user>/.local/bin/s3-cred-oidc.sh --oidc-profile infn-cloud
endpoint_url = https://rgw.cloud.infn.it
; On some systems you may need to also specify the following:
; request_checksum_calculation = when_required
; response_checksum_validation = when_required

If you have set up a password when creating the oidc-agent profile, you have to manually load the profile after each reboot:

oidc-add infn-cloud

Since backups usually run unattended (e.g. via cron), you can instead store the password in a file readable only by you and pass it to the script, so that the profile is loaded automatically:

mkdir -p ~/.backup-secrets && chmod 700 ~/.backup-secrets
read -rsp "oidc-agent profile password: " pw && echo && \
    printf '%s\n' "$pw" > ~/.backup-secrets/oidc-pass && unset pw
chmod 600 ~/.backup-secrets/oidc-pass

[default]
credential_process = /home/<user>/.local/bin/s3-cred-oidc.sh --oidc-profile infn-cloud --oidc-pw-file /home/<user>/.backup-secrets/oidc-pass
endpoint_url = https://rgw.cloud.infn.it

Configure and run Restic

Restic is a fast, secure, and efficient backup program that supports deduplication, encryption, and data integrity verification. It is designed to be easy to use and to work well with cloud storage providers.

To install Restic, we suggest you to use the package manager of your distribution of choice. For Ubuntu, you can use the following commands to download and install Restic:

sudo apt update
sudo apt install restic

For other distributions, you can find more detailed installation instructions in the official Restic documentation.

To initialize a new repository with Restic, you can use the following command:

restic -r s3://<username>/<dir-name> init

Where <username> is your INFN AAI username, which is the name of your personal bucket in the object storage, and <dir-name> is the name of a directory of your choice inside your bucket. This will create a new Restic repository in the specified directory. You will be prompted to enter a password for encrypting the repository. Make sure to remember this password, as you will need it to access the data in the future.

To avoid being prompted for the password each time you run a Restic command, you can store it in a file readable only by you and point the RESTIC_PASSWORD_FILE environment variable to it. In this guide we keep all the backup secrets in the ~/.backup-secrets directory. The RESTIC_REPOSITORY variable defines the repository path.

mkdir -p ~/.backup-secrets && chmod 700 ~/.backup-secrets
read -rsp "Restic repository password: " pw && echo && \
    printf '%s\n' "$pw" > ~/.backup-secrets/restic-pass && unset pw
chmod 600 ~/.backup-secrets/restic-pass
export RESTIC_PASSWORD_FILE="$HOME/.backup-secrets/restic-pass"
export RESTIC_REPOSITORY="s3://<username>/<dir-name>"

To create a backup with Restic, you can use the following command:

restic backup /path/to/be/backed-up

Where /path/to/be/backed-up is the local directory you want to backup. Restic will create a backup of the specified directory in the repository.

To list the snapshots in the repository, you can use the following command:

restic snapshots

To restore a backup, you can use the following command:

restic restore <snapshot_id> --target /path/where/to/restore

Where <snapshot_id> is the ID of the snapshot you want to restore, and /path/where/to/restore is the directory where you want to restore the backup. Restic will then restore the specified snapshot to the specified directory.

See Restore a backup for a step-by-step guide on what you need and how to restore your data.

To verify the integrity of the repository, you can run:

restic check

We recommend running it periodically and, from time to time, restoring a snapshot to a temporary directory to make sure that your backups can actually be recovered.

Restic keeps every snapshot until you explicitly remove it. To limit the space used in your bucket, you can define a retention policy and delete the snapshots that don't match it, for example:

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

This keeps the last 7 daily, 4 weekly and 6 monthly snapshots, removes the others and frees the space they used. Adapt the policy to your needs; see the Restic documentation for all the available options.

For more information on the available commands and options, you can consult the Restic documentation.

Configure and run Duplicity

Duplicity is a software suite that provides encrypted, digitally signed, versioned, local or remote backup of files. On the first run it performs a complete (full) backup, then any subsequent (incremental) backups only add differences from the latest version of the backup. Duplicity can use GPG to encrypt the backups and supports a wide range of backends, including an S3 object storage like the INFN Cloud one.

There are multiple ways to install Duplicity, depending on the distribution you are using. For Ubuntu, you can use the official PPA and install it with the following commands:

sudo add-apt-repository ppa:duplicity-team/duplicity-release-git
sudo apt update
sudo apt install duplicity

For other distributions, you can either download the official release from the official GitHub repository and manually install it, or use the pip package manager:

pip3 install -U duplicity

You can find more detailed installation instructions in the "Download" section of the official Duplicity documentation.

To use the INFN Cloud object storage, Duplicity needs the boto3 Python library (version 1.28 or later). Install it with your package manager (e.g. sudo apt install python3-boto3) or, if you installed Duplicity with pip, with:

pip3 install -U boto3

We will now assume that you can successfully run the duplicity command in your terminal.

To execute a backup with Duplicity, using the INFN Cloud object storage as a backend, you can simply run the following command:

duplicity /path/to/be/backed-up s3://<username>/<dir-name>

Where /path/to/be/backed-up is the path of the directory you want to put under backup, <username> is your INFN AAI username (or the name of the bucket in the object storage). The command will create a full backup of the directory on the first run, and then only the differences on subsequent runs. Duplicity can also encrypt and sign the backup with a GPG key, you just need to provide the ID of the key. You can create a new GPG key by running the following command:

gpg --gen-key

and follow the on screen prompts. After the key is generated, you can list the keys with the following command:

gpg --list-keys --keyid-format LONG

The output will look like this:

/home/$USER/.gnupg/pubring.kbx
----------------------------------
pub   rsa3072/KEY_ID 2024-03-15 [SC] [expires: 2026-03-15]
uid                 [ultimate] Name <email address>
sub   rsa3072/SUBKEY_ID 2024-03-15 [E] [expires: 2026-03-15]

The ID we need is the one of the primary key, on the pub line, reported here as KEY_ID. You can now use this key to encrypt and sign the backup by adding the proper option to the duplicity command:

read -rsp "GPG key passphrase: " PASSPHRASE && echo && export PASSPHRASE
duplicity --encrypt-sign-key=KEY_ID /path/to/be/backed-up s3://<username>/<dir-name>

Duplicity reads the passphrase of the GPG key used to encrypt and sign the backup from the PASSPHRASE environment variable. Reading it with read -s avoids writing it in your shell history. Remember to unset PASSPHRASE when you are done.

If you want to restore the backup, you can use the following command:

duplicity restore s3://<username>/<dir-name> /path/where/to/restore

This will restore the latest version of the backup to /path/where/to/restore. If you want to restore a specific version of the backup, you can add the --time option followed by the date of the backup you want to restore:

duplicity restore --time 1D s3://<username>/<dir-name> /path/where/to/restore

This will restore the backup as it was 1 day ago. You can also use other time formats, such as 1W for 1 week ago, 1M for 1 month ago, etc. You can also use the --force option to force the restore, even if the target directory already exists, overwriting its content. For more information on the available options, you can consult the Duplicity documentation.

We recommend periodically restoring the backup to a temporary directory, or running duplicity verify s3://<username>/<dir-name> /path/to/be/backed-up, to make sure that your backups can actually be recovered.

Bash script to automate the backup process

To automate the backup process, you can create a bash script that runs the Duplicity command and then schedule it with cron. Create a new file, we will call it backup.sh in this example, in your home directory.

The script needs the oidc-agent profile password, stored in ~/.backup-secrets/oidc-pass as described in Configure S3 credentials, and the GPG key passphrase. Store the passphrase in the same directory, so that it is readable only by you and not written in the script:

read -rsp "GPG key passphrase: " pw && echo && \
    printf '%s\n' "$pw" > ~/.backup-secrets/gpg-pass && unset pw
chmod 600 ~/.backup-secrets/gpg-pass

Then create backup.sh with the following content:

#!/bin/bash

eval `oidc-agent-service use`
oidc-add --pw-file="$HOME/.backup-secrets/oidc-pass" infn-cloud

DIR_TO_BACKUP="/path/to/be/backed-up"
DEST_BUCKET="s3://<username>/<dir-name>"
LOG_DIR="$HOME/.duplicity"

GPG_KEY="YOUR_KEY_ID"
export PASSPHRASE="$(cat "$HOME/.backup-secrets/gpg-pass")"

mkdir -p "$LOG_DIR"
duplicity --verbosity info --encrypt-sign-key="$GPG_KEY" \
          --log-file "$LOG_DIR/info.log" \
          --full-if-older-than=1M "$DIR_TO_BACKUP" "$DEST_BUCKET"
duplicity --force remove-all-but-n-full 3 "$DEST_BUCKET"

unset PASSPHRASE

The script will run the Duplicity command to create a full backup of the directory on first run and every time the last backup is older than one month and then remove all but the last 3 full backups.

You can make the script executable by running the following command:

chmod 700 ~/backup.sh

To schedule the backup, you can use the cron daemon. You can edit the crontab file by running the following command:

crontab -e

This will open the crontab file in your default text editor. You can then add a new line with the following content:

0 4 * * * $HOME/backup.sh

This will run the backup script every day at 4:00 AM. You can customize the schedule by changing the values in the first five fields. For more information on the crontab syntax, you can consult the cron documentation.