Deploy Sync&Share aaS (sys-admin nomination required)

Sync&Share aaS is offered to INFN Cloud users as a means to easily share documentation and user data among a project or a scientific collaboration. Different tools should be used for scientific data archival and distribution.

Prerequisites

The user has to be registered in the IAM system for INFN-CLOUD https://iam.cloud.infn.it/login. Only registered users can login into the INFN-CLOUD dashboard https://my.cloud.infn.it.

At the time being additional authorizations are needed to submit a Sync&Share deployment on INFN Cloud. Please contact the support team in order to be granted access to the INFN-Cloud Object storage backend before launching your application.

• The solution described in this guide consists on deployment of a Sync&Share solution on top of a Virtual Machine instantiated on INFN-CLOUD infrastructure. The instantiation of a VM comes with the responsibility of maintaining it and all the services it hosts.

  Please read the INFN Cloud AUP in order to understand the responsibilities you have in managing this service.

• Production level Sync&Share aaS instances should not be deployed within the infn-catch-all project as the archived data are potentially accessible by extraneous users. If you need to deploy a Sync & Share service for production use make sure to do it within your experiment project or to ask for a dedicated one.

Sync&Share aaS Configuration

Once done, select the "Sync&Share aaS" card in the service catalog and click on the Configure button.

Select either Automatic or Manual scheduling as shown below:

In the first case, the Orchestrator will take care of choosing the best available provider, in the other case it will be performed a direct submission towards one of the providers available, to be selected from the drop-down menu. In the case of manual scheduling, the flavors displayed on the next page will be those offered by the chosen provider.

The configuration parameters are split into three tabs:

Basic configuration tab

The following parameters can be set in order to customize the service:

• DOCKER STORAGE SIZE: size of the volume to be mounted in /var/lib/docker
• CONTACT EMAIL: contact for receiving notifications (e.g. alerts from the monitoring system)
• ADMIN USERNAME: username for admin access (default is admin)
• ADMIN PASSWORD: password for admin user. A password is proposed by default, following secure password best-practices.
• MONITORING ADMIN USERNAME: username for the admin user of the monitoring service (default is admin)
• MONITORING ADMIN PASSWORD: password for the admin user of the monitoring service
• BACKUP PASSPHRASE: passphrase used for encrypting the backup data
• DATA BUCKET NAME: name of the bucket where data will be stored (max 27 characters)
• BACKUP BUCKET NAME: name of the bucket where backup data will be stored (max 27 characters)
• IAM URL: URL of the IAM (default: https://iam.cloud.infn.it) for enabling user authentication/authorization (see IAM integration section)
• IAM AUTHORIZED GROUP: IAM group whose members will be authorized to access the service
• VM FLAVOR: The VM "flavor": 2 CPU and 4GB RAM for small projects involving a few dozen users, 4 CPUs and 8GB RAM for middle sized projects involving up to hundreds of users.

Default values, if provided, can be changed; required inputs are highlighted in red if not provided. The deployment description field has to be filled with an arbitrary label that will identify your service in the list of your deployments (Figure 4).

Implementation (advanced) tab

Here you can choose the type of service and whether to add an IAM group authorized to access the service as admins (applicable only with nextcloud). If there are no special needs, the use of ownCloud is recommended.

Advanced configuration tab

The Advanced tab allows to configure further properties of the deployment:

• you can set a timeout for the deployment creation;
• you can decide to preserve the machine in case of failure in the service configuration (mostly used for debugging purposes).
• you can enable or disable an e-mail to be sent when the deployment is completed.

In general, you do not need to change the settings provided by default in the Advanced tab, unless you have specific needs.

Clicking on the Submit button, the deployment request is sent to the INFN Cloud Orchestration system. The deployment request can be monitored from the "My deployments" page. As soon as your deployment is ready, you will receive a notification by email.

If the creation of a deployment fails, an additional option (retry) is introduced in the dropdown menu, allowing the user to resubmit the deployment with the same parameters:

If the deletion of a deployment fails, resulting in the status being set to DELETE_FAILED, the "delete (force)" button is displayed in the list of available actions, allowing the user to force the deletion of the deployment:

Deployment outputs

Clicking on the deployment identifier you can retrieve the details of your deployment (Figure 5) split in three tabs:

• Overview tab (Figure 5): overall information about the deployment (status, creation time, provider name, etc.);
• Input values tab (Figure 6): the values used to configure the service;
• Output values tab (Figure 7): the information needed to access the deployed services.

Access your services

The "Output values" tab, as shown in the previous figure, provides the following information:

• the IP and ssh account (that coincides with your AAI username) to access the VM where the services are running

• the list of endpoints to access the web UI of the deployed services

  • field URL of ownCloud web UI. The login credentials (username/password) are those provided in the configuration phase and are also available in the "Input values" tab;

    

  • field status_service_endpoint: URL of the monitoring dashboard. The login credentials (username/password) are those provided in the configuration phase and are also available in the "Input values" tab;

    

Manage your service

All the configuration files needed to run the service can be found on the VM in the folder /opt/storageservice:

storageservice
├── .client-iam.json
├── .env
├── docker-compose.yaml
├── oidc.config.php
└── traefik
    └── tls.toml

In particular,

• .env file contains a list of environment variables used to customize the service (including the input parameters provided through the configuration form),
• oidc.config.php and .client-iam.json files are relevant for the IAM integration (see next paragraph),
• traefik/tls.toml file defines some SSL configuration options

In general, you do not need to change any settings in these files unless you have received a specific notification from the INFN Cloud Support Team/Security Incident Team.

Integration with IAM

The service is automatically integrated with IAM by registering a new client dynamically. The information about the client are saved in the .client-iam.json file; in this file you can find the registration access token that can be used to modify (if needed) the client properties in IAM:

{
  "client_id": "REDACTED",
  "client_secret": "REDACTED",
  "client_secret_expires_at": 0,
  "client_id_issued_at": 1646126535,
  "registration_access_token": "eyJ...REDACTED",
  "registration_client_uri": "https://iam.cloud.infn.it/register/b6deeb49-2259-44a7-a8b5-347ffa069159",
  "redirect_uris": [
    "https://data.90.147.174.123.myip.cloud.infn.it/apps/openidconnect/redirect"
  ],
  "client_name": "oc-client",
  "contacts": [
    "username@infn.it"
  ],
  "token_endpoint_auth_method": "client_secret_basic",
  "scope": "openid email profile",
  "grant_types": [
    "authorization_code"
  ],
  "response_types": [
    "code"
  ]
}

From the MitreID Dashboard of IAM you can edit your client as shown in the following screenshots:

Fill in the client id and the registration access token provided in the .client-iam.json file (see above):

Edit the client and save your changes:

Sync&Share aaS Backup

A periodic remote backup of the database and of all configuration files is already set up and working when you launch your service. The status endpoint gives information about the backup status.

If you need to check for backup manually do the following:

• Enter your VM

ssh 192.135.24.21

• Become root

sudo -i

• Enter the backup container

docker exec -it backup bash

• Run exactly the following command for verifying what has been backed up and when

duplicity --archive-dir /duplicity-metadata list-current-files rclone://remote:/${S3_BACKUP_BUCKET}

• This is an example of restoring a single file to its original location

duplicity --archive-dir /duplicity-metadata --force --file-to-restore logs/backup.log rclone://remote:/${S3_BACKUP_BUCKET} /backup/logs/backup.log

The duplicity man page has full usage documentation and examples on how to manage backups.

Sync&Share aaS Monitoring

The INFN-Cloud "Sync&Share aaS" is made of different sub-services, each living inside a separate environment (docker container). These sub-services are monitored internally by a process called Nagios. The following image represents the overview of the monitored services, when everything is working correctly (all green). If any sub-system should fail, the corresponding status label will become yellow or red and a mail alert will be sent to the service owner (email provided in the configuration phase - input field contact_email).

IN THAT CASE, SOME ACTIONS HAVE TO BE TAKEN!!!

How to access your data via cli

ownCloud offers a user-friendly web interface for managing and sharing your data. In some cases, you may need to access your data from your environment, e.g. mounting your storage space on your local machine. Rclone is a powerful command-line tool that allows to manage files on a cloud storage. In this guide you can find some useful tips on how to use RClone.

How to change the authorized IAM group

ownCloud

If you deployed an ownCloud instance and want to change the name of the IAM group that users must be members of to have access granted, you need to update the file located in /opt/storageservice/oidc.config.php. Here is an example of its content:

<?php
$CONFIG = [
  'http.cookie.samesite' => 'None',
  'openid-connect' => [
        'provider-url' => 'https://iam.cloud.infn.it',
        'client-id' => 'REDACTED',
        'client-secret' => 'REDACTED',
        'loginButtonName' => 'INFN Cloud IAM',
        'auto-provision' => [
                'enabled' => true,
                'email-claim' => 'email',
                'display-name-claim' => 'name',
                'provisioning-claim' => 'groups',
                'provisioning-attribute' => 'users/example',
        ],
        'mode' => 'userid',
        'search-attribute' => 'preferred_username',
  ]
];

The value to be updated to change the group the user authorization is based on is that on line 14, i.e. the value at $CONFIG.openid-connect.auto-provision.provision-attribute that is the users/example string.

Furthermore, to make the change effective, a restart of the service has to be performed:

cd /opt/storageservice/
docker-compose restart || docker compose restart

NextCloud

If you deployed an NextCloud instance and want to change the name of the IAM group that users must be members of to have access granted, you need to modify or add groups through the service's webui at the link https://data.<ip>.myip.cloud.infn.it/settings/admin/sociallogin, in the section Custom OpenID Connect --> add group mapping

Troubleshooting

Authorization failure

In case the submission of your deployment request is not accepted due to authorization issues, you will get a warning message as shown in the figure below. Please contact the support team in order to be granted access to the backend INFN-Cloud Object storage.