How To: Deploy Sync&Share aaS (sys-admin nomination required)

Sync&Share aaS is offered to INFN Cloud users as a means to easily share documentation and user data among a project or a scientific collaboration. Different tools should be used for scientific data archival and distribution.

Prerequisites

The user has to be registered in the IAM system for INFN-CLOUD https://iam.cloud.infn.it/login. Only registered users can login into the INFN-CLOUD dashboard https://my.cloud.infn.it.

At the time being additional authorizations are needed to submit a Sync&Share deployment on INFN Cloud. Please contact the support team in order to be granted access to the INFN-Cloud Object storage backend before launching your application.

Important

  • The solution described in this guide consists on deployment of a Sync&Share solution on top of a Virtual Machine instantiated on INFN-CLOUD infrastructure. The instantiation of a VM comes with the responsibility of maintaining it and all the services it hosts.

    Please read the INFN Cloud AUP in order to understand the responsabilities you have in managing this service.

  • Production level Sync&Share aaS instances should not be deployed within the infn-catch-all project as the archived data are potentially accessible by extraneous users. If you need to deploy a Sync & Share service for production use make sure to do it within your experiment project or to ask for a dedicated one.

Sync&Share aaS Configuration

Note

If you belong to multiple projects, aka multiple IAM-groups, after login into the dashboard, from the upper right corner, select the one to be used for the deployment you intend to perform. Not all solutions are available for all projects. The resources used for the deployment will be accounted to the respective project, and impact on their available quota. See figure below.

../_images/howto1_01.png

Once done, select the “Sync&Share aaS” card in the service catalog and click on the Configure button.

to be filled in

Figure 1: The main page of the dashboard allows to select the service to be deployed. Move the cursor on the service card and click the Configure button to access the service configuration page.

The configuration parameters are split into two tabs:

Basic configuration tab

The basic configuration is shown here:

to be filled in

Figure 2: Basic configuration tab overview. This form allows to customize Sync&Share aaS service.

The following parameters can be set in order to customize the service:

  • contact email for receiving notifications (e.g. alerts from the monitoring system)
  • username for ownCloud admin access (default is admin)
  • password for the ownCloud admin user. A password is proposed by default, following secure password best-practices.
  • username for the admin user of the monitoring service (default is admin)
  • username for the admin user of the monitoring service
  • passphrase used for encrypting the backup data
  • URL of the IAM (default: https://iam.cloud.infn.it) for enabling user authentication/authorization (see IAM integration section)
  • IAM group whose members will be authorized to access the ownCloud service
  • The VM “flavor”: 1 cpu and 2GB RAM for small projects involving a few users, 2 cpus and 4GB RAM for middle sized projects involving up to 100 users.
to be filled in

Figure 2b: You can select the VM flavor.

Default values, if provided, can be changed; required inputs are highlighted in red if not provided. The deployment description field has to be filled with an arbitrary label that will identify your service in the list of your deployments (Figure 4).

Advanced configuration tab

The Advanced tab allows to configure further properties of the deployment:

  • you can manually select the provider where the service will be instantiated;
  • you can set a timeout for the deployment creation;
  • you can decide to preserve the machine in case of failure in the service configuration (mostly used for debugging purposes).
  • you can enable or disable an e-mail … to be sent when the deployment is completed.

In general, you do not need to change the settings provided by default in the Advanced tab, unless you have specific needs.

to be filled in

Figure 3: Advanced configuration tab.

Clicking on the Submit button, the deployment request is sent to the INFN Cloud Orchestration system. The deployment request can be monitored from the “My deployments” page. As soon as your deployment is ready, you will receive a notification by email.

to be filled in

Figure 4: Deployment list.

Deployment outputs

Clicking on the deployment identifier you can retrieve the details of your deployment (Figure 5) split in three tabs:

  • Overview tab (Figure 5): overall information about the deployment (status, creation time, provider name, etc.);
  • Input values tab (Figure 6): the values used to configure the service;
  • Output values tab (Figure 7): the information needed to access the deployed services.
to be filled in

Figure 5: Deployment details - Overview tab

to be filled in

Figure 6: Deployment details - Input values tab

to be filled in

Figure 7: Deployment details - Output values tab

Access your services

The “Output values” tab, as shown in the previous figure, provides the following information:

  • the IP and ssh account (that coincides with your AAI username) to access the VM where the services are running

  • the list of endpoints to access the web UI of the deployed services

    • field url of ownCloud web UI. The login credentials (username/password) are those provided in the configuration phase and are also available in the “Input values” tab;

      to be filled in
    • field status_service_endpoint: url of the monitoring dashboard. The login credentials (username/password) are those provided in the configuration phase and are also available in the “Input values” tab;

      to be filled in

Manage your service

All the configuration files needed to run the service can be found on the VM in the folder /opt/storageservice:

storageservice
├── .client-iam.json
├── .env
├── docker-compose.yaml
├── oidc.config.php
└── traefik
    └── tls.toml

In particular,

  • .env file contains a list of environment variables used to customize the service (including the input parameters provided through the configuration form),
  • oidc.config.php and .client-iam.json files are relevant for the IAM integration (see next paragraph),
  • traefik/tls.toml file defines some SSL configuration options

In general, you do not need to change any settings in these files unless you have received a specific notification from the INFN Cloud Support Team/Security Incident Team.

Integration with IAM

The service is automatically integrated with IAM by registering a new client dynamically. The information about the client are saved in the .client-iam.json file; in this file you can find the registration access token that can be used to modify (if needed) the client properties in IAM:

{
"client_id": "b6deeb49-2259-44a7-a8b5-347ffa069159",
"client_secret": "AILJOjMlqVUzVi19HaWQQKOavdLU5PEEsZnCKLf86gaJAlj34yFTzJoY0fa77oNgQpXEldHMsk_hXbrV0wkr12o",
"client_secret_expires_at": 0,
"client_id_issued_at": 1646126535,
"registration_access_token": "eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJhdWQiOiJiNmRlZWI0OS0yMjU5LTQ0YTctYThiNS0zNDdmZmEwNjkxNTkiLCJpc3MiOiJodHRwczpcL1wvaWFtLmNsb3VkLmluZm4uaXRcLyIsImlhdCI6MTY0NjEyNjUzNiwianRpIjoiYWNkMmY2OTAtMzE3NC00NTcwLTllMGUtNThmY2Q4MDA4YjQ2In0.RaXYN_GIZa-lvlYuKOc4xFr31EwD8bXoGEElk1QCTSd0FlEebTKir7J6nZmCPqa7ma1U0B0h8zxVFxgwD08qjsnYR4KYJ8xl67JVOWGtIBhJakS5aAS6mBWKO-KiQ4P33pFDYTaMQh1XcjgiF6koGDj0YZhSLYQWryDhw60ZYWU",
"registration_client_uri": "https://iam.cloud.infn.it/register/b6deeb49-2259-44a7-a8b5-347ffa069159",
"redirect_uris": [
    "https://data.90.147.174.123.myip.cloud.infn.it/apps/openidconnect/redirect"
],
"client_name": "oc-client",
"contacts": [
    "antonacci@infn.it"
],
"token_endpoint_auth_method": "client_secret_basic",
"scope": "openid email profile",
"grant_types": [
    "authorization_code"
],
"response_types": [
    "code"
]
}

From the MitreID Dashboard of IAM you can edit your client as shown in the following screenshots:

to be filled in
to be filled in

Fill in the client id and the registration access token provided in the .client-iam.json file (see above):

to be filled in

Edit the client and save your changes:

to be filled in

Sync&Share aaS Backup

A periodic remote backup of the database and of all configuration files is already set up and working when you launch your service. The status endpoint gives information about the backup status.

If you need to check for backup manually do the following:

  • Enter your VM

    ssh 192.135.24.21
    
  • Become root

    sudo -i
    
  • Enter the backup container

    docker exec -it backup bash
    
  • Run exactly the following command for verifying what has been backed up and when

    duplicity --archive-dir /duplicity-metadata list-current-files rclone://remote:/${S3_BACKUP_BUCKET}
    
  • This is an example of restoring a single file to its original location

    duplicity --archive-dir /duplicity-metadata --force --file-to-restore logs/backup.log rclone://remote:/${S3_BACKUP_BUCKET} /backup/logs/backup.log
    

The duplicity man page has full usage documentation and examples on how to manage backups.

Sync&Share aaS Monitoring

Note

The monitoring service will be available five minutes after the Sync & Share service has started.

The INFN-Cloud “Sync&Share aaS” is made of different sub-services, each living inside a separate environment (docker container). These sub-services are monitored internally by a process called Nagios. The following image represents the overview of the monitored services, when everything is working correctly (all green). If any sub-system should fail, the corresponding status label will become yellow or red and a mail alert will be sent to the service owner (email provided in the configuration phase - input field “contact_email”).

IN THAT CASE, SOME ACTIONS HAVE TO BE TAKEN!!!

to be filled in

How to access your data via cli

ownCloud offers a user-friendly web interface for managing and sharing your data. In some cases, you may need to access your data from your environment, e.g. mounting your storage space on your local machine. Rclone is a powerful command-line tool that allows to manage files on a cloud storage. In the following guide you can find some useful tips on how to use Rclone:

https://guides.cloud.infn.it/docs/users-guides/en/latest/users_guides/howto9.html

Troubleshooting

Authorization failure

In case the submission of your deployment request is not accepted due to authorization issues, you will get a warning message as shown in Figure 8. Please contact the support team in order to be granted access to the backend INFN-Cloud Object storage.

to be filled in

Figure 12: Authorization may be needed to deploy Sync&Share aaS