INFN Cloud object storage service

Table of contents

INFN Cloud object storage service

Introduction

This guide provides a short description about the INFN-Cloud object storage and how to use the centrally managed service based on rados-Gateway, https://s3webui.cloud.infn.it. The object storage is the persistent storage of INFN-Cloud.

Prerequisites

The user has to be registered in the IAM system for INFN-Cloud https://iam.cloud.infn.it/login. Only authorized users can access the storage service, obtaining a personal private bucket and an area under /scratch in readonly mode visible to all registered users.

For more details regarding the registration process please see Getting Started. To use this service you don't need the "system admin" nomination.

The Object Storage

INFN-Cloud provides a distributed object storage service, centrally managed, based on rados-Gateway. This storage is the persistent one for INFN-Cloud. The object storage is replicated in the two sites of backbone, Bari and CNAF. In this way the redundancy of data is guaranteed. There is not a backup of this data. Only authorized users can access the storage service. Its configuration allows users to have a personal bucket labeled as the INFN AAI username and an area under /scratch. Data stored in the personal bucket are private, data under /scratch are visible and downloadable by all the registered users. The quota for each user is 200GB.

INFN-Cloud uses the rados-gateway software to provide access to the storage, create buckets and manage files. It provides also S3 api for the object storage access via code.

The personal bucket can by created by the user accessing the rados-gateway webui otherwise it is automatically created the first time user instantiates a notebook. Instead the user area under /scratch/<INFN_AAI_username> has to be created by the user.

All the services running notebooks mount in automatic way the personal bucket and the /scratch areas, so they are visible in the notebook file system as posix directories (under the cloud-storage dir).

To log into the Object Storage, follow the endpoint https://s3webui.cloud.infn.it and click on "Login with INDIGO IAM"

/users_guides/img/objectstorage/gw_login_dark.png — Figure 1: Log in page

The user logs in to the system via IAM using INFN-AAI credentials

/users_guides/img/objectstorage/iam_login.png — Figure 2: IAM login homepage

/users_guides/img/objectstorage/infn_aai_login.png — Figure 3: Using INFN-AAI identity

and then authorizes the client:

/users_guides/img/objectstorage/authorize_client.png — Figure 4: Client authorization

Creation of the personal bucket

The user can create the personal bucket clicking (1) on the button "Buckets" on the left side menu and then (2) selecting the icon "+ Create Bucket". The name of the bucket must be the INFN-AAI username.

/users_guides/img/objectstorage/create_bucket.png — Figure 5: Personal bucket creation

Creation of the `/scratch/<username>` directory

The user can create his/her directory under /scratch, clicking (1) on the icon "Browser" on the left menu and then (2) on "scratch".

/users_guides/img/objectstorage/create_area1.png — Figure 6: How to select `/scratch` area

At this point select the icon "New path"

/users_guides/img/objectstorage/create_area2.png — Figure 7: How to select the New path

Enter the name of the directory (1) that must be identical to the INFN-AAI username and then (2) click the botton to confirm. A file must be uploaded to this area immediately otherwise the empty area is removed.

/users_guides/img/objectstorage/create_area3.png — Figure 8: How to give the name to the new area under `/scratch`

In the following it is reported how it looks the read only area in /scratch visible to all registered users:

/users_guides/img/objectstorage/created_area.png — Figure 9: Readonly area in `/scratch` visible to all registered users

Upload and download files

The user can upload and download files from the personal bucket. He/she can also upload and download files from the "personal scratch area" and download files from the "scratch area" of other users.

For the upload of files use the "Upload File" icon then select the file to be uploaded.

/users_guides/img/objectstorage/upload_file.png — Figure 10: Upload file to the "personal scratch area"

To download files, click on the selected files and then click on the "Download" botton.