How To: Access cloud storage from a scientific environment

Introduction

Cloud storage data can be mounted directly as a folder or synced on your computer or on a cloud environment. This can be achieved, for example, using Rclone (https://rclone.org).

Installing Rclone

Installing rclone on you computer is easy on linux systems:

curl https://rclone.org/install.sh | sudo bash

The Rclone site has full instruction describing different approaches for installing Rclone on different operating systems (https://rclone.org/install/).

Configuring Rclone

Once Rclone is installed, you need to configure it to access your cloud data. One way is to set the following variables:

export RCLONE_CONFIG_MYDATA_TYPE=webdav
export RCLONE_CONFIG_MYDATA_URL=my_owncloud_url
export RCLONE_CONFIG_MYDATA_VENDOR=owncloud
export RCLONE_CONFIG_MYDATA_USER=my_username
export RCLONE_CONFIG_MYDATA_ENV_AUTH=true
export RCLONE_CONFIG_MYDATA_PASS=`rclone obscure “my_password”`

In this case MYDATA is the uppercase name of ther Rclone remote storage you are creating. The cloud storage url (my_owncloud_url) can be obtained by clicking on the “Settings” button on the bottom left on the ownCloud web page and copying the value of the “WEBDav” field

Fig 1: Obtaining the cloud storage url, step 1
Fig 2: Obtaining the cloud storage url, step 2

If you do not want to leave your cloud storage password in a file or in an environment variable, you can obtain an app password from ownCloud itself

Fig 3: Obtaining an app password, step 1
Fig 4: Obtaining an app password, step 2
Fig 5: Obtaining an app password, step 3

Mounting Rclone folders

With the said variables set, mounting an OwnCloud folder is as easy as:

$ mkdir my_owncloud_data
$ rclone mount mydata:Documents my_owncloud_data/

The content of the Documents folder in OwnCloud is now available on your laptop or on your cloud instance under my_owncloud_data/.

Syncing Rclone folders

If you want to just sync one ownCloud folder to your laptop it is enough to

$ mkdir my_owncloud_data
$ rclone sync mydata:Documents my_owncloud_data/

After your work is done you may want to copy the new files to the cloud storage:

$ rclone sync my_owncloud_data mydata:Documents

Conclusions

The previous are just two basic examples of what rclone can do for you, just go to https://rclone.org if you want to learn more. Also, Rclone is only one of many tools you can integrate with your cloud storage.