Run MATLAB Notebooks with persistence on a JupyterHub (sys-admin nomination required)

Prerequisites

Make sure you are registered to the IAM system for INFN-CLOUD https://iam.cloud.infn.it, as described in the Getting Started guide. Only registered users can login into the INFN-CLOUD dashboard https://my.cloud.infn.it.

Access to the INFN-CLOUD dashboard enables users to instantiate a JupyterHub service on a single VM, providing Notebooks with data persistence.

This solution requires the instantiation of a JupytherHub service on top of a newly created virtual machine (VM). You will have complete control, administration rights, on the respective service and VM becoming a service administrator.

Please read the INFN Cloud AUP in order to understand the responsibilities you have in managing this service.

How to deploy and access the JupyterHub service with a MATLAB kernel

Step 1 - Connecting and authenticating to the INFN-CLOUD dashboard

Connect to the INFN-CLOUD dashboard (https://my.cloud.infn.it/ ):

Press the login button and the IAM authentication page should open

Select "Sign in with INFN CCR-AAI" button and use your AAI credential.

Step 2 - Select and Configure the Jupyter + Matlab (with persistence for Notebooks) service

---

First check that you are working in the intended project, among those you belong to, by looking at the active one in the bottom left of the dashboard (under your name).

Then go over the Jupyter + Matlab (with persistence for Notebooks) card and click on the Configure button.

Select either Automatic or Manual scheduling as shown below:

In the first case, the Orchestrator will take care of choosing the best available provider, in the other case it will be performed a direct submission towards one of the providers available, to be selected from the drop-down menu. In the case of manual scheduling, the flavors displayed on the next page will be those offered by the chosen provider.

After that you will have to configure your deployment. The deployment definition window consists of three tabs: "GENERAL", "AUTHORIZATIONS" and "ADVANCED". Before continuing please fill the first mandatory field - the "DEPLOYMENT DESCRIPTION". You will not be able to submit your deployment without it!

"GENERAL" tab

In this tab, all fields have default values but can be changed if desired. You can fill the following fields:

• NUM_CPUS
  • Number of virtual CPUs for the VM that will host the Jupyter hub. The default value is 2.
• MEM_SIZE
  • Amount of memory for the same VM, expressed in GB. The default value is 4.
• ENABLE_MONITORING
  • Option to enable the monitoring of the VM. It is disabled by default.
• JUPYTER_IMAGES
  • The image used to create the Jupyter Hub on the VM.
  • Default value is "harbor.cloud.infn.it/datacloud-templates/jupyter_matlab". If you want to build and use your own Jupyter Hub images, it is better to use the default image as a base image and add your customization on top of it.
• JUPYTERLAB_COLLABORATIVE
  • Enable the new collaborative editing feature that allows collaboration in real-time between multiple users. It is disabled by default. See JupyterLab documentation for more information.
• JUPYTERLAB_COLLABORATIVE_IMAGE
  • The image used to create the Jupyter Hub where it's possible to host a collaborative Jupyter Lab instance.
  • Default image for JupyterLab collaborative feature is "harbor.cloud.infn.it/datacloud-templates/collaborative_matlab".
• CONTACT_EMAIL
  • Email address of the administrator of the VM, it will be used for the certificate of the VM.
• PORTS
  • List of additional ports to be opened. By default, and you don't need to specify them, the deployment will have the following TCP ports accessible: 22 (ssh to the host VM), 3000 (for grafana dashboard), 8888 (for JupyterHub), 8889 (for jupyter collaborative)
• FLAVOR
  • Number of vCPUs and memory size of the Virtual Machine

"AUTHORIZATIONS" tab

You can decide to authorize INFN Cloud user groups by filling:

• IAM_GROUPS
  • user groups that are allowed to access jupyterhub services.
• IAM_ADMIN_GROUPS
  • user groups that are allowed to administrate jupyterhub.

"ADVANCED" tab

Advanced parameters can be configured here:

• Deployment creation timeout (minutes)
  • If specified the deployment will fail when the timeout is reached
• Do not delete the deployment in case of failure
  • It will leave the deployment in the dashboard even if it failed. Useful to check the errors in the log.
• Send a confirmation email when complete
  • The system will send an email to the address provided in the contact_email field under the General tab.

Step 3 - Submitting the deployment

After pressing the Submit button, you will be redirected to the list of your deployments from where you can follow the evolution of this new deployment. Please wait for the status to change from CREATE_IN_PROGRESS to CREATE_COMPLETE

If the creation of a deployment fails, an additional option (retry) is introduced in the dropdown menu, allowing the user to resubmit the deployment with the same parameters:

If the deletion of a deployment fails, resulting in the status being set to DELETE_FAILED, the "delete (force)" button is displayed in the list of available actions, allowing the user to force the deletion of the deployment:

Step 4 - Access your application

After the deployment is complete (Status change to CREATE_COMPLETE), you can click on the Details button.

A new page will appear with three tabs: "Overview", "Input Values" and "Output Values".

Open the "Output values" tab and click on the jupyter_endpoint link.

If this is the first time connecting to this endpoint, the jupyterhub login page should appear asking to Sign-in with OAuth 2.0. After pressing the orange button the IAM login form should open. Login with your INFN-AAI credentials and approve the request for the jh-client.

The home page should now appear. Here it is possible to initialize a jupyterlab instance by filling the following fields:

• desired image: field should be filled with the proper image, the default one can be chosen by clicking on the field and selecting the value that pops-up.
• desired memory size: you can select multiple options depending on your needs for the particular jupyterhub instance; please be aware that the maximum amount depends on the total memory assigned to the machine deployed in the first place.
• GPU: if available, a GPU can be added here.

When all the fields are correctly filled it is possible to start the jupyterhub instance. The Hub homepage should now appear. Click the "Open MATLAB" button in order to start the MATLAB Online instance, needed to run the MATLAB kernel inside the notebook.

The MathWorks login page should now open. Insert as email your AAI username followed by @infn.it domain and click next. The AAI login page should appear where you can use again your AAI credentials. Once logged in please wait for MATLAB Status to be "Complete" and the MATLAB Online instance to open.

Now it is possible to go back to the JupyterHub home page and launch a notebook with a MATLAB Kernel by clicking on the proper button.