How To: Deploy an Apache Mesos cluster (sys-admin nomination required)

Prerequisites

The user has to be registered in the IAM system for INFN-Cloud https://iam.cloud.infn.it/login. Only registered users can login into the INFN-Cloud dashboard https://my.cloud.infn.it/login. Through the INFN-Cloud dashboard users can instantiate in a simple way infrastructures that cover different use cases, from the simple VMs up to clusters for big data processing.

User responsabilities

Important

The solution described in this guide consists on the deployment of a Mesos Cluster on top of Virtual Machines instantiated on INFN-CLOUD infrastructure. The instantiation of a VM comes with the responsibility of maintaining it and all the services it hosts.

Please read the INFN Cloud AUP in order to understand the responsabilities you have in managing this service.

Apache Mesos cluster configuration

Note

If you belong to multiple projects, aka multiple IAM-groups, after login into the dashboard, from the upper right corner, select the one to be used for the deployment you intend to perform. Not all solutions are available for all projects. The resources used for the deployment will be accounted to the respective project, and impact on their available quota. See figure below.

../_images/howto1_01.png

After the selection of the project choose the “Apache Mesos cluster” button from the solutions available, then press “configure”. The configuration menu is shown. Parameters are splitted in two pages: “Configuration” and “Advanced”.

Basic configuration

The user has to fill the following parameters:

  • the description of the cluster
  • the password for the Mesos admin user: you will use this password to access the Mesos Web UI/APIs;
  • the password for the Marathon admin user: you will use this password to access the Marathon Web UI/APIs;
  • the number of slave nodes (default: 2);
  • the admin email: this email will be used for certificate expiration notifications;
  • the flavor (cores, ram) of the master node;
  • the flavor (cores, ram) of each slave node.

By default the provider where the cluster will be istantiated is automatically selected by PaaS system, through the scheduling mechanism of the Orchestrator that will deploy the cluster on the site that better fits your requirements.

When all parameters have been set the user can submit the cluster deployment request.

Advanced configuration

The user can select also:

  • the scheduling mechanism: automatic (default) done by the PaaS orchestrator or manual, selecting the provider where the cluster will be created
  • the timeout for the deployment creation
  • to not delete the deployment in case of failure: for debbuginig purposes
  • to be notified by e-mail when the cluster deployment has been completed (this is the default behaviour).

Deployment result

To check the status of the deployment and its details select the “deployments” button. Here all the user’s deployments are reported with a list of fields: “description”, “deployment uuid”, “status”, “creation time” and “provider”. An example is provided in the screenshot below.

Deployment Mesos

For each deployment the button “Details” allows:

  • to retrieve the deployment log file that contains info and eventual error messages in case of failure
  • to show the tosca template used by the Orchestrator to deploy the cluster
  • to lock the depoyment in order to avoid accidental deletion
  • to get the details of the VMs composing the cluster: public and private IP addresses, CPU, RAM, disk, operating system
  • to delete the cluster

Clicking on the deployment uuid, you can access your deployment details split into three tabs:

  • Overview tab: it provides generic information about the deployment (creation/update time, status, site hosting the cluster)
  • Input Values tab: it reports a summary of the input parameters used to customize the cluster
  • Output Values tab: it provides the information needed to access the cluster and its endpoints, e.g. the Mesos/Marathon dashboards. An example is provided in the screenshot below:
Mesos Output

Among the deployment outputs there is also the mesos_lb_ip used to expose (outside the cluster) the services deployed by the user on top of the Mesos cluster (e.g. if you want to run an nginx server and make it accessible from the external network).The endpoints of both Mesos and Marathon are provided on https: a Let’sencrypt certificate is automatically generated.

This is an example of a Mesos dashboard responding to the endpoint https://131.154.96.207.myip.cloud.infn.it:5050

Mesos Dashboard

This is an example of the Marathon dashboard responding to the endpoint https://131.154.96.207.myip.cloud.infn.it:8443

Marathon Dashboard CreateApp

IMPORTANT: if you want to expose your services using the Load Balancer mentioned above you need to set the label HAPROXY_GROUP to external. An example is shown below where an nginx container is deployed on Marathon:

SimpleAppTest

Connecting with a browser to the mesos_lb_ip and port 10000 (servicePort assigned by the cluster) you can access your nginx instance.

Troubleshooting

The successful creation of the cluster depends on the provider resources availability. If the quota is exhausted, you may get a “no quota” error. In that case, please contact the support team.