Requirements for OpenStack-based cloud infrastructures

This chapter discusses the federation of resource providers using OpenStack as cloud middleware framework.

Table of contents

Required OpenStack services

Sites using OpenStack need to operate at least the following services:

  • Keystone (authentication and authorization)
  • Glance (images)
  • Nova (computing)
  • Cinder (block storage)
  • Neutron (networking)

At the moment there are no strict requirements on the OpenStack version that can be used, but we recommend to use a fully supported version.

General requirements

To enable the integration of an OpenStack site into the INFN-Cloud federation, it is necessary:

  • to enable the INFN-Cloud IAM instance as one of the Cloud identity provider
  • to enable the desired federated users
  • to support the creation of INFN-Cloud service instances on the local resources
  • to configure the needed accounting services

Access to resources

The instantiation of services using the core services of the INFN-Cloud PaaS is the blessed mechanism that must be supported by the resource provider. Unless other specific agreements, the resource provider is not bound to provide INFN-Cloud users also with direct access to its resources at the IaaS level. This basically means that it is not mandatory to provide direct access to the resources to the INFN-Cloud users through the OpenStack dashboard or through the OpenStack APIs.

Supporting the creation of INFN-Cloud service instances

The INFN-Cloud services can be grouped in two categories:

  • services instantiated on public networks
  • services instantiated on private networks.

The service instances of the first group are exposed on the Internet and are therefore directly accessible by their users.

Service instances on private networks can instead be accessed by their users only through a VPN.

A site can support both types of services, or just the services of one of the two categories. Details about the configurations needed to support the two types of services are reported below.

Enabling the INFN-Cloud IAM instance

The procedure to enable the INFN-Cloud IAM is detailed in this presentation.

Enabling multiple IAMs with ESACO

The ESACO software can be used in those cases when a site need to configure multiple IAMs. ESACO is a daemon that checks the validity and the signatures of OAuth tokens for registered trusted OAuth authorization servers. Thus, it is typically used to integrate multiple IAM instances with software that do not naturally support multiple token issuers, such as Openstack Keystone, but it works fine even in other scenarios. More information can be found in the software repository, from which most of the information available here is taken.

Prerequisites

The general procedure described in the previous section titled "Enabling the INFN-Cloud IAM instance" for the particular case of INFN-Cloud IAM, should be followed to properly configure each IAM that will be integrated with ESACO. In summary:

  • Each IAM should be already properly set-up and fully functional
  • Keystone should be configured as a web app inside an Apache virtual host (as described in the presentation linked above)
  • Apache module mod_auth_openidc should be installed and configured to work with the desired IAMs (as described in the presentation linked above), and the introspection endpoint should be changed to that exposed by ESACO
    OIDCOAuthIntrospectionEndpoint <ESACO_ADDRESS>/introspect
OIDCOAuthClientID <ID>
OIDCOAuthClientSecret <SECRET>

Finally, ESACO is distributed as a docker image, thus a full docker installation is needed.

Configuration

To properly configure the software two files needs to be passed to the container at creation:

  • a YAML file, named ⁠application.yml containing the configuration for all the trusted IAM in which ESACO should be registered as a client
  • a file containing all the needed environmental variables, usually called ⁠esaco.env

Authorization server configuration

To be be used by client applications as a gateway for token validation and introspection, ESACO needs to be registered as a client in each IAM, the procedure is described in details here. After registering a new client, each IAM should generate an id-secret credentials pair. This information should be passed to ESACO via a YAML file, named ⁠application.yml and with the following structure:

oidc:
clients:
    - issuer-url: https://iam.example
      client-id: iam.example.client-id
      client-secret: iam.example.client-secret
    - issuer-url: https://iam2.example
      client-id: iam2.example.client-id
      client-secret: iam2.example.client-secret

Each client entry in the file should be dedicated to one of the IAMs to enable.

Environmental variables

Port and address By default ESACO listens on port 8156 on all interfaces, and the introspection endpoint answers at ⁠/introspect. This behavior can be changed by setting, respectively, the ⁠ESACO_BIND_PORT and ESACO_BIND_ADDRESS variable.

Cache management ESACO uses an internal in-memory cache to cache results of token introspection and userinfo calls. The size and the time duration of the cache can be set with ⁠ESACO_CACHE_SPEC variable. By default, the cache is setup with a maximum size of 500 entries and the records are cleared after 60 seconds. Cache can be totally disabled by setting the ESACO_CACHE environment variable to ⁠none.

Authentication Introspection requests to ESACO should be authenticated using a username and password. The default values of which are 'user' and 'password' respectively. They can be set by assigning the ESACO_USER_NAME and ESACO_USER_PASSWORD environment variables.

Reverse Proxy ESACO should be deployed behind a reverse proxy used to terminate TLS. In this case ESACO_USE_FORWARD_HEADERS environment variable should be set to true.

Example configuration file The following is a environmental file template used to configure ESACO.

⁠# ESACO client will bind on this port
ESACO_BIND_PORT=8156

# ESACO client will bind on this address
ESACO_BIND_ADDRESS=0.0.0.0

# Set this to true when deploying behind a reverse proxy (nginx)
ESACO_USE_FORWARD_HEADERS=false

# X.509 trust anchors location
X509_TRUST_ANCHORS_DIR=/etc/grid-security/certificates/

# X.509 trust anchors refresh interval (in msec)
X509_TRUST_ANCHORS_REFRESH=14400

# or use a single-file CA bundle without CRLs
#X509_TRUST_ANCHORS_BUNDLE=/etc/ssl/certs/ca-bundle.crt
#X509_TRUST_ANCHORS_TYPE=BUNDLE

# User name credential requested from clients introspecting tokens
ESACO_USER_NAME=user

# Password  credential requested from clients introspecting tokens
ESACO_USER_PASSWORD=password

# Enables caching of the results of introspection and userinfo calls
# To disable the cache set ESACO_CACHE=none
ESACO_CACHE=caffeine

# The size and eviction time policies for the cache
ESACO_CACHE_SPEC=maximumSize=500,expireAfterWrite=60s

# TLS version
ESACO_TLS_VERSION=TLSv1.2

For more detailed and up to date information it is advised to check the ESACO repository .

Usage

After creating the two aforementioned configuration files, the service can be started with the following command:

docker run --env-file=esaco.env -v application.yml:/esaco/config/application.yml:ro indigoiam/esaco:latest

ESACO integration example

In order to see ESACO in action in a test environment, containing all the necessary software, a docker-compose has been developed and it can be found here . In particular, this compose deploys:

  • two test IAMs
  • Apache httpd
  • nginix reverse proxy
  • ESACO

In depth updated instructions and details can be found in the README of the software available here .

Managing users

Once the INFN-Cloud IAM instance is enabled, the site admin must enable the desired federated users.

The users in the INFN-Cloud IAM are organized in groups:

  • groups in the form of /admins/<VO> refer to users that can create instances on public networks;
  • groups in the form of /priv-admins/<VO> refer to users that can create instances on private networks.

The site admin can enable the desired IAM groups, mapping them to local OpenStack projects, considering that:

  • different IAM groups must be mapped to different OpenStack local projects
  • proper isolation must be implemented between users (e.g. a user must be prevented from deleting an instance or a volume created by another user of the same project), which also means that IAM users must not have an ADMIN role in OpenStack.

The procedure to map a IAM group into a OpenStack local project is described in this presentation.

Service accounts

It is mandatory to enable the ops IAM user, which must be mapped to a local OpenStack project. This is used to test the proper functionality of the site, by instantiating new virtual machines, by creating volumes, etc. Such temporary resources are deleted once the test is completed.

It is necessary to assign at least two floating IPs to the local project mapped to the ops IAM user.

The ops IAM user does also perform monitoring and accounting measurements.

In particular it is needed by:

  • the Cloud Information Provider (CIP);
  • the INFN-Cloud security-scan central service, which is used to detect possible vulnerabilities in the services deployed on INFN-Cloud resources.

This user simply retrieves the list of images and flavors available in all federated projects (for the cloud information provider) and the list of instances (for the security scan service). No special actions are needed by the site to enable such functionality.

The ops user must not have admin privileges on any OpenStack project.

Configuring projects for services on public networks

A project mapped to a /admins/<VO> will be used to create service instances on the public network.

Such project must therefore be given the needed quota of public floating IP addresses: in general one floating IP is needed for each service instance.

To support the deployment of the high-level services and to allow their access by the users, the resource-provider firewall has to be configured to allow inbound access to the following ports belonging to the public IP addresses allocated for such services:

  • port 22
  • port 80
  • port 443
  • any port greater than 1024 except for the following ones, that must be kept closed: * 1080 (socks proxy) * 1191 (gpfs) (udp+tcp) * 2049, 4045, 4046, 4049, 20048, 20049 (nfs) (udp+tcp) * 3260 (iscsi) * 3389 (rdp) * 5900 (vnc) * 5800 (jvr) * 10000 (webmin) * 6000 to 6023 (X11)

Any other port below 1025 must be kept closed, unless properly documented and authorized. If a resource provider for some reasons needs to have other ports open, it must contact the INFN-Cloud security team (WP4) providing all the details related to the request. This must be done using the INFN-Cloud service desk (https://servicedesk.cloud.infn.it/) and then selecting "Technical support". All the motivations and details related with the request must be specified in the ticket. The INFN-Cloud security team will then start a discussion with the proponent to determine the best solution.

The public IP addresses provided by the resource center (needed for the services instantiated by the INFN-Cloud users) will be monitored by the INFN-Cloud security incident team, in order to promptly detect possible vulnerabilities. If a problem is found, both the user who instantiated the service and the admin of the site where the service was instantiated are notified.

Configuring projects for services on private networks

A project mapped to a /priv-admins/<VO> will be used to create service instances on the private network.

Such project doesn't need public floating IP addresses except one for a special virtual machine (usually to be created on the same project) must be configured by the site administrator. This special instance must implement two functionalities:

  • proxy-host, to allow the configuration of services by the INFN-Cloud PaaS layer services;
  • VPN server, to allow users to access their services.

The VPN server must be integrated with the INFN-Cloud IAM, so that only the members of the IAM group mapped to this project are authorized to use this VPN server.

This means that this special virtual machine:

  • must have a public IP address

  • must have port 22 open to the following networks:

    • 90.147.174.0/24
    • 192.135.24.0/24
    • 90.147.176.0/24

  • must be able to access port 22 of all the virtual machines created in its project

  • must have port 1194 (the one used by the VPN) open.

To install and configure this special virtual machine, first of all please create an Ubuntu 22.04 instance. The flavor of this virtual machine depends on how many users it needs to support. A VM with 2 VCPUs, 4 GB of RAM and 20 GB of root disk should be able to support most use cases. This machine must have a public floating IP and its network must be configured as stated above.

Then the site admin needs to register a client in IAM. Client registration using the command line is described at this page. Details on how to perform the IAM client registration using the IAM GUI follow.

Go to https://iam.cloud.infn.it/login and, after the login, select MitreID Dashboard. Then select Self-service client registration to register a new client. Specify the following parameters:

  • Main --> Client name: write here a string to identity the client (e.g. "VPN on ")
  • Main --> Redirect URI(s): this field is mandatory but it is not used in this case (you can set it e.g. to "https://<server ip>")
  • Access --> Scope: you must enable 'openid' and 'profile'
  • Access --> Grant Types: you must enable urn:ietf:params:oauth:grant-type:device_code

Then select Save to register the new client. Please be sure to copy the following information (that are needed later):

  • Client ID
  • Client Secret
  • Registration Access Token

Configure the VPN server (integrated with IAM). These are the relevant instructions (also available at this page):

First of all the site admin must install the needed packages:

wget https://github.com/maricaantonacci/pam_oauth2_device/releases/download/v0.0.3/pam-oauth2-device_0.0.3_all.deb
sudo dpkg -i pam-oauth2-device_0.0.3_all.deb
wget -O - https://swupdate.openvpn.net/repos/repo-public.gpg | sudo apt-key add -
sudo echo "deb http://build.openvpn.net/debian/openvpn/release/2.5 focal main" > /etc/apt/sources.list.d/openvpn-aptrepo.list
sudo apt update

Then the site admin has to properly configure the PAM module for OAuth 2.0 Device flow by setting the /etc/pam_oauth2_device/config.json file. An example is provided here below:

{
 "oauth": {
     "client": {
         "id": "a142c390-78c9-658d-c329-f9c60564ba96",
         "secret": "CJKSDVCHyy8Sj5H2jjdpaDS4E6HLCYr3MWMG1NigFU__vmVGnW7xVFKHN6MwhoMVW3dt-pHVYJPLyalLbPekrKQ"
     },
     "scope": "openid profile",
     "device_endpoint":"https://iam.cloud.infn.it/devicecode",
     "token_endpoint": "https://iam.cloud.infn.it/token",
     "userinfo_endpoint": "https://iam.cloud.infn.it/userinfo",
     "username_attribute": "preferred_username",
     "groups": ["developers"]
 },

 "enable_email" : true,
 "send_mail" : {
     "smtp_server_url": "smtp://smtp-cc.infn.it:587",
     "smtp_username": "sgaravat",
     "smtp_password": "PASSWORD",
     "smtp_insecure": false,
     "smtp_ca_path": "",
     "from_address": "sgaravat@infn.it",
     "from_username": "noreply@cloud.infn.it",
     "cc_address": ""
 },
 "qr": {
     "error_correction_level": 0
 }
}

The site admin has to modify the following fields:

  • id: write here the id of the IAM client you registered
  • secret: write here the secret of the IAM client you registered
  • groups: write here the name of the IAM group you are authorizing
  • smtp_username: ask the INFN-Cloud WP1 team for this information
  • smtp_password: ask the INFN-Cloud WP1 team for this information
  • from_address: this must be <smtp_username>@infn.it
  • from_username: this can be set to noreply@cloud.infn.it

Then, to configure the OpenVPN server:

wget https://baltig.infn.it/infn-cloud/site-admins/-/raw/main/openvpn-infncloud-install.sh
chmod +x /openvpn-infncloud-install.sh
./openvpn-infncloud-install.sh

Then, create the /etc/pam.d/openvpn file with this content:

auth required pam_oauth2_device.so
account sufficient pam_oauth2_device.so

Edit the /etc/openvpn/server/server.conf file, by adding the following lines:

push "route 172.30.6.0 255.255.255.0"

where "172.30.6.0/24" is the private network used by the instances

An example of the /etc/openvpn/server/server.conf file is provided here:

local 212.189.205.136
port 1194
proto tcp
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
auth SHA512
tls-crypt tc.key
topology subnet
server 10.8.0.0 255.255.255.0
#push "redirect-gateway def1 bypass-dhcp"
push "route 172.30.6.0 255.255.255.0"
ifconfig-pool-persist ipp.txt
#push "dhcp-option DNS 8.8.8.8"
#push "dhcp-option DNS 8.8.4.4"
keepalive 10 120
cipher AES-256-CBC
user nobody
group nogroup
persist-key
persist-tun
verb 7
crl-verify crl.pem
plugin /usr/lib/x86_64-linux-gnu/openvpn/plugins/openvpn-plugin-auth-pam.so openvpn
duplicate-cn
setenv deferred_auth_pam 1
reneg-sec 0
hand-window 300
username-as-common-name

Finally, enable and restart the OpenVPN server:

systemctl restart openvpn-server@server.service
systemctl enable openvpn-server@server.service

The openvpn-infncloud-install.sh script will also create in your home directory the VPN client file (client.ovpn). This file must be uploaded in https://baltig.infn.it/infn-cloud/vpnconfiles. Its name must have the following format:

vpn-<IP address of the VPN server>-client.ovpn

To configure this special virtual machine also as proxy-host, the site admin needs to create a im account, that must be configured to authorize access with this SSH key. It is recommend to configure the SSH service to allow authentication only through SSH keys.

Other network requirements

Firewall configuration for the OpenStack services

The endpoints exposing the OpenStack services must be reachable at least by the core services of the INFN-Cloud PaaS and by the INFN-Cloud monitoring servers. This means that they must be reachable by the following networks:

  • 90.147.174.0/24
  • 192.135.24.0/24
  • 90.147.176.0/24

OpenStack network service quota

Each time a service is deployed through the INFN-CLOUD services, multiple security groups are created (they are then deleted when the service is canceled). This means that, for each federated project, the quota values for security groups and security rules need to be increased since the default values are usually pretty low. We suggest setting a default value of 300 security groups and 1000 security rules for the federated projects.

Images

The site must provide all projects federated to INFN-Cloud with images at least for the following operating systems:

  • Ubuntu 20.04
  • Ubuntu 22.04
  • Debian 12
  • AlmaLinux 9

Images available in the official Ubuntu repositories are usually compliant with such requirement.

There are no requirements on the names of the images, but each image exposed to INFN-Cloud users must be tagged with the infn-cloud tag. Moreover for such image it is necessary to set the os_distro and os_version metadata, according to the following table:

Operating System os_distro os_version
Ubuntu 20.04 ubuntu 20.04
Ubuntu 22.04 ubuntu 22.04
Debian 12.x debian 12
AlmaLinux 9.x almalinux 9

E.g. for an ubuntu 22.04 image, it will be necessary to set:

openstack image set --tag infn-cloud <image-id>
openstack image set --property os_distro=ubuntu --property os_version=22.04 <image-id>

Images should be updated by the site administrator quite often. In particular they must be updated whenever there is a vulnerability to be addressed.

When an image is updated the infn-cloud tag should be added to the new image, and removed from the old image. To remove the infn-cloud from an image, you can use the following command:

openstack image unset --tag infn-cloud <image-id>

The old image can then be disabled, so it won't be used in new instances. The safest way to do that is to hide the image using one of the following commands (the first one could not work in old versions of OpenStack):

openstack image set --hidden <imageid>

or:

glance image-update --hidden True <imageid>

Preparing and Configuring Debian 12 (Bookworm)

This section outlines the steps required to prepare and configure a Debian 12 QCOW2 image.

Prerequisites

Required tools:

  • wget
  • qemu-utils]
  • virt-sparsify (from libguestfs-tools)

Procedure

Choose a directory to save and modify the image. Set the path as follows (replace /path/to/destination with your desired path):

#!/bin/bash                                                                                                                          
DESTINATION_PATH=/path/to/destination
mkdir -p $DESTINATION_PATH

Download the Debian 12 Image

Download the latest official Debian 12 (Bookworm) image in QCOW2 format:

#!/bin/bash                                                                                                                          
wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2 -O $DESTINATION_PATH/debian-12-generic-amd64.qcow2

Prepare the Environment

Load the nbd (Network Block Device) kernel module to access the image:

#!/bin/bash                                                                                                                          
sudo modprobe nbd

Connect the image to the an unused nbd device (for example /dev/nbd0):

#!/bin/bash                                                                                                                          
sudo qemu-nbd --connect=/dev/nbd0 $DESTINATION_PATH/debian-12-generic-amd64.qcow2

Mount the File System

Mount the primary partition of the image:

#!/bin/bash                                                                                                                          
sudo mkdir /mnt/debian
sudo mount /dev/nbd0p1 /mnt/debian

Mount the virtual filesystems required for the chroot environment:

#!/bin/bash                                                                                                                          
sudo mount -t proc proc /mnt/debian/proc
sudo mount -o bind /dev /mnt/debian/dev
sudo mount -o bind /run /mnt/debian/run

Configure the System

Enter the chroot environment:

#!/bin/bash                                                                                                                          
sudo chroot /mnt/debian

Update the system:

#!/bin/bash                                                                                                                          
apt update
apt -y upgrade

Install essential packages:

#!/bin/bash                                                                                                                          
apt -y install rsyslog openssh-server vim

Enable [rsyslog] and [ssh] services to start on boot:

#!/bin/bash                                                                                                                          
systemctl enable rsyslog
systemctl enable ssh

Exit the chroot environment:

#!/bin/bash                                                                                                                          
exit

Unmount and Disconnect the Image

Unmount the virtual filesystems and the primary partition:

#!/bin/bash                                                                                                                          
sudo umount /mnt/debian/dev
sudo umount /mnt/debian/run
sudo umount /mnt/debian/proc
sudo umount /mnt/debian
sudo rmdir /mnt/debian

Disconnect the NBD device:

#!/bin/bash                                                                                                                          
sudo qemu-nbd --disconnect /dev/nbd0

Optimize and Sparsify the Image

Reduce the image size and compress it for distribution: lib #!/bin/bash
sudo mv $DESTINATION_PATH/debian-12-generic-amd64.qcow2 $DESTINATION_PATH/debian-12-generic-amd64.qcow2.uncompressed sudo virt-sparsify --tmp tmp --compress $DESTINATION_PATH/debian-12-generic-amd64.qcow2.uncompressed $DESTINATION_PATH/debian-12-generic-amd64.qcow2 sudo rm -f $DESTINATION_PATH/debian-12-generic-amd64.qcow2.uncompressed

Upload

After completing the configuration:

  • Upload the compressed image to the images catalog in Glance;
  • Ensure the image complies with the guidelines outlined in this section, paying particular attention to the initial part.

Disabling weak SSH ciphers, MACs and KexAlgorithms on images using the vendordata mechanism

The idea is to use the vendordata mechanism to execute a script, downloaded from a web server, once a new instance starts.

This is achieved creating the file /etc/nova/vendor-data.json with this content:

{"cloud-init": "#cloud-boothookn#!/bin/bashn/usr/bin/curl -m20 -s http://<fqdn>/cloud-vendordata-script.sh | /bin/bashn/usr/bin/wget  --timeout 10 --tries 2 -O - http://&lt;fqdn&gt;/cloud-vendordata-script.sh | /bin/bashn"}

and modifying the nova.conf file of the nova api server(s) adding these two lines in the "[api]" section:

vendordata_providers = StaticJSON
vendordata_jsonfile_path = /etc/nova/vendor-data.json

"<fqdn>" in the /etc/nova/vendor-data.json file should be replaced with the fully qualified domain name of your web server hosting the "cloud-vendordata-script.sh" script.

Once a new instance starts, the script published as http://<fqdn>/cloud-vendordata-script.sh is downloaded via curl or wget (the hypothesis we make is that at least one of the two commands is provided) and executed.

The implementation of the cloud-vendordata-script.sh script, to disable weak ssh ciphers, MACs and KexAlgorithms, can be the following one:

#!/bin/bash                                                                                                                          
export SSH_CONFIG=/etc/ssh/sshd_config
export CIPHERS="Ciphers aes128-ctr,aes128-gcm@openssh.com,aes192-ctr,aes256-ctr,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com"
export KEXALGOS="KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1"
export MACS="MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com"
if [ -e /etc/redhat-release ]
then
  export LINUX_DISTRIBUTION=RedHat
else
  export LINUX_DISTRIBUTION=`lsb_release -si`
fi
case $LINUX_DISTRIBUTION in
  RedHat|Ubuntu)
       # Disable system wide macs policy for sshd if is RHEL 9 based distribution
       if grep -q 9 /etc/redhat-release ; then
         if ! grep -q '#MACs' /etc/crypto-policies/back-ends/opensshserver.config ; then
           sed -i 's/^MACs/#MACs/' /etc/crypto-policies/back-ends/opensshserver.config
           logger "SSHD disable system wide macs policy for sshd in RHEL9 based distributions"
         fi
       fi
       # Disable weak ssh ciphers and KexAlgorithms on RedHat systems and restart ssh if needed
       if ! grep -q "${CIPHERS}" ${SSH_CONFIG}
       then
           sed -i 's/^Ciphers/#Ciphers/' ${SSH_CONFIG}
           echo "${CIPHERS}" >> ${SSH_CONFIG}
           echo "" >> ${SSH_CONFIG}
           pidof sshd && service sshd restart
       fi
       if ! grep -q "${KEXALGOS}" ${SSH_CONFIG}
       then
         sed -i 's/^KexAlgorithms/#KexAlgorithms/' ${SSH_CONFIG}
         echo "${KEXALGOS}" >> ${SSH_CONFIG}
         echo "" >> ${SSH_CONFIG}
        pidof sshd && service sshd restart
       fi
       if ! grep -q "${MACS}" ${SSH_CONFIG}
       then
         sed -i 's/^MACs/#MACs/' ${SSH_CONFIG}
         echo "${MACS}" >> ${SSH_CONFIG}
         echo "" >> ${SSH_CONFIG}
        pidof sshd && service sshd restart
       fi
       logger "Vendor data injected on $LINUX_DISTRIBUTION host"
       ;;
  *)
       logger "$LINUX_DISTRIBUTION not managed by this script"
       ;;
esac

The vendordata mechanism can be used also to perform other actions, e.g. to enable the instances to use a central log server, as described in Logging.

Logging

OpenStack logs must be kept for at least 90 days and for a maximum of 1 year.

It is also mandatory to collect and centralize the logs of the virtual machines hosting the INFN-Cloud service instances. They must be kept at least for 6 months.

This can be implemented e.g. relying on the vendordata mechanism already introduced in section Disabling weak SSH ciphers, MACs and KexAlgorithms on images using the vendordata mechanism

The implementation of the vendordata script, to send log instances to a central log server, can be something like:

#!/bin/bash
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
export instance_id="`cat /var/lib/cloud/data/instance-id`"
export LOGSERVER=fdqn.log.server

if [ -e /etc/redhat-release ] && ! grep -q "6." /etc/redhat-release ;
then
  export LINUX_DISTRIBUTION=RedHat
else
  export LINUX_DISTRIBUTION=`lsb_release -si`
fi

case $LINUX_DISTRIBUTION in
  Ubuntu|Debian)
       echo "$template CloudFormat, "%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%n"" > /etc/rsyslog.d/99-cloudveneto.conf
       echo "*.* @${LOGSERVER};CloudFormat" >> /etc/rsyslog.d/99-cloudveneto.conf
       # Restart rsyslog if needed
       pidof rsyslogd && service rsyslog restart
       logger "Vendor data injected on $LINUX_DISTRIBUTION host"
       ;;
  Scientific)
      # New format. Doesn't work for CentOS6
      # echo "template(name="CloudFormat" type="string" string= "%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%n")" > /etc/rsyslog.d/99-cloudveneto.conf
      # Legacy (old) format
       echo "$template CloudFormat, "%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%n"" > /etc/rsyslog.d/99-cloudveneto.conf
       echo "*.* @${LOGSERVER};CloudFormat" >> /etc/rsyslog.d/99-cloudveneto.conf
       # Restart rsyslog if needed
       pidof rsyslogd && service rsyslog restart
       ;;
  RedHat)
      # New format. Doesn't work for CentOS6
      # echo "template(name="CloudFormat" type="string" string= "%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%n")" > /etc/rsyslog.d/99-cloudveneto.conf
      # Legacy (old) format
       echo "$template CloudFormat, "%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%n"" > /etc/rsyslog.d/99-cloudveneto.conf
       echo "*.* @${LOGSERVER};CloudFormat" >> /etc/rsyslog.d/99-cloudveneto.conf
       # Restart rsyslog if needed
       pidof rsyslogd && service rsyslog restart
       logger "Vendor data injected on $LINUX_DISTRIBUTION host"
       ;;
  *)
       logger "$LINUX_DISTRIBUTION not managed by this script"
       ;;
esac

Debugging/Troubleshooting

In this section we report the instructions to be applied for some relevant troubleshooting use cases.

Find the instance that connected to an external server at a given time

Let's suppose that you need to find the instance that connected to an external server at a given time. You will need to get this information in particular in case of a security incident.

If the relevant instance had a floating IP, then please see this section.

If instead the instance had a private fixed IP, then it is not trivial to find this information, since the instance is NAT-ted (usually it uses the address of the OpenStack virtual router, and in general multiple instances refer to the same OpenStack virtual router).

One possible solution is to use the 'conntrack' tool to log the connections initiated from the Cloud instances. For each virtual router you will need to use a command such as this one:

ip netns exec qrouter-1450abf3-5513-4916-aa96-254c66f96888 conntrack -E -e NEW,DESTROY -s 10.63.0.0/16 | logger

where:

  • 1450abf3-5513-4916-aa96-254c66f96888 is the ID of OpenStack router
  • 10.63.0.0/16 represents the instances that need to be monitored

Once you have found the fixed IP of the 'guilty' VM, please see this section to find this VM.

Find the instance associated to a certain floating IP at a given time

Let's suppose that we want to find the instance that was using a certain floating IP (e.g. 90.147.77.102) at a certain time.

Floating IP associations and disassociations are logged in the server.log file of the neutron node:

#  grep "77.102" /var/log/neutron/server.log
2022-04-28 12:39:36.266 5898 INFO neutron.db.l3_db [req-509272a8-093a-4f7e-a573-04dff6d3e846 e237e43716fb490db5bda4b777835669 b38a0dab349e42bdbb469274b20a91b4 - default default] Floating IP 28bf4ef6-6be1-4ac8-9a7e-dab0c52396ea associated. External IP: 90.147.77.102, port: 52e650bf-b163-4bd5-8095-7aea71901c4b.
2022-04-28 13:07:50.976 5909 INFO neutron.db.l3_db [req-a064c6d1-d1c7-4c80-bd17-6153ff846a99 e237e43716fb490db5bda4b777835669 b38a0dab349e42bdbb469274b20a91b4 - default default] Floating IP 28bf4ef6-6be1-4ac8-9a7e-dab0c52396ea disassociated. External IP: 90.147.77.102, port: 52e650bf-b163-4bd5-8095-7aea71901c4b.
2022-04-28 13:08:11.401 5895 INFO neutron.db.l3_db [req-ad02529b-3ea0-40ca-80f8-0558492314fa e237e43716fb490db5bda4b777835669 b38a0dab349e42bdbb469274b20a91b4 - default default] Floating IP 28bf4ef6-6be1-4ac8-9a7e-dab0c52396ea associated. External IP: 90.147.77.102, port: d7b764fc-5f74-4d8c-9571-de3221505483.

Now we need to find the instance associated to the specific port (e.g. d7b764fc-5f74-4d8c-9571-de322150548). If the relevant VM is still alive, you can simply query the neutron database:

mysql> select * from ports where id='d7b764fc-5f74-4d8c-9571-de3221505483';
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+--------------------------------------+--------------+------------------+---------------+
| project_id                       | id                                   | name | network_id                           | mac_address       | admin_state_up | status | device_id                            | device_owner | standard_attr_id | ip_allocation |
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+--------------------------------------+--------------+------------------+---------------+
| b38a0dab349e42bdbb469274b20a91b4 | d7b764fc-5f74-4d8c-9571-de3221505483 |      | 0a1b566c-5479-44dc-8cd8-0e396605b806 | fa:16:3e:c7:de:9b |              1 | ACTIVE | 2303896a-61ec-4128-ace4-63672ffff8ea | compute:nova |          1309804 | immediate     |
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+--------------------------------------+--------------+------------------+---------------+
1 row in set (0.00 sec)

If the VM was deleted in the meantime, the information can be found in the dhcp-agent.log, e.g.:

# grep d7b764fc-5f74-4d8c-9571-de3221505483 /var/log/neutron/dhcp-agent.log

2022-04-28 13:07:11.178 4263 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=, binding:profile=, binding:vif_details=, binding:vif_type=unbound, binding:vnic_type=normal, created_at=2022-04-28T11:07:10Z, description=, device_id=2303896a-61ec-4128-ace4-63672ffff8ea, device_owner=, extra_dhcp_opts=[], fixed_ips=[{'subnet_id': '1f32039f-f426-42f7-8d1a-88cd2862af11', 'ip_address': '10.63.15.74'}], id=d7b764fc-5f74-4d8c-9571-de3221505483, mac_address=fa:16:3e:c7:de:9b, name=, network=admin_state_up=True, availability_zone_hints=[], availability_zones=['nova'], description=None, id=0a1b566c-5479-44dc-8cd8-0e396605b806, ipv4_address_scope=None, ipv6_address_scope=None, mtu=1458, name=AdminTesting-wan, project_id=b38a0dab349e42bdbb469274b20a91b4, provider:network_type=gre, pro...

Another method to see which instance was using a certain floating IP is modifying the neutron database creating some triggers, as described in https://github.com/FranceGrilles/openstack-triggers. Then you can simply query the floatingip_actions table of the neutron database:

mysql> select * from floatingip_actions where ip_address='90.147.77.102';
+-------+---------------+--------------------------------------+--------------+---------------------+
| id    | ip_address    | device_id                            | action       | start_date          |
+-------+---------------+--------------------------------------+--------------+---------------------+
| 17226 | 90.147.77.102 | NULL                                 | create       | 2021-09-23 13:23:21 |
| 17229 | 90.147.77.102 | f23f4685-9d1a-4d91-b389-4f3ba10dfd4f | associate    | 2021-09-23 13:25:11 |
| 17232 | 90.147.77.102 | f23f4685-9d1a-4d91-b389-4f3ba10dfd4f | disassociate | 2021-09-23 13:27:15 |
| 17235 | 90.147.77.102 | f23f4685-9d1a-4d91-b389-4f3ba10dfd4f | associate    | 2021-09-24 17:08:58 |
| 17238 | 90.147.77.102 | f23f4685-9d1a-4d91-b389-4f3ba10dfd4f | disassociate | 2021-09-24 17:10:49 |
| 17241 | 90.147.77.102 | f23f4685-9d1a-4d91-b389-4f3ba10dfd4f | associate    | 2021-09-24 17:12:01 |
| 17244 | 90.147.77.102 | f23f4685-9d1a-4d91-b389-4f3ba10dfd4f | disassociate | 2021-09-24 17:13:29 |
| 17379 | 90.147.77.102 | fbf80aaa-bf42-4038-b5e8-e0889edf25e7 | associate    | 2021-12-02 17:26:13 |
| 17406 | 90.147.77.102 | fbf80aaa-bf42-4038-b5e8-e0889edf25e7 | disassociate | 2022-01-03 13:55:36 |
| 17467 | 90.147.77.102 | f23f4685-9d1a-4d91-b389-4f3ba10dfd4f | associate    | 2022-04-28 12:39:35 |
| 17470 | 90.147.77.102 | f23f4685-9d1a-4d91-b389-4f3ba10dfd4f | disassociate | 2022-04-28 13:07:50 |
| 17473 | 90.147.77.102 | 2303896a-61ec-4128-ace4-63672ffff8ea | associate    | 2022-04-28 13:08:11 |
+-------+---------------+--------------------------------------+--------------+---------------------+

12 rows in set (0.01 sec)

Find the instance associated to a certain fixed (private) IP at a given time

Le'ts suppose you want to find the VM which was using a certain private IP (e.g. 10.63.31.231) at a certain time (that instance could have been deleted in the meantime). :::

First of all, grep for that IP on the syslog of the network node for the relevant period, looking for the DHCP requests:

# grep DHCP.*10.63.31.231 messages-20220102
Dec 26 12:12:26 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 26 12:12:26 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 26 23:17:46 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 26 23:17:46 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 27 09:56:26 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 27 09:56:26 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 27 21:01:04 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 27 21:01:04 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 28 08:02:55 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 28 08:02:55 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 28 18:49:06 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 28 18:49:06 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 29 05:42:14 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 29 05:42:14 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 29 16:20:34 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 29 16:20:34 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 30 03:15:19 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 30 03:15:19 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 30 14:27:16 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 30 14:27:16 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 31 01:04:22 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 31 01:04:22 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 31 12:18:36 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 31 12:18:36 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Dec 31 23:21:04 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Dec 31 23:21:04 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Jan  1 10:26:50 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Jan  1 10:26:50 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231
Jan  1 21:10:03 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPREQUEST(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd
Jan  1 21:10:03 cld-ctrl-01 dnsmasq-dhcp[17579]: DHCPACK(tap3add8c38-53) 10.63.31.231 fa:16:3e:56:a0:cd host-10-63-31-231

This will return a MAC address (fa:16:3e:56:a0:cd in our example).

Querying the nova DB, you can find the UUID of the relevant instance:

mysql> select * from virtual_interfaces where address REGEXP 'fa:16:3e:56:a0:cd';
+---------------------+------------+---------------------+--------+--------------------------------------------------------+------------+-----------------------  ---------------+--------------------------------------+---------+------+
| created_at          | updated_at | deleted_at          | id     | address                                                | network_id | uuid                                   | instance_uuid                        | deleted | tag  |
+---------------------+------------+---------------------+--------+--------------------------------------------------------+------------+-----------------------  ---------------+--------------------------------------+---------+------+
| 2021-09-23 10:10:54 | NULL       | 2022-02-07 15:17:06 | 798143 | fa:16:3e:56:a0:cd/6e970b28-e88f-4bd6-8499-ecf5c52a6600 |       NULL | 6e970b28-e88f-4bd6-8499-ecf5c52a6600 | 9ad1947d-b7e1-43c4-8131-305cfee3ee7c |  798143 | NULL |
+---------------------+------------+---------------------+--------+--------------------------------------------------------+------------+--------------------------------------+--------------------------------------+---------+------+
1 row in set (1.36 sec)

In this example, 9ad1947d-b7e1-43c4-8131-305cfee3ee7c is the UUID of the instance using the IP '10.63.31.231' in the considered time period.

Find the information associated to a VM

If you need to find information about a VM (even if this was cancelled in the meantime) you can simply query the 'instances' table of the nova database, e.g.:

mysql> select display_name,created_at,deleted_at,user_id from instances where uuid='9ad1947d-b7e1-43c4-8131-305cfee3ee7c';
+-------------------------------------------------------+---------------------+---------------------+------------------------------------------------------------------+
| display_name                                          | created_at          | deleted_at          | user_id                                                          |
+-------------------------------------------------------+---------------------+---------------------+------------------------------------------------------------------+
| k8s-slave-server-7f75d9d0-1c56-11ec-8cdf-fa163e5257a7 | 2021-09-23 10:10:42 | 2022-02-07 15:17:08 | 95c417826447f19ef245c49a3054e5d43b42e41ac161c9d7db9a5bce97514447 |
+-------------------------------------------------------+---------------------+---------------------+------------------------------------------------------------------+
1 row in set (0.00 sec)

If you use the 'nova-manage db archive_deleted_rows' to archive the records of the deleted instances, you will need to query the shadow_instances table.