VacProject       Vac       Vac-in-a-Box       Vcycle       VacMon       GridPP      

Vcycle Admin Guide 01.00pre

Andrew McNab <Andrew.McNab AT cern.ch>

Vcycle is a VM factory system which implements the Vacuum Platform described in HSF-TN-2016-04, using a cloud service such as OpenStack to carry out its VM instantiation decisions.

Contents

Quick start

By following this quick start recipe you can verify that your cloud service will work with Vcycle and see it creating and destroying virtual machines.

To follow the quick start, we assume you have access to an OpenStack service and a Scientific Linux 6 (or CentOS 6 or RHEL 6) machine to use as a VM factory on which to run Vcycle. This factory machine can itself be a VM, but don't choose a name which starts with "vcycle-" . You must be able to perform Nova VM management operations from the command line on this machine. This may require changes to the OpenStack and/or local firewall configuration. Make sure you can list VMs in the tenancy/project you plan to use, whether they are created with the Horizon web dashboard or from the command line too.

Then install the vcycle RPM. This installs a copy of this guide and some example files in /usr/share/doc/vcycle-VERSION and the vcycle.conf man page in the usual place.

Copy /usr/share/doc/vcycle-VERSION/example.vcycle.conf into /etc/vcycle.d the Vcycle configuration directory. You don't need to change the name, and it must end in ".conf" .

Make sure that root on the VM factory machine has an RSA ssh keypair. Run the command ssh-keygen if /root/.ssh/id_rsa does not exist already. This key pair will be authorized to ssh into the VMs which is very useful for debugging. You may need to copy the keypair to another machine if the factory is not allowed ssh access to the OpenStack VMs. If this is completely barred, then production use of Vcycle is still possible as this feature is just used for debugging.

Change the following values in the [space vcycle01.example.com] section of example.vcycle.conf:

url
This is the KeyStone URL for your OpenStack service. This is given by the Identity field when looking at the API Access tab of Access &Security in the Horizon web dashboard.
tenancy_name
Is the OpenStack tenancy or project name. Spaces are allowed and quotes are not needed (or allowed).
username
Is the username for API access.
password_base64
Is the corresponding password, encoded in Base64. You can do this with something like: echo -n 'PASSword' | base64
max_machines
If there are already VMs in the tenancy/project, then you should increase this from 1 to one more than the number of existing VMs. Otherwise leave it at 1 for now.

The next options are for the the section [machinetype vcycle01.example.com example]

flavor_name
The flavor of the VMs which will be created.
user_data_option_cvmfs_proxy
It should be set to the URL of an HTTP cache you have access to. If you are already using cvmfs for grid worker nodes, you can use the same value. CernVM will mostly work without this option so you can leave it commented out for now. However, if you go into production without a cache set, you will get random filesystem errors!

Now start vcycle with service vcycled start

You will see messages appear in /var/log/vcycled.log and you see a VM appear in your OpenStack tenancy once a message saying Vcycle has created a VM is logged. If you are able to ssh to VMs, then you should be able to ssh into the VM.

If everything is working, you can see from the log file how a CernVM image is fetched from https://repo.gridpp.ac.uk/vacproject/example/cernvm3.iso and uploaded to OpenStack, and a user_date template is fetched from https://repo.gridpp.ac.uk/vacproject/example/user_data and used to create the VM.

If you look in the subdirectories of /var/lib/vcycle/machines you will see files used to create the VM and record its state changes.

Configuration step-by-step

This part of the guide covers the same ground as the quick start guide but in a lot more detail. It's intended to help you choose how best to configure your site.

The configuration files /etc/vcycle.conf and /etc/vcycle.d/*conf use the Python ConfigParser syntax, which is similar to MS Windows INI files. The files are divided into sections, with each section name in square brackets. For example: [space vcycle01.example.com]. Each section contains a series of option=value pairs. Values can continue onto the next line by starting that line with some white space. Sections with the same name are merged and if options are duplicated, later values overwrite values given earlier. Any configuration file ending in .conf in the directory /etc/vcycle.d is read. These files are read in alphanumeric order, and then /etc/vcycle.conf is read if present.

Based on this ordering in /etc/vcycle.d/, options from space.conf would override any given in site.conf, but themselves be overwritten by options from vm.conf .

You may find it convenient to use multiple configuration files if you have a complicated setup. For instance: space01.conf, space01-machinetype01.conf, space01-machinetype02.conf, space02.conf, space02-machinetype01.conf each containing one [space ...] or [machinetype ... ...] section.

Apache httpd

Vcycle uses an Apache httpd running on the VM factory to publish Machine/Job Features values according to HSF-TN-2016-02 and to receive $JOBOUTPUTS files according to HSF-TN-2016-04 which is essential for VMs using the heartbeat and shutdown_message mechanisms.

You can meet this requirement by installing the standard httpd RPM from SL6 (or CentOS 6 or RHEL 6) and copying two files from /usr/share/doc/vcycle-VERSION:

  • vcycle.httpd.conf to /etc/httpd/conf/httpd.conf
  • vcycle.httpd.inc to /etc/httpd/includes/vcycle.httpd.inc
and then (re)start httpd. You must ensure that firewall rules allow access to TCP port 443 on the VM factory from the VMs.

If this port access is not possible, then you should replace 443 in both configuration files with a suitable value, and set the https_port option in the [space ...] sections. This will cause the $MACHINEFEATURES, $JOBFEATURES, and $JOBOUTPUTS URLs to include this port number. See the vcycle.conf man page for more details.

CernVM images

Any ISO or HDD image which is compatible with your OpenStack service shoudl also work with Vcycle. However, Vcycle works best with images which look for a user_data contextualization file, and CernVM boot images are particularly convenient due to their very small size.

If you need to download an image, they can be found on the CernVM downloads page.

However, most experiments will supply you with their own URL from which Vcycle can automatically fetch their current designated image version using the user_data option of the [machinetype ... ...] section, which Vcycle caches in /var/lib/vcycle/imagecache .

Installation of Vcycle: tar vs RPM

RPM is the recommended installation procedure, and RPMs are available from the Vcycle downloads area.

It is also possible to install Vcycle from a tar file, using the install Makefile target.

Configuration of the Vcycle space

Each [space ...] section must include a space name, which is also used as the virtual CE name. The section name is of the format [space vcycle01.example.com] .

The space must have a value for the option api which defines how to talk to the service (such as api = openstack), and then other API specific options. For OpenStack, these are url, tenancy_name, username, and password_base64.

See the vcycle.conf man page for more details.

GOCDB and GGUS

Vcycle is designed to work within the WLCG/EGI grid model of sites composed of one or more CEs. Each Vcycle space name corresponds to one CE within a site, and can co-exist with conventional CREAM or ARC CEs. If you are at a site registered in the GOCDB, you should add your space name(s) to your site in GOCDB as services. There is a registered service type (uk.ac.gridpp.vcycle) for Vcycle spaces.

Problems encountered during the operation of Vcycle in production may appear as tickets in GGUS. The Vac/Vcycle Support Unit appears under "Second Level - Software" on the GGUS "Assign ticket to support unit" menu.

Vcycle writes APEL accounting records as described below. The GOCDB site name given by gocdb_sitename in the [space ...] section is included in these records. To avoid the risk of polluting the central APEL database with incorrect site names, please use your real GOCDB sitename for this option.

Setting up machinetypes

One [machinetype ... ...] section must exist for each machinetype in the system, with the name of the space and machinetype given in the section name, such as [machinetype vcycle01.example.com example] . A machinetype name must only consist of lowercase letters, numbers, and hyphens. The vcycle.conf man page lists the options that can be given for each machinetype.

The target_share option for the machinetype gives the desired share of the total VMs available in this space for that machinetype. The shares do not need to add up to 1.0, and if a share is not given for a machinetype, then it is set to 0. The creation of new VMs can be completely disabled by setting all shares to 0. Vcycle consults these shares when deciding which machinetype to start as VM slots become available.

For ease of management, the target_shares options can be grouped together in a separate file in /etc/vcycle.d apart from the main [machinetype ... ...] sections, which is convenient if shares are generated automatically or frequently edited by hand. For example: 

[machinetype vcycle01.example.com example1]
target_share = 5.0
[machinetype vcycle01.example.com example2]
target_share = 6.0
[machinetype vcycle01.example.com example3]
target_share = 7.0

The experiment or VO responsible for each machinetype should supply step by step intructions on how to set up the rest of the [machinetype ... ...] section and how to create any hostcert.pem and hostkey.pem pair to give to the VM.

Starting and stopping vcycled

The Vcycle daemon, vcycled, is started and stopped by /etc/rc.d/init.d/vcycled on conjunction with the usual service and chkconfig commands. As the configuration files are reread at the start of each cycle (by default, one per minute) it is not necessary to restart vacd after changing the configuration.

Furthermore, as vcycled rereads the current state of the VMs from status files and the cloud service at the start of each cycle, vcycled can be restarted without disrupting running VMs or losing information about their state. In most cases it will even be possible to upgrade vcycled from one patch level to another within the same minor release without having to drain the site of running VMs. If problems arise during upgrades, the most likely outcome is that Vcycle will fail to create new VMs until the configuration is fixed, but the existing VMs will continue to run. ("We want Vcycle failures to look like planned draining.")

APEL accounting

When Vcycle detects that a VM has run for at least fizzle_seconds and now finished, it writes a copy of the APEL accounting message to subdirectories of /var/lib/vcycle/apel-archive . If you have set gocdb_sitename in [space ...], then the file is also written to /var/lib/vcycle/apel-outgoing .

Vcycle uses the UUID of the VM as the local job ID, the VM factory hostname as the local user ID, and the machinetype name as the batch queue name. A unique user DN is constructed from the components of the Vcycle space name. For example, vac01.example.com becomes /DC=com/DC=example/DC=vac01 . If the accounting_fqan option is present in the [machinetype ... ...] section, then for VMs of that type the value of that option is included as the user FQAN, which indicates the VO associated with the VM. The GOCDB sitename field is either the value you gave explicitly or the Vcycle space name as a placeholder.

These accounting messages are designed to be published to the central APEL service using the standard APEL ssmsend command, which can be run on each factory machine from cron. Please see the APEL SSM client documentation for details. To submit records you should agree use of APEL with the APEL team, have your certificate authorized, set up the correct APEL entries in GOCDB, and do any requested tests.

Setting fizzle_seconds and backoff_seconds

For each machinetype you can set values of fizzle_seconds and backoff_seconds to make the most efficient use of your resources when deciding to start VMs.

You may be able to optimize the value of fizzle_seconds by looking for VMs which fail to find work and seeing how long they ran for. However, for most VM architectures and VM factories a value of 600 seconds will work perfectly well and you do not need to spend a lot of effort arriving at an ideal number.

The value of backoff_seconds is a matter for your site policy about how quickly to recover from periods when each experiment has no work available. If you know a particular experiment usually has plenty of work, then you could set a low value of 600 seconds, so that Vcycle will try creating one VM every 10 minutes or so to see if work is available again after a short idle period. Alternatively, if you know that an experiment usually has no work for you, then you could set much larger values of many hours between creating VMs. In either case, once Vcycle identifies that VMs for the experiment are indeed passing fizzle_seconds and finding work to do, then the backoff_seconds option is ignored and VMs are created as VM slots become free in line with the machinetype's target_share.