vcycle.conf

NAME
DESCRIPTION
[SPACE ...] SECTIONS
OPENSTACK SPACE SECTIONS
EC2 SPACE SECTIONS
MACHINE / JOB FEATURES HTTP(S) SERVER
[MACHINETYPE ... ...] SECTIONS
USER_DATA SUBSTITUTIONS
AUTHOR
SEE ALSO

NAME

vcycle.conf − vcycle configuration file

DESCRIPTION

vcycled is a daemon which implements VM lifecycle management on IaaS Cloud systems in a way inspired by Vac. vcycled reads its configuration from /etc/vcycle.conf and .conf files in /etc/vcycle.d

The configuration files use the Python ConfigParser syntax, which is similar to MS Windows INI files. The files are divided into sections, with each section name in square brackets. Each section contains a series of option=value pairs. Sections with the same name are merged and if options are duplicated, later values overwrite values given earlier.

For ease of management, any configuration file ending in .conf in the directory /etc/vcycle.d will be read, in alphanumeric order by name, and then /etc/vcycle.conf is read if present.

[SPACE ...] SECTIONS

One [space ...] section must exist for each tenancy, project, or account in which VMs will be created, with the Vcycle space name for the space given in the section name, such as [space project1.example.com]. The Vcycle admin has a free choice of what names to give each space, but the name must only consist of lowercase letters, numbers, periods, and hyphens so it can be used as the DNS name of a virtual CE elsewhere in the system.

api is required and specifies the lowercase name of the API to use when contacting the IaaS service associted with this space.

max_machines prevents new machines from being created if the number of machines for this space in any state exceeds the given limit.

gocdb_sitename gives the GOCDB site name to use when writing APEL accounting record files to /var/lib/vcycle/apel-outgoing and /var/lib/vcycle/apel-archive. Please use your official site name to avoid the risk of misnamed records getting into the central APEL database. If gocdb_sitename is not given, then records are only written to apel-archive and the Vcycle space name is used as a placeholder in the files.

https_port gives the port number used to contact the Vcycle HTTPS server from within the VMs. This only needs to be changed if there is an intervening firewall. Default 443.

OPENSTACK SPACE SECTIONS

OpenStack spaces are enabled with api = openstack and the following options are then available.

url is the URL of the identity (KeyStone) endpoint for this OpenStack service.

tenancy_name is the tenancy or project name.

username is a user with access to the tenancy.

password_base64 is the base64 encoded password for the user. Vcycle will decode the value to obtain the true password. This encoding is to avoid revealing memorable passwords when casually viewing or editing configuration files. You can encode the real password with something like:

echo -n ’PASSword’ | base64

When creating VMs in OpenStack spaces, Vcycle will create "machinefeatures", "jobfeatures", and "joboutputs" metadata keys with the URLs of the corresponding directories for the VM on the Vcycle machine’s HTTP(S) server.

EC2 SPACE SECTIONS

As well as OpenStack, Vcycle natively supports the EC2 API. However, you should use the OpenStack API if available. Spaces accessed with the EC2 API are enabled with api = ec2 and the following options are then available.

url is the URL of the endpoint of this EC2 service.

access_key and secret_key are the credentials needed to access the service.

version explicitly gives the API version to be passed to EC2. Default "2010-08-31".

region explicitly gives the region to be passed to EC2. Default "us-east-1".

service explicitly gives the service name to be passed to EC2. Default "openstack".

When creating VMs in EC2 spaces, Vcycle will create "machinefeatures", "jobfeatures", and "joboutputs" metadata tags with the URLs of the corresponding directories for the VM on the Vcycle machine’s HTTP(S) server.

MACHINE / JOB FEATURES HTTP(S) SERVER

Vcycle will create the machinefeatures and jobfeatures files according to the WLCG MJF protocol described in HSF-TN-2016-02 in machinefeatures and jobfeatures subdirectories of the VMs directory in /var/lib/vcycle/machines . Using the vcycle.httpd.inc Apache configuration file supplied in /usr/share/doc/vcycle-VERSION, these files are publish via HTTP and HTTPS.

In addition, if x509dn is set for the VM’s machinetype, then the script vcycle-cgi will allow the VM to write heartbeat, logging, and shutdown_message files to the VM’s $JOBOUTPUTS directory.

[MACHINETYPE ... ...] SECTIONS

One [machinetype ... ...] section must exist for each machinetype in each space, with the space name and name of the machinetype given in the section name, such as [machinetype project1.example.com example]. A machinetype name must only consist of lowercase letters, numbers, and hyphens. Each of these sections contain option=value pairs that are specific to that machinetype. The same machinetype name can appear in different spaces and will be managed separately.

flavor_name gives the name of a flavor previously defined in the IaaS system which represents a particular CPU, memory, and disk geometry.

hs06 gives the total HEPSPEC06 power of each virtual machine created for this flavor in this machinetype. If set, this is used as the value $MACHINEFEATURES/hs06 and $JOBFEATURES/hs06_job supplied to the VM. It is also used when calculating target shares and when writing APEL accounting records, and for both of these a default of 1.0 is used if not set explicitly.

cpu_per_machine gives the number of processors of each virtual machine created for this flavor in this machinetype. This is used as the value of $MACHINEFEATURES/total_cpu and $JOBFEATURES/allocated_cpu supplied to the VM. If the api plugin can positively determine the number from metadata about the flavor, it will be used in preference to the value given here. Default 1.

target_share gives the desired share of the capacity available in this space for this machinetype. The shares do not need to add up to 1.0, and if a share is not given for a machinetype, then it is set to 0. Vcycle consults these shares when deciding which machinetype to start as VM capacity becomes available. Shares are weighted by the hs06 value of the machinetype.

backoff_seconds is the delay after a VM of this machinetype aborts. If a VM aborts, then no new VMs of this type will be created for this amount of time. This can be used to prevent the unnecessary creation of many VMs when no work is available, and avoid overloading the matcher or task queue of the VO.

fizzle_seconds is used in three places within the backoff procedure and in two other parts of Vcycle:
(1) First, if a VM finishes without producing a shutdown message code and has lasted less than fizzle_seconds, then it is treated as aborted.
(2) Secondly, after the backoff_seconds time has expired for a VM abort, once at least one VM has been started in this Vcycle space, then no more new VMs can be started for another fizzle_seconds.
(3) Additionally, when writing the accounting log files, any VMs which run for less than fizzle_seconds are excluded.
(5) Finally, the heartbeat file checking is only carried out once an initial period of fizzle_seconds has passed.

accounting_fqan is used to specify a FQAN to include when writing APEL accounting records, to associate usage with particular experiments.

max_machines prevents new machines from being created if the number of machines for this machinetype in any state exceeds the given limit.

max_wallclock_seconds gives the maximum lifetime of a VM. Vcycle will create $MACHINEFEATURES/shutdowntime inside the VM using this value to communicate it to the VM. Vcycle will destroy the VM if it is still running after this amount of time. Default 86400.

heartbeat_file allows the machinetype to nominate a file which will be created in the $JOBOUTPUTS directory before fizzle_seconds has passed. If this file is not created by then and maintained for the lifetime of the VM, the VM will be destroyed.

heartbeat_seconds gives the frequency at which the heartbeat_file must be updated after fizzle_seconds has passed. If the file is not updated for heartbeat_seconds then the VM will be destroyed. If heartbeat_seconds is 0, then only the existence of the file will be checked. Default 0.

x509dn is an optional X.509 DN which will be used by the vcycle-cgi script to control writing to VMs’ $JOBOUTPUTS directories on the local HTTPS server.

log_joboutputs can be set to True to enable recording of all the files from local $JOBOUTPUTS directories for VMs, to subdirectories of /var/lib/vcycle/joboutputs when the VMs finish or are killed. The subdirectories are in a hierarchy of the space name, machinetype name, and then hostname of the VM. Default False.

joboutputs_days sets the expiration time in days for per-VM directories created under /var/lib/vcycle/joboutputs.

remote_joboutputs_url sets a base URL on a remote HTTPS server to which VMs of this machinetype can write. The value of $JOBOUTPUTS will be the VM name chosen by Vcycle appended as a directory name to the URL given by this option.

For the remaining options, if the file name begins with ’/’, then it will be used as an absolute path; otherwise the path will be interpreted relative to the machinetype’s subdirectory of /var/lib/vcycle/spaces/SPACE/machinetypes/MACHINETYPE/files where SPACE is the parent space name and MACHINETYPE is the name of this machinetype.

remote_joboutputs_cert and remote_joboutputs_key give filesnames of an X.509 client certificate and key to use when requesting $JOBOUTPUTS/shutdown_message and any heartbeat file in $JOBOUTPUTS. If both are contained in the same file then the same value can be given to both options.

root_image identifies the image file from which the VM will boot. If the cloud service already has the desired image, then it can be referenced by prefixing the service’s native image ID with "image:".
For the OpenStack API, root_image can be the path to the image file itself on the local filesystem. Alternatively, it can also be a remote HTTP or HTTPS URL which Vcycle will cache in /var/lib/vcycle/imagecache. The remote server must supply a Last-Modified timestamp and Vcycle will re-request the image each time a VM starts using an If-Modified-Since request to minimise network load. Alternatively, the images may be files in the local filesystem. If root_image ends in .iso , then the image will be declared as ISO format (a CD-ROM image), otherwise as a raw HDD image.

cernvm_signing_dn is used to specify a regular expression to match the DN of an X.509 certificate used to verify the authenticity of the root image. Vcycle attempts to obtain the certificate and signature from a CernVM Signature Block at the end of the image file, verifies the certificate using the CA files in /etc/grid-security/certificates, and compares the certificate DN to cernvm_signing_dn. If this option is given, all these verification steps must be satisified for the image to be used. As of 2016, CernVM images are signed with a DN matching the regular expression /CN=cvm-sign01\.cern\.ch$

root_public_key is the file name of a public key which Vcycle will set up on the IaaS system and supply to the VMs to allow root ssh access. Setting this option to /root/.ssh/id_rsa.pub will give access from the factory machine.

user_data is the path of a contextualization file provided by the VO and perhaps modified by the site. If the path is a remote HTTP or HTTPS URL, Vcycle will fetch it over the network each time a VM is started. However the file is obtained, Vcycle will apply a series of default and locally defined ##user_data___## substitutions to it. See USER_DATA SUBSTITUTIONS below for a list of the default substitutions.

user_data_option_XXX and user_data_file_XXX are locally defined substitutions which will be applied to the user_data file before the VM is started. user_data_option_XXX takes the string to be substituted. user_data_file_XXX takes the relative or absolute path to a file whose contents will be substituted for the pattern in the user_data file.

user_data_proxy_cert and user_data_proxy_key are the locations of files containing X.509 certificate(s) and an RSA private key, all in PEM format, which will be used to make a limited X.509 proxy. The same file can be given for both options if desired.

legacy_proxy can be set to True to generate Globus legacy proxies rather than RFC 3820 proxies. Default False.

USER_DATA SUBSTITUTIONS

Before the user_data file is used in starting a VM, several pattern based substitutions are performed by Vcycle. These patterns are in the form ##user_data___##. String values given to the option user_data_option_XXX replace patterns of the form ##user_data_option_XXX##. The contents of the files given to user_data_file_XXX options also replace patterns of the form ##user_data_option_XXX##. In both cases XXX are arbitrary strings consisting of letters, numbers, and underscores.

The pattern ##user_data_x509_proxy## is replaced by the proxy created if the user_data_proxy_cert and user_data_proxy_key options are given.

In addition, the following substitutions are performed automatically by Vcycle using data it holds internally:

##user_data_space## is the Vcycle space name.
##user_data_machinetype## and ##user_data_vmtype## (deprecated) are the name of the machinetype of this VM.
##user_data_machine_hostname## and ##user_data_vm_hostname## (deprecated) are the hostname given to the VM by Vcycle.
##user_data_manager_version## and ##user_data_vmlm_version## (deprecated) have the form "Vcycle v.v.v" where v.v.v is the Vcycle version.
##user_data_manager_hostname## and ##user_data_vmlm_hostname## (deprecated) are the hostname of the machine on which the Vcycle daemon is running.

AUTHOR

Andrew McNab <Andrew.McNab@cern.ch>

vcycled is part of Vcycle: https://www.gridpp.ac.uk/vcycle/