vac.conf

NAME
DESCRIPTION
[SETTINGS] OPTIONS
[MACHINETYPE ...] SECTIONS
USER_DATA SUBSTITUTIONS
SUPERSLOTS AND MULTIPROCESSOR VMs
VACUUM PIPES
AUTHOR
SEE ALSO

NAME

vac.conf − Vac configuration file

DESCRIPTION

vacd is a daemon which implements the Vacuum model on a factory (hypervisor) machine. vacd reads its configuration from /etc/vac.conf and .conf files in /etc/vac.d and these files are also read by the vac utility command to find default values.

The configuration files use the Python ConfigParser syntax, which is similar to MS Windows INI files. The files are divided into sections, with each section name in square brackets. For example: [settings]. Each section contains a series of option=value pairs. Sections with the same name are merged and if options are duplicated, later values overwrite values given earlier.

For ease of management, any configuration file ending in .conf in the directory /etc/vac.d will be read, in alphanumeric order by name, and then /etc/vac.conf is read if present.

[SETTINGS] OPTIONS

The [settings] section is required and has options which apply to all virtual machines.

vac_space is required and gives the name of this Vac space. A single space will be sufficient for many sites. The space name should be a fully qualified domain name, like vac01.example.com and may be used as a virtual CE name in some monitoring and accounting systems. It should not be the canonical hostname of any of the factory machines.

factories is a space separated list of the fully qualified domain names of all the factories in this Vac space, including this factory. The factories are queried using UDP when a factory needs to decide which machinetype to start. The Vac responder process on the factories replies to these queries with a summary of the VM and the outcome of recent attempts to run a VM of each machinetype.

udp_timeout_seconds is how long to wait before giving up on more UDP replies. Defaults to 10.0 seconds.

mb_per_processor sets the memory allocated for each processor in a VM in MiB (1024^2). If enough VMs will underuse memory or KSM is enabled, then it may be more efficient to increase this value and overcommit the factory’s physical memory to prevent the VM kernels from believing they need to swap. Defaults to 2048.

hs06_per_processor gives the HEPSPEC06 power of each processor in a virtual machine. This is used to calculate the total values $MACHINEFEATURES/hs06 and $JOBFEATURES/hs06_job inside the VM, when writing APEL accounting records, and when comparing running VMs to target shares. If not set, the default value 1.0 is used internally, but the $MACHINEFEATURES and $JOBFEATURES values are not set and only the /var/lib/vac/apel-archive copy of the APEL accounting record for each VM is created.

processors_per_superslot sets size of the superslots, as the total number of logical processors which can be allocated to single or multiprocessor VMs in each superslot. See SUPERSLOTS AND MULTIPROCESSOR VMs for more details. Defaults to 1, disabling the superslot mechanism and multiprocessor VMs.

shutdown_time can be set to apply a limit on the shutdown time for all VMs. This value is used if it is ever earlier than the shutdown time calculated from the machinetype’s max_wallclock_seconds. In this case $MACHINEFEATURES/shutdowntime is updated for the VM and Vac will kill the VM if that shutdown time is reached. As shutdown_time is approached, VMs will be created with shorter and shorter lifetimes if their min_wallclock_seconds settings permit. shutdown_time is given in Unix seconds since 00:00:00 UTC on 1st of January 1970.

gocdb_sitename gives the GOCDB site name to use when writing APEL accounting record files to /var/lib/vac/apel-outgoing and /var/lib/vac/apel-archive. Please use your official site name to avoid the risk of misnamed records getting into the central APEL database. If gocdb_sitename is not given, then records are only written to apel-archive and the domain name of the Vac space name is used as a placeholder in the files.

description optionally gives a short description of the space which can be included in logging and monitoring.

version_logger can be set to 0 to disable the logging of the version in use. Normally Vac sends one UDP factory_status message per day to vac-version-logger.gridpp.ac.uk on port 8884 to report the Vac version number. This is used by the Vac developers to target patches for security and bugs at the versions currently in use. Defaults to 1.

vacmon_hostport If set, this option gives a space-separated list of HOST:PORT to send VacQuery UDP messages to. This can be used to monitor the ongoing status of factories and VMs via site or central VacMon services. The central GridPP VacMon service is vacmon.gridpp.ac.uk:8884

total_processors is derived from /proc/cpuinfo by default and does not usually need to be set explicitly. If set, then it provides an additional limit on the number of processors which may be assigned to virtual machines. In turn this limits the number of virtual machines which can be created. This allows you to reduce the number of running VMs on a factory machine without having to kill running VMs. Instead, they are allowed to finish one by one and are only recreated if total_processors allows.

overload_per_processor sets the level of load per processor on the factory machine which will temporarily prevent the creation of more VMs. The number of processors is counted using /proc/cpuinfo and the one minute load average is taken from /proc/loadavg. VMs typically generate high loads during their initial set up, and this mechanism throttles the VM creation rate in response to the current overall load figure. Values of around 2.0 are ok with well-behaved VMs, but the default is more cautious. Default 1.25.

volume_group can be used to set the volume group in which a logical volume will be created for each VM. The logical volumes will have the VMs’ fully qualified domain names as their names. For example, /dev/vac_volume_group/factory1-00.example.com/ would be used by the VM factory1-00.example.com. Defaults to vac_volume_group.

disk_gb_per_processor explicitly sets the size of disks to create for VMs in GB (1000^3). For logical volumes, Vac normally calculates the disk size using the space available to Vac in volume_group, total_processors, and the number of processors allocated to the virtual machine. For file-backed virtual disks, the default value of 40 GB is used unless overriden by disk_gb_per_processor.

root_public_key is the file name of a public key which will be offered to the VM if it exists. For most machinetypes this will allow root ssh access from the VM factory. Default /root/.ssh/id_rsa.pub

fix_networking can be set to False to stop Vac trying to reset the lowlevel networking state if the vac_169.254.0.0 virtual network does not exist and cannot be created with libvirt. /var/log/vacd-factory receives details of what Vac detects and tries to do. Default True.

forward_dev can be used to specify which network interface on the factory will be used for outgoing network connections from the VMs to other machines. For example, eth1. This option is only needed if you have multiple network interfaces on the factory and only one should be used for traffic originating from the VMs.

user_data_option_XXX and user_data_file_XXX are locally defined substitutions which will be applied to all machinetypes’ user_data files, unless the value is overridden by a per-machinetype value elsewhere in the configuration. See [MACHINETYPE ...] SECTIONS for the full syntax of these options.

The VM names are formed by adding a hyphen and the VM number to the hostname component of its fully qualified domain name. For example, factory1.example.com would have factory1-00.example.com, factory1-01.example.com, ... as its VMs. MAC addresses are formed with the prefix 56:4D as the first two bytes, and the four bytes of the IP address as the remaining four bytes.

[MACHINETYPE ...] SECTIONS

One [machinetype ...] section must exist for each machinetype in the system, with the name of the machinetype given in the section name, such as [machinetype example]. A machinetype name must only consist of lowercase letters, numbers, and hyphens. Each of these sections contain option=value pairs that are specific to that machinetype. Typically, only taget_share and vacuum_pipe_url are required.

target_share gives the desired share of the total VMs available in this space for this machinetype. The shares do not need to add up to 1.0, and if a share is not given for a machinetype, then it is set to 0. Vac factories consult these shares when deciding which machinetype to start as VMs become available.

vacuum_pipe_url gives the HTTP(S) URL of a remote Vacuum Pipe file supplied by the VO which contains details of the boot image, contextualization, and other options needed to create VMs of this machinetype.

The following options may be given in the Vac configuration to override defaults or values set in the Vacuum Pipe file. This should not be required in normal operation.

machines_dir_days sets the expiration time in days for per-VM directories created under /var/lib/vac/machines. Default 3.

backoff_seconds is the delay after a VM of this machinetype aborts. If a VM aborts, then no new VMs of this type will be created for this amount of time. This can be used to prevent the unnecessary creation of many VMs when no work is available, and avoid overloading the matcher or task queue of the VO.

fizzle_seconds is used in three places within the backoff procedure and in two other parts of Vac:
(1) First, if a VM finishes without producing a shutdown message code and has lasted less than fizzle_seconds, then it is treated as aborted.
(2) Secondly, after the backoff_seconds time has expired for a VM abort, once at least one VM has been started in this Vac space, then no more new VMs can be started for another fizzle_seconds.
(3) Thirdly, these new VMs are identified because they are still in the starting phase of creating files, or because they have been running for less than fizzle_seconds.
(4) Additionally, when writing the accounting log files, any VMs which run for less than fizzle_seconds are excluded.
(5) Finally, the heartbeat file checking is only carried out once an initial period of fizzle_seconds has passed.

max_wallclock_seconds gives the maximum lifetime of a VM. Vac will set $MACHINEFEATURES/shutdowntime for the VM using this value to communicate it to the VM. Vac will destroy the VM if it is still running after this amount of time. Default 86400.

min_wallclock_seconds gives the minimum remaining time required when creating a VM. This can be used to stop Vac creating VMs with short lifetimes when shutdown_time has been set or when building superslots. Default max_wallclock_seconds.

min_processors and max_processors give the minimum and maximum number of logical processors which can be allocated to VMs of this type when they are created.

accounting_fqan is used to specify a FQAN to include when writing APEL accounting records, to associate usage with particular experiments.

machine_model is required and tells Vac how to configure the virtual hardware seen by the VMs of this machinetype. Currently cernvm3 or vm-raw. Default cernvm3.

heartbeat_file allows the machinetype to nominate a file which will be created in $JOBOUTPUTS before fizzle_seconds has passed. If this file is not created by then and maintained for the lifetime of the VM, the VM will be destroyed.

heartbeat_seconds gives the frequency at which the heartbeat_file must be updated after fizzle_seconds has passed. If the file is not updated for heartbeat_seconds then the VM will be destroyed. If heartbeat_seconds is 0, then only the existence of the file will be checked. Default 0.

image_signing_dn is used to specify a regular expression to match the DN of an X.509 certificate used to verify the authenticity of the root image. Vac attempts to obtain the certificate and signature from a CernVM Signature Block at the end of the image file, verifies the certificate using the CA files in /etc/grid-security/certificates, and compares the certificate DN to image_signing_dn. If this option is given, all these verification steps must be satisified for the image to be used. As of 2016, CernVM images are signed with a DN matching the regular expression /CN=cvm-sign01\.cern\.ch$

root_device is the device name exposed to the VM that is associated with the root disk image. Default vda.

scratch_device is the device name exposed to the VM that is associated with a scratch logical volume in the vm-raw model. Ignored for CernVM. Default vdb.

legacy_proxy can be set to True to generate Globus legacy proxies rather than RFC 3820 proxies. Default False.

user_data_proxy set to true causes the files x509cert.pem and x509key.pem in the machinetype’s subdirectory of /var/lib/vac/machinetypes to be used to make a limited X.509 proxy. The two files can be identical if desired, and the X.509 certificate and RSA private key will be extracted from the files as appropriate. (Note that this location is one level above the files subdirectory in which the following options look by default.)

For the remaining options, if the file name begins with ’/’, then it will be used as an absolute path; otherwise the path will be interpreted relative to the files subdirectory of the machinetype’s subdirectory of /var/lib/vac/machinetypes . For values supplied in a remote Vacuum Pipe file, only filenames without

root_image is the path to the image file from which the VM will boot. With the cernvm3 machine_model, this can also be a remote HTTP or HTTPS URL which Vac will cache in /var/lib/vac/imagecache. The remote server must supply a Last-Modified timestamp and Vac will re-request the image each time a VM starts using an If-Modified-Since request to minimise network load. Alternatively, the images may be files in the local filesystem. With cernvm3 machine_model, the files are ISO CDROM-style boot images; with the cernvm2 machine_model, they are the root hard disk image itself.

user_data is the path of a contextualization file provided by the VO and perhaps modified by the site. If the path is a remote HTTP or HTTPS URL, Vac will fetch it over the network each time a VM is started. However the file is obtained, Vac will apply a series of default and locally defined ##user_data___## substitutions to it. See USER_DATA SUBSTITUTIONS below for a list of the default substitutions.

user_data_option_XXX and user_data_file_XXX are locally defined substitutions which will be applied to the user_data file before the VM is started. user_data_option_XXX takes the string to be substituted. user_data_file_XXX takes the relative or absolute path to a file whose contents will be substituted for the pattern in the user_data file.

USER_DATA SUBSTITUTIONS

Before the user_data file is used in starting a VM, several pattern based substitutions are performed by Vac. These patterns are in the form ##user_data___##. String values given to the option user_data_option_XXX replace patterns of the form ##user_data_option_XXX##. The contents of the files given to user_data_file_XXX options also replace patterns of the form ##user_data_option_XXX##. In both cases XXX are arbitrary strings consisting of letters, numbers, and underscores.

The pattern ##user_data_option_x509_proxy## is replaced by the proxy created if the user_data_proxy_cert and user_data_proxy_key options are given.

In addition, the following substitutions are performed automatically by Vac using data it holds internally:

##user_data_uuid## is the UUID assigned to the VM by Vac.
##user_data_space## is the Vac space name.
##user_data_url## is the HTTP(S) from which the user_data template was obtained. Only given if the template was retrieved by HTTP(S) rather from a local path.
##user_data_machinefeatures_url## and ##user_data_jobfeatures_url## and ##user_data_joboutputs_url## are the values of $MACHINEFEATURES, $JOBFEATURES, and $JOBOUTPUTS to set within the VM.
##user_data_machinetype## and ##user_data_vmtype## (deprecated) are the name of the machinetype of this VM.
##user_data_machine_hostname## and ##user_data_vm_hostname## (deprecated) are the hostname given to the VM by Vac.
##user_data_manager_version## and ##user_data_vmlm_version## (deprecated) have the form "Vac v.v.v" where v.v.v is the Vac version.
##user_data_manager_hostname## and ##user_data_vmlm_hostname## (deprecated) are the hostname of the Vac factory machine.

SUPERSLOTS AND MULTIPROCESSOR VMs

By setting processors_per_superslot in [settings] to a value greater than one, Vac will attempt to create VMs in groups with the same finishing time. This causes groups of processors to become available at the same time which enables the creation of VMs which require multiple virtual CPUs. When creating these VMs, the max_processors and min_processors values from the relevant machinetype section determine the VM’s requirements. processors_per_superslot also limits the largest number of processors which may be assigned to a single VM (which will occupy a whole superslot.) The min_wallclock_seconds value is used to determine whether there is sufficient time left to create a VM of that machinetype. max_wallclock_seconds determines whether a sufficiently long-lived VM can be created to match the superslot.

VACUUM PIPES

If vacuum_pipe_url is given within a machinetype as described above, then the corresponding file is fetched via HTTP(S) and used to obtain default values supplied by the VO associated with that machinetype. The following options are supported within Vacuum Pipe files: accounting_fqan, backoff_seconds, fizzle_seconds, heartbeat_file, heartbeat_seconds, image_signing_dn, legacy_proxy, machine_model, max_processors, max_wallclock_seconds, min_processors, min_wallclock_seconds, root_device, root_image, scratch_device, user_data, user_data_option_XXXX, user_data_file_XXXX, user_data_proxy. In addition cache_seconds in the Vacuum Pipe file sets the maximum time the file may be cached, and defaults to 3600 seconds if not set or the file has never been fetched successfully. As explained above, options referring to files on the VM factory may not specify filesystem paths if obtained from Vacuum Pipe file: only filenames within the /var/lib/vac/machinetypes/MACHINETYPE/files directory are acceptable.

AUTHOR

Andrew McNab <Andrew.McNab@cern.ch>

More about Vac: http://www.gridpp.ac.uk/vac/