Difference between revisions of "Manchester Multicore Torque Configuration"

Revision as of 14:10, 31 July 2014

Dynamic Partitioning with Nikhef scripts

This method relies on a custom python script - wirtten by Jeff templon - to create dynamic partitions in torque. It needs a specific queue for mcore jobs to be created. It also requires to change the (so far) standard properties of the nodes as it uses the resources_default.neednodes queue attribute to create the partitions. I eventually put everything in puppet but below are the basic steps to install.

Scripts Installation

You can download the scripts from the Nikhef SVN repository. There is the main script *mcfloat* and 3 python modules to install. I've opted to install everything in /usr/local. The script will also require a $HOME/tmp directory for a file used to contain the list of nodes that are too empty and should be moved back to the single core partition.

Python Modules

mkdir -p $HOME/tmp
mkdir -p /usr/local/lib/python2.6/site-packages
cd /usr/local/lib/python2.6/site-packages 
wget https://ndpfsvn.nikhef.nl/cgi-bin/viewvc.cgi/pdpsoft/trunk/nl.nikhef.ndpf.tools/pjobstats/"torqueJobs.py?revision=2698" -O torqueJobs.py
wget https://ndpfsvn.nikhef.nl/cgi-bin/viewvc.cgi/pdpsoft/trunk/nl.nikhef.ndpf.tools/pjobstats/"torqueAttMappers.py?revision=2526" -O torqueAttMappers.py
wget https://ndpfsvn.nikhef.nl/cgi-bin/viewvc.cgi/pdpsoft/nl.nikhef.pdp.dynsched-pbs-plugin/trunk/torque_utils.py?revision=2028 -O torque_utils.py

• NOTE 1: I've put the wget command to make it easy but of course to do that the version has to be set. If you want to make sure you have the latest version you'll need to check it via your broser and download that.
• Note 2: With python2.6 torque_utils.py gives a syntax error and you need to move the line below above the version line or remove it as it is apparently only needed in python 2.2.

from __future__ import generators # only needed in Python 2.2

• NOTE 3: /usr/local/lib/python2.6/site-packages will need to be added to the PYTHONPATH. I've done it in the cron job at the end to avoid adding an extra profile script.

mcfloat

cd /usr/local/bin
wget https://ndpfsvn.nikhef.nl/cgi-bin/viewvc.cgi/pdpsoft/nl.nikhef.ndpf.mcfloat/trunk/"mcfloat?revision=2708" -O mcfloat

You need to edit mcfloat to set 5 things

• Torque server

TORQUE              = "<torque-server-fqdn>"

• Initial set of WN to use my nodes don't have sequential names so I replaced the elegant for loop to build the array with a plainer comma separated list of nodes names. Do not forget the quotes around the names.

CANDIDATE_NODES = [ 'node-0%02d.domain' % (n) for n in range(1,19) ]

• Queue name you can leave it or replace it. I've replace it with a less experiment oriented name

MCQUEUE  = 'atlasmc'

• MAXDRAIN and MAXFREE these depend on the size of your nodes and cluster, you may want to play with it. I reduced the number of MAXDRAIN to 4 for example but I'm probably going to be review it to tune it.

MAXDRAIN = 7             # max num of nodes allowed to drain
MAXFREE  = 49            # max num of free slots to tolerate

Torque settings

nodes properties

mcfloat will rely on the nodes properties in /var/lib/torque/server_priv/nodes. If you are still using YAIM they are usually set to lcgpro. The mcfloat script makes use of *el6* for the nodes to use for single core jobs and *mc* for the nodes to use for multicore. If you want to use something else you need to edit the mcfloat script. I've opted for a smooth sed command

sed -i.old 's/lcgpro/el6/g' /var/lib/torque/server_priv/nodes

qmgr commands

Now you need to create the queue for multicore. You need to limit the access to those groups that will run multicore. For the moment for me it is only atlas production, but some sites may need to add cms too so I put the additional line as an example.

qmgr 
create queue mcore
set queue mcore queue_type = Execution
set queue mcore resources_max.cput = 48:00:00
set queue mcore resources_max.walltime = 72:00:00
set queue mcore resources_default.cput = 48:00:00
set queue mcore resources_default.neednodes = mc
set queue mcore resources_default.walltime = 72:00:00
set queue mcore acl_group_enable = True
set queue mcore acl_groups = atlprd
set queue mcore acl_groups += cmsprd
set queue mcore enabled = True
set queue mcore started = True

you also need to set resource_default.neednodes for the other queues for the partitioning to work. So for example at my site the other main queue would be set to

set queue long resources_default.neednodes = el6

other queues have their own parameters.

cron job

Finally you need to setup a cron job to run mcfloat. Here is mine

cat /etc/cron.d/mcfloat.cron
PYTHONPATH=/usr/local/lib/python2.6/site-packages:${PYTHONPATH}
*/10 * * * * root python /usr/local/bin/mcfloat >> /var/log/mcfloat.log 2>&1

YAIM

If you are still using YAIM you will need to add the extra queue there too. We create the queue with puppet but the BDII is still handled by YAIM.

MCORE_GROUP_ENABLE="/atlas/ROLE=production"
QUEUES="long mcore"