LCG-on-x86 64

Worker Nodes

It is possible to install and configure the LCG software distribution on worker nodes running a bi-arch 32bit / 64bit operating system, such as RedHat Enterprise Linux 3 (RHEL3). This document describes an example production configuration in operation at IC-LeSC that uses the relocatable tarball distribution.

• Install the TAR-WN relocatable tarball distribution in some centrally-accessible location. In our case, this is an NFS-mounted volume, ~lt2-lcg/LCG-$VERSION/, but equally you could build a standard configuration one one machine and replicate it to all other worker nodes though some other mechanism.
• Build and install a 32bit version of perl in the LCG software volume. This is necessary because the system perl installation and libraries as provided by Redhat are 64bit binaries, and so will cannot be linked against the 32bit binary libraries used by LCG tools or experiments.
  • At IC-LeSC, we have installed our 32bit perl interpreter in ~lt2-lcg/prereqs/perl, and created symlinks to this location from inside ~lt2-lcg/LCG-$VERSION/.
  • Because the 32bit perl libraries are not automatically located in their non-standard location, we created a small wrapper around perl to add the appropriate -I flags to the interpreter's command-line.
  • Experiments that have hardcoded #!/usr/bin/perl into their scripts, rather than search the path (eg via #!/usr/bin/env perl) will still use the system perl binary, and thus will fail if they depend on compiled 32-bit libraries. However, in practice, this doesn't appear to be a problem.
  • Note that it is not necessary to also build a 32bit python interpreter, as one is already provided in the TAR-WN distributions. (At least, there is one in LCG-2_7_0 -- I haven't looked at gLite-3.0 yet.)
• Modify the initial working environment of all LCG-submitted jobs so that they will source the top-level LCG configuration shell script, ~lt2-lcg/LCG-$VERSION/etc/profile.d/grid-env.sh (or .csh if using a C-shell).
  • At IC-LeSC, we use the job pre-amble facility in our SGE JobManager to insert this source instruction into every job script.
  • Because we install each relocatable distribution in it's own sub-directory, we can install a new TAR-WN distribution whilst the existing one is being used; when satisfied that it is installed and working correctly, we can simply update the live job preamble configuration file to configure all newly-submitted jobs to use the new installation.