Difference between revisions of "VomsSnooper Tools"

From GridPP Wiki
Jump to: navigation, search
 
m (Changed the displayed size of the poster image)
 
Line 72: Line 72:
 
=== Poster ===
 
=== Poster ===
  
[[Image:VomsSnooper.gif]]
+
[[Image:VomsSnooper.gif|200px]]
  
 
=== High Level Design ===
 
=== High Level Design ===

Latest revision as of 14:58, 9 March 2022

Tools for managing VOMs records

This document describes a process for converting XML data to VOMs records, suitable for use with Yaim. The latest RPM containing the tools used in the process is available here: http://www.sysadmin.hep.ac.uk/rpms/fabric-management/RPMS.vomstools

The tools developed to support the conversion of XML to VOMs records are also suitable for many additional sys admin tasks, such as verifying a site's VOMs records, version control and documentation automation.

BACKGROUND

System Context

A tool, Yaim, is used to configure the middle-ware software at grid sites. Yaim processes a set of input files containing variables that influence the configuration activity. The main inputs we are concerned about here are the "site-info.def" file, and the "vo.d" directory, which will be described in more detail later.

The processes described herein are concerned with those Yaim variables relating to VOMs records (Virtual Organisation Membership Service) that determine which VOMS servers are used to authenticate and authorise users of a particular VO. Examples of such variables follow:

VO_ALICE_VOMS_SERVERS="'vomss://voms.cern.ch:8443/voms/alice?/alice' " 
VO_ALICE_VOMSES="'alice voms.cern.ch 15000 \
        /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch alice' " 
VO_ALICE_VOMS_CA_DN="'/DC=ch/DC=cern/CN=CERN Trusted Certification Authority' "

The data that is used to determine the content of these variables is administered centrally, at a location known as the "Operations Portal" . The data is made available in XML format.

Notes on SIDs and VODs

It is important to note that these Yaim variables can be given to Yaim in one of two styles.

  • SIDs style - the records are included directly in the site-info.def file (hence the name) in the style shown above in the example for ALICE.
  • VODs style - each VOs records are stored in their own separate files in a vo.d directory, which is located beside the site-info.def file. The files are named after the VO concerned.

Either format may be used. However it is best practise to select one style, not both. It is not proscribed to use both styles at once, but if a VO is represented twice, the VOD style takes precedence.

The VOD standard is favoured because it separates out the concerns of each VO. It is also more readable, less cluttered and better suited to VO names in the DNS style (with dots).

The SID style is viable, but monolithic, so a use case is provided to convert a site to VOD style VOMS records.

Requirements

The usecases described here provide a bridge between the XML data and the Yaim inputs in order to ensure they are correct with the minimum amount of user intervention.

The shell scripts of the usecases, which were developed at Liverpool, are provided as examples that can be modified to work at any other grid site to ease the burden of VOMs records maintenance. They are fully described in the USE section, below.

The usecases to be supported are :

  • ConvertSidsToVods - This tool has been provided to allow conversion of all VOs from SIDs into VODs format. This simplifies the management of VOMS records.
  • newVomsRecsForMySite - This usecase allows the site admin to generate new records automatically, which can then be merged (by hand or semi-automatically) into the site's configuration. Furthermore, it allows the site-admin to correct the config when a site is not in exact correspondence with the Operations Portal XML. Previously, this involved manual transcripts/proof reading.
  • checkMySite - Previously, it has not been feasible to easily check that the records in a site's Yaim configuration correspond with the data from the Operations Portal. This usecase provides a means to check the level of correspondence and highlight any problems.
  • fixApprovedVos - Prior to these processes, and for the interim period, a document (GridPP approved VOs) is used to disseminate the Yaim variables relating to VOMs records. The data was manually transcribed and the document was prone to becoming stale. This usecase describes a means to keep that document current with regard to the newest XML available from the Operations Portal, without significant manual intervention. Sites can then use the existing procedures with greater confidence than they have in the past.
  • genResourceTable - This is used to create a table of required resources for each VO, to be included in the Approved VOs document.
  • makeTarball - This is used to create a tar ball of all the records for all approved VOs, in both SIDs and VODs format. A link to the tarball will be made from the Approved VOs document.
  • getLSCRecords - Yaim is nominally used to create LSC files from the VOMs records created above, but not everyone uses Yaim to configure their servers - other processes can have good results. For sites that use some other process, it is necessary to bypass the site-info.def/vo.d material, and directly create a set of LSC files. This usecase allows the site admin to generate new LSC records automatically from XML.
  • checkMyLSCRecords - As with getLSCRecords, this usecase allows the site admin to generate new LSC records automatically from XML. It also compares the new LSC files with the site copies in the /etc/grid-security/vomsdir directory.
  • ListFQANs - This usecases allows the user to list a VO's FQANs.
  • MakeGroupsConf - This is used to create the groups.conf file, useful for grid security.
  • checkMyVomsRecords - This is an alternative to "checkMySite", above. It serves the same purposes and it is only documented for the sake completeness.
  • instantUI - A tool for setting up a VO without much fuss.

DESIGN

Poster

VomsSnooper.gif

High Level Design

The design of the main software components is illustrated below, in a class diagram.


VomsSnooper class.png


The software has several entry points (mains).

  • VomsSnooper parses the Operations Portal XML and writes out formatted data records in a standard, sorted way suitable for Yaim.
  • SidFormatter parses existing Yaim files (site-info.def, vo.d) and prints them out in a standard, sorted manner.
  • SiteChecker parses existing Yaim files (site-info.def, vo.d) and the Operations Portal XML using a Sax parser. It then compares both data sources and prints out a report on the differences.
  • CicToLsc also uses the SAX parser, but only includes functionality for making LSC files. It is for the convenience of people who wish to bypass the Yaim configuration process.
  • VoResources uses the SAX Parser to extract resource requirements for each VO and pack them into a wiki table.

All mains take various arguments to control the flow. The arguments are discussed in more detail below, and the detailed usecase descriptions below explain how the arguments can be combined to fill the requirements.

Flags and options for CicToLsc

Copyright (c) The University of Liverpool, 2012 (Licensed under the Academic Free License version 3.0)

Version 1.36

The CicToLsc tool takes an XML file from the CIC portal, and 
creates a set of LSC files from the data, bypassing Yaim.

Mandatory arguments: 
  --xmlfile       f       # Input XML file downloaded from CIC portal
  --approvedvos   f       # File of names of VOs that I support
  --vomsdir               # Where to print LSC Files 
Optional arguments: 
  --help                  # Prints this info

Flags and options for SidFormatter

Copyright (c) The University of Liverpool, 2012 (Licensed under the Academic Free License version 3.0)

Version 1.36

The SidFormatter tool takes an existing populated site-info.def file and vo.d directory, and 
formats it into a standard, sorted manner. It writes its output into a new, 
unpopulated site-info.def file and vo.d directory.

Mandatory arguments: 
  --oldsiddir dir   # Some existing directory that contains a populated site-info.def file and vo.d directory
  --newsiddir dir   # Some existing directory that contains an unpopulated site-info.def file and vo.d directory
  --myvos   f       # Names of VOs that I support
Optional arguments: 
  --flat            # Print all the records out in a site-info.def, even when they are silly sids
  --help            # Prints this info

Flags and options for SiteChecker

Copyright (c) The University of Liverpool, 2012 (Licensed under the Academic Free License version 3.0)

Version 1.37

The SiteChecker tool checks if a site complies with the CIC portal XML

Mandatory arguments: 
  --sidfile f       # Location of site-info.def file
Optional arguments: 
  --xmlurl  f       # URL of XML file (def: http://operations-portal.egi.eu/xml/voIDCard/public/all/true)
  --help            # Prints this info

Flags and options for VomsSnooper

Copyright (c) The University of Liverpool, 2012 (Licensed under the Academic Free License version 3.0)

Version 1.36

The VomsSnooper tool takes an XML file from the CIC portal, and 
formats it into a standard, sorted manner for Yaim. 

Mandatory arguments: 
  --xmlfile f       # Input XML file downloaded from CIC portal
  --myvos   f       # Names of VOs that I support
  --vodfile f       # Names of VOs that must be output in VOD format
  --outfile f       # Where to write SIDs (records that can be represented in a site-info.def)
Optional arguments: 
  --help            # Prints this info
  --voddir  d       # Where to write VODs (records that cannot be represented in a site-info.def)
  --printvodtitle   # When printing VODs, put the name of the VOD file in the output
  --nosillysids     # When printing SIDs, reject ones with silly names (DNS/dot style)
  --extrafields     # Print some extra fields (not recommended)
  --contactsfile    # Where to print the VO contacts (not recommended)
  --fqans           # File where to print VO FQANs 
  --vomsdir         # Where to print LSC files 
  --ignorecernrule           # Ignore the cern rule (use only voms.cern.ch)
  --dontuseisvomsadminserver # Ignore whether VOMS is an admin server (not recommended)

Flags and Options for VoResources

Copyright (c) The University of Liverpool, 2012 (Licensed under the Academic Free License version 3.0)

Version 1.36

The VoResources tool takes an XML file from the CIC portal, and 
creates a set of LSC files from the data, bypassing Yaim.

Mandatory arguments: 
  --xmlfile       f       # Input XML file downloaded from CIC portal
  --myvos         f       # File of names of VOs that I support
  --res           f       # File to print resources to
Optional arguments: 
  --help                  # Prints this info

BUILD AND INSTALL

The software is open source, and it can be downloaded from this location. Please choose the newest version.

http://www.sysadmin.hep.ac.uk/rpms/fabric-management/RPMS.vomstools/

The software is distributed as an rpm file that can be installed on your system as root, e.g.:

# rpm -i VomsSnooper-<1-someversion>.noarch.rpm

No build needs to be done. The tools are distributed as relocatable JAR files that reside under /opt/GridDevel/. To test the installation:

# export PATH=/opt/GridDevel/bin:$PATH
# cd /opt/GridDevel/bin/
# ./siteChecker.sh

The software can be installed with yum. To do so, put the following yum repo in /etc/yum.repos.d

[root@hepgrid95 yum.repos.d]# cat sysadmin.hep.ac.uk.repo
[sysadmin.hep.ac.uk]
baseurl=http://www.sysadmin.hep.ac.uk/rpms/fabric-management/RPMS.vomstools/
enabled=1
priority=15
[root@hepgrid95 yum.repos.d]# 

Then use "yum install VomsSnooper"

Note: the package is (as yet) unsigned. If yum complains about that, use --nogpgcheck

USE

The usecase examples can all be found in /opt/GridDevel/vomssnooper/usecases They were developed at Liverpool, and match the site layout there. You'll need to edit some of the shell scripts suit your own grid site.

convertSidsToVods

This usecase allows the site admin to convert a site-info.def file that contains old, SID format VOMs records into the VOD style that uses discrete files under the vo.d directory. It does not use any of the main VomsSnooper components.

Process

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/convertSidsToVods

Now run the convertSidsToVods.pl script, as follows.

# ./convert_sids_to_vods.pl -s /root/glitecfg/site-info.def 

The script converts the data in-situ, so you may wish to keep backups first!

Principles of operation

The script backs up the site-info.def file into a temporary file in the same directory. Then it writes new VOD format files into the vo.d directory, finally writing out a new site-info.def minus the old SID format records. You can also use the dry-run features (-keepsids 1 -novods 1) to prevent changes. If any VO represented in the site-info.def is already represented in the vo.d directory, the script prints a message and exits.

newVomsRecsForMySite

This usecase allows the sites admin to download a new set of records for inclusion (by hand or automatically) into the local site-info.def file. At present, the usecase only works if the site has converted all its VOMs records into VOD format. Another usecase is provided for that purpose.

Process

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/newVomsRecsForMySite

There are some files in here that you have to edit. In ExtraVOIDCardInfo.txt, put some XML for any sites that you support, but which are not in the Operations Portal, i.e. local groups etc. By default, it contains some data pertaining to Liverpool. Also edit void/myvos.txt and void/vod.txt to contain the VOs you support, and the ones to print in VOD format (should be all), respectively.

Edit the voidRun.sh script, replacing the example (hepgrid6) with the location of your standard site-info.def file/vo.d directory. The script uses this as the input to a merge operation - the new records are merged into that and written out to new files. If new VOMs records are created for a VO that is not yet configured at the site, a message is printed, and some fields are populated with default data.

Now run the voidRun.sh script, as follows.

# ./voidRun.sh


Check that the merged data, in void/merged/vo.d, looks rational. If so, it can be used as input to Yaim.

Principles of operation

The script gets the newest Operations Portal XML, and merges the local XML into it. It then uses VomsSnooper to convert these into SID and VOD records, suitable for site configuration files.

Last, the script downloads your own site-info.def/vo.d configuration, and uses a perl script to merge the new data into that, printing the new material out into a new directory structure.

checkSite

This usecase gets the newest records from the Operations Portal XML and makes sure your current site-info.def/vo.d tallies with those records.

Process

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/checkSite

Vi the checkSite.sh script to make it comply with your local set-up. And now run the checkSite.sh script, as follows.

# ./checkSite.sh

The script reports any discrepancies if finds to the screen.

Principles of operation

The checkSite.sh script syncs a local directory with a copy of the site-info.def/vo.d config from a cream CE. It then uses a tool, SiteChecker, to read XML from the Operations Portal, it reads SID data for the site-info.def/vo.d and finally compares the records.

fixApprovedVos

This usecase is used by a Core Task member for documentation to maintain the Approved VOs document.

The usecase gets the newest records from the Operations Portal XML and merges them into the wiki page of approved VOs, keeping it up to date.

Process:

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/fixApprovedVos

In another screen, start a browser and make your way to:

https://www.gridpp.ac.uk/wiki/GridPP_approved_VOs#Approved_Global_VOs

Click the edit tab, logging in if necessary. Use the clipboard to select and copy all the text. Deposit this, via some editor of your choice, into a file called wiki.txt:

# vi /opt/GridDevel/vomssnooper/usecases/fixApprovedVo/wiki.txt

Now run the makeSidVods.sh script, as follows.

# ./makeSidVods.sh

Some output scrolls past, and (if all is well) a new file called new.wiki.txt is made. This is supposed to be the same as the old wiki.txt, but all the VOMs records are updated, in both SID and VOD format (for VOMs records that can't be represented as SID records, the SID lines are still printed, but in comments).

Check by eye the new.wiki.txt file, and use diff on it to see if it has any weird errors. If not, clip the text in your editor (e.g. nedit) and paste it back into the browser. Show the preview to make sure it's not horribly mangled, then save the changes if it looks good.

This usecase also builds in the wiki table that lists each VOs resource requirements.

Principles of operation:

The makeSidVods.sh script downloads a copy of the newest Operations Portal XML. It then uses VomsSnooper twice. On the first run, it parses the XML and prints out all the data in VOD format. On the second run, it prints out all the data in SID format. Finally, all that data is put together and formatted by a perl script, assemble_content.pl, which also takes the old wiki.txt content and merges the new content into it, creating a new wiki page.

It finally uses the genResourceTable usecase to make the twiki table that lists each VOs resource requirements.

genResourceTable

This usecase is used by the Core Task member for documentation to maintain the Approved VOs document.

The usecase is used to create a twiki table that lists the resource requirements for each VO from the XML, suitable for inclusion in the Approved VOs document. It is run in conjunction with the fixApprovedVos usecase, above.

Process

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/genResourceTable

Now run the ./genResourceTable.sh script, as follows.

# ./genResourceTable.sh 

Some output scrolls past, and (if all is well) a file called res.txt is created containing the twiki table.

Principles of operation

The script downloads a copy of the newest Operations Portal XML. It then uses VoResources to make the twiki table from the XML file.

--------

makeTarball

This is used to create a tar ball of all the records for all approved VOs, in both SIDs and VODs format. A link to the tarball will be made from the Approved VOs document.

Process

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/makeTarball

Now run the ./makeTarball.sh script, as follows.

# ./makeTarball.sh 

Some output scrolls past, and (if all is well) a file called glitecfg.tar is created containing the data. Both SIDs and VODs are made, but the SIDs (in the site-info.def.frag file) are commented out - VODs take precedence anyway. Take care not to over write your operational records.

The fragment of site-info.def contains a VOS line, as well as the SIDs. It is intended that you incorporate this fragment into your operational site-info.def file.

Site admins are able to edit the supplied myvos.txt file, to include only those records that are supported at a particular site.

Principles of operation

The script downloads a copy of the newest Operations Portal XML. It then uses VomsSnooper to make SIDs and VODs of all the VOs in the Approved list. It then formats the data and inserts default values for some fields.

These default values can be configured in the addDefault.pl script, which is provided. In particular, some VOs use CVMFS - the script provides defaults for those VOs as well.

getLSCRecords

This usecase allows the site admin to generate new LSC records automatically from XML, suitable for inclusion in a /etc/grid-security/vomsdir directory, by-passing the usual yaim steps.

Process

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/getLSCRecords

Now run the getLSCRecords.sh script, as follows.

# ./getLSCRecords.sh 

Some output scrolls past, and (if all is well) a directory called vomsdir is created containing the latest and greatest LSC files.

Principles of operation

The script downloads a copy of the newest Operations Portal XML. It then uses CicToLsc to make a vomsdir consisting of all newest LSC files for your site.


checkMyLSCRecords

This usecase compares the LSC records at a site to the ones that exists in the Operations Portal XML file.

Process

In a shell:

# cd /opt/GridDevel/vomssnooper/usecases/checkMyLSCRecords/

Vi the checkMyLSCRecords.sh script to make it comply with your local set-up. And now run the checkMyLSCRecords.sh script, as follows.

# checkMyLSCRecords.sh

Ignore the warnings about "silly SIDs". The script makes a file, diffs.txt, that shows any discrepancies it finds.

Principles of operation

The script makes a set of LSC records from the Operations Portal XML, then makes another set from the data at a site. It then does an MD5 sum of all the data, and prints out any diffs into a text file.

listFQANs

This experimental usecase allows the user to list a VOs FQANs.

Process

In a shell:

# export PATH=/opt/GridDevel/bin:$PATH
# cd  /opt/GridDevel/vomssnooper/usecases/newVomsRecsForMySite
# vomsSnooper.sh --xmlfile   \
   http://operations-portal.egi.eu/xml/voIDCard/public/all/true   --myvos void/myvos.txt \
   --outfile /dev/null --fqans fqans

Check that the file, fqans, contains the list of FQANs for each VO.

Principles of operation

The VomsSnooper tool contains a functions to collect and print the FQANs from the XML.

makeGroupsConf

This experimental usecase allows the user to create a groups.conf file, that is required for grid security.

Process

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/MakeGroupsConf
# ./makeGroupsConf.sh

Check that the file, groups.conf, shows up.

Principles of operation

The script first uses the VomsSnooper tool to download the FQANs of the VOs. It then uses a perl script to convert them into a groups.conf file.

checkMyVomsRecords

This usecase allows you to compare your set-up with the Operations Portal XML and make sure they match well, or not, as the case may be. This is superseded by the checkMySite use case,

Process

In a shell:

# cd  /opt/GridDevel/vomssnooper/usecases/checkMyVomsRecords

Vi the checkMySite.sh script, replacing the example (hepgrid6) with the directory location of your standard site-info.def file/vo.d directory. The script uses this as the basis to compare with the XML from the Operations Portal. Also, ensure the myvos.txt contains the names of all your VOs.

Now run the checkMySite.sh script, as follows.

# ./checkMySite.sh

Some output scrolls past, and (if all is well) a new file called diff.txt is made. This contains a list of differences between your site and the Operations Portal XML.

Principles of operation

The script downloads a copy of the newest Operations Portal XML. It then uses VomsSnooper to make a site-info.def consisting of all SIDs, no VODs. This format allows easy comparison to be made.

Next, the script synchronises a local directory with the contents of your standard site-info.def/vo.d configuration. It uses SidFormatter with the --flat switch to write these records out in pure SID format. Some output in printed about "sillysids"; pay no heed to that. Last, a simple diff is made to highlight discrepancies.

instantUI

This is used to setup a new UI with the minimum of fuss. To use it, see https://www.gridpp.ac.uk/wiki/User_Interface_%28UI%29_to_support_approved_VOs .

Process

There are some prerequisites.

Note: a decent overview of the grid account set up process is avaliable in: https://edms.cern.ch/file/722398/1.4/gLite-3-UserGuide.pdf‎

In brief, to host a User Interface (UI), you need to set up a standard Linux system using SL6 or RHEL6. Each grid user needs to have a user account on the system, and each grid user must be a member of at least one Virtual Organisation (VO). But, prior to joining any VO, a user needs a grid certificate. Here's the run down.

  • Create the Linux user
  • Get a grid certificate for the user
See the user guide. Additional advice on acquiring a UK grid certificate is given here: https://ca.grid-support.ac.uk
  • Install the user's certificate
See the user guide. Once a grid certificate is received for the user, install it as per the instructions here: https://twiki.cern.ch/twiki/bin/view/LHCb/FAQ/Certificate
  • Use the grid certificate to join a VO
Once users have a grid certificate, they can join a VO. Advice on joining an example VO, dteam, is given here: https://wiki.egi.eu/wiki/Dteam_vo . However, each VO will have its own membership process. Home pages for most of the Approved VOs is available here: GridPP_approved_VOs.

Once that is done, the UI can be installed and configured as follows.

Using some tool such a yum or rpm, install these packages:

  1. emi-ui (http://repository.egi.eu/sw/production/umd/3/sl6/x86_64)
  2. VomsSnooper (http://www.sysadmin.hep.ac.uk/rpms/fabric-management/RPMS.vomstools)

Note: if using yum, the following repositories are suggested:

[root@hepgrid93]# cd /etc/yum.repos.d/
[root@hepgrid93 yum.repos.d]# cat sysadmin.hep.ac.uk.repo 
[sysadmin.hep.ac.uk]
name=sysadmin.hep.ac.uk
baseurl=http://www.sysadmin.hep.ac.uk/rpms/fabric-management/RPMS.vomstools/
enabled=1
gpgcheck=0
priority=100

[root@hepgrid93 yum.repos.d]# cat UMD-3-base.repo 
[UMD-3-base]
name=UMD-3-base
baseurl=http://map2.ph.liv.ac.uk/yum/umd/3/sl6/x86_64/base
enabled=1
protect=1
priority=75

[root@hepgrid93 yum.repos.d]# cat UMD-3-updates.repo 
[UMD-3-updates]
name=UMD-3-updates
baseurl=http://map2.ph.liv.ac.uk/yum/umd/3/sl6/x86_64/updates
enabled=1
protect=1
priority=75

Next, as root,

# cd /opt/GridDevel/vomssnooper/usecases/instantUI/


Run the ./installUI script.

# ./installUI

If you give it no options, it sets up the UI to use all the VOs that are approved for use at the time of writing. You can also give specific VOs as options, e.g.

# ./installUI zeus t2k.org

In this case, the UI will only support zeus and t2k.org.

Note: Any VO must be present in the Operations Portal: http://operations-portal.egi.eu/

Principles of operation

Instant UI merges a template config file (site-info.def) with data acquired from the Grid Operations Portal and the grid Information System to create a suitable config for any of the GridPP Approved VO. It then runs yaim to configure the UI system.

END OF DOCUMENT