Difference between revisions of "New VO deployment"

From GridPP Wiki
Jump to: navigation, search
(Removed chris from the responsibility list)
(Update. Removed link to 10yo doc.)
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Creating a New VO ==<!--The general procedure is sketched out in the section [[Instruction for VO administrators]]. The process is still under development, and anyone wishing to create a new VO should contact the [https://www.gridpp.ac.uk/deployment/contact.html Deployment Team] for further help and information (in particular the Production Manager, Security Officer and VOMS Manager).-->
+
This page covers both the ''creation'' and ''deployment'' of new VOs. Its target audience is new VO (technical) managers and infrastructure sysadmins and support contacts for VOs.
Please follow the instructions on [http://www.ngs.ac.uk/How-to-Join#joinAsVo http://www.ngs.ac.uk/How-to-Join#joinAsVo] to apply for a VO to be hosted on the UK NGI VOMS servers.
+
The page contains a link that gets you to the VO hosting application form.
+
  
What follows below is general information that you should be aware of if you want to start a VO.
+
== Creating a New VO ==
 +
 
 +
A ''Virtual Organisation'' or VO is a group or collaboration with a common purpose. It may be a particular project with funding, it may be a common purpose, or it may be a specific group of people collaborating on a particular task. Typically, a VO will be able to share data with every member in the VO, and they will have resources allocated to them as a VO.
 +
 
 +
It is also possible to join an existing VO, of course, if there is one with similar goals: see the [http://operations-portal.egi.eu/vo/search EGI VO registration portal].
 +
 
 +
VOs can be local (supported at a single site), national (supported by several sites but in the same country, e.g. on a national infrastructure), and international. What follows below is generally about international VOs as even regional (local/national) VOs can grow to become international. For regional VOs, some of the information requested below may not be needed.
  
 
===Information needed===
 
===Information needed===
The VO will need to provide some information, partly for security reasons and partly to let system administrators judge what resources the VO will be likely to need. Useful information would include:
+
The VO will need to provide some information, partly for practical and security reasons and partly to let system administrators judge what resources the VO will be likely to need.  
* VO name. This should be reasonably short, distinctive, and must not clash with any existing VO. A lower-case name is recommended, and generally no more than five or six characters (letters and numbers are allowed in the name, but most other characters are not). There is a recommendation to base VO names on DNS names to avoid name clashes, so that  GridPP VOs should have names like xxx.gridpp.ac.uk.
+
 
* VO support contacts - both specific responsible people and various experiment mailing lists.
+
* Name of the VO. This should be reasonably short, distinctive, and must not clash with any existing VO. A VO will typically have two names, a short name (usually lower case), say "poohsticks" (an experiment running poohsticks simulations), and a DNS style name, such as vo.poohsticks.org (assuming they own the DNS name poohsticks.org.)
* Security contacts - ideally at least two people who can respond quickly in the event of a security incident relating to a member of the VO, or to the VO as a whole.
+
* VO management: a VO manager (who can decide membership of VO, roles and responsibilities), plus usually at least one deputy or co-manager. You will need a mailing list (e.g. poohsticks-management@example.com)
* VO/VOMS server, file catalogue etc. end-points (see below).
+
* VO support contacts (see below) - you can choose to register support people with the helpdesks; this is recommended for larger VOs or for VOs that have VO-specific software.
* Hardware requirements - memory size, disk space etc.
+
* Security contacts - ideally at least two people who can respond quickly in the event of a security incident relating to a member of the VO, or to the VO as a whole. This also needs to have a mailing list.
 +
* A VOMS server. It manages the VO's memberships and the members' roles and subgroups. If you don't have a VOMS server, you can request to use GridPP's - for this, you will typically need ''authorisation'' from the prinicpal investigator (PI).
 +
** Optionally, roles and/or subgroups of the VO - see Security Considerations below.
 +
* Hardware requirements - memory size, disk space etc, types of storage required (working repositories, archives, databases)
 
* Software requirements - any software beyond the basic Linux tools/libraries, including things which are part of standard distributions as they may not be installed by default.
 
* Software requirements - any software beyond the basic Linux tools/libraries, including things which are part of standard distributions as they may not be installed by default.
 
* Typical usage pattern - expected job frequency and variation over time, job length, data read and written per job etc.
 
* Typical usage pattern - expected job frequency and variation over time, job length, data read and written per job etc.
* Glue schema fields used - this would give an idea of what is really used in the information system and needs to be ensured to be properly set and maintained.
 
 
* General procedures - for example if the site has to request the installation of VO software.
 
* General procedures - for example if the site has to request the installation of VO software.
 
* Size of the VO (i.e how many users), to give a guide to how many pool accounts to create.  
 
* Size of the VO (i.e how many users), to give a guide to how many pool accounts to create.  
  
 
See the [http://www.phenogrid.dur.ac.uk/howto/config Phenogrid] web site for an example of the sort of thing required. You can also have a look at a [https://www.gridpp.ac.uk/deployment/users/questionnaire.html questionnaire] which EGEE has used to start discussions with new VOs.
 
See the [http://www.phenogrid.dur.ac.uk/howto/config Phenogrid] web site for an example of the sort of thing required. You can also have a look at a [https://www.gridpp.ac.uk/deployment/users/questionnaire.html questionnaire] which EGEE has used to start discussions with new VOs.
 
The EGEE operations group has developed a standardised [http://operations-portal.egi.eu/aboutportal/map VO ID card] to provide this kind of information. Most of the entries are well explained.
 
  
 
===Security considerations===
 
===Security considerations===
The VO will need to provide administrators who take responsibility for adding users into the VO, checking that they understand their responsibilities, and if necessary removing them from the VO if they abuse the system. VOs should define what constitutes acceptable use for their members (in addition to the general acceptable use policies applicable to all grid users).
+
* The VO managers are responsible for setting the aims of the VO: describe which work is in scope and which is not. Users must do only work which is in scope for the VO.
 +
* Some of the [https://www.gridpp.ac.uk/deployment/security/policies/index.html security policy documents] are relevant to VO creation and operation, and the VO administrators need to ensure that they comply with the relevant policies.
 +
* The VO security contacts (via the mailing list) are expected to deal in a timely manner with breaches of policy. An incident may lead to the errant member of the VO being banned from the infrastructure, but if the infrastructure admins/coordinators don't feel the incident is being dealt with appropriately by the VO, the whole VO may get banned.
 +
* If the VO has data security issues, these should be discussed with GridPP during setup of the VO.
 +
** A typical VO will have every file readable by everyone in the VO, but define a ''role'' of users who are allowed to create, update, and delete files.
 +
** Grid middleware does support stronger security models.
  
Some of the [https://www.gridpp.ac.uk/deployment/security/policies/index.html security policy documents] are relevant to VO creation and operation, and the VO administrators need to ensure that they comply with the relevant policies.
+
===VO software===
 +
Each VO will typically need some VO-specific services. Some of these are grid-based, such as file catalogues, resource brokers, compute and storage elements. There may be additional software required by the VO, and there will be means of making this available.
  
===VO services===
 
Each VO will need some VO-specific services. At a minimum you need a VO/VOMS server to store the list of VO users, but file catalogues, resource brokers and perhaps other services may also be needed. These may be run by the VO itself or, by negotiation, as part of the general GridPP infrastructure. In particular a [https://voms.gridpp.ac.uk:8443/vomses/ GridPP VOMS server] is run by Manchester for the use of the GridPP community; [https://www.gridpp.ac.uk/deployment/contact.html contact] the VOMS manager for more information.
 
 
===Getting the VO enabled at sites===
 
Enabling a VO is a relatively easy process, and sites which are directly associated with the VO (including sites in other countries) should be able to do it given the information described above. To get further resources from other GridPP sites, contact the [https://www.gridpp.ac.uk/deployment/contact.html Deployment Team].
 
 
===VO software installation===
 
 
There are various models for dealing with the installation of VO-specific software. If only a few dedicated sites are involved the software can be directly installed by the administrators. If the software is relatively compact it can be shipped with the job in the sandbox, or downloaded from a Storage Element or a web site.
 
There are various models for dealing with the installation of VO-specific software. If only a few dedicated sites are involved the software can be directly installed by the administrators. If the software is relatively compact it can be shipped with the job in the sandbox, or downloaded from a Storage Element or a web site.
  
There is also a more general [http://grid-deployment.web.cern.ch/grid-deployment/eis/docs/ExpSwInstall/sw-install.html method to install software] in VO-specific disk space visible from the Worker Nodes.
+
[[RAL Tier1]] uses [[RAL Tier1|CVMFS]] to manage software for different VOs, particularly when they conflict (e.g. requiring different versions of python or libraries, etc.) Alternatively ask the admins on the sites that support your VO what they provide.
  
 
===Support procedures===
 
===Support procedures===
To some extent these are still in development. VOs should be prepared to support their users at least in the use of VO-specific software. More general Grid support will be provided by GridPP as a whole, including community support by users themselves.
+
* VOs should be prepared to support their users at least in the use of VO-specific software.
 +
* More general Grid support will be provided by GridPP as a whole, including community support by users themselves.
 +
* VO support liaison should sign up for a mailing list called [http://www.jiscmail.ac.uk/lists/GRIDPP-SUPPORT.html GRIDPP-SUPPORT].
 +
* It is '''strongly''' recommended to make GridPP technical support contacts members of the VO at least initially, in order to help shoot any trouble that might arise.
  
 
The standard support route for all Grid users is the GGUS portal, as described [http://www.gridpp.ac.uk/deployment/users/support.html here]. For regional (e.g. GridPP-specific) VOs the tickets will generally be directed back to the UK Grid helpdesk. There is also a [mailto:GRIDPP-USERS@JISCMAIL.AC.UK GridPP Users] mailing list (see the [http://www.jiscmail.ac.uk/lists/GRIDPP-USERS.html JISCmail] web site for subscription information).
 
The standard support route for all Grid users is the GGUS portal, as described [http://www.gridpp.ac.uk/deployment/users/support.html here]. For regional (e.g. GridPP-specific) VOs the tickets will generally be directed back to the UK Grid helpdesk. There is also a [mailto:GRIDPP-USERS@JISCMAIL.AC.UK GridPP Users] mailing list (see the [http://www.jiscmail.ac.uk/lists/GRIDPP-USERS.html JISCmail] web site for subscription information).
 +
 +
The [http://egi.eu EGI] [[Virtual Organisation|VO]] Registration Form can be found [http://operations-portal.egi.eu/vo/registrationWelcome here]. It also provides a list of documents you should consult before [[Start Here - Creating a new VO|creating a new VO]].
 +
 +
The general procedure is sketched out in the section [[Instruction for VO administrators]]. The process is still under development, and anyone wishing to create a new VO should contact the [https://www.gridpp.ac.uk/deployment/contact.html Deployment Team] for further help and information (in particular the Production Manager, Security Officer and VOMS Manager).
 +
 +
 +
== VO Deployment ==
 +
 +
Once a VO is set up, sites should be requested to support it - VOs typically do this via their GridPP support contact(s). At this point the VO should have:
 +
* Acceptable use policies defined
 +
* A VOMS server with VO information
 +
* Mailing lists
 +
* Resources allocated to it - CPU hours, disk storage, tape storage
 +
 +
Sites will then need to:
 +
* Add the VO to the list of supported VOs (e.g. gridmap files)
 +
* Add the VO to the UIs, if applicable (so users can do voms-proxy-init locally)
 +
* Allocate the resources to the VO, e.g. space tokens, disk pools
 +
 +
It is recommended that the VO attends the following meetings:
 +
* Tier 1 liaison meeting (if using Tier 1) - VOs should attend regularly
 +
 +
 +
 +
 
[[Category:VOMS]]
 
[[Category:VOMS]]
  
{{KeyDocs|responsible=Jens Jensen|reviewdate=2014-10-16|accuratedate=2014-10-16|percentage=80}}
+
{{KeyDocs|responsible=Jens Jensen|reviewdate=2016-01-19|accuratedate=2016-01-19|percentage=93}}

Revision as of 15:31, 19 January 2016

This page covers both the creation and deployment of new VOs. Its target audience is new VO (technical) managers and infrastructure sysadmins and support contacts for VOs.

Creating a New VO

A Virtual Organisation or VO is a group or collaboration with a common purpose. It may be a particular project with funding, it may be a common purpose, or it may be a specific group of people collaborating on a particular task. Typically, a VO will be able to share data with every member in the VO, and they will have resources allocated to them as a VO.

It is also possible to join an existing VO, of course, if there is one with similar goals: see the EGI VO registration portal.

VOs can be local (supported at a single site), national (supported by several sites but in the same country, e.g. on a national infrastructure), and international. What follows below is generally about international VOs as even regional (local/national) VOs can grow to become international. For regional VOs, some of the information requested below may not be needed.

Information needed

The VO will need to provide some information, partly for practical and security reasons and partly to let system administrators judge what resources the VO will be likely to need.

  • Name of the VO. This should be reasonably short, distinctive, and must not clash with any existing VO. A VO will typically have two names, a short name (usually lower case), say "poohsticks" (an experiment running poohsticks simulations), and a DNS style name, such as vo.poohsticks.org (assuming they own the DNS name poohsticks.org.)
  • VO management: a VO manager (who can decide membership of VO, roles and responsibilities), plus usually at least one deputy or co-manager. You will need a mailing list (e.g. poohsticks-management@example.com)
  • VO support contacts (see below) - you can choose to register support people with the helpdesks; this is recommended for larger VOs or for VOs that have VO-specific software.
  • Security contacts - ideally at least two people who can respond quickly in the event of a security incident relating to a member of the VO, or to the VO as a whole. This also needs to have a mailing list.
  • A VOMS server. It manages the VO's memberships and the members' roles and subgroups. If you don't have a VOMS server, you can request to use GridPP's - for this, you will typically need authorisation from the prinicpal investigator (PI).
    • Optionally, roles and/or subgroups of the VO - see Security Considerations below.
  • Hardware requirements - memory size, disk space etc, types of storage required (working repositories, archives, databases)
  • Software requirements - any software beyond the basic Linux tools/libraries, including things which are part of standard distributions as they may not be installed by default.
  • Typical usage pattern - expected job frequency and variation over time, job length, data read and written per job etc.
  • General procedures - for example if the site has to request the installation of VO software.
  • Size of the VO (i.e how many users), to give a guide to how many pool accounts to create.

See the Phenogrid web site for an example of the sort of thing required. You can also have a look at a questionnaire which EGEE has used to start discussions with new VOs.

Security considerations

  • The VO managers are responsible for setting the aims of the VO: describe which work is in scope and which is not. Users must do only work which is in scope for the VO.
  • Some of the security policy documents are relevant to VO creation and operation, and the VO administrators need to ensure that they comply with the relevant policies.
  • The VO security contacts (via the mailing list) are expected to deal in a timely manner with breaches of policy. An incident may lead to the errant member of the VO being banned from the infrastructure, but if the infrastructure admins/coordinators don't feel the incident is being dealt with appropriately by the VO, the whole VO may get banned.
  • If the VO has data security issues, these should be discussed with GridPP during setup of the VO.
    • A typical VO will have every file readable by everyone in the VO, but define a role of users who are allowed to create, update, and delete files.
    • Grid middleware does support stronger security models.

VO software

Each VO will typically need some VO-specific services. Some of these are grid-based, such as file catalogues, resource brokers, compute and storage elements. There may be additional software required by the VO, and there will be means of making this available.

There are various models for dealing with the installation of VO-specific software. If only a few dedicated sites are involved the software can be directly installed by the administrators. If the software is relatively compact it can be shipped with the job in the sandbox, or downloaded from a Storage Element or a web site.

RAL Tier1 uses CVMFS to manage software for different VOs, particularly when they conflict (e.g. requiring different versions of python or libraries, etc.) Alternatively ask the admins on the sites that support your VO what they provide.

Support procedures

  • VOs should be prepared to support their users at least in the use of VO-specific software.
  • More general Grid support will be provided by GridPP as a whole, including community support by users themselves.
  • VO support liaison should sign up for a mailing list called GRIDPP-SUPPORT.
  • It is strongly recommended to make GridPP technical support contacts members of the VO at least initially, in order to help shoot any trouble that might arise.

The standard support route for all Grid users is the GGUS portal, as described here. For regional (e.g. GridPP-specific) VOs the tickets will generally be directed back to the UK Grid helpdesk. There is also a GridPP Users mailing list (see the JISCmail web site for subscription information).

The EGI VO Registration Form can be found here. It also provides a list of documents you should consult before creating a new VO.

The general procedure is sketched out in the section Instruction for VO administrators. The process is still under development, and anyone wishing to create a new VO should contact the Deployment Team for further help and information (in particular the Production Manager, Security Officer and VOMS Manager).


VO Deployment

Once a VO is set up, sites should be requested to support it - VOs typically do this via their GridPP support contact(s). At this point the VO should have:

  • Acceptable use policies defined
  • A VOMS server with VO information
  • Mailing lists
  • Resources allocated to it - CPU hours, disk storage, tape storage

Sites will then need to:

  • Add the VO to the list of supported VOs (e.g. gridmap files)
  • Add the VO to the UIs, if applicable (so users can do voms-proxy-init locally)
  • Allocate the resources to the VO, e.g. space tokens, disk pools

It is recommended that the VO attends the following meetings:

  • Tier 1 liaison meeting (if using Tier 1) - VOs should attend regularly

This page is a Key Document, and is the responsibility of Jens Jensen. It was last reviewed on 2016-01-19 when it was considered to be 93% complete. It was last judged to be accurate on 2016-01-19.