Difference between revisions of "GridPP VO Incubator"

From GridPP Wiki
Jump to: navigation, search
(EUCLID)
(LSST)
Line 392: Line 392:
 
|
 
|
 
|In progress
 
|In progress
|2015-12-11
+
|2016-02-22
|Understand how LHC experiments do it, understand how dirac does it, test jobs longer than a day with dirac, talk to FNAL about longer proxies. Ticket with FNAL [https://fermi.service-now.com/nav_to.do?uri=sc_req_item.do%3Fsys_id=22ae3abe6f270200000131012e3ee40d%26sysparm_stack=sc_req_item_list.do%3Fsysparm_query=active=true  RITM0302478] need authentication.
+
|Ticket with FNAL [https://fermi.service-now.com/nav_to.do?uri=sc_req_item.do%3Fsys_id=22ae3abe6f270200000131012e3ee40d%26sysparm_stack=sc_req_item_list.do%3Fsysparm_query=active=true  RITM0302478]. The mechanisms of renewal and how LHC experiments do it is now understood. Using dirac may help, but this needs to be tested. See VO-LSST-18
  
 
|-
 
|-
Line 417: Line 417:
 
|AF, JZ
 
|AF, JZ
 
|
 
|
|In progress
+
|Closed
|2016-02-07
+
|2016-02-22
|https://its.cern.ch/jira/browse/GRIDPP-30<br>Found way to register files, but still cannot get them back. Problem is with the naming scheme assumptions on the storages and in the catalogue for existing files. More info in<br>https://groups.google.com/forum/#!topic/diracgrid-forum/sclcLrQBPFY
+
|https://its.cern.ch/jira/browse/GRIDPP-30<br>More info in<br>https://groups.google.com/forum/#!topic/diracgrid-forum/sclcLrQBPFY.<br>Files transferred to Manchester with correct directory naming scheme is working. See VOI-LSST-015.
  
 
|-
 
|-
Line 439: Line 439:
 
|Requires direct job submission and longer proxies.<br>https://ggus.eu/index.php?mode=ticket_info&ticket_id=119265<br>https://ggus.eu/index.php?mode=ticket_info&ticket_id=119322<br>https://ggus.eu/index.php?mode=ticket_info&ticket_id=119323.<br>Joe has submitted his jobs and the majority run. There  
 
|Requires direct job submission and longer proxies.<br>https://ggus.eu/index.php?mode=ticket_info&ticket_id=119265<br>https://ggus.eu/index.php?mode=ticket_info&ticket_id=119322<br>https://ggus.eu/index.php?mode=ticket_info&ticket_id=119323.<br>Joe has submitted his jobs and the majority run. There  
 
were problems in Bristol and Cambridge which are under investigation. Joe was recommended to remove  
 
were problems in Bristol and Cambridge which are under investigation. Joe was recommended to remove  
them from the list for now and resubmit the jobs.
+
them from the list for now and resubmit the jobs. With this
  
 
|-
 
|-
Line 449: Line 449:
 
| 2016-02-08
 
| 2016-02-08
 
| The local time out problem with the data access was solved and data is available now for jobs and transfers without timing out
 
| The local time out problem with the data access was solved and data is available now for jobs and transfers without timing out
 +
 +
|-
 +
| VOI-LSST-18
 +
| Adapt Joe's ganga and bash scripts to submit to dirac and use the dirac file catalog
 +
| AF
 +
|
 +
| In progress
 +
| 2016-02-22
 +
| the current scripts use gfal which is incompatible with dirac job submission. This was a stumbling block that forced us to go back to use gridpp and direct job submission. The scripts are modified and are currently running two test jobs in UKI-SOUTHGRID-RALPP (ARC).
  
 
|}
 
|}

Revision as of 13:51, 22 February 2016

This page is for monitoring the progress of new(ish) GridPP VOs.

  • PoC - Point of Contact

All new VOs

These tasks will need to be completed for all

Action ID Action description Owner Target date Status Date closed Notes
VOI-GEN-001 Deploy test software to RVO CernVM-FS repositories. Duncan, Daniela, Gareth, Alessandra, Ewan 2015-05-26 Open N/A New users in the Regional VOs will need to run test jobs using software in the RVO CernVM-FS repositories. This test software will need to be uploaded by the RVO admins. Instructions for doing this can be found here.
VOI-GEN-002 Write up the VO registration procedure Tom 2015-05-31 In progress N/A Guide started here - comments and feedback appreciated.

DEAP3600

PoC: Jeremy Coles (JC)

  • Update requested February 2016.

vo.DiRAC.ac.uk

PoC: Jens Jensen, Brian Davies


Action ID Action description Owner Target date Status Date closed Notes
VOI-DIRAC-001 Set up VO in GridPP JJ 2015-04-30 Closed
VOI-DIRAC-002 Register DiRAC with EGI JJ 2015-05-31 Closed [1]
VOI-DIRAC-003 Write up DiRAC site setup document LH 2015-08-31 Closed Version 1.3 circulated to DIRAC-USERS
VOI-DIRAC-004 Re-evaluate data packaging method JJ 2015-11-30 In Progress Need a new method. Sam contributing.
VOI-DIRAC-005 Restart transfers with new method (VOI-DIRAC-004) LH 2015-12-10 In Progress Successful large data transfer over new year 2016, http://gridpp-storage.blogspot.co.uk/2016/01/update-on-vodiracacuk-data-mopvement.html slight error in script (leading to some failures) being updated
VOI-DIRAC-006 Get Leicester ready for transfer JJ 2015-11-30 In Progress Leicester now have certificates again. Need to await solution to VO-DIRAC-004
VOI-DIRAC-007 Request or requirement for ACL between sites. BD 2015-11-30 In Progress Awaiting to here if it is a request or a requirement to keep data access seperate bewteen DIRAC sites. I fneeded new voms roles and castor setup may need to be enabled.



EUCLID

PoC: Andrew Lahiff (AL)

Action ID Action description Owner Target date Status Date closed Notes
VOI-EUC-001 Enable /cvmfs/euclid.in2p3.fr on RAL worker nodes AL 2016-02-24 Open
VOI-EUC-002 Setup /cvmfs/euclid-uk.egi.eu repository CC 2016-02-26 Open
VOI-EUC-003 Setup accounts on RAL UIs AL 2016-02-26 Open

GalDyn

PoC: Tom Whyntie (TW)

Action ID Action description Owner Target date Status Date closed Notes
VOI-GAL-001 Assist GalDyn users with CernVM creation and testing. TW 2015-02-18 Closed 2015-02-18 GalDyn users have successfully instantiated CernVMs for accessing the grid.
VOI-GAL-002 Assist GalDyn users with running test jobs on the Imperial DIRAC instance. TW 2015-02-23 Closed 2015-02-23 GalDyn users have successfully run test jobs on the Imperial DIRAC instance via a GridPP CernVM.
VOI-GAL-003 Assist GalDyn users with compiling user software on the CernVM. TW 2015-03-08 Closed 1016-02-15 The code compiles and runs, but needs to be put in a grid/CernVM-FS-friendly format.
VOI-GAL-004 Create the GalDyn CernVM-FS repository TW, CC 2015-05-05 Closed 2015-05-05 New CernVM-FS repository galdyn.egi.eu has been created on the RAL Stratum-1 for the GalDyn VO.

LIGO

PoC: Catalin Condurache (CC)
LIGO: Paul Hopkins (PH)
Other people: Andrew Lahiff (AL)

Action ID Action description Owner Target date Status Date closed Notes
VOI-LIGO-001 Create the LIGO CernVM-FS repository CC 2014-12-01 Closed 2015-02-15 New CernVM-FS repository ligo.egi.eu has been created on the RAL Stratum-1 for the LIGO VO.
VOI-LIGO-002 Assist LIGO users with using Condor + nordugrid to access ARC-CE@RAL. AL, CC 2015-12 Closed 2016-02-12 Test jobs submitted from LIGO Condor instance to ARC-CE service at RAL were successful.
VOI-LIGO-003 Plan to run proper analyses jobs using scientists involvement PH, CC 2016-02-12 Open

LOFAR

PoC: George Ryall (GR)

LSST

PoC: Alessandra Forti (AF)
Other people: Joe Zuntz (JZ), Andy Washbrook (AW), Ewan McMahon (EM), Steve Jones (SJ), Catalin Condurache (CC), Daniela Bauer (DB), Marcus Ebert (ME), Kashif Mohammad (KM)

Action ID Action description Owner Target date Status Last update Notes
VOI-LSST-001 Ganga direct job submission using Northgrid AF, JZ 2015-01-31 Closed 2015-12-11 Do direct job submission testing using northgrid infrastructure and ganga
VOI-LSST-002 Get European users on the LSST VOMS server at FNAL AF, JZ 2015-02-28 Closed 2015-12-11
VOI-LSST-003 Enable LSST at sites AF, AW, EM, SJ 2015-06-30 Closed 2015-12-11 Get the correct configuration. Sites get the info from Operations portal which has some obsolete information about VOMS. We should ask to fix it.
VOI-LSST-004 Find which LSST CVMFS stratum0 is usable by us AF 2015-08-10 Closed 2015-12-11 3 instances, in France, OSG and FNAL. Chosen FNAL
VOI-LSST-005 Get the repository at FNAL and replicated at RAL AF, CC 2015-08-25 Closed 2015-12-11 Repository automatically mounted at EGI sites as part of OSG EGI agreement, but was not replicated on any EGI stratum1. https://ggus.eu/index.php?mode=ticket_info&ticket_id=115335


VOI-LSST-006 Run jobs using LSST CVMFS using direct job submission and ganga JZ 2015-09-30 Closed 2015-12-11 Joe uploaded the software and used it to run jobs with direct job submission using lsst VO
VOI-LSST-007 Enable LSST on Dirac AF, DB, AW, EM, SJ 2015-09-30 Closed 2015-12-11 Got the Dirac pilot DN assigned to the pilot role on VOMS, enabled the VO on Dirac, enabled pilot at sites, tested submission and fixed misconfigured sites. JIRA
1) https://its.cern.ch/jira/browse/GRIDPP-22
2) https://ggus.eu/index.php?mode=ticket_info&ticket_id=117585
3) https://ggus.eu/index.php?mode=ticket_info&ticket_id=117586
VOI-LSST-008 Test Ganga Dirac setup AF Closed 2015-12-11 1) https://its.cern.ch/jira/browse/GRIDPP-22
2) https://github.com/ganga-devs/ganga/issues/45
3) https://its.cern.ch/jira/browse/GRIDPP-29

since gfal_util doesn't work instead of debugging it I've started to look into the dirac file catalogue clients, which have better compatiblity chances on top of having the file catalogue. See VO-LSST-014

VOI-LSST-009 voms-proxy-init EMI-3 not working AF, DB Closed 2015-01-11 FNAL upgraded the VOMS to a EMI-3 clients compatible version. Ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=117587 closed.
VOI-LSST-010 Update the Operations portal with correct VOMS info AF Closed 2015-01-05 LSST US managers added voms1 and voms2 to the ops portal. http://operations-portal.egi.eu/vo/view/voname/lsst
VOI-LSST-011 Long lived proxies AF In progress 2016-02-22 Ticket with FNAL RITM0302478. The mechanisms of renewal and how LHC experiments do it is now understood. Using dirac may help, but this needs to be tested. See VO-LSST-18
VOI-LSST-012 Check with NERSC how to use gridftp, authentication and authorisation mechanisms JZ, ME, AF Closed 2015-01-11 NERSC account required to do the transfers. Marcus has now an account.
VOI-LSST-013 Test gridtp transfers with NERSC JZ, ME Closed 2015-01-25 Data copied from NERSC with globus-url-copy
VOI-LSST-014 Investigate Dirac file catalogue usage AF, JZ Closed 2016-02-22 https://its.cern.ch/jira/browse/GRIDPP-30
More info in
https://groups.google.com/forum/#!topic/diracgrid-forum/sclcLrQBPFY.
Files transferred to Manchester with correct directory naming scheme is working. See VOI-LSST-015.
VOI-LSST-015 Spread the LSST data on different sites AF In progress 2016-02-09 https://its.cern.ch/jira/browse/GRIDPP-31
All data on a test SE instance for 20,000 foreseen jobs. Currently only 3 SEs (Ed, Man, Liv) enabled and mostly on older hardware. Replication will also allow to solve the registration problem by copying to a subdirectory <storage_path>/lsst/lsst. First leg of the distribution done. All files copied to Manchester.
VOI-LSST-016 Move to use gridpp VO because GridPP management requested to run an analysis with any "means possible" AF, JZ, RF, DB, KM In progress 2016-02-16 Requires direct job submission and longer proxies.
https://ggus.eu/index.php?mode=ticket_info&ticket_id=119265
https://ggus.eu/index.php?mode=ticket_info&ticket_id=119322
https://ggus.eu/index.php?mode=ticket_info&ticket_id=119323.
Joe has submitted his jobs and the majority run. There

were problems in Bristol and Cambridge which are under investigation. Joe was recommended to remove them from the list for now and resubmit the jobs. With this

VOI-LSST-17 Edinburgh LSST data access ME Closed 2016-02-08 The local time out problem with the data access was solved and data is available now for jobs and transfers without timing out
VOI-LSST-18 Adapt Joe's ganga and bash scripts to submit to dirac and use the dirac file catalog AF In progress 2016-02-22 the current scripts use gfal which is incompatible with dirac job submission. This was a stumbling block that forced us to go back to use gridpp and direct job submission. The scripts are modified and are currently running two test jobs in UKI-SOUTHGRID-RALPP (ARC).

LZ

PoC: David Colling (DC)

PRaVDA

PoC: Mark Slater/ Matt Williams (MS/MW)

SNO+

Action ID Action description Owner Target date Status Last update Notes
VOI-SNO+-001 Check on progress via GridPP-Support list PG 2016-02-17 On going


SuperNEMO

PoC: Pete Gronbech

Action ID Action description Owner Target date Status Last update Notes
VOI-SuperNEMO-001 Check on progress via GridPP-Support list PG 2016-02-17 On going

UKQCD

PoC: Jeremy Coles (JC)

  • Update requested February 2016.