Difference between revisions of "GridPP VO Incubator"
(→LSST) |
|||
Line 415: | Line 415: | ||
The [https://www.gridpp.ac.uk/wiki/LSST_UK LSST UK] page. <br> | The [https://www.gridpp.ac.uk/wiki/LSST_UK LSST UK] page. <br> | ||
+ | All '''Closed''' action items can be found in the [LSST_archived_actions archive]. | ||
''PoC: Alessandra Forti (AF)''<br> | ''PoC: Alessandra Forti (AF)''<br> |
Revision as of 11:19, 3 October 2017
This page is for monitoring the progress of new(ish) GridPP VOs.
- PoC - Point of Contact
Contents
All new VOs
These tasks will need to be completed for all
Action ID | Action description | Owner | Target date | Status | Date closed | Notes |
---|---|---|---|---|---|---|
VOI-GEN-001 | Deploy test software to RVO CernVM-FS repositories. | Duncan, Daniela, Gareth, Alessandra, Ewan | 2015-05-26 | Closed | 2016-08-23 | New users in the Regional VOs will need to run test jobs using software in the RVO CernVM-FS repositories. This test software will need to be uploaded by the RVO admins. Instructions for doing this can be found here. Tested for vo.londongrid.ac.uk (--Daniela) |
VOI-GEN-002 | Write up the VO registration procedure | Tom | 2015-05-31 | Closed | 2016-08-23 | Guide started here - comments and feedback appreciated. Use gridpp guide. |
DEAP3600
PoC: Jeremy Coles (JC)
- Update requested February 2016.
- No response as of 21st March 2016
- 24th May: Awaiting main local user at RHUL to begin activities.
- 23rd Aug: DEAP3600 will generate around 10TB of (calibrated) data per year for 5 years, starting this year I think. The original (much larger) raw data are backed up on tape in Canada, but the calibrated data are not. For reasons of backup and access, we were hoping it would it be possible to get these 50TB calibrated data stored at the Tier0 at RAL.
The model would be only to use RAL for custodial data storage and to copy data as needed to Tier2 sites such as RHUL for analysis. There will also be around 60 TB (possible x 2 generations) of MC which will be kept only at a Tier2 because it can be regenerated in case it is lost.
- 25th Sept 17: No current activity. Query sent.
vo.DiRAC.ac.uk
PoC: Jens Jensen, Brian Davies
- 03/05/16: Durham have now moved 940TB in 6 months. Expect ~2.5PB in total from Durham.
- 26/09/17: The number of 8TB tapes being used by Cambridge/Durham/Edinburgh/Leicester are 45/149/41/16
- 26/09/17: Mark Wilkinson gave talk at GRIDPP39 https://indico.cern.ch/event/656544/contributions/2710500/attachments/1523457/2381245/DiRAC_GridPP_Meeting.pdf
Action ID | Action description | Owner | Target date | Status | Date closed | Notes |
---|---|---|---|---|---|---|
VOI-DIRAC-001 | Set up VO in GridPP | JJ | 2015-04-30 | Closed | ||
VOI-DIRAC-002 | Register DiRAC with EGI | JJ | 2015-05-31 | Closed | [1] | |
VOI-DIRAC-003 | Write up DiRAC site setup document | LH | 2015-08-31 | Closed | Version 1.3 circulated to DIRAC-USERS | |
VOI-DIRAC-004 | Re-evaluate data packaging method | JJ | 2015-11-30 | Closed | 2016-01-31 | Lydia re-engineered the packaging, focusing on tar, after extensive testing with GridPP |
VOI-DIRAC-005 | Restart transfers with new method (VOI-DIRAC-004) | LH | 2015-12-10 | Closed | 2016-01-31 | Successful large data transfer over new year 2016, http://gridpp-storage.blogspot.co.uk/2016/01/update-on-vodiracacuk-data-mopvement.html slight error in script (leading to some failures) being updated |
VOI-DIRAC-006 | Get Leicester ready for transfer | JJ | 2015-11-30 | Closed | 2016-07-05 | Currently (March '16) debugging script
05/07/16 Believe all work at Leicester is complete |
VOI-DIRAC-007 | Request or requirement for ACL between sites. | BD | 2015-11-30 | Closed | 2016-12-13 | Awaiting to here if it is a request or a requirement to keep data access separate between DIRAC sites. If needed new voms roles and castor setup may need to be enabled.
03/05/16 update: VO does indeed need separation. requires voms-proxy usage on transfers and groups to be enacted in voms/. Also needs setting namespace configuratiuon to separate different users. 23/05/16 Now plan ( if acceptable to VO) to explicitly amend gridmapfile to assign specific DNs associated with each site to their own uid within castor. 28/06/16 This is acceptable to VO. 05/07/16 Awaiting deployment by T1. 29/11/16 Still in progress, Further sites closer to needing data trnasfer. other sites need to be implemented. 13/12/2016 Implemented. |
VOI-DIRAC-008 | Identify next (third) DiRAC site | LH | 2016-03-22 | Closed | 2017-02-28 | 28/06/16 LH contacted to find out who contacts at other sites ( BD points out that these are Cambridge and ECDF so may know them or their colleagues from GridPP
07/02/17 Contacts detail update: Contact at Durham is Lydaia Heck. Contact at Cambridge (HPC) is Stuart Rankin. Contact at Edinburgh (EPCC) is Linda Dewar. Contact at Leicester is Jon Wakelin. Contact for Cambridge (DAMTP) may well be Juha Jäykkä. (Need to confirm this second conatct with LH.) |
VOI-DIRAC009 | Renewal of Robot certificate at Durham | LH | 2016-08-05 | Closed | 2017-03-28 | Robot certificate needs to be renewed at Durham (due in one month). LH unsure of methodology. Transfer completing last week so cert must have been updated. |
VOI-DIRAC010 | Track Data transfer from further Sites | BD | 2016-11-29 | Open | Following vo.dirac.ac.uk transfer working group meeting. dirac have identified initial data from non Durham sites which need to be transferred. ( ~700TB per site this needs verifying.)
07/02/17 First test transfers from Cambridge and Leicester now succeeding. 23/05/2017 Edinburgh now transferring ~2TB/day. Durham testing recall from RAL to Durham. 25/09/2017 In last three months, Cambridge/Edinburgh/Leicester have transferred 49/114/2633 files; equating to volumes of 6.5/10.4/41.6 TB of data. Issue raised at GRIDPP39 for mechanism of how to update data needs to be investigated.
|
EUCLID
PoC: Andrew Lahiff (AL)
Action ID | Action description | Owner | Target date | Status | Date closed | Notes |
---|---|---|---|---|---|---|
VOI-EUC-001 | Enable /cvmfs/euclid.in2p3.fr on RAL worker nodes | AL | 2016-02-24 | Open | [2016-04-01] Now works on a test worker node. Unable to deploy changes to all worker nodes due to an issue with Aquilon. [2016-04-13] No longer have access to Edinburgh stratum 1. [2016-05-24] No change, but seems to be not needed yet. | |
VOI-EUC-002 | Setup /cvmfs/euclid-uk.egi.eu repository | CC | 2016-02-26 | Open | [2016-05-25] Not needed yet. [2017-02-28] Giuseppe Congedo given write access to the Scotgrid CVMFS repository so he can run his code on UK grid sites. | |
VOI-EUC-003 | Setup accounts on RAL UIs | AL | 2016-02-26 | Closed | 2016-05-05 | All users who requested accounts have access to our UIs. |
VOI-EUC-004 | Run jobs at RAL | AL | 2016-02-26 | Closed | 2016-05-10 | Tom Kitching has successfully run jobs. |
VOI-EUC-005 | Ganga + HTCondor on RAL Tier-1 | AL/TW | 2016-11-15 | Closed | 2016-11-17 | Giuseppe Congedo (Edinburgh, EUCLID/GridPP has, after an issue with running Ganga on the NFS-based Tier-1 cluster, got going with the UserGuide (now based on Ganga). Thanks to AL + Ganga team for support and prompt responses to ensure as smooth operation as possible. |
GalDyn
- PoC: Tom Whyntie (TW)
- UCLan: Adam Clarke (AC)
Action ID | Action description | Owner | Target date | Status | Date closed | Notes |
---|---|---|---|---|---|---|
VOI-GAL-001 | Assist GalDyn users with CernVM creation and testing. | TW | 2015-02-18 | Closed | 2015-02-18 | GalDyn users have successfully instantiated CernVMs for accessing the grid. |
VOI-GAL-002 | Assist GalDyn users with running test jobs on the Imperial DIRAC instance. | TW | 2015-02-23 | Closed | 2015-02-23 | GalDyn users have successfully run test jobs on the Imperial DIRAC instance via a GridPP CernVM. |
VOI-GAL-003 | Assist GalDyn users with compiling user software on the CernVM. | TW | 2015-03-08 | Closed | 1016-02-15 | The code compiles and runs, but needs to be put in a grid/CernVM-FS-friendly format. |
VOI-GAL-004 | Create the GalDyn CernVM-FS repository | TW, CC | 2015-05-05 | Closed | 2015-05-05 | New CernVM-FS repository galdyn.egi.eu has been created on the RAL Stratum-1 for the GalDyn VO. |
VOI-GAL-005 | (Re)new grid certificate for AC | AC | 2015-02-22 | Closed | 2015-03-15 | UK CA managed to renew the old certificate. Work on hold - user preparing for PhD viva! |
VOI-GAL-006 | Creating an account for a UCLan student on the Lancaster cluster | TW/Robin Long (Lancaster) | 2016-09-15 | Defunct | The student (visiting from China) will pick up Adam's work on grid deployment for an upcoming paper. They will have a UCLan computing account but an account on the Lancaster cluster would further speed things up. Under discussion. 2016-11-16: TW emailed Victor D (group head) for an update. | |
VOI-GAL-007 | Recontact GalDyn once new student starts | Matt D | 2017-09-27 | Open | Victor D from GalDyn was contacted and Victor was interested in continuing the GridPP work. Waiting on a new student to start to take up the work. Will contact GalDyn towards the end of October to build some momentum. |
LIGO
PoC: Catalin Condurache (CC)
LIGO: Paul Hopkins (PH)
Other people: Andrew Lahiff (AL)
Action ID | Action description | Owner | Target date | Status | Date closed | Notes |
---|---|---|---|---|---|---|
VOI-LIGO-001 | Create the LIGO CernVM-FS repository | CC | 2014-12-01 | Closed | 2015-02-15 | New CernVM-FS repository ligo.egi.eu has been created on the RAL Stratum-1 for the LIGO VO. |
VOI-LIGO-002 | Assist LIGO users with using Condor + nordugrid to access ARC-CE@RAL. | AL, CC | 2015-12 | Closed | 2016-02-12 | Test jobs submitted from LIGO Condor instance to ARC-CE service at RAL were successful. |
VOI-LIGO-003 | Plan to run proper analyses jobs using scientists involvement | PH, CC | 2016-02-12 | Open | [2016-05-24]Still chasing scientists to run analyses jobs. Some promises. | |
VOI-LIGO-004 | Get file storage working via the GridPP CernVM. | PH, CC | 2016-02-24 | Closed | 2016-03-08 | PH managed to get file transfers working with the GridPP CernVM using bridged networking and getting the VM registered on the university network. |
VOI-LIGO-005 | Enable the OSG VO on RAL CEs and batch system. | AL | 2017-05-06 | Closed | 2017-05-23 | Job successfully being submitted to RAL, but have requested that single-core pilots are used rather than multi-core pilots, as LIGO are only running single-core jobs. We don't need to do anything else however. |
VOI-LIGO-006 | Get access to LIGO data via secure CVMFS working on RAL workernoes | CC,AL | 2017-06 | Open |
LOFAR
PoC: George Ryall (GR), from April 2016 - Alex Dibbo (AD)
21/03/16: LOFAR should be in a posetion to perform an analysis run on a limited number of VMs with real data in the next few weeks. (GR)
25/05/16: Note that LOFAR is a VO supported under 'STFC' not GridPP. Communication with SCD cloud users is good. A recent issue with the cloud storage/CEPH has led to less recent activity.
LSST
The LSST UK page.
All Closed action items can be found in the [LSST_archived_actions archive].
PoC: Alessandra Forti (AF)
Other people: Joe Zuntz (JZ), Andy Washbrook (AW), Steve Jones (SJ), Catalin Condurache (CC), Daniela Bauer (DB), Marcus Ebert (ME), Kashif Mohammad (KM), Dan Traynor (DT), Andrew Lahiff (AL), Gareth Roy (GR), Matt Doidge (MD), Peter Love (PL), Pavlo Svirin (PS), Rob Currie (RC)
Action ID | Action description | Owner | Target date | Status | Last update | Notes
|
---|---|---|---|---|---|---|
VOI-LSST-026 | USLSST Panda job submission: extend to other sites. | AF, PL, PS | Open | 2017-09-26 | First tests: Working: RALPP, Liverpool, QMUL, Glagow Not working: Oxford and Brunel (wrong queue used), IC (didn't enable multicore for small VOs. It is now enabled but need better configuration in panda), ECDF (storage problems, still to be revised). Pavlo will start again testing this week. There has been a general slow down over the summer. | |
VOI-LSST-027 | Help UK users to run Joe's ganga workflow | RC | Open | 2017-09-26 | Edinburgh and Southampton users are using the ganga workflow. Rob as has been helping them with it. However ganga/gridpp doesn't work anymore out of the box since Tom left. Documentation needs updating.
|
LZ
In Production: GridPP LZ page.
dune
In production: Link to the GridPP DUNE page.
na62.vo.gridpp.ac.uk
In production:
NA62 Monitoring server.
They welcome more sites supporting them.
October 2017: NA62 wants to use resources at CERN, which requires the GridPP DIRAC to submit to HTCondor. This requires major changes to DIRAC server and liaising with CERN to get NA62 enabled correctly. In progress.
PRaVDA
- PoC: Mark Slater/Matt Williams (MS/MW)
- End User: Tony Price (TP)
- Update requested 3rd Feb., 19th Feb. 2016 by TW. TP replied 2016-03-21 - they have been busy building the actual device!
- TP in the process of changing roles. Need to finalise the new end user.
- 20th Feb: Asked TP about status - currently processing data so need of simulations reduced. Will pick up again when more simulation required.
Action ID | Action description | Owner | Target date | Status | Date closed | Notes |
---|---|---|---|---|---|---|
VOI-PRA-001 | Get PRaVDA up and running with DIRAC and Ganga. | MS/MW | 2015-10-01 | Closed | 2016-03-21 | TP has successfully got simulations running using DIRAC and Ganga. |
VOI-PRA-002 | Issues with DIRAC, Ganga and LFN names when copying data back. | MS/MW | 2015-03-21 | In progress | MS/MW assisting on the Ganga side. | |
VOI-PRA-003 | TP changing roles. Need to make contact with new end user. | MS | 2016-05-23 | In progress | ||
VOI-PRA-004 | Waiting on data processing to be completed and more simulations to be required. | MS | 2017-02-20 | In progress |
SKA Regional Centre
- PoC: Andrew McNab (AM)
- SKA: David Mulcahy (DM, leaving Oct 17) / Rohini Joshi (RJ, joined Sep 17 funded by AENEAS)
- VO: skatelescope.eu
Action ID | Action description | Owner | Target date | Status | Last update | Notes |
---|---|---|---|---|---|---|
VOI-SKA-001 | LOFAR tests for SKA with DIRAC | DM/AM | 2017-12-31 | Open | 2017-09-05 | LOFAR application now running from cvmfs at LCG.UKI-NORTHGRID-MAN-HEP.uk, with input jobs matched to input data in GridPP DIRAC File Catalog, stored on DPM. Working on mass input of data from Groningen via grid jobs (wget with password then DIRAC data management commands) |
VOI-SKA-002 | LOFAR tests for SKA on DIRAC/OpenStack | AM | 2017-12-31 | Open | 2017-08-29 | GridPP DIRAC SAM tests for skatelescope.eu running at DataCentred. Manchester storage associated with DataCentred in GridPP DIRAC so it can match SKA/LOFAR jobs |
VOI-SKA-003 | Add more sites with skatelescope.eu VO / GridPP DIRAC | AM | 2017-12-31 | Open | 2017-10-03 | QMUL joins Manchester, Imperial, Cambridge in passing SAM tests and providing storage. More volunteer sites welcome: need 10-20TB on grid storage that is set up in GridPP DIRAC. |
SNO+
- PoC: Pete Gronbech (PG)
- SNO+: David Auty (DA)
Action ID | Action description | Owner | Target date | Status | Last update | Notes |
---|---|---|---|---|---|---|
VOI-SNO+-001 | Check on progress via GridPP-Support list. | PG | 2016-02-17 | Closed | 2016-03-10 | See VOI-SNO+-003 - success, closing this for now. |
VOI-SNO+-002 | MM to join GridPP Storage meeting | PG | 2016-02-24 | Closed | 2016-02-24 | MM joined the meeting to discuss requirements and various options. See minutes. |
VOI-SNO+-003 | MM to transfer files out of the SNO+ cavern via an FTP server. | PG | 2016-02-17 | Closed | 2016-03-10 | Success after fantastic support/discussion on the GRIDPP-SUPPORT mailing list. |
VOI-SNO+-004 | Check on progress via GridPP-Support list (16thMay). | PG | 2016-05-31 | Closed. Will be reopened if there is demand. | 2017-06-06 | Help requested for setting up Condor submission tools. Dialogue started on the mailing list. See also: https://ggus.eu/index.php?mode=ticket_info&ticket_id=119167 |
- 23rd August 2016: Condor installed on test server. Like other tools it is not contained - not versioned or quarantined so unable to run on production server. Not sure how to move to production. Solution is a hack and may not scale. No time to progress on IC support side. Good to resolve as US sites have this submission route.
SuperNEMO
- PoC: Pete Gronbech (PG)
- SuperNEMO: Ben Morgan (BM)
Action ID | Action description | Owner | Target date | Status | Last update | Notes |
---|---|---|---|---|---|---|
VOI-SuperNEMO-001 | Check on progress via GridPP-Support list | PG | 2016-02-17 | On going | 2016-02-17 | |
VOI-SuperNEMO-002 | Resurrect the SuperNEMO VO. | PG | 2016-02-17 | Open | 2016-03-22 | |
VOI-SuperNEMO-003 | Establish the feasibility of using the Ganga interface to the Atlas Metadata Interface (AMI) | PG, MS | 2016-02-24 | On going | 2016-03-22 | |
VOI-SuperNEMO-004 | PG asked if more help required (10th April) | PG | 2016-05-31 | On going | 2016-05-23 | UK SuperNEMO are having a meeting in May so will get back with an update then. |
UKQCD
PoC: Jeremy Coles (JC)
- Update requested February 2016.
- Update 21st March: "Hoping to do more with the gridpp resources".
- Preliminary result in conference proceedings - "Investigating Some Technical Improvements to Glueball Calculations" e-Print: arXiv:1511.09303.
- 24th May: Will try and leverage some of the international lattice data grid stuff. Nothing immediate planned.
- 22nd August: No recent activity or planned activities.
- 25th Sept 17: No further work planned.