| General updates
|
Tuesday 25th May
- VO updates:
- LIGO issue looked like DNS resolution issue. Sam following up this week. Noted DIRAC errors truncated. Suggested trying non-IC disk. Tom reported: CERN@school-enabled disks - QMUL-disk, GLASGOW-disk and LIVERPOOL-disk worked on every network tried with various instances of the GridPP CernVM with the DIRAC UI installed... so issue elsewhere.
- LZ: Andy W: Met Paolo (Edinburgh) from LZ to walk through the basics (in parallel with LZ computing on the Grid being established). AW joined the GridPP VO to assist.
- gridpp-support at jiscmail ac uk created.
- Friday WMS issue?: svr024.gla.scotgrid.ac.uk has been removed from the Nagios configuration. Most CEs quickly returned from 'unknown' state, and others came back to normal in the next round of tests.
- Tier-1problems with secondary database system for Castor - resolved quickly.
Tuesday 19th May
- There was a GDB last week. The summary is available.
- The summary of the pre-GDB about batch systems is available.
- GridPP contacts for other VOs established (these are a current priority). Contacts expected to provide weekly updates on progress and status.
- DIRAC: Jens Jensen (-> Brian Davies?) – vo being created
- LIGO: Catalin Condurache – vo being created
- LOFAR: George Ryall
- LSST: Alessandra Forti
- LZ: David Colling
- UKQCD: Jeremy Coles
- Glexec: Matt is redirecting efforts from coming up with a relocatable glexec tarball, to a recipe that sites could follow. He comments that this would be a lot more involved than he would like for a tarball install, but thinks that it's the only way to proceed with any confidence.
- gstat is not supported. Note this ticket.
- The network issues resolution process/procedure...
- Assessment of the impact on User Communities/NGIs of the EGI core activities 2015 (results uploaded to the meeting page)
- The EGI conference is taking place this week - link to the detailed agenda.
- A reminder that the HEPSYSMAN & security training meeting is taking place 1st-3rd June.
- STFC (through Catalin Condurache) are interested in investigating joining EGI Fed Cloud
Tuesday 12th May
|
| WLCG Operations Coordination - Agendas
|
Tuesday 26th May
- Minutes of the 21st May WLCG ops coordination meeting are available. A snapshot of the updates:
- Baselines: New EMI release with Storm 1.11.8. dCache 2.10.28/2.12.8 verified and set as baseline. Torque 2.5.13 added to baselines table.
- MW issues: NTR.
- T0 services: CASTOR updated to 2.1.15. SRM validation ongoing. xroot is the main access protocol.
- T1 services: KIT, IN2P3 and RRC-KI-T1 dCahce upgrades.
- T0 news: LFC to be stopped 22nd June. CMS pilot role mappings changed from static to pool account mapping (for tests and scheduling to work properly).
- T1 feedback: NTR
- T2 feedback: NTR
- ALICE: Normal-high activity. No ops issues.
- ATLAS: >200k running jobs - requires single/multi-core job mix. Final data transfer tests this week. Request T1s to avoid major downtimes until the summer.
- CMS: Started Run2 DIGI-RECO. Several EOS issues. Increased use/dependence on global xrootd re-director at CERN (so criticality impact has increased).
- LHCb: Computing workshop last week. Problems with SARA SRM.
- glexec: NTR
- RFC proxies: SAM-Nagios refresh_proxy probe fixed in UMD-3. Once sites check ok, central services of the experiments will be checked.
- Machine job features: NTR
- MW readiness: dCahce v2.10.28 done. StoRM in progress. Progress on MW Readiness App.
- Multi-core:
- ATLAS goal is 80% production resources MC enabled. Looking to increase job lengths to 10-15hrs for 8-core slots to improve efficiency (also looking at a global fairshare). Sites should give more priority to multi-core over single core jobs. Sites that dynamically cap should adjust their cap to respect the 80% share.
- CMS consolidating deployment to T1s. Looking into multicore pilots inefficiencies.
- IPv6: NTR
- Squid monitoring/HTTP proxy: No news.
- Network & transfer metrics WG: perfSONAR (be aware of logjam vulnerability). Network performance new GGUS SU from 24th June (WLCG network throughput). Next meeting 3rd June.
- HTTP deployment: First meeting took place. Next: identification of an appropriate monitoring solution and the compilation of a full list of functionality that the experiments would like to see delivered via HTTP.
|
| Tier-1 - Status Page
|
- A reminder that there is a weekly Tier-1 experiment liaison meeting.
- The agenda follows this format:
- 1. Summary of Operational Status and Issues
- 2. Highlights/summary of the Tier1 Monday operations meeting (Grid Services; Fabric; CASTOR and Other)
- 3. Experiment plans and operational issues (CMS; ATLAS; LHCb; ALICE and Others)
- 4. Special presentations
- 5. Actions
- 6. Highlights for Operations Bulletin Latest
- 7. AoB
Tuesday 26th May
- The problems with our primary network router are still being followed up. Planned intervention this week (Thursday morning) to change routing for some non-Tier1 services. This will ease our ability to make future changes.
- We moved the rack containing our Castor standby databases last Wednesday. There were some problems and Castor was in a 'warning' state in the GOC DB for some time - although in the end the service was not affected.
- Our plans to update to the version of the Oracle database behind Castor are delayed so that we avoid interventions for this cause during at least the first part of LHC Run 2.
|
| Storage & Data Management - Agendas/Minutes
|
Wednesday 27 May
- Working on troubleshooting DIRAC data for/with LIGO (not to be confused with DiRAC or with any of the other things called DiRAC)
- Working on setting up DiRAC at Tier 1 (not to be confused with DIRAC or Dirac or with any other thing called Dirac)
- New secret user support list!
Tuesday 18th May
Tuesday 21st April
- Has there been any Tier-1 contact with DiRAC?
- Proposal to setup an 'other VOs' users list. GridPP-Users is too tied with WLCG projects.
Wednesday 15 April
- Backing up data from DiRAC to GridPP (tape)
- More case studies on supporting non-LHC VOs on GridPP: we have a lot of great stuff that can do great stuff - non-LHC VOs tend to have less regimented data models so maybe we need more case studies.
|
| Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06
|
Tuesday 26th May
- Delay noted for Sheffield.
Tuesday 12th May
- Issues noted with sync for Brunel, Liv, ECDF (see EGI ticket 113473). Message broker issues (memory related) are likely the underlying EGI problem.
- Need to check on VAC sync publishing.
|
| Documentation - KeyDocs
|
See the worst KeyDocs list for documents needing review now and the names of the responsible people.
Tuesday 21st April
- The Approved VOs document has been updated to take account of changes to the Ops Portal VOID cards.For SNOPLUS.SNOLAB.CA, the port numbers for voms02.gridpp.ac.uk and voms03.gridpp.ac.uk have both been updated from 15003 to 15503. Sites that support SNOPLUS.SNOLAB.CA should ensure that their configuration conforms to these settings: Approved VOs
- KeyDocs still need updating since agreements reached at last core ops meeting.
- New section in Wiki called "Project Management Pages".
The idea is to cluster all Self-Edited Site Tracking Tables
in here. Sites should keep entries in Current Activities
up to date. Once a Self-Edited Site Tracking Tables has
served its purpose, PM to move it to Historical Archive
or otherwise dispose of the table.
|
| Interoperation - EGI ops agendas
|
Tuesday 21st April
- There was an EGI ops meeting on Monday 20th.
- David updated the UK SL5 response.
- Please review the agenda/minutes.
Monday 9th March
- The agenda for February's EGI ops meeting is here. Minutes are here
- APEL 1.4.0
- Added Month and Year columns to primary key of CloudSummaries table in cloud schema.
- DPM-Xrootd 3.5.2 is in EPEL stable - this is the first version of the component compatible with xrootd4
- gLExec-wn - v. 1.2.3: lcmaps-plugins-c-pep 1.3.0-1 & mkgltempdir 0.0.5-1
- "The lcmaps-plugins-c-pep-1.3.0-1 preferably needs the argus-pep-api-c-2.3.0. This version will be released into EMI & UMD repositories in a near future."
- UMD 3.11.0 released on 16.02.2014, UMD 3.11.1 released on 4.03.2014
- lcg-CA 1.62 noted with an intention to broadcast these as they occur as opposed to monthly.
- EGI looking at the decommissioning of SL5, possibly by end of 2015, as a byproduct of adding CentOS 7 to UMD. NGIs to make a note if extended SL5 support is required.
- Vincenzo Spinoso has joined EGI Ops team from NGI_IT. Vincenzo will chair EGI Ops.
- Next meeting is April 20th.
|
| Monitoring - Links MyWLCG
|
Tuesday 31st March
Monday 7th December
|
| On-duty - Dashboard ROD rota
|
Monday 11th May
- Rota responses awaited from Andrew and Daniela.
- Handover summary should be uploaded to the bulletin please.
Tuesday 28th April
- Glasgow: A GLUE2 problem is transient and doesn't have a short-term solution (if the service status was checked a little more frequently it would help). Currently on hold. IC sometimes see this too.
- UCL: No change to the on-going situation. UCL has hopped from one downtime to another this week. Note – AM visiting UCL this week to setup VAC. Services will be decommissioned after this step.
Tuesday 21st April
- UCL have put themselves into a downtime until the 21st April. (Start of next week). Noted this in their outstanding tickets.
- Birmingham's availability has steadily recovered over the week - and the low availability ticket against them should be closable next week.
|
| Rollout Status WLCG Baseline
|
Tuesday 12th May
- MW Readiness WG meeting Wed May 6th at 4pm. Attended by Raul, Matt, Sam and Jeremy.
Tuesday 17th March
- Daniela has updated the [ https://www.gridpp.ac.uk/wiki/Staged_rollout_emi3 EMI-3 testing table]. Please check it is correct for your site. We want a clear view of where we are contributing.
- There is a middleware readiness meeting this Wednesday. Would be good if a few site representatives joined.
- Machine job features solution testing. Fed back that we will only commence tests if more documentation made available. This stops the HTC solution until after CHEP. Is there interest in testing other batch systems? Raul mentioned SLURM. There is also SGE and Torque.
References
|
| Security - Incident Procedure Policies Rota
|
Tuesday 18th May
- EGI SVG and CSIRT Advisory "Critical/Low?". "VENOM: QEMU vulnerability (CVE-2015-3456)
- Issue with VM appliance - image ships with ...
- EGI SVG Advisory 'High' Risk - Dirac SQL injection vulnerability [EGI-SVG-2014-7553]
- IGTF is about to release an update to the trust anchor repository (1.64)
Tuesday 12th May
|
|
| Services - PerfSonar dashboard | GridPP VOMS
|
- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).
Tuesday 12th May
- LHCOPN & LHCONE joint meeting at LBL June 1st & 2nd. Agenda taking shape.
Tuesday 31st March
Tuesday 10th March
- From the recent WLCG meeting, two slides (1 & 2) give the direction of the network monitoring and metrics progress: integration of perfSONAR event types into experiment monitoring and an architecture for data to get from RSV probes to client. Components described on slide 3.
- The next LHCOPN and LHCONE joint meeting will take place on Monday 1st and Tuesday 2nd of June 2015 in Berkeley (US) (hosted by LBL and ESnet).
|
| Tools - MyEGI Nagios
|
Tuesday 17th February
- Another period where message brokers were temporarily unavailable seen yesterday. Any news on the last follow-up?
Tuesday 27th January
- Unscheduled outage of the EGI message broker (GRNET) caused a short-lived disruption to GridPP site monitoring (jobs failed) last Thursday 22nd January. Suspect BDII caching meant no immediate failover to stomp://mq.cro-ngi.hr:6163/ from stomp://mq.afroditi.hellasgrid.gr:6163/
|
| VOs - GridPP VOMS VO IDs Approved VO table
|
Tuesday 19th May
- There is a current priority for enabling/supporting our joining communities.
Tuesday 5th May
- We have a number of VOs to be removed. Dedicated follow-up meeting proposed.
Tuesday 28th April
- For SNOPLUS.SNOLAB.CA, the port numbers for voms02.gridpp.ac.uk and voms03.gridpp.ac.uk have both been updated from 15003 to 15503.
Tuesday 31st March
- LIGO are in need of additional support for debugging some tests.
- LSST now enabled on 3 sites. No 'own' CVMFS yet.
|
| Site Updates
|
Tuesday 24th February
- Next review of status today.
Tuesday 27th January
- Squids not in GOCDB for: UCL; ECDF; Birmingham; Durham; RHUL; IC; Sussex; Lancaster
- Squids in GOCDB for: EFDA-JET; Manchester; Liverpool; Cambridge; Sheffield; Bristol; Brunel; QMUL; T1; Oxford; Glasgow; RALPPD.
Tuesday 2nd December
- Multicore status. Queues available (63%)
- YES: RAL T1; Brunel; Imperial; QMUL; Lancaster; Liverpool; Manchester; Glasgow; Cambridge; Oxford; RALPP; Sussex (12)
- NO: RHUL (testing); UCL; Sheffield (testing); Durham; ECDF (testing); Birmingham; Bristol (7)
- According to our table for cloud/VMs (26%)
- YES: RAL T1; Brunel; Imperial; Manchester; Oxford (5)
- NO: QMUL; RHUL; UCL; Lancaster; Liverpool; Sheffield; Durham; ECDF; Glasgow; Birmingham; Bristol; Cambridge; RALPP; Sussex (14)
- GridPP DIRAC jobs successful (58%)
- YES: Bristol; Glasgow; Lancaster; Liverpool; Manchester; Oxford; Sheffield; Brunel; IC; QMUL; RHUL (11)
- NO: Cambridge; Durham; RALPP; RAL T1 (4) + ECDF; Sussex; UCL; Birmingham (4)
- IPv6 status
- Allocation - 42%
- YES: RAL T1; Brunel; IC; QMUL; Manchester; Sheffield; Cambridge; Oxford (8)
- NO: RHUL; UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; Bristol; RALPP; Sussex
- Dual stack nodes - 21%
- YES: Brunel; IC; QMUL; Oxford (4)
- NO: RHUL; UCL; Lancaster; Glasgow; Liverpool; Manchester; Sheffield; Durham; ECDF; Birmingham; Bristol; Cambridge; RALPP; Sussex, RAL T1 (15)
Tuesday 21st October
- High loads seen in xroot by several sites: Liverpool and RALT1... and also Bristol (see Luke's TB-S email on 16/10 for questions about changes to help).
Tuesday 9th September
- Intel announced the new generation of Xeon based on Haswell.
|
|
