| General updates
|
- pre-GDB: https://indico.cern.ch/event/319819/
- GDB: https://indico.cern.ch/event/319745/overview
- downtime of the UK eScience CA Services on Wed 4th March from around 08:30 till about 10:00. They should be considered "at risk" until the afternoon.
- Atlas offline software releases in AGIS (lancaster)
Tuesday 17th February
- There was a GDB last week. See the agenda and minutes.
- Introduction
- Sys-on-Chip
- Ops coordination
- LHCOPN/ONE
- pre-GDB on cloud traceability
- HSF workshop
- INDIGO datacloud
- Those needing an LCG associated CERN account should reply to Jeremy's mail to TB-SUPPORT (12/02/14).
- The final EGI-InSPIRE review took place last week. Feedback so far is very positive. EGI.eu is now working on its 2015-2020 strategy and 'open commons' is playing a large role.
- Adam Huffman has left Imperial. Thanks to Adam for his contributions.
- Working with DiRAC ...
- There was a GridPP technical meeting last Friday. Any minutes?
- GridPP: Low availability/reliability sites may be contacted regarding options for adopting a lightweight setup. Sites not meeting experiment needs will be reviewed against suitability for Tier-2 inclusion - note ATLAS focus is moving to sites in the 80%-90% ASAP region.
Tuesday 10th February
|
| WLCG Operations Coordination - Agendas
|
Tuesday 17th February
Tuesday 10th February
- There was an ops coordination meeting last Thursday. Agenda. Minutes.
- News: An updated version of the agenda for the Okinawa workshop will be soon produced.
- MW Baselines: frontier/squid 2.7.STABLE9-22.
- MW Issues: Ghost vulnerability. ARGUS failures with latest Java (disable SSLv3 by default) – workaround in place.
- MW Services: Various upgrades mentioned for T0 and T1s.
- T0 news: Latest VOMS-admin in test. If no issues VOMRS off from 16th Feb. AFS UI closed 2nd Feb.
- T1: NTR
- T2: Asked about CERN accounts for T2 sysadmins. Should be possible!
- ALICE: Busy. ALICE Xrootd transfers to dCache can result in dark data when the destination disk server is very busy and the upload then times out on a checksum calculation.
- ATLAS: NTR - software week.
- CMS: Bigger run 2 MC started. Doing staging/consistency checks. >50% T1 capacity MC –partional slot model so slots will be used either way.
- LHCb: Run-1 legacy stripping almost done. Problems due to SARA disk failure. Some issues CERN simulation jobs. APEL issues 2 sites. VOMS tests.
- Glexec: With panda - 55 sites covered. T1 and biggest T2s. Issue resolved at BNL. Further ramping ATLAS.
- SHA-2: old VOMS retirement – will keep special firewall/router configs until VOMRS gone. VOMS config reminder broadcast soon.
- MJF: Site testing II starting – documentation question.
- MW readiness: Some steady progress (Condor-G; dCache Prometheus; Storm; DPM (xrootd 4 for CMS); EOD and M/W reporter.
- IPv6: Thanks to Duncan a dual-stack PS mesh is available.
- Squid mon + HTTP proxy discovery TF: Progress. Not all instances squids (154/178) registered so will broadcast again. Generate ATLAS/CMS pages for monitoring. Contributors busy. Additional CVMFS repos now available. Some sites put restriction.
- Net+Transfer metrics WG: Waiting for ATLAS input. Use case discussions on 18th February. Final deadline for PS 3.4.1 readiness is 16th Febraury - tickets created thereafter. Production [ http://pfomd.grid.iu.edu/ PS monitoring page] available (or is it!?).
Tuesday 3rd February
- Next meeting this Thursday 5th Febraury.
- Tier-2 feedback so far: Has there been any progress on CERN accounts or material access?
|
| Tier-1 - Status Page
|
Tuesday 17th February
- Some racks in the datacentre have to be moved to make way for latest procurement. This will affect one generation of production diskservers. Details are still being finalised but likely to be next Tuesday (24th Feb).
- The problems with our primary network router are still being followed up. There will be a test made on Thursday morning (19th Feb) between 07:45 and 08:30. During this test we anticipate a 5-minute break in connectivity to the Tier1.
|
| Documentation - KeyDocs
|
See the worst KeyDocs list for documents needing review now and the names of the responsible people.
Tuesday 17th February
- KeyDocs review continues. Deadline for removals requests 25th February.
Tuesday 10th February
- Survey of KeyDocs has started. If you own a document please take note. Documents remaining after this exercise will need to be brought fully up-to-date.
- A reminder that Steve created this Current Activities capture page. Only ARGUS at the moment!?
- See also the documents from Tom linked in the 'Other VOs' section.
Tuesday 3 Feb
New VO, LSST, in Approved VOs.
Tuesday 27th January
- Reviewed briefly at the core ops meeting last week. Doc owners are to be requested to review the usefulness of their keydocs and report any they believe should be removed from keydoc status.
Monday 8th December
- Chris Walker is handing over several 'other VO' documents. Some aspects of the role are being taken on by a combination of Duncan and Daniela... but the documents need reviewing in a core-ops meeting (next Thursday @ 11am being likely).
Tuesday 4th Nov
- New section in Wiki called "Project Management Pages".
The idea is to cluster all Self-Edited Site Tracking Tables
in here. Sites should keep entries in Current Activities
up to date. Once a Self-Edited Site Tracking Tables has
served its purpose, PM to move it to Historical Archive
or otherwise dispose of the table.
|
| Interoperation - EGI ops agendas
|
Monday 9th February
- The agenda for February's EGI ops meeting is here.
- Argus PAP update (1.6.2) out imminently. This should fix the SSL problems for sites affected after a java update. Hopefully this will solve Raul's problems in his ticket 111505.
- Other update notifications include APEL 1.4.0 (with a bunch of new features), Storm 1.11.6, UNICORE v7.2.0 and a new version of Globus in EPEL testing. UMD release 3.11.0 being prepared.
- Multicore republishing was discussed - if sites need to republish multicore data previously parsed as single core then they need to reparse the logs. Any site wanting to republish is asked to submit a ticket to apel support first.
- Operational issues were discussed whereby sites saw problems running ARGUS and CREAM on the same node, special care is taken if you want to take this mad path. See tickets 111464 and 111373.
- The decommissioning of the EMI2 APEL service and the UMD repositories was mentioned.
- Sites are advised to avoid using MySQL v.5.0 where possible.
- A comment was passed that the EPEL5 repository was becoming harder to keep up to date, bringing the effective EOL of SL5 closer for the community
- SL7 is a topic that still needs to be discussed in earnest in this forum.
- Next meeting is March 9th - effort to keep the number of meetings down.
Monday 12th January
- There was an EGI ops meeting on Monday 12th January. Minutes to follow.
- STORM 1.11.5
- SR: Another reminder to check your contacts
- Call for sites to test FTS3 and SQUID (Glasgow is testing the new squid)
- Multicore accounting: As per broadcast, sites have been asked to publish number of cores used by jobs but updating apel config where appropriate: instructions given to test publishing in accounting-devel.
- EGI forum in May: slightly delayed because they were waiting for approval to start fine-grained definition of programme. They will circulate the programme soon (end of the month).
|
| On-duty - Dashboard ROD rota
|
Monday 16th February
- Uneventful week. Nothing to report to ops meeting.
Monday 9th February
- A quiet week. The ECDF availability ticket should be closable next week.
- Shift: Gareth -> Daniela.
|
| Rollout Status WLCG Baseline
|
Tuesday 17th February
- A message from Maria Dimou: A request to you and the sites you collaborate with, to please install the MW Package Reporter.
The current version, documented in the wiki , it
is very easy to install and addresses all site and security requirements.
- Quick check on: 1. ARC CE readiness testing. 2. Machine job features testing.
Tuesday 20th January
- From Cristina's GDB talk last week note that EMI repositories will be frozen and the product team releases will become UMD-preview (repository content and webpages similar but now managed).
References
|
| Security - Incident Procedure Policies Rota
|
Tuesday 17th February
- The IGTF is about to release (23/2) an update to the trust anchor repository (1.62)
Tuesday 27th January
- Keep watching paikiti - WNs are popping up (seemingly) unpatched at some sites!
Tuesday 20th January
- DPM configuration instructions.
- Status of CVE-2014-9322.
- Steve's old perf package false positive.
|
|
| Services - PerfSonar dashboard | GridPP VOMS
|
- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).
Tuesday 3rd February
Tuesday 20th January
|
| Tickets
|
Monday 16th February 2015, 14.30 GMT
Only 19 open UK tickets today.
TIER 1
111120(12/1)
An atlas ticket concerning transfer failures between RAL and BNL. Brian mentioned last week that the lack of recent failures is due to atlas not attempting to transfer any older data recently. Perhaps this could do with being put into the ticket (and the ticket being put On Hold, or prodded some more)? Waiting for reply (29/1)
Also
111800 (17/2)
ARC CE issues at RAL detected.
RALPP
111703(11/2)
Atlas running glexec hammerclouds - having trouble at RALPP (and RAL, see 111699). The glexec experts have gotten involved on this one, and asked to take a peek at a proxy - not sure about anyone else, but I'd feel a tad uncomfortable sharing proxies, even with known and trusted experts as in this case. Am I being overly paranoid? Either way the ticket has gone a bit quiet. In progress (11/2)
100IT
108356 & 111333
Both the 100IT tickets are in Waiting for reply - the oldest one for quite a while - David asked a question a while back and no answer. The newest one asks if the 100IT logs made it to apel safely - I think what David has to do is submit a ticket with this question to the apel support team - have I got the right end of the stick?
Biomed Tickets at Manchester and Imperial
111356 & 111357
FYI There's a note at the bottom of both of these tickets that the version of CREAM that should fix this has been delayed until the end of February(ish).
Update - I read those updates wrong - the cream update has been released and these tickets have been (perhaps erroneously) closed.
|
| Tools - MyEGI Nagios
|
Tuesday 17th February
- Another period where message brokers were temporarily unavailable seen yesterday. Any news on the last follow-up?
Tuesday 27th January
- Unscheduled outage of the EGI message broker (GRNET) caused a short-lived disruption to GridPP site monitoring (jobs failed) last Thursday 22nd January. Suspect BDII caching meant no immediate failover to stomp://mq.cro-ngi.hr:6163/ from stomp://mq.afroditi.hellasgrid.gr:6163/
|
| VOs - GridPP VOMS VO IDs Approved VO table
|
Tuesday 17th February
Two changes to approved VOs (https://www.gridpp.ac.uk/wiki/GridPP_approved_VOs)
- LSST uses port 15003 (had been 15002, clashing with dzero)
- t2k has included a note that it';s software is now distrinuted via CVMFS.
Tuesday 17th February
- Some interest from UK LIGO users. Catalin setting up CVMFS. Issue with VOMS. "Our users have access to CILogon basic CA certificates, which I understand can in principal be entered into VOMS to provide a VO identity."
Tuesday 10th February
- There is now a Quick Guide to CVMFS.
- Part of Tom's quick guide series! There is also a Quick Guide to DIRAC.
- Feedback on both welcome... on the CVMFS front some mapping issues for regional VOs are being examined.
- GalDyn (UCLan galaxy simulations): test jobs have been submitted via DIRAC; now moving onto data storage testing via the official DIRAC tutorials.
- CERN@school has its first student on the grid with a grid certificate! LUCID data arriving.
- The CERN VM mechanism looks like it could provide an interesting way for new users to get a grid-ready UI by installing VirtualBox (or similar), downloading the CERN VM image, and applying a GridPP-specific context.
|
| Site Updates
|
Tuesday 27th January
- Squids not in GOCDB for: UCL; ECDF; Birmingham; Durham; RHUL; IC; Sussex; Lancaster
- Squids in GOCDB for: EFDA-JET; Manchester; Liverpool; Cambridge; Sheffield; Bristol; Brunel; QMUL; T1; Oxford; Glasgow; RALPPD.
Tuesday 2nd December
- Multicore status. Queues available (63%)
- YES: RAL T1; Brunel; Imperial; QMUL; Lancaster; Liverpool; Manchester; Glasgow; Cambridge; Oxford; RALPP; Sussex (12)
- NO: RHUL (testing); UCL; Sheffield (testing); Durham; ECDF (testing); Birmingham; Bristol (7)
- According to our table for cloud/VMs (26%)
- YES: RAL T1; Brunel; Imperial; Manchester; Oxford (5)
- NO: QMUL; RHUL; UCL; Lancaster; Liverpool; Sheffield; Durham; ECDF; Glasgow; Birmingham; Bristol; Cambridge; RALPP; Sussex (14)
- GridPP DIRAC jobs successful (58%)
- YES: Bristol; Glasgow; Lancaster; Liverpool; Manchester; Oxford; Sheffield; Brunel; IC; QMUL; RHUL (11)
- NO: Cambridge; Durham; RALPP; RAL T1 (4) + ECDF; Sussex; UCL; Birmingham (4)
- IPv6 status
- Allocation - 42%
- YES: RAL T1; Brunel; IC; QMUL; Manchester; Sheffield; Cambridge; Oxford (8)
- NO: RHUL; UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; Bristol; RALPP; Sussex
- Dual stack nodes - 21%
- YES: Brunel; IC; QMUL; Oxford (4)
- NO: RHUL; UCL; Lancaster; Glasgow; Liverpool; Manchester; Sheffield; Durham; ECDF; Birmingham; Bristol; Cambridge; RALPP; Sussex, RAL T1 (15)
Tuesday 21st October
- High loads seen in xroot by several sites: Liverpool and RALT1... and also Bristol (see Luke's TB-S email on 16/10 for questions about changes to help).
Tuesday 9th September
- Intel announced the new generation of Xeon based on Haswell.
|
|
