| General updates
|
Tuesday 17th March
- John Gordon mentions that he will report to the WLCG MB this afternoon about multicore accounting. The UK looks good in general, but there are two sites who are still not publishing. UCL (no plan) and Birmingham (following up). A number of others (IC, Glasgow, Liverpool) are publishing for the minority of their CEs. Please change all CEs, not just those queues running multicore.
- Andrew M has created a table to track cloud/VM status and plans. Please check it is up-to-date.
- Looking at [(http://planet.gridpp.ac.uk our blogs] it would be good if everyone reviews if there work/findings might benefit others and if so considers sharing it via blog postings. The latest postings in each blog are: Storage - 9th March 2015 (Storage accounting); NorthGrid - 24th February 2015 (Condor); Tier-1 - 22nd January 2015 (CEPH); SouthGrid - 14th October 2014 (Nagios monitoring non-LHC VOs); LondonGrid - 8th October 2014 (xrootD & ARGUS); ScotGrid - 24th March 2014 (3 coordinators).
- T2 draft reliability & availability figures for February have been circulated: ALICE; ATLAS; CMS and LHCb.
- Wahid leaves us this week - thanks for all your contributions!
Tuesday 10th March
- CERN computing seminars.
- Status of WNs in CVMFS
- Suggestion to write actions to Vidyo chat window
- Problem with dashboards – following announced GOCDB changes last week.
- cvmfs 2.1.20 is currently being certified at CERN and RAL. A small update to cvmfs-puppet is required to deploy cvmfs 2.1.20 clients.
- A new version of the CA Portal was released last week. It supports email address updates.
- JK requests those with host certificates with embedded email addresses to phase them out.
- CERN accounts: Note WLCG office acting as CERN guarantor for everyone, so by default everyone will be”‘LCG"’d in the database (new HR rules mean Externals automatically take on the organic-unit of the guarantor).
- RAL T1: Dell Force10 Z9000 Firmware update. Was on version 9.2.0.0 running on BIOS 3.0.0.3 and boot code 3.0.1.1. Testing version is 9.6.0.0P3 which runs on the same BIOS but requires boot code 3.0.1.4.
- For LIGO, Rob Quick informs JC that OSG has recently re-engaged with the VO and will update us on their VOMS situation in the next week.
- ARC CONDOR CPU time fix - see Steve Jones's TB-S email on 3rd March. (related to ATLAS submitting some longer running Multi-Core jobs which get killed with the original configuration?)
- A reminder that on Tuesday 10th there is a pre-GDB: agenda. AM: HEP and other sciences. PM: Cloud issues.
- Wednesday 11th there is a GDB: Agenda covering EGI/WLCG work, European procurements, WLCG operational costs and supporting other sciences.
|
| WLCG Operations Coordination - Agendas
|
Tuesday 17th February
- There is a WLCG ops coordination meeting this Thursday 14:30 (UK time): agenda: twiki/notes. Please could someone attend to represent the UK (JC is unavailable).
- An http deployment task force has been approved and will meet soon. Sam S will represent GridPP sites.
Monday 9th March
- There was a WLCG ops meeting on Thursday: Agenda | Minutes.
- In summary:
- News: Looking at survey – GDB & Workshop.
- M/W news: Baselines – new FTS 3.3.32 (shares fix).
- M/W issues: RAL problem upgrading gfal2 in WN ( conflict with gfal). New vulnerability (SSL).
- T0/T1 services: Various FTS and SE upgrades at CERN, RAL and BNL...
- Question about WN and UI status and maintenance wrt CVMFS.
- T0 news: LHCb LFC migration fine. Checking who uses LFC. VOMS-admin in place - some issues require follow-up.
- T1/T2: NTR
- ALICE: High activity. T1/T2 workshop. Some VOMS-admin problems.
- ATLAS: Taking cosmics. MC15 started (consider start of Run2). Care with maintenance requested. Transatlantic link problems encountered.
- CMS: Taking cosmics. Moderate load. Tier-1 tape tests continue. VOMRS migration in progress. Migration to a single global Condor pool for Analysis and Production (For T2s: 80% VOMS pilot; 10% VOMS production; 10% other).
- LHCb: "Run1 Legacy stripping" finished. Some VOMRS migration issues. LFC to DFC migration done.
- glexec: Panda campaign at 63 sites (+9)
- RFC proxies: Presentation at GDB. Checking experiment usage. SAM-Nagios fix easy. Default later this year.
- MJ features: NTR
- M/W readiness: Recent Storm/dCache/DPM validations done, more happening. ARC-CE tests for CMS in preparation. Next meeting 18th March.
- Multicore: Main objectives achieved. Entering 'passive' mode.
- IPv6: NTR
- Squid proxy & http proxy discovery: NTR
- Network & transfers WG: [(https://indico.cern.ch/event/372546/ Meeting 18th Feb]. Deadline for 3.4.1 passed. Campaign to correct some configurations. Testbed for 3.4.2rc. New meshes (inc. IPv6). Testing and evaluation of the pilot instances for esmond/maddash ongoing. Production instance: psomd.grid.iu.edu. LHCb pilot project to provide experiment agnostic prototype to access central datastore (esmond). Extending ATLAS FTS performance study to CMS and LHCb. Next meeting 18th March.
- Temporary ATLAS solution has been found to publish the HTCondor CEs in the BDII and OIM in a way that satisfies both ATLAS and SAM needs.
- Next meeting 19th March.
Monday 2nd March
|
| Tier-1 - Status Page
|
Tuesday 17th March
- The problems with our primary network router are still being followed up. We replaced the unit last Thursday (12th) - but still had a failure very quickly. We will be making further investigations this coming Thursday (19th) between 07:45 and 08:30 during which there may be a couple of short (few minute) breaks in connectivity.
|
| Storage & Data Management - Agendas/Minutes
|
Tuesday 17th March
- An annual DPM collaboration board will take place in coming weeks. Are there issues that sites want raised (in relation to things like roadmap, concerns about DPM, requests etc.)?
Wedn 11 March
- Few odd things encountered with DPM 1.8.9 upgrades. Sites not using puppet hack workarounds.
- An audience with LIGO
Tuesday 9th March
- DPM with puppet issues (trying this documented approach.
- Command for backing up DPM (see email to TB-SUPPORT on 3rd) - is it documented!?
|
| Documentation - KeyDocs
|
See the worst KeyDocs list for documents needing review now and the names of the responsible people.
Tuesday 17th March
- Documents reviewed at core-ops meeting last week. Agreed to remove 4 documents and add 2 (tarball status).
Tuesday 3rd March
- Proposals for about 4 keydocs to be removed/downgraded. For discussion at next core-ops or focus meeting.
Tuesday 24th February
- New section in Wiki called "Project Management Pages".
The idea is to cluster all Self-Edited Site Tracking Tables
in here. Sites should keep entries in Current Activities
up to date. Once a Self-Edited Site Tracking Tables has
served its purpose, PM to move it to Historical Archive
or otherwise dispose of the table.
|
| Interoperation - EGI ops agendas
|
Monday 9th March
- The agenda for February's EGI ops meeting is here. Minutes are here
- APEL 1.4.0
- Added Month and Year columns to primary key of CloudSummaries table in cloud schema.
- DPM-Xrootd 3.5.2 is in EPEL stable - this is the first version of the component compatible with xrootd4
- gLExec-wn - v. 1.2.3: lcmaps-plugins-c-pep 1.3.0-1 & mkgltempdir 0.0.5-1
- "The lcmaps-plugins-c-pep-1.3.0-1 preferably needs the argus-pep-api-c-2.3.0. This version will be released into EMI & UMD repositories in a near future."
- UMD 3.11.0 released on 16.02.2014, UMD 3.11.1 released on 4.03.2014
- lcg-CA 1.62 noted with an intention to broadcast these as they occur as opposed to monthly.
- EGI looking at the decommissioning of SL5, possibly by end of 2015, as a byproduct of adding CentOS 7 to UMD. NGIs to make a note if extended SL5 support is required.
- Vincenzo Spinoso has joined EGI Ops team from NGI_IT. Vincenzo will chair EGI Ops.
- Next meeting is April 20th.
|
| On-duty - Dashboard ROD rota
|
Monday 16th March
- Low availability alarm for UCL, but no alarm for the actual underlying cause (disk full on DPM).
- Ticket now issued.
- Kashif -> Daniela
Monday 9th March
- There was a problem with the upgrade of the dashboard when it lost the scope of the ROD role.
- Generally quiet.
- Low availability alarm for RALPP.
- Gareth->Kashif
|
| Rollout Status WLCG Baseline
|
Tuesday 17th March
- Daniela has updated the [ https://www.gridpp.ac.uk/wiki/Staged_rollout_emi3 EMI-3 testing table]. Please check it is correct for your site. We want a clear view of where we are contributing.
- There is a middleware readiness meeting this Wednesday. Would be good if a few site representatives joined.
- Machine job features solution testing. Fed back that we will only commence tests if more documentation made available. This stops the HTC solution until after CHEP. Is there interest in testing other batch systems? Raul mentioned SLURM. There is also SGE and Torque.
Tuesday 17th February
- A message from Maria Dimou: A request to you and the sites you collaborate with, to please install the MW Package Reporter.
The current version, documented in the wiki , it
is very easy to install and addresses all site and security requirements.
- Quick check on: 1. ARC CE readiness testing. 2. Machine job features testing.
References
|
| Security - Incident Procedure Policies Rota
|
Tuesday 17th March
- There will be a security team meeting this week. Are there any site issues to be raised?
- Note a new OpenSSL release is coming.
Monday 9th March
- Trust anchor release 1.62 (missing announcement!).
- From DG: "1.62 is indeed out and was released by the IGTF on Feb 23rd. The EGI release by an unfortunate coincidence got linked to a VOMS update for UMD (v3.3.3), so was released only today."
|
|
| Services - PerfSonar dashboard | GridPP VOMS
|
- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).
Tuesday 10th March
- From the recent WLCG meeting, two slides (1 & 2) give the direction of the network monitoring and metrics progress: integration of perfSONAR event types into experiment monitoring and an architecture for data to get from RSV probes to client. Components described on slide 3.
- The next LHCOPN and LHCONE joint meeting will take place on Monday 1st and Tuesday 2nd of June 2015 in Berkeley (US) (hosted by LBL and ESnet).
Tuesday 3rd March
Tuesday 3rd February
Tuesday 20th January
|
| Tickets
|
Monday 16th March 2015, 15.30 GMT
16 Open UK tickets this week. Half Red, Half Green.
RALPP and TIER 1 glexec HC tickets
111703 (RALPP)
111699 (TIER 1)
No news on these tickets since it was expected that a new stable HC job would be released last Tuesday. All very quiet.
OXFORD
112011(25/2)
Similar for this CMS glexec ticket - no news after Kashif asked for some more information way back. Waiting for reply (27/2)
TIER 1
109694(28/10/14)
The Sno+ gfal copying ticket. A lot of people are working on this, and attempts to recreate the problems seem to occasionally be devolving into "did we get this complicated command right?". At some point it might be necessary to get the gfal devs involved (are there gfal devs?). Waiting for reply (11/3)
Also there's the poor 100IT ticket, still waiting for a reply. The JET ticket also needs wrapping up, I put a reminder in to the end of it.
|
| Tools - MyEGI Nagios
|
Tuesday 17th February
- Another period where message brokers were temporarily unavailable seen yesterday. Any news on the last follow-up?
Tuesday 27th January
- Unscheduled outage of the EGI message broker (GRNET) caused a short-lived disruption to GridPP site monitoring (jobs failed) last Thursday 22nd January. Suspect BDII caching meant no immediate failover to stomp://mq.cro-ngi.hr:6163/ from stomp://mq.afroditi.hellasgrid.gr:6163/
|
| VOs - GridPP VOMS VO IDs Approved VO table
|
Monday 9th March
- SIXT VOMS parameters have changed.
Tuesday 17th February
Two changes to approved VOs (https://www.gridpp.ac.uk/wiki/GridPP_approved_VOs)
- LSST uses port 15003 (had been 15002, clashing with dzero)
- t2k has included a note that it';s software is now distrinuted via CVMFS.
Tuesday 17th February
- Some interest from UK LIGO users. Catalin setting up CVMFS. Issue with VOMS. "Our users have access to CILogon basic CA certificates, which I understand can in principal be entered into VOMS to provide a VO identity."
Tuesday 10th February
- There is now a Quick Guide to CVMFS.
- Part of Tom's quick guide series! There is also a Quick Guide to DIRAC.
- Feedback on both welcome... on the CVMFS front some mapping issues for regional VOs are being examined.
- GalDyn (UCLan galaxy simulations): test jobs have been submitted via DIRAC; now moving onto data storage testing via the official DIRAC tutorials.
- CERN@school has its first student on the grid with a grid certificate! LUCID data arriving.
- The CERN VM mechanism looks like it could provide an interesting way for new users to get a grid-ready UI by installing VirtualBox (or similar), downloading the CERN VM image, and applying a GridPP-specific context.
|
| Site Updates
|
Tuesday 24th February
- Next review of status today.
Tuesday 27th January
- Squids not in GOCDB for: UCL; ECDF; Birmingham; Durham; RHUL; IC; Sussex; Lancaster
- Squids in GOCDB for: EFDA-JET; Manchester; Liverpool; Cambridge; Sheffield; Bristol; Brunel; QMUL; T1; Oxford; Glasgow; RALPPD.
Tuesday 2nd December
- Multicore status. Queues available (63%)
- YES: RAL T1; Brunel; Imperial; QMUL; Lancaster; Liverpool; Manchester; Glasgow; Cambridge; Oxford; RALPP; Sussex (12)
- NO: RHUL (testing); UCL; Sheffield (testing); Durham; ECDF (testing); Birmingham; Bristol (7)
- According to our table for cloud/VMs (26%)
- YES: RAL T1; Brunel; Imperial; Manchester; Oxford (5)
- NO: QMUL; RHUL; UCL; Lancaster; Liverpool; Sheffield; Durham; ECDF; Glasgow; Birmingham; Bristol; Cambridge; RALPP; Sussex (14)
- GridPP DIRAC jobs successful (58%)
- YES: Bristol; Glasgow; Lancaster; Liverpool; Manchester; Oxford; Sheffield; Brunel; IC; QMUL; RHUL (11)
- NO: Cambridge; Durham; RALPP; RAL T1 (4) + ECDF; Sussex; UCL; Birmingham (4)
- IPv6 status
- Allocation - 42%
- YES: RAL T1; Brunel; IC; QMUL; Manchester; Sheffield; Cambridge; Oxford (8)
- NO: RHUL; UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; Bristol; RALPP; Sussex
- Dual stack nodes - 21%
- YES: Brunel; IC; QMUL; Oxford (4)
- NO: RHUL; UCL; Lancaster; Glasgow; Liverpool; Manchester; Sheffield; Durham; ECDF; Birmingham; Bristol; Cambridge; RALPP; Sussex, RAL T1 (15)
Tuesday 21st October
- High loads seen in xroot by several sites: Liverpool and RALT1... and also Bristol (see Luke's TB-S email on 16/10 for questions about changes to help).
Tuesday 9th September
- Intel announced the new generation of Xeon based on Haswell.
|
|
