Operations Bulletin 271014

Bulletin archive

Week commencing 20th October 2014

Task Areas

General updates Monday 27th October There is an EGI OMB this Thursday 30th October. Tuesday 21st October There is now a schedule of pre-GDB meetings to be aware of in coming months: November 11: volunteer computing December 8-9: a 2-day pre-GDB on Condor. Looking at the feedback of sites who recently migrated to Condor with some OSG Condor experts and developers. This is an outcome of HEPiX discussions which prompted wishes to share experiences and discuss advanced features and needs like multi-core job supports, containers and cloud support. January 13: pre-GDB on data management focused on data preservation impacts on sites and reducing the protocol zoo. A post-GDB ARGUS developer meeting will take place on 11 December. The following ROC profile SAM tests have been removed: ch.cern.FTS-ChannelList; ch.cern.FTS-InfoSites and org.gstat.SanityCheck. ATLAS FAX monitoring configurations need updating for EU privacy law compliance (see TB-S email on 30th Sept). A power cut at CERN on 16th October has had ongoing intermittent impacts on computer services. David has pruned a version of the blogs aggregator with a focus on our technical blog posts. Please take a look through the uploaded 55 HEPiX talks for items of interest! Alex Owen mentioned a call for FLOSSUK papers for the Spring 2015 conference in York. Monday 13th October UMD 3.8.1 was released on 9th October. There was a GridPP technical meeting last Friday. Minutes are available. The CHEP 2015 abstract deadline has moved to 25th October. Who has or will be submitting something? HEPiX is taking place this week in Nebraska. Minutes and actions have been made available from the October GDB. WLCG Operations Coordination - Agendas Tuesday 21st October There was a WLCG ops coordination meeting last Thursday 16th October. Minutes are available. The brief summary: News: A survey to investigate operational efficiency areas is being prepared - it covers managing changes, communication channels, monitoring, grid service administration. Expect it to be ready for public input soon. MW news: dCache 2.6.35 and 2.10.7 have been released - update is a priority. perfSONAR 3.4 has many fixes - SSLv3.0 vulnerability TBC - but baseline at 3.3.4 until docs ready. CESNET will co-maintain (classads) package needed by CREAM, WMS, L&B, UI, WN (was removed from EPEL). T0&T1 services: FTS and dCache upgrades. Oracle: DB migrations timetable is work in progress. T0: VOMS-ADMIN test cluster available again. Here is a link. lxplus5 now off. Job efficiency differences remain unexplained. High Availability FTS3 being discussed. Quattor End of Service (EoS) now end November. T1: NTR T2: NTR ALICE: Improved job eff at CERN. Operational instability noted on 15th Oct. ATLAS: Full Chain test considered successful. Reco campaign finished. Had 2 FTS3 problems. CMS: PromptRECO will need Tier-1 sites in addition to CERN. Still need SLS and AFS-UI after Oct. Encourage dCache update. LHCb: Proposes to re-visit the list of critical services before Run2. Setting up a prototype installation for http access and access federation where 70 % of the storage endpoints are accessible. gLExec: NTR Machine/Job features: Concluded on a single architecture for cloud and batch implementations. MW readiness: Last met 1st Oct. Verification process now well documented. More volunteer sites needed. DB designed to hold verification results. HTCondor testing request from ATLAS. MW Package Reporter designs discussed. Next meeting 19th November. Multicore: CMS started testing. SHA-2: Ops and ALICE VOMS opened last Monday. LHCb TBC. CMS reminded. ATLAS tests progressing. IPv6: All T1s should join HEPiX IPv6 WG. Dual-stack SAM to be put in place. Deploy PS in dualstack in 2015. OSG infrastructure ready. Squid and HTTP proxy: Squid registration campaign pushed to November. Network & transfer metrics WG: Request for sites to await 3.4 instructions. Internal security audit performed. Recommend all sites running 3.3 to temporarily disable SSL3 - (as at 16th Oct) patches coming. Other: Vidyo ticket opened. Tuesday 14th October The next WLCG Network and Transfer Metrics WG meeting is on Wednesday 15th October. The next ops coordination meeting is on Thursday 16th. Any T2 issues to raise? Tier-1 - Status Page Tuesday 21st October The intervention to put our Castor Oracle database configuration to its 'normal' state was completed OK last Tuesday. This was necessary following problems with one of the disk arrays hosting the primary database. We are having problems with our ARC CEs at the moment. The FTS3 service was updated yesterday, Monday 20th Oct. (To version 3.2.29-1) Storage & Data Management - Agendas/Minutes Wedn 22 Oct Martin Bly (T1) - HEPiX report Wedn 15 Oct Report on T1 CEPH plans from Alistair Dewhurst Feedback from last week's DPM collaboration meeting in Naples Wedn 01 Oct Summary of all the exciting events in Amsterdam last week - EUDAT, EGI big data, RDA DPM 1.8.9 early testing, and (separately) xroot4 early-ish testing. Supporting multiple VOs in one xroot server. Wedn 17 Sept iRODS - what it is and why it should choose to collapse on Betelgeuse 7. Technical problems with Vidyo Wedn 10 Sept. High load at L'pool causing low throughput - how to throttle xroot transfers (and is the load necessary or a bug?) Still testing WebFTS Prep for DPM workshop Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06 Tuesday 21st October Sussex publishing of accounting records looks to have stopped a few weeks ago. Tuesday 7th October GridPP metrics need updating for CMS. Any comments on the metrics page at the moment? APEL issues for Birmingham and Sussex, and the portal appears to stop at 1st October (being followed-up). APEL status: An issue at Sheffield? Documentation - KeyDocs See the worst KeyDocs list for documents needing review now and the names of the responsible people. Tuesday 7th October Keydocs were reviewed at the core-ops meeting last week. The situation with updates is improving. Main GridPP website expected to use Wordpress with a plug-in to cover the gridsite aspects. Interoperation - EGI ops agendas Tuesday 21st October There was an EGI ops meeting on Monday 20th October. [Minutes are available. URT: dCache server v. 2.6.35 verified by WLCG as baseline DPM 1.8.9 in EPEL-testing SR: If sites have been using/testing EMI-WN 3.1.0 please get in touch to help with verification. They seem keen for people to test this. New VOMS servers rollout: NGI SAMs being notified for reconfiguration as of yesterday. MySQL 5.0 EOL campaign: note progress in agenda. Monday 6th October There was a meeting today - link: https://wiki.egi.eu/wiki/Agenda-06-10-2014 EMI-WN 3.1.0 in SR: if anyone is running this in production please get in touch to help get this past rollout MySQL 5.0 noted to be under Oracle Lifetime Sustaining Support (for some time now). See agenda for guidance on middleware consequences classads "retired" from EPEL repos See Kashif's ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=108878 In the short term these are in the EMI 3 third party repo long term seek to return these to EPEL SL/SLC/CentOS 5 Support Lifetime This was highlighted, though not suggested to be urgent Monitoring - Links MyWLCG Tuesday 14th October Meeting last Friday, agenda: https://indico.cern.ch/event/343515/ minutes: https://indico.cern.ch/event/343515/material/minutes/minutes.html Presentation on FTS3 monitoring: https://indico.cern.ch/event/343515/contribution/5/material/slides/0.pdf SAM2 to be turned off 1st December (pre-prod 1st November): https://indico.cern.ch/event/343515/contribution/6/material/slides/1.pdf Next meeting ~ 31st October On-duty - Dashboard ROD rota Tuesday 21st October Possible issue with slow updates to GGUS tickets as viewed via ROD Dashboard (or is it a RAL cache issue?) EFDA - Recurring alarms owing to site availability. Just waiting for the bad period to age out of the 30-day window. SUSSEX - Ongoing problem. Matt has a ticket open with middleware developers. Monday 29th September Rota being updated. Monday 22nd September Quiet week - little to report. EGI is looking for people to join the ops portal review and testing TF. Rollout Status WLCG Baseline Tuesday 26th August EMI3 WN tarball update needed soon (GGUS 107869) Monday 28th July UMD v.3.8.0 was released on 24th July. References Staged Rollout pages (now separated into EMI1 & 2), and the page listing the deployed versions is extractable from the bdii, so they should all be reasonably up-to-date: http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout.html http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout_emi2.html http://www.hep.ph.ic.ac.uk/~dbauer/grid/state_of_the_nation.html Security - Incident Procedure Policies Rota Tuesday 21st October The IGTF has an update which introduced rather unexpected changes in the trust anchors used by Comodo for the TCS. There is now an additional set of SHA-2 intermediate CAs in addition to the old ones. SSL3 (Poodle) follow-up! Test your browser. Test your webserver. Tuesday 14th October Shellshock updates and follow-up Banning challenge status The EGI security dashboard.

Services - PerfSonar dashboard | GridPP VOMS - This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update). Tuesday 21st October There was a problem with the OPS VOMS on 18th October. Still waiting for PS 3.4 documentation to be updated. Also now SSL3 patches. Monday 13th October perfSONAR 3.4 has been released. Clear documentation on what to do (clean reinstall) coming this week together with information on mesh updates. See the GDB presentation slides 13 and 14. RIPE have sent a reminder to connect probes that have been handed out (some weeks ago now). Please could the following sites check their status: Lancaster; Brunel; Sussex; and ECDF. 20599 at RAL has never properly connected (DHCP issue?). Tickets Monday 20th October 2014, 14.30 BST Non-LHC VO Nagios Failures: VO Nagios Liverpool, Lancaster (we're getting better), Sheffield, EFDA-JET, The Tier 1, Bristol (in downtime) and Cambridge are on "the list". Most are transient, load based errors. gridpp, pheno and southgrid seem to be the VOs having most problems. We're up to 30 Open UK Tickets this week. TIER 1 109276(11/10) Submissions to the FTS3 REST interface was failing for some, probably after the certs or crls got stale. Andrew L suggested implementing an httpd restart which Maarten suggested was overkill - but anyhoo the submitter has come back to say that he hasn't seen a problem all week, so this ticket can likely be closed. In progress (20/10) 108845(27/9) Just a heads up that this atlas transfer failure ticket has been reopened. Reopened (18/10) RALPP 109360(15/10) This SNO+ ticket, about failing nagios tests at RALPP, hasn't been noticed yet. Assigned (15/10) Update - it was a ticket meant for the Tier 1 all along, In Progress now - actually, waiting for reply SHEFFIELD 109207(8/10) SNO+ would like the VO_SW_DIR environmental variable to point to cvmfs - I know Elena has looked at this, any progress? In progress (9/10) Similar story with another Sno_ ticket at Sheffield: 109223(9/10) BRUNEL 109379(16/10) SRM Nagios test failures. It looks like Brunels SE is in a dodgey state - too many ftp connection failures have been seen in the gridftp logs, httpd causing heavy load, possible SELinux problems after DB move. I'm sure if anyone has any input on this it would be appreciated. In progress (17/10) IMPERIAL/DIRAC 108723(23/9) I think this ticket from Chris W, containing questions for the DIRAC team, can be closed in favour of the new line of communication Daniela set up (https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users). Waiting for reply (7/10) Update - closed. ECDF AND GLASGOW Two very similar LHCB cvmfs tickets at these sites, any chance of a link? Or perhaps just a coincidence? ECDF: 109440 GLASGOW: 109439 Update - probably not, the Edinburgh ticket is now closed. Another Update I think that the SnoPlus ticket asking for srmcp help for SUSE users (107880) can be closed now, thanks to Duncan's tip for getting srmcp to work (num_steams=1). Another another Update The 100IT ticket (108356), about making fedcloud.egi.eu available at the site, has been updated by David B after some silence - currently waiting for a reply. Perhaps a sign that the process needs better documentation, in the event any sites go down this rabbit hole? Tools - MyEGI Nagios Tuesday 21st October VOMS servers for OPS based at CERN were down on Saturday 18th October for around 12 hours. Nagios tests started failing after existing proxy expired. Availibilty figures will be slightly affected but outage will be considered as unknown. Blog about VO Nagios http://southgrid.blogspot.co.uk/2014/10/nagios-monitoring-for-non-lhc-vos.html Tuesday 16th Sep Multi VO nagios maintained at Oxford has been upgraded to add ARC CE tests. https://vo-nagios.physics.ox.ac.uk/nagios/ It is currently monitoring gridpp, pheno, t2k.org, snoplus.snolab.ca, vo.southgrid.ac.uk Should we start monitoring it more actively and open ticket for sites failing tests ? VOs - GridPP VOMS VO IDs Approved VO table Thursday 23 October 2014 CVMFS keys - new cvmfs-keys package cvmfs-keys-1.5 Part of decoupling of CVMFS from CERN - support for keys from various repositories <voname>.gridpp.ac.uk -> <voname>.egi.eu Please update and change VO_<VONAME>_SW_DIR to point to new directory Monday 11th August Steve J sent an email to hyperk on 7th regarding "software directory for Hyperk (CVMFS)" and entries in the VO ID card. "Monday 14th July 2014" HyperK.org will initially use remote storage (irods at QMUL) - so CPU resources would be appreciated. "Monday 30 June 2104" HyperK.org request for support from other sites 2TB storage requested. CVMFS required Cernatschool.org WebDAV access to storage -world read works at QMUL. ideally will configure federated access with DFC as LFC allows. Impact Citation policy (https://www.gridpp.ac.uk/acknowledging.html) Site Updates Tuesday 21st October High loads seen in xroot by several sites: Liverpool and RALT1... and also Bristol (see Luke's TB-S email on 16/10 for questions about changes to help). Tuesday 9th September Intel announced the new generation of Xeon based on Haswell. Tuesday 20th May Various sites but notably Oxford have ARGUS problems. 100s of requests seen per minute. Performance issues have been noted after initial installation at RAL, QMUL and others.

Meeting Summaries

Project Management Board - MembersMinutes Quarterly Reports Empty GridPP ops meeting - Agendas Actions Core Tasks Empty RAL Tier-1 Experiment Liaison Meeting (Wednesday 13:30) Agenda Meeting takes place on Vidyo. Wednesday 22nd October 2014 Operations report There are ongoing problems with the ARC CEs. The production FTS3 service was upgraded to version 3.2.29-1 on Monday (20th Oct). The OGMA database system (Atlas3D/Frontier) has been updated and switched to using Oracle GoldenGate for updates from CERN. WLCG Grid Deployment Board - Agendas MB agendas Empty

NGI UK - Homepage CA Empty Events UK ATLAS - Shifter view News & Links Empty UK CMS Empty UK LHCb Empty UK OTHER N/A To note N/A