Operations Bulletin 130415

From GridPP Wiki
Jump to: navigation, search

Bulletin archive


Week commencing 6th April 2015
Task Areas
General updates

Tuesday 7th April

  • Proton beams are back in the LHC
  • DIRAC (framework) users survey circulated.
  • ATLAS: Reprocessing of full 2012 dataset - resources should have been busy over the Easter weekend.
  • Training Course, run by the authors, on ISO 16363 for Auditors and Managers of Digital Repositories - CERN 15th-19th June.


Tuesday 31st March

  • Winnie has an issue in that 'SL5 (works) vs SL6 (not) emi-3 UI'.
  • There is a DPM Collaboration Board this Tuesday.
  • The GridPP34 agenda is now being created; are there specific topics anyone wishes to put forward?
  • NGI-FR report Argus cannot run with the last version of the Java JDK which requires argus-pap-1.6.2-1.el6.noarch.rpm only available in EMI not UMB repos. (See tickets 111505 and 112606).
  • A look through the HEPiX talks given last week is recommended.
  • There was an EGI OMB last Thursday 26th March: agenda.
    • Operations
    • Request to register iRODS endpoints in GOCDB.
    • Advisory groups being created for ops and accounting portals.
    • Some EGI/NGI ARGUS servers are 'exposed'. Please check. Also NGI Argus service type should be "emi.argus" not "ngi.argus".
    • Plans to use ARGO for monitoring of central series
    • Updated technology provider UA (please review)
    • Reminder of EGI conference 18th-22nd May.
    • Security: Some FedCloud security packaged defined; OpenSSL Debian patch impacted VOMS API
    • Some RODS are re-enabling suspended sites without fixing them! Process change in order.
    • Action: Comment on change to make it mandatory for sites to download and run Pakiti on highlighted WNs. (see slides)
    • EGI-Engage update: started in March. New platforms and business models being explored. See slides for details.
    • Developing the infrastructure: includes plans for HPC provisioning, data preservation and a storage federation pilot. Testbed to explore areas such as persistent IDs.
    • EGI Commons: Talk explained the new architecture and how core services will evolve. EGI Engage WP3 will meet in Amsterdam 14th-15th Arpil.
    • Survey of core activities: In progress till 24th April.



Tuesday 17th March

  • John Gordon mentions that he will report to the WLCG MB this afternoon about multicore accounting. The UK looks good in general, but there are two sites who are still not publishing. UCL (no plan) and Birmingham (following up). A number of others (IC, Glasgow, Liverpool) are publishing for the minority of their CEs. Please change all CEs, not just those queues running multicore.
  • Andrew M has created a table to track cloud/VM status and plans. Please check it is up-to-date.
  • Looking at [(http://planet.gridpp.ac.uk our blogs] it would be good if everyone reviews if there work/findings might benefit others and if so considers sharing it via blog postings. The latest postings in each blog are: Storage - 9th March 2015 (Storage accounting); NorthGrid - 24th February 2015 (Condor); Tier-1 - 22nd January 2015 (CEPH); SouthGrid - 14th October 2014 (Nagios monitoring non-LHC VOs); LondonGrid - 8th October 2014 (xrootD & ARGUS); ScotGrid - 24th March 2014 (3 coordinators).
  • T2 draft reliability & availability figures for February have been circulated: ALICE; ATLAS; CMS and LHCb.
  • Wahid leaves us this week - thanks for all your contributions!


WLCG Operations Coordination - Agendas

Monday 6th April

  • There was a WLCG ops coordination meeting on Thursday 2nd April: Agenda | Minutes.
    • News: Ops survey responses reviewed. Results overview available.
    • Data Preservation Training: Need more WLCG sites involved to justify central funding. See slides.
    • Baselines: missing ARGUS PAP 1.6.2 in UMD.
    • MW Issues: NTR
    • T0 & T1 services: CERN CASTOR upgrades 6th and 7th April. T1s various SE upgrades.
    • T0 news/T1/T2 feedback: NTR
    • ALICE: High activity. Most VO Boxes now use RFC proxies. Various CERN CASTOR instabilities. CRL URL being firewalled for IPv6.
    • ATLAS: Cosmics export smooth. Grid should be full. Triumf-RAL link problems 50kB/s. WhiteHat challenge comments.
    • CMS: Cosmic run finished. T1s and T2s busy. Problems with frontier/squid infrastructure. Scheduling (conflict) of SAM tests. T1 tape stage tests done.
    • LHCb: NTR
    • gLExec: 61/94 panda site coverage. Pilot to be enhanced - RAL/RALPP results to be investigated.
    • SHA-2: Old VOMS servers to be retired (kept to forward user requests). Exceptions (till 5th May) to stop lcg-voms.cern.ch queries hanging. Propose to remove aliases on 27th April.
    • Machine/Job features: NTR
    • M/W readiness WG: Setup working well - and finding issues. Actions list updated. Next meeting 6th May. Need to clarify ECDF DPM contribution.
    • Multicore: NTR.
    • IPv6: FTS3 testbed operational (KIT & IC). 8 sites now activated IPv6. OSG testing glideinWMS. There is an IPv6 dualstack mesh.
    • Squid/HTTP proxy: NTR
    • Network & transfers WG: CVE released for cassandra. PS 3.4.2 released - anyone at 3.4.1 encouraged to check disk space and enable auto-update. Full mesh trace paths now at 80%. Network performance follow-up proposal
      • New mailing list and GGUS SU will be established to follow up (wlcg-network-throughput)
      • Experiments can report to the GGUS SU/mailing list potential network performance incidents/degradations, WLCG perfSONAR support unit will investigate and confirm if this is a network related issue.
      • Affected sites will be contacted. Sites observing a network performance problem should follow their standard procedure, i.e. report to their network team and if necessary escalate to their network provider while informing the wlcg-network-throughput mailing list.
      • Issues will be tracked in the twiki.
      • Next WG meeting 8th April.
    • HTTP deployment: First meeting will be on 29th April.
    • Next meeting 23rd April.
Tier-1 - Status Page

Tuesday 7th April

  • Castor update to 2.1.14-15 planned for tomorrow (8th April).
  • RAL has been running CVMFS 2.1.20 for the past few weeks as an early deployment site.
  • The problems with our primary network router are still being followed up.
Storage & Data Management - Agendas/Minutes

Wednesday 8 April

Tuesday 7th April

  • There was a DPM collaboration meeting last week. (Blog update)

Tuesday 31st March

Tuesday 17th March

  • An annual DPM collaboration board will take place in coming weeks. Are there issues that sites want raised (in relation to things like roadmap, concerns about DPM, requests etc.)?


Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06

Tuesday 10th March

  • Possible delay at Birmingham. Worth checking.

Tuesday 24th February

  • UKI-SOUTHGRID-SUSX now highlighted as not publishing for 3 months.
Documentation - KeyDocs

See the worst KeyDocs list for documents needing review now and the names of the responsible people.

Tuesday 17th March

  • Documents reviewed at core-ops meeting last week. Agreed to remove 4 documents and add 2 (tarball status).

Tuesday 3rd March

  • Proposals for about 4 keydocs to be removed/downgraded. For discussion at next core-ops or focus meeting.

Tuesday 24th February

  • New section in Wiki called "Project Management Pages".
The idea is to cluster all Self-Edited Site Tracking Tables
in here. Sites should keep entries in Current Activities
up to date. Once a Self-Edited Site Tracking Tables has
served its purpose, PM to move it to  Historical Archive 
or otherwise dispose of the table.
Interoperation - EGI ops agendas

Monday 9th March

  • The agenda for February's EGI ops meeting is here. Minutes are here
    • APEL 1.4.0
      • Added Month and Year columns to primary key of CloudSummaries table in cloud schema.
    • DPM-Xrootd 3.5.2 is in EPEL stable - this is the first version of the component compatible with xrootd4
    • gLExec-wn - v. 1.2.3: lcmaps-plugins-c-pep 1.3.0-1 & mkgltempdir 0.0.5-1
      • "The lcmaps-plugins-c-pep-1.3.0-1 preferably needs the argus-pep-api-c-2.3.0. This version will be released into EMI & UMD repositories in a near future."
    • UMD 3.11.0 released on 16.02.2014, UMD 3.11.1 released on 4.03.2014
    • lcg-CA 1.62 noted with an intention to broadcast these as they occur as opposed to monthly.
    • EGI looking at the decommissioning of SL5, possibly by end of 2015, as a byproduct of adding CentOS 7 to UMD. NGIs to make a note if extended SL5 support is required.
    • Vincenzo Spinoso has joined EGI Ops team from NGI_IT. Vincenzo will chair EGI Ops.
    • Next meeting is April 20th.


Monitoring - Links MyWLCG

Tuesday 31st March

Monday 7th December

On-duty - Dashboard ROD rota

Monday 31st March

  • Dashboard has been problematic - following recent UI changes. Alarms not appearing.
  • ROD rota to be updated this week.

Monday 16th March

  • Low availability alarm for UCL, but no alarm for the actual underlying cause (disk full on DPM).
  • Ticket now issued.
  • Kashif -> Daniela
Rollout Status WLCG Baseline

Tuesday 17th March

  • Daniela has updated the [ https://www.gridpp.ac.uk/wiki/Staged_rollout_emi3 EMI-3 testing table]. Please check it is correct for your site. We want a clear view of where we are contributing.
  • There is a middleware readiness meeting this Wednesday. Would be good if a few site representatives joined.
  • Machine job features solution testing. Fed back that we will only commence tests if more documentation made available. This stops the HTC solution until after CHEP. Is there interest in testing other batch systems? Raul mentioned SLURM. There is also SGE and Torque.


Tuesday 17th February

  • A message from Maria Dimou: A request to you and the sites you collaborate with, to please install the MW Package Reporter.

The current version, documented in the wiki , it is very easy to install and addresses all site and security requirements.

  • Quick check on: 1. ARC CE readiness testing. 2. Machine job features testing.

References


Security - Incident Procedure Policies Rota

Tuesday 7th April

  • EGI Trust Anchor release 1.63-1 (see announcement 30th March).

Tuesday 31st March

Tuesday 17th March

  • There will be a security team meeting this week. Are there any site issues to be raised?
  • Note a new OpenSSL release is coming.

Monday 9th March

  • Trust anchor release 1.62 (missing announcement!).
  • From DG: "1.62 is indeed out and was released by the IGTF on Feb 23rd. The EGI release by an unfortunate coincidence got linked to a VOMS update for UMD (v3.3.3), so was released only today."


Services - PerfSonar dashboard | GridPP VOMS

- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).

Tuesday 31st March

Tuesday 10th March

  • From the recent WLCG meeting, two slides (1 & 2) give the direction of the network monitoring and metrics progress: integration of perfSONAR event types into experiment monitoring and an architecture for data to get from RSV probes to client. Components described on slide 3.
  • The next LHCOPN and LHCONE joint meeting will take place on Monday 1st and Tuesday 2nd of June 2015 in Berkeley (US) (hosted by LBL and ESnet).
Tickets

Tuesday 7th April

Monday 23rd March 2015, 15.30 GMT
19 Open tickets this week.

BIRMINGHAM
https://ggus.eu/?mode=ticket_info&ticket_id=112550 (23/3)
A ticket fresh off the ROD dashboard - the Birmingham CREAMs aren't being matched ("BrokerHelper: no compatible resources"). Matt W has double checked their setup and can't spot anything wrong - they've been running "normal" atlas/lhcb etc jobs fine over the last few weeks. Any advice appreciated. In progress (23/3)

TIER 1
https://ggus.eu/?mode=ticket_info&ticket_id=112495 (21/3)
MICE problems running jobs at RAL, which Andrew L discovered coincided with WMS problems that he fixed. Probably should be in "Waiting for Reply/Seeing if the problem's evaporated". In progress (23/3)

https://ggus.eu/?mode=ticket_info&ticket_id=112350 (14/3)
The cause of this Sno+ ticket, about a recent user not being able to access files due to not being in the gridmap, has been discovered. As Robert F sagely pointed out the latest version of the mkgridmap rpm is required to talk to the voms server. Just waiting on the time for it to updated at now. In progress (17/3)

100IT
https://ggus.eu/?mode=ticket_info&ticket_id=108356 (10/9/14)
This 100IT ticket is still waiting for a reply (since mid-January). The question needs to be answered by someone familiar with the technologies and terminologies? Is anyone up on vmcatcher? Anyone know what other channel to pass David's query onto? Waiting for reply (15/1)

Tools - MyEGI Nagios

Tuesday 17th February

  • Another period where message brokers were temporarily unavailable seen yesterday. Any news on the last follow-up?

Tuesday 27th January

  • Unscheduled outage of the EGI message broker (GRNET) caused a short-lived disruption to GridPP site monitoring (jobs failed) last Thursday 22nd January. Suspect BDII caching meant no immediate failover to stomp://mq.cro-ngi.hr:6163/ from stomp://mq.afroditi.hellasgrid.gr:6163/


VOs - GridPP VOMS VO IDs Approved VO table

Tuesday 31st March

  • LIGO are in need of additional support for debugging some tests.
  • LSST now enabled on 3 sites. No 'own' CVMFS yet.

Monday 9th March

  • SIXT VOMS parameters have changed.

Tuesday 17th February Two changes to approved VOs (https://www.gridpp.ac.uk/wiki/GridPP_approved_VOs)

  • LSST uses port 15003 (had been 15002, clashing with dzero)
  • t2k has included a note that it';s software is now distrinuted via CVMFS.
Site Updates

Tuesday 24th February

  • Next review of status today.

Tuesday 27th January

  • Squids not in GOCDB for: UCL; ECDF; Birmingham; Durham; RHUL; IC; Sussex; Lancaster
  • Squids in GOCDB for: EFDA-JET; Manchester; Liverpool; Cambridge; Sheffield; Bristol; Brunel; QMUL; T1; Oxford; Glasgow; RALPPD.

Tuesday 2nd December

  • Multicore status. Queues available (63%)
    • YES: RAL T1; Brunel; Imperial; QMUL; Lancaster; Liverpool; Manchester; Glasgow; Cambridge; Oxford; RALPP; Sussex (12)
    • NO: RHUL (testing); UCL; Sheffield (testing); Durham; ECDF (testing); Birmingham; Bristol (7)
  • According to our table for cloud/VMs (26%)
    • YES: RAL T1; Brunel; Imperial; Manchester; Oxford (5)
    • NO: QMUL; RHUL; UCL; Lancaster; Liverpool; Sheffield; Durham; ECDF; Glasgow; Birmingham; Bristol; Cambridge; RALPP; Sussex (14)
  • GridPP DIRAC jobs successful (58%)
    • YES: Bristol; Glasgow; Lancaster; Liverpool; Manchester; Oxford; Sheffield; Brunel; IC; QMUL; RHUL (11)
    • NO: Cambridge; Durham; RALPP; RAL T1 (4) + ECDF; Sussex; UCL; Birmingham (4)
  • IPv6 status
    • Allocation - 42%
    • YES: RAL T1; Brunel; IC; QMUL; Manchester; Sheffield; Cambridge; Oxford (8)
    • NO: RHUL; UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; Bristol; RALPP; Sussex
  • Dual stack nodes - 21%
    • YES: Brunel; IC; QMUL; Oxford (4)
    • NO: RHUL; UCL; Lancaster; Glasgow; Liverpool; Manchester; Sheffield; Durham; ECDF; Birmingham; Bristol; Cambridge; RALPP; Sussex, RAL T1 (15)


Tuesday 21st October

  • High loads seen in xroot by several sites: Liverpool and RALT1... and also Bristol (see Luke's TB-S email on 16/10 for questions about changes to help).

Tuesday 9th September

  • Intel announced the new generation of Xeon based on Haswell.



Meeting Summaries
Project Management Board - MembersMinutes Quarterly Reports

Empty

GridPP ops meeting - Agendas Actions Core Tasks

Empty


RAL Tier-1 Experiment Liaison Meeting (Wednesday 13:30) Agenda Meeting takes place on Vidyo.

Wednesday 18th March 2015

  • Operations report
  • The replacement of the problematic Tier1 network router with another unit last Thursday failed to solve the problem. Further tests will be carried out tomorrrow morning (19th March).
  • Proposed date for Castor upgrade to version 2.1.14-15 is the 8th April.
  • Five additional disk servers have been deployed into Castor storage for ALICE_Disk. This roughly doubles the disk space available to ALICE which is now approximately 390TB.
WLCG Grid Deployment Board - Agendas MB agendas

Empty



NGI UK - Homepage CA

Empty

Events
UK ATLAS - Shifter view News & Links

Atlas S&C week 2-6 Feb 2015

Production

• Prodsys-2 in production since Dec 1st

• Deployment has not been transparent , many issued has been solved, the grid is filled again

• MC15 is expected to start soon, waiting for physics validations, evgen testing is underway and close to finalised.. Simulation expected to be broadly similar to MC14, no blockers expected.

Rucio

• Rucio in production since Dec 1st and is ready for LHC RUN-2. Some fields need improvements, including transfer and deletion agents, documentation and monitoring.

Rucio dumps available.

Dark data cleaning

files declaration . Only Only DDM ops can issue lost files declaration for now, cloud support needs to fill a ticket.

• Webdav panda functional tests with Hammercloud are ongoing

Monitoring

Main page

DDM Accounting

space

Deletion

ASAP

• ASAP (ATLAS Site Availability Performance) in place. Every 3 months the T2s sites performing BELOW 80% are reported to the International Computing Board.


UK CMS

Empty

UK LHCb

Empty

UK OTHER
  • N/A
To note

  • N/A