Difference between revisions of "Operations Bulletin Latest"

From GridPP Wiki
Jump to: navigation, search
()
()
Line 401: Line 401:
 
'''TIER 1'''<br />
 
'''TIER 1'''<br />
 
[https://ggus.eu/?mode=ticket_info&ticket_id=112721 112721] (28/3)<br />
 
[https://ggus.eu/?mode=ticket_info&ticket_id=112721 112721] (28/3)<br />
ATLAS FTS failures too RAL. A rucio issue causing double-transfers has been discovered (https://its.cern.ch/jira/browse/ATLDDMOPS-4939), which would explain the behaviour seen. No news since this revelation. In progress (16/4)
+
ATLAS FTS failures too RAL. A rucio issue causing double-transfers has been discovered ([https://its.cern.ch/jira/browse/ATLDDMOPS-4939 here]), which would explain the behaviour seen. No news since this revelation. In progress (16/4)
  
 
There are a number of other Tier 1 tickets that could do with either an update or On Holding
 
There are a number of other Tier 1 tickets that could do with either an update or On Holding

Revision as of 15:05, 27 April 2015

Bulletin archive


Week commencing 27th April 2015
Task Areas
General updates

Monday 20th April

  • A pre-GDB on batch systems will take place in May.
  • The VOMS ports for SNO+ have changed to 15503.
  • Catalin mentioned there will be a dedicated CVMFS session on Tuesday 19 May 17:00 to 18:00 ("The EGI CernVM-FS Infrastructure - Status, Developments, Opportunities") at the EGI conference.
  • Reminder: Advance warning there will be no cert signing or CRL signing Wed 22 and Thu 23 April due to a RAL away-days.
  • UKQCD have indicated an intention to restart their use of the grid. Please consider supporting their VO (currently only Glasgow does!).

Tuesday 14th April

  • There was a WLCG workshop over the weekend in Okinawa: agenda.
  • CHEP has started. Many interesting talks. Timetable/Talks. Will review at GridPP34.
  • GridPP34 agenda is developing.
  • GOCDB has a number of Service Types that appear to be unused. We need to confirm for the following list: OpsTool; dg.TargetSystemFactory; dg.ARC-CE; eu.egi.cloud.broker.proprietary.slipstream; globus-RLS; RGMA-IC; eu.egi.egran; SRM.online; egi.VODashboard; CUSTOM.egi.HTTPserver; egi.NetworkPortal; ch.cern.cvmfs.stratum.1; CUSTOM.pl.plgrid.Bazaar.
  • User Community DIRAC meeting that will take place in May. Survey in progress for all communities who use DIRAC. Let Jeremy know if you need the URL.
  • A potentially useful LHCb monitoring page.
  • Reminder of Tom's VO cleanup campaign.
  • Quarterly reports for Q115 now due.
  • March 2015 WLCG A/R results circulated:
    • ALICE. No problems.
    • ATLAS. Brunel: 85%:96%; QMUL: 62%:62%; UCL: 27%:27%; Sussex: 39%:39.
    • CMS Brunel: 86%:96%
    • LHCb. All good.


WLCG Operations Coordination - Agendas

Tuesday 21st April

  • The next WLCG ops coordination meeting will be on Thursday 23rd. Jeremy will try to attend, but please could one or two others from core ops provide backup representation.

Tuesday 14th April

  • No updates this week.

Monday 6th April

  • There was a WLCG ops coordination meeting on Thursday 2nd April: Agenda | Minutes.
    • News: Ops survey responses reviewed. Results overview available.
    • Data Preservation Training: Need more WLCG sites involved to justify central funding. See slides.
    • Baselines: missing ARGUS PAP 1.6.2 in UMD.
    • MW Issues: NTR
    • T0 & T1 services: CERN CASTOR upgrades 6th and 7th April. T1s various SE upgrades.
    • T0 news/T1/T2 feedback: NTR
    • ALICE: High activity. Most VO Boxes now use RFC proxies. Various CERN CASTOR instabilities. CRL URL being firewalled for IPv6.
    • ATLAS: Cosmics export smooth. Grid should be full. Triumf-RAL link problems 50kB/s. WhiteHat challenge comments.
    • CMS: Cosmic run finished. T1s and T2s busy. Problems with frontier/squid infrastructure. Scheduling (conflict) of SAM tests. T1 tape stage tests done.
    • LHCb: NTR
    • gLExec: 61/94 panda site coverage. Pilot to be enhanced - RAL/RALPP results to be investigated.
    • SHA-2: Old VOMS servers to be retired (kept to forward user requests). Exceptions (till 5th May) to stop lcg-voms.cern.ch queries hanging. Propose to remove aliases on 27th April.
    • Machine/Job features: NTR
    • M/W readiness WG: Setup working well - and finding issues. Actions list updated. Next meeting 6th May. Need to clarify ECDF DPM contribution.
    • Multicore: NTR.
    • IPv6: FTS3 testbed operational (KIT & IC). 8 sites now activated IPv6. OSG testing glideinWMS. There is an IPv6 dualstack mesh.
    • Squid/HTTP proxy: NTR
    • Network & transfers WG: CVE released for cassandra. PS 3.4.2 released - anyone at 3.4.1 encouraged to check disk space and enable auto-update. Full mesh trace paths now at 80%. Network performance follow-up proposal
      • New mailing list and GGUS SU will be established to follow up (wlcg-network-throughput)
      • Experiments can report to the GGUS SU/mailing list potential network performance incidents/degradations, WLCG perfSONAR support unit will investigate and confirm if this is a network related issue.
      • Affected sites will be contacted. Sites observing a network performance problem should follow their standard procedure, i.e. report to their network team and if necessary escalate to their network provider while informing the wlcg-network-throughput mailing list.
      • Issues will be tracked in the twiki.
      • Next WG meeting 8th April.
    • HTTP deployment: First meeting will be on 29th April.
    • Next meeting 23rd April.
Tier-1 - Status Page

Tuesday 21st April

  • Castor update to 2.1.14-15 successfully took place on 15th April.
  • There will be an interruption to the LFC on Tuesday next week (28th April) for an update to Oracle on the back end database.
  • We will turn off our remainining CREAM CEs on 5th May.
  • The problems with our primary network router are still being followed up.
Storage & Data Management - Agendas/Minutes

Tuesday 21st April

  • Has there been any Tier-1 contact with DiRAC?
  • Proposal to setup an 'other VOs' users list. GridPP-Users is too tied with WLCG projects.

Wednesday 15 April

  • Backing up data from DiRAC to GridPP (tape)
  • More case studies on supporting non-LHC VOs on GridPP: we have a lot of great stuff that can do great stuff - non-LHC VOs tend to have less regimented data models so maybe we need more case studies.

Wednesday 8 April

Tuesday 7th April

  • There was a DPM collaboration meeting last week. (Blog update)

Tuesday 31st March

Tuesday 17th March

  • An annual DPM collaboration board will take place in coming weeks. Are there issues that sites want raised (in relation to things like roadmap, concerns about DPM, requests etc.)?


Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06

Tuesday 21st April

  • (Slight) Accounting delays seen for: UCL; Sheffield; QMUL & RALPP.

Tuesday 14th April

  • APEL delays for UCL; Sheffield; RALPP and Bristol
Documentation - KeyDocs

See the worst KeyDocs list for documents needing review now and the names of the responsible people.

Tuesday 21st April

  • The Approved VOs document has been updated to take account of changes to the Ops Portal VOID cards.For SNOPLUS.SNOLAB.CA, the port numbers for voms02.gridpp.ac.uk and voms03.gridpp.ac.uk have both been updated from 15003 to 15503. Sites that support SNOPLUS.SNOLAB.CA should ensure that their configuration conforms to these settings: Approved VOs
  • KeyDocs still need updating since agreements reached at last core ops meeting.
  • New section in Wiki called "Project Management Pages".
The idea is to cluster all Self-Edited Site Tracking Tables
in here. Sites should keep entries in Current Activities
up to date. Once a Self-Edited Site Tracking Tables has
served its purpose, PM to move it to  Historical Archive 
or otherwise dispose of the table.
Interoperation - EGI ops agendas

Tuesday 21st April

  • There was an EGI ops meeting on Monday 20th.
  • David updated the UK SL5 response.
  • Please review the agenda/minutes.

Monday 9th March

  • The agenda for February's EGI ops meeting is here. Minutes are here
    • APEL 1.4.0
      • Added Month and Year columns to primary key of CloudSummaries table in cloud schema.
    • DPM-Xrootd 3.5.2 is in EPEL stable - this is the first version of the component compatible with xrootd4
    • gLExec-wn - v. 1.2.3: lcmaps-plugins-c-pep 1.3.0-1 & mkgltempdir 0.0.5-1
      • "The lcmaps-plugins-c-pep-1.3.0-1 preferably needs the argus-pep-api-c-2.3.0. This version will be released into EMI & UMD repositories in a near future."
    • UMD 3.11.0 released on 16.02.2014, UMD 3.11.1 released on 4.03.2014
    • lcg-CA 1.62 noted with an intention to broadcast these as they occur as opposed to monthly.
    • EGI looking at the decommissioning of SL5, possibly by end of 2015, as a byproduct of adding CentOS 7 to UMD. NGIs to make a note if extended SL5 support is required.
    • Vincenzo Spinoso has joined EGI Ops team from NGI_IT. Vincenzo will chair EGI Ops.
    • Next meeting is April 20th.


Monitoring - Links MyWLCG

Tuesday 31st March

Monday 7th December

On-duty - Dashboard ROD rota

Tuesday 21st April

  • UCL have put themselves into a downtime until the 21st April. (Start of next week). Noted this in their outstanding tickets.
  • Birmingham's availability has steadily recovered over the week - and the low availability ticket against them should be closable next week.


Rollout Status WLCG Baseline

Tuesday 17th March

  • Daniela has updated the [ https://www.gridpp.ac.uk/wiki/Staged_rollout_emi3 EMI-3 testing table]. Please check it is correct for your site. We want a clear view of where we are contributing.
  • There is a middleware readiness meeting this Wednesday. Would be good if a few site representatives joined.
  • Machine job features solution testing. Fed back that we will only commence tests if more documentation made available. This stops the HTC solution until after CHEP. Is there interest in testing other batch systems? Raul mentioned SLURM. There is also SGE and Torque.


Tuesday 17th February

  • A message from Maria Dimou: A request to you and the sites you collaborate with, to please install the MW Package Reporter.

The current version, documented in the wiki , it is very easy to install and addresses all site and security requirements.

  • Quick check on: 1. ARC CE readiness testing. 2. Machine job features testing.

References


Security - Incident Procedure Policies Rota

Tuesday 21st April

  • EGI Alert 'High' risk - Xen Vulnerability Hypervisor memory corruption due to x86 emulator flaw CVE-2015-2151 [EGI-ADV-20150415]

Tuesday 7th April

  • EGI Trust Anchor release 1.63-1 (see announcement 30th March).

Tuesday 31st March


Services - PerfSonar dashboard | GridPP VOMS

- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).

Tuesday 31st March

Tuesday 10th March

  • From the recent WLCG meeting, two slides (1 & 2) give the direction of the network monitoring and metrics progress: integration of perfSONAR event types into experiment monitoring and an architecture for data to get from RSV probes to client. Components described on slide 3.
  • The next LHCOPN and LHCONE joint meeting will take place on Monday 1st and Tuesday 2nd of June 2015 in Berkeley (US) (hosted by LBL and ESnet).
Tickets

Monday 27th April 2015, 14.45 BST

A quick check of the Other VO Nagios page.

26 Open UK tickets this week.

ITWO Decommissioning
Three of the tickets are to the VOMS sites (Manchester, Oxford, IC), concerning the decommissioning of the ITWO VO. Just an FYI to y'all.

ECDF
113293 (26/4)
There was an APEL problem last month where a lot of sites needed to republish their data for the month. I think Edinburgh are the only UK site that suffered this problem, but another FYI ticket. Assigned (26/4)

113181(21/4)
Atlas production jobs not running at ECDF. Andy noticed that analysis jobs were running fine, and believes that this might be a problem scheduling pilots in time. Perhaps a multicore issue if this is only effecting (affecting?) production jobs. In progress (22/4)

ATLAS GLEXEC HAMMERCLOUD PROBLEMS
111699 (TIER 1)
111703 (RALPP)
It was discovered that there was a problem in the test code, so the ball is very much in atlas' court for this one. The problem has been fixed and the tests are being rebuilt and resubmitted.

TIER 1
112721 (28/3)
ATLAS FTS failures too RAL. A rucio issue causing double-transfers has been discovered (here), which would explain the behaviour seen. No news since this revelation. In progress (16/4)

There are a number of other Tier 1 tickets that could do with either an update or On Holding

Tools - MyEGI Nagios

Tuesday 17th February

  • Another period where message brokers were temporarily unavailable seen yesterday. Any news on the last follow-up?

Tuesday 27th January

  • Unscheduled outage of the EGI message broker (GRNET) caused a short-lived disruption to GridPP site monitoring (jobs failed) last Thursday 22nd January. Suspect BDII caching meant no immediate failover to stomp://mq.cro-ngi.hr:6163/ from stomp://mq.afroditi.hellasgrid.gr:6163/


VOs - GridPP VOMS VO IDs Approved VO table

Tuesday 31st March

  • LIGO are in need of additional support for debugging some tests.
  • LSST now enabled on 3 sites. No 'own' CVMFS yet.

Monday 9th March

  • SIXT VOMS parameters have changed.

Tuesday 17th February Two changes to approved VOs (https://www.gridpp.ac.uk/wiki/GridPP_approved_VOs)

  • LSST uses port 15003 (had been 15002, clashing with dzero)
  • t2k has included a note that it';s software is now distrinuted via CVMFS.
Site Updates

Tuesday 24th February

  • Next review of status today.

Tuesday 27th January

  • Squids not in GOCDB for: UCL; ECDF; Birmingham; Durham; RHUL; IC; Sussex; Lancaster
  • Squids in GOCDB for: EFDA-JET; Manchester; Liverpool; Cambridge; Sheffield; Bristol; Brunel; QMUL; T1; Oxford; Glasgow; RALPPD.

Tuesday 2nd December

  • Multicore status. Queues available (63%)
    • YES: RAL T1; Brunel; Imperial; QMUL; Lancaster; Liverpool; Manchester; Glasgow; Cambridge; Oxford; RALPP; Sussex (12)
    • NO: RHUL (testing); UCL; Sheffield (testing); Durham; ECDF (testing); Birmingham; Bristol (7)
  • According to our table for cloud/VMs (26%)
    • YES: RAL T1; Brunel; Imperial; Manchester; Oxford (5)
    • NO: QMUL; RHUL; UCL; Lancaster; Liverpool; Sheffield; Durham; ECDF; Glasgow; Birmingham; Bristol; Cambridge; RALPP; Sussex (14)
  • GridPP DIRAC jobs successful (58%)
    • YES: Bristol; Glasgow; Lancaster; Liverpool; Manchester; Oxford; Sheffield; Brunel; IC; QMUL; RHUL (11)
    • NO: Cambridge; Durham; RALPP; RAL T1 (4) + ECDF; Sussex; UCL; Birmingham (4)
  • IPv6 status
    • Allocation - 42%
    • YES: RAL T1; Brunel; IC; QMUL; Manchester; Sheffield; Cambridge; Oxford (8)
    • NO: RHUL; UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; Bristol; RALPP; Sussex
  • Dual stack nodes - 21%
    • YES: Brunel; IC; QMUL; Oxford (4)
    • NO: RHUL; UCL; Lancaster; Glasgow; Liverpool; Manchester; Sheffield; Durham; ECDF; Birmingham; Bristol; Cambridge; RALPP; Sussex, RAL T1 (15)


Tuesday 21st October

  • High loads seen in xroot by several sites: Liverpool and RALT1... and also Bristol (see Luke's TB-S email on 16/10 for questions about changes to help).

Tuesday 9th September

  • Intel announced the new generation of Xeon based on Haswell.



Meeting Summaries
Project Management Board - MembersMinutes Quarterly Reports

Empty

GridPP ops meeting - Agendas Actions Core Tasks

Empty


RAL Tier-1 Experiment Liaison Meeting (Wednesday 13:30) Agenda Meeting takes place on Vidyo.

Wednesday 15th April 2015

  • Operations report
  • On Wednesday 8th April a minor network change triggered a wider problem across the Tier1 network and caused significant disruption of services. The planned Castor upgrade for that day had to be abandoned. A post mortem is being generated for this incident.
  • Castor was successfully upgraded to version 2.1.14-15 this morning.
  • The CREAM CEs will be turned off on 5th May. (Leaving only ARC CEs).
WLCG Grid Deployment Board - Agendas MB agendas

Empty



NGI UK - Homepage CA

Empty

Events
UK ATLAS - Shifter view News & Links

Atlas S&C week 2-6 Feb 2015

Production

• Prodsys-2 in production since Dec 1st

• Deployment has not been transparent , many issued has been solved, the grid is filled again

• MC15 is expected to start soon, waiting for physics validations, evgen testing is underway and close to finalised.. Simulation expected to be broadly similar to MC14, no blockers expected.

Rucio

• Rucio in production since Dec 1st and is ready for LHC RUN-2. Some fields need improvements, including transfer and deletion agents, documentation and monitoring.

Rucio dumps available.

Dark data cleaning

files declaration . Only Only DDM ops can issue lost files declaration for now, cloud support needs to fill a ticket.

• Webdav panda functional tests with Hammercloud are ongoing

Monitoring

Main page

DDM Accounting

space

Deletion

ASAP

• ASAP (ATLAS Site Availability Performance) in place. Every 3 months the T2s sites performing BELOW 80% are reported to the International Computing Board.


UK CMS

Empty

UK LHCb

Empty

UK OTHER
  • N/A
To note

  • N/A