Operations Bulletin Latest

From GridPP Wiki
Jump to: navigation, search

Bulletin archive


Week commencing 13th April 2015
Task Areas
General updates

Tuesday 14th April

  • There was a WLCG workshop over the weekend in Okinawa: agenda.
  • CHEP has started. Many interesting talks. Timetable/Talks. Will review at GridPP34.
  • GridPP34 agenda is developing.
  • GOCDB has a number of Service Types that appear to be unused. We need to confirm for the following list: OpsTool; dg.TargetSystemFactory; dg.ARC-CE; eu.egi.cloud.broker.proprietary.slipstream; globus-RLS; RGMA-IC; eu.egi.egran; SRM.online; egi.VODashboard; CUSTOM.egi.HTTPserver; egi.NetworkPortal; ch.cern.cvmfs.stratum.1; CUSTOM.pl.plgrid.Bazaar.
  • User Community DIRAC meeting that will take place in May. Survey in progress for all communities who use DIRAC. Let Jeremy know if you need the URL.
  • A potentially useful LHCb monitoring page.
  • Reminder of Tom's VO cleanup campaign.
  • Quarterly reports for Q115 now due.
  • March 2015 WLCG A/R results circulated:
    • ALICE. No problems.
    • ATLAS. Brunel: 85%:96%; QMUL: 62%:62%; UCL: 27%:27%; Sussex: 39%:39.
    • CMS Brunel: 86%:96%
    • LHCb. All good.



Tuesday 7th April

  • Proton beams are back in the LHC
  • DIRAC (framework) users survey circulated.
  • ATLAS: Reprocessing of full 2012 dataset - resources should have been busy over the Easter weekend.
  • Training Course, run by the authors, on ISO 16363 for Auditors and Managers of Digital Repositories - CERN 15th-19th June.


Tuesday 31st March

  • Winnie has an issue in that 'SL5 (works) vs SL6 (not) emi-3 UI'.
  • There is a DPM Collaboration Board this Tuesday.
  • The GridPP34 agenda is now being created; are there specific topics anyone wishes to put forward?
  • NGI-FR report Argus cannot run with the last version of the Java JDK which requires argus-pap-1.6.2-1.el6.noarch.rpm only available in EMI not UMB repos. (See tickets 111505 and 112606).
  • A look through the HEPiX talks given last week is recommended.
  • There was an EGI OMB last Thursday 26th March: agenda.
    • Operations
    • Request to register iRODS endpoints in GOCDB.
    • Advisory groups being created for ops and accounting portals.
    • Some EGI/NGI ARGUS servers are 'exposed'. Please check. Also NGI Argus service type should be "emi.argus" not "ngi.argus".
    • Plans to use ARGO for monitoring of central series
    • Updated technology provider UA (please review)
    • Reminder of EGI conference 18th-22nd May.
    • Security: Some FedCloud security packaged defined; OpenSSL Debian patch impacted VOMS API
    • Some RODS are re-enabling suspended sites without fixing them! Process change in order.
    • Action: Comment on change to make it mandatory for sites to download and run Pakiti on highlighted WNs. (see slides)
    • EGI-Engage update: started in March. New platforms and business models being explored. See slides for details.
    • Developing the infrastructure: includes plans for HPC provisioning, data preservation and a storage federation pilot. Testbed to explore areas such as persistent IDs.
    • EGI Commons: Talk explained the new architecture and how core services will evolve. EGI Engage WP3 will meet in Amsterdam 14th-15th Arpil.
    • Survey of core activities: In progress till 24th April.


WLCG Operations Coordination - Agendas

Tuesday 14th April

  • No updates this week.

Monday 6th April

  • There was a WLCG ops coordination meeting on Thursday 2nd April: Agenda | Minutes.
    • News: Ops survey responses reviewed. Results overview available.
    • Data Preservation Training: Need more WLCG sites involved to justify central funding. See slides.
    • Baselines: missing ARGUS PAP 1.6.2 in UMD.
    • MW Issues: NTR
    • T0 & T1 services: CERN CASTOR upgrades 6th and 7th April. T1s various SE upgrades.
    • T0 news/T1/T2 feedback: NTR
    • ALICE: High activity. Most VO Boxes now use RFC proxies. Various CERN CASTOR instabilities. CRL URL being firewalled for IPv6.
    • ATLAS: Cosmics export smooth. Grid should be full. Triumf-RAL link problems 50kB/s. WhiteHat challenge comments.
    • CMS: Cosmic run finished. T1s and T2s busy. Problems with frontier/squid infrastructure. Scheduling (conflict) of SAM tests. T1 tape stage tests done.
    • LHCb: NTR
    • gLExec: 61/94 panda site coverage. Pilot to be enhanced - RAL/RALPP results to be investigated.
    • SHA-2: Old VOMS servers to be retired (kept to forward user requests). Exceptions (till 5th May) to stop lcg-voms.cern.ch queries hanging. Propose to remove aliases on 27th April.
    • Machine/Job features: NTR
    • M/W readiness WG: Setup working well - and finding issues. Actions list updated. Next meeting 6th May. Need to clarify ECDF DPM contribution.
    • Multicore: NTR.
    • IPv6: FTS3 testbed operational (KIT & IC). 8 sites now activated IPv6. OSG testing glideinWMS. There is an IPv6 dualstack mesh.
    • Squid/HTTP proxy: NTR
    • Network & transfers WG: CVE released for cassandra. PS 3.4.2 released - anyone at 3.4.1 encouraged to check disk space and enable auto-update. Full mesh trace paths now at 80%. Network performance follow-up proposal
      • New mailing list and GGUS SU will be established to follow up (wlcg-network-throughput)
      • Experiments can report to the GGUS SU/mailing list potential network performance incidents/degradations, WLCG perfSONAR support unit will investigate and confirm if this is a network related issue.
      • Affected sites will be contacted. Sites observing a network performance problem should follow their standard procedure, i.e. report to their network team and if necessary escalate to their network provider while informing the wlcg-network-throughput mailing list.
      • Issues will be tracked in the twiki.
      • Next WG meeting 8th April.
    • HTTP deployment: First meeting will be on 29th April.
    • Next meeting 23rd April.
Tier-1 - Status Page

Tuesday 14th April

  • The Castor update to 2.1.14-15 that was planned for the 8th April did not take place owing to a local network problem. It will now take place tomorrow (15th April).
  • Severe problems during Wednesday 8th April. These were triggered by a minor network change.
  • We will turn off our remainining CREAM CEs on 5th May.
  • The problems with our primary network router are still being followed up.
Storage & Data Management - Agendas/Minutes

Wednesday 8 April

Tuesday 7th April

  • There was a DPM collaboration meeting last week. (Blog update)

Tuesday 31st March

Tuesday 17th March

  • An annual DPM collaboration board will take place in coming weeks. Are there issues that sites want raised (in relation to things like roadmap, concerns about DPM, requests etc.)?


Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06

Tuesday 14th April

  • APEL delays for UCL; Sheffield; RALPP and Bristol

Tuesday 10th March

  • Possible delay at Birmingham. Worth checking.

Tuesday 24th February

  • UKI-SOUTHGRID-SUSX now highlighted as not publishing for 3 months.
Documentation - KeyDocs

See the worst KeyDocs list for documents needing review now and the names of the responsible people.

Tuesday 17th March

  • Documents reviewed at core-ops meeting last week. Agreed to remove 4 documents and add 2 (tarball status).

Tuesday 3rd March

  • Proposals for about 4 keydocs to be removed/downgraded. For discussion at next core-ops or focus meeting.

Tuesday 24th February

  • New section in Wiki called "Project Management Pages". 
The idea is to cluster all Self-Edited Site Tracking Tables
in here. Sites should keep entries in Current Activities
up to date. Once a Self-Edited Site Tracking Tables has
served its purpose, PM to move it to  Historical Archive 
or otherwise dispose of the table.
Interoperation - EGI ops agendas

Monday 9th March

  • The agenda for February's EGI ops meeting is here. Minutes are here
    • APEL 1.4.0
      • Added Month and Year columns to primary key of CloudSummaries table in cloud schema.
    • DPM-Xrootd 3.5.2 is in EPEL stable - this is the first version of the component compatible with xrootd4
    • gLExec-wn - v. 1.2.3: lcmaps-plugins-c-pep 1.3.0-1 & mkgltempdir 0.0.5-1
      • "The lcmaps-plugins-c-pep-1.3.0-1 preferably needs the argus-pep-api-c-2.3.0. This version will be released into EMI & UMD repositories in a near future."
    • UMD 3.11.0 released on 16.02.2014, UMD 3.11.1 released on 4.03.2014
    • lcg-CA 1.62 noted with an intention to broadcast these as they occur as opposed to monthly.
    • EGI looking at the decommissioning of SL5, possibly by end of 2015, as a byproduct of adding CentOS 7 to UMD. NGIs to make a note if extended SL5 support is required.
    • Vincenzo Spinoso has joined EGI Ops team from NGI_IT. Vincenzo will chair EGI Ops.
    • Next meeting is April 20th.


Monitoring - Links MyWLCG

Tuesday 31st March

Monday 7th December

On-duty - Dashboard ROD rota

Monday 31st March

  • Dashboard has been problematic - following recent UI changes. Alarms not appearing.
  • ROD rota to be updated this week.

Monday 16th March

  • Low availability alarm for UCL, but no alarm for the actual underlying cause (disk full on DPM).
  • Ticket now issued.
  • Kashif -> Daniela
Rollout Status WLCG Baseline

Tuesday 17th March

  • Daniela has updated the [ https://www.gridpp.ac.uk/wiki/Staged_rollout_emi3 EMI-3 testing table]. Please check it is correct for your site. We want a clear view of where we are contributing.
  • There is a middleware readiness meeting this Wednesday. Would be good if a few site representatives joined.
  • Machine job features solution testing. Fed back that we will only commence tests if more documentation made available. This stops the HTC solution until after CHEP. Is there interest in testing other batch systems? Raul mentioned SLURM. There is also SGE and Torque.


Tuesday 17th February

  • A message from Maria Dimou: A request to you and the sites you collaborate with, to please install the MW Package Reporter.

The current version, documented in the wiki , it is very easy to install and addresses all site and security requirements.

  • Quick check on: 1. ARC CE readiness testing. 2. Machine job features testing.

References


Security - Incident Procedure Policies Rota

Tuesday 7th April

  • EGI Trust Anchor release 1.63-1 (see announcement 30th March).

Tuesday 31st March

Tuesday 17th March

  • There will be a security team meeting this week. Are there any site issues to be raised?
  • Note a new OpenSSL release is coming.

Monday 9th March

  • Trust anchor release 1.62 (missing announcement!).
  • From DG: "1.62 is indeed out and was released by the IGTF on Feb 23rd. The EGI release by an unfortunate coincidence got linked to a VOMS update for UMD (v3.3.3), so was released only today."


Services - PerfSonar dashboard | GridPP VOMS

- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).

Tuesday 31st March

Tuesday 10th March

  • From the recent WLCG meeting, two slides (1 & 2) give the direction of the network monitoring and metrics progress: integration of perfSONAR event types into experiment monitoring and an architecture for data to get from RSV probes to client. Components described on slide 3.
  • The next LHCOPN and LHCONE joint meeting will take place on Monday 1st and Tuesday 2nd of June 2015 in Berkeley (US) (hosted by LBL and ESnet).
Tickets

Monday 13th April 2015, 14.00 BST
24 Open tickets this week - going over all of them this week, site by site.

Fresh in this morning - 113010 and 113011 - Sno+ tickets concerning the RAL and Glasgow WMSes not updating job statuses.

RALPP
111703(11/2)
Atlas glexec hammercloud tests failing. There's been a lot of waiting on atlas to build new HC jobs. The most recent exchange (delayed due to Easter), was asking about SELinux - but no news since the first. In progress (1/4)

BIRMINGHAM
112875(7/4)
Low availability ROD ticket. Availability is crawling back up, just need it to go green. On hold (13/4)

GLASGOW
112967(10/4)
Another ROD ticket for bdii errors at Glasgow. Gareth has been doing everything right investigating this. Kashif recommended ticketed the midmon unit, but Gareth has spotted that the errors correspond to high load on their ARC CE - so it might be a site problem after all - Gareth asks for clarification. Waiting for reply (13/4)

EDINBURGH
95303 (1/7/13)
Tarball glexec ticket. No news (sorry). End of April I believe was the "deadline" I set for having this made. On Hold (9/3)

LANCASTER
100566 (27/1/14)
Lancaster's poor perfsonar performance. I'm not believing quite what I was seeing with the tests I performed so I'm aiming to rerun them. On hold (13/4)

95299 (1/7/13)
Lancaster's tarball glexec ticket. Same as ECDF. On hold (9/3)

BRUNEL
112966 (13/3)
A ROD cream job submit ticket, freshly assigned this afternoon. It's a bit mean of me to bring notice to it. Assigned (13/4) And POW, Raul closed this after kicking torque into shape - solved

100IT
112948 (10/4)
100IT needed to upgrade to the latest CA release. They've done this, but there are still authentication problems. In progress (13/4)

108356 (10/9/14)
Deploying vmcatcher at 100IT. After David's questions falling on deaf ears for a while it has been advised that the ticket be closed as this issue will be dealt with elsewhere. Whether or not it is to be "solved" or "unsolved" is open to debate! In progress (can possibly be closed) (13/4)

TIER 1
108944 (1/10/14)
CMS AAA tests failing at RAL. After a lot of work and new xrootd redirectors problems persist. It's looking to be a problem that needs the CASTOR and/or xrootd devs to look at. In progress (30/3)

112713 (27/3)
CMS asking to clean up the "unmerged area". Andrew conjured up a list of files and asked if they could be deleted - CMS responded with a "yes please then close the ticket". Has the deed been done? In progress (31/3)

109694 (28/10/14)
The Sno+ gfal copy ticket. Matt M still sees gfal-copy hang for files at RAL when he uses the GUID (SURL works). A Castor oddity perhaps? Matt asks a question about what problems like this (coupled with the move away from lcg tools) will mean for VOs that rely on the LFC. In progress (31/3)

112977 (10/3)
CMS high job failure rate at RAL. Related to 112896 (below) - the jobs all want that file! In progress (13/3)

https://ggus.eu/index.php?mode=ticket_info&ticket_id=112896 (9/4)
CMS Dataset access problems - caused by over a million access attempts on a single file over a 18 hour period. Andrew L comments that CMS needs to have a think about how they access pileup datasets. In progress (9/4)

111699 (10/2)
Tier 1 counterpart to 111703. A new HC stress test was submitted near the end of March, but no news on how it did. In progress (23/3)

112866 (7/4)
A different "lots of CMS job failures" ticket. Again a "hot file" seems to be the root cause. In progress (7/4)

112721 (28/3)
An atlas file access ticket, seemingly caused by some odd FTS behaviour. No answers to Shaun's question about this odd occurrence or much noise at all till today. Waiting for reply (13/4)

UCL
UCL has 6 tickets - 4 just "assigned". I'll just list them in the interests of brevity.
112371 (ROD low availiability, On Hold)
112841 (atlas 0% transfer efficiency, assigned)
112873 (ROD srm put failures, assigned)
95298 (glexec ticket, on hold)
112722 (atlas checksum timeouts, in progress)
112966 (ROD job submit failures, assigned)

Tools - MyEGI Nagios

Tuesday 17th February

  • Another period where message brokers were temporarily unavailable seen yesterday. Any news on the last follow-up?

Tuesday 27th January

  • Unscheduled outage of the EGI message broker (GRNET) caused a short-lived disruption to GridPP site monitoring (jobs failed) last Thursday 22nd January. Suspect BDII caching meant no immediate failover to stomp://mq.cro-ngi.hr:6163/ from stomp://mq.afroditi.hellasgrid.gr:6163/


VOs - GridPP VOMS VO IDs Approved VO table

Tuesday 31st March

  • LIGO are in need of additional support for debugging some tests.
  • LSST now enabled on 3 sites. No 'own' CVMFS yet.

Monday 9th March

  • SIXT VOMS parameters have changed.

Tuesday 17th February Two changes to approved VOs (https://www.gridpp.ac.uk/wiki/GridPP_approved_VOs)

  • LSST uses port 15003 (had been 15002, clashing with dzero)
  • t2k has included a note that it';s software is now distrinuted via CVMFS.
Site Updates

Tuesday 24th February

  • Next review of status today.

Tuesday 27th January

  • Squids not in GOCDB for: UCL; ECDF; Birmingham; Durham; RHUL; IC; Sussex; Lancaster
  • Squids in GOCDB for: EFDA-JET; Manchester; Liverpool; Cambridge; Sheffield; Bristol; Brunel; QMUL; T1; Oxford; Glasgow; RALPPD.

Tuesday 2nd December

  • Multicore status. Queues available (63%)
    • YES: RAL T1; Brunel; Imperial; QMUL; Lancaster; Liverpool; Manchester; Glasgow; Cambridge; Oxford; RALPP; Sussex (12)
    • NO: RHUL (testing); UCL; Sheffield (testing); Durham; ECDF (testing); Birmingham; Bristol (7)
  • According to our table for cloud/VMs (26%)
    • YES: RAL T1; Brunel; Imperial; Manchester; Oxford (5)
    • NO: QMUL; RHUL; UCL; Lancaster; Liverpool; Sheffield; Durham; ECDF; Glasgow; Birmingham; Bristol; Cambridge; RALPP; Sussex (14)
  • GridPP DIRAC jobs successful (58%)
    • YES: Bristol; Glasgow; Lancaster; Liverpool; Manchester; Oxford; Sheffield; Brunel; IC; QMUL; RHUL (11)
    • NO: Cambridge; Durham; RALPP; RAL T1 (4) + ECDF; Sussex; UCL; Birmingham (4)
  • IPv6 status
    • Allocation - 42%
    • YES: RAL T1; Brunel; IC; QMUL; Manchester; Sheffield; Cambridge; Oxford (8)
    • NO: RHUL; UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; Bristol; RALPP; Sussex
  • Dual stack nodes - 21%
    • YES: Brunel; IC; QMUL; Oxford (4)
    • NO: RHUL; UCL; Lancaster; Glasgow; Liverpool; Manchester; Sheffield; Durham; ECDF; Birmingham; Bristol; Cambridge; RALPP; Sussex, RAL T1 (15)


Tuesday 21st October

  • High loads seen in xroot by several sites: Liverpool and RALT1... and also Bristol (see Luke's TB-S email on 16/10 for questions about changes to help).

Tuesday 9th September

  • Intel announced the new generation of Xeon based on Haswell.



Meeting Summaries
Project Management Board - MembersMinutes Quarterly Reports

Empty

GridPP ops meeting - Agendas Actions Core Tasks

Empty


RAL Tier-1 Experiment Liaison Meeting (Wednesday 13:30) Agenda Meeting takes place on Vidyo.

Wednesday 18th March 2015

  • Operations report
  • The replacement of the problematic Tier1 network router with another unit last Thursday failed to solve the problem. Further tests will be carried out tomorrrow morning (19th March).
  • Proposed date for Castor upgrade to version 2.1.14-15 is the 8th April.
  • Five additional disk servers have been deployed into Castor storage for ALICE_Disk. This roughly doubles the disk space available to ALICE which is now approximately 390TB.
WLCG Grid Deployment Board - Agendas MB agendas

Empty



NGI UK - Homepage CA

Empty

Events
UK ATLAS - Shifter view News & Links

Atlas S&C week 2-6 Feb 2015

Production

• Prodsys-2 in production since Dec 1st

• Deployment has not been transparent , many issued has been solved, the grid is filled again

• MC15 is expected to start soon, waiting for physics validations, evgen testing is underway and close to finalised.. Simulation expected to be broadly similar to MC14, no blockers expected.

Rucio

• Rucio in production since Dec 1st and is ready for LHC RUN-2. Some fields need improvements, including transfer and deletion agents, documentation and monitoring.

• Rucio dumps available.

• Dark data cleaning

• files declaration . Only Only DDM ops can issue lost files declaration for now, cloud support needs to fill a ticket.

• Webdav panda functional tests with Hammercloud are ongoing

Monitoring

• Main page

• DDM Accounting

• space

• Deletion

ASAP

• ASAP (ATLAS Site Availability Performance) in place. Every 3 months the T2s sites performing BELOW 80% are reported to the International Computing Board.


UK CMS

Empty

UK LHCb

Empty

UK OTHER
  • N/A
To note

  • N/A



Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Operations_Bulletin_Latest&oldid=8160"