Difference between revisions of "Operations Bulletin Latest"

Monday 8th September

• Be ready for the new CERN and ops VOMS. Compare the prod and preprod instances for:
  • ALICE:preprod and prod.
  • ATLAS:preprod and prod.
  • CMS:preprod and prod.
  • LHCb:preprod and prod.
• An EMI3 WN tarball update has been done by Matt (see also GGUS 107869.
• There is an LHCONE/LHCOPN meeting next week on 16th and 17th (agenda). It would be good to have some remote participation.
• Website redesign - please complete this survey.
• For multicore - a reminder for sites running multicore and CREAM that there is an option in APEL to account multicore/multicpu. By default it is off.

Monday 1st September

• A/R results for August have been released.
  • ALICE: All good.
  • ATLAS: Durham (89%:98%) - very close! Sussex (45%:86%) - downtime for various updates. Problems with CE for WMS jobs only, so fine for ATLAS.
  • CMS: All good.
  • LHCb: All good. Northgrid and London perfect!
• EGI A/R results have also been uploaded to this table. July's results show the UK at 96% overall. UCL, Durham and Birmingham had a couple of issues that affected them.
• There is a UK CA TAG on 3rd September. Please let Jeremy know if you have any CA related issues or comments.
• There has been discussion about lock-up problems with 2.6.32-431 kernels on supermicro kit. Any conclusions?
• VOMS updates checks (mixed amongst pre-prod critical alarms):
  • CMS: Bristol, RALPP, RHUL.
  • LHCb: ECDF, EFDA.
  • ATLAS: UCL, ECDF, Oxford, RALPP.

Monday 25th August

• GridPP33 took place in Ambleside last week.
• Our thanks to Sam and Mohit, Year in Industry students, who have now finished. There will be less ticket prompts until new students are in place.
• Reminder of 17th March message: new VOMS servers for Ops and LHC experiments. The deadline is Monday 15th September. The experiment pre-prod instances will switch earlier. Already started: ALICE 23rd July; LHCb 22nd August. Pending ATLAS 28th August; CMS 28th August.
• For CMS: transition Savannah to GGUS (CMS Computing Operations): September 1st - Disable submission of new tickets; September 30th - Close Savannah (still open issues will be transferred to GGUS).
• ATLAS RIPE probes handed out to some GridPP sites; at these a welcome notification message should have been received.
• A UK CA TAG meeting is planned for 3rd September. One discussion item concerns an opportunity to migrate the UK e-Science CA to a new commercial CA as part of a JANET agreement.
• On 18th August the main DNS servers associated to the egi.eu domain were switched from Nikhef to CESNET.

Monday 8th September

• There will be a multi-core meeting on Tuesday 9th at 14:30 (CERN time). Covering reviews of the UGE setup for multicore jobs at CCIN2P3 and of the method to passing job requirement arguments to batch systems via CE. (Agenda)
• A review of last week's ops meeting....

Tuesday 2nd September

• The next WLCG ops coordination meeting is this Thursday 4th September.
• There will be a Tier-1/2 feedback section in the agenda IF there is feedback/input. Do we have any items to raise?

Tuesday 26th August

• There was a WLCG coordination meeting last Thursday. [1] are available.
• News: CERN-IT to terminate the SLC5-based interactive and batch services (lxplus5 and lxbatch5) soon. The current target date is 30 September 2014.
• A study to assess how operational effort in WLCG is used and could be optimised will launch in the next weeks. This will cover the management of sites and site services. It will (generally) not cover the experiment computing operations.
• MW baselines: No recent updates
• MW Issues: Storm and Argus integration issues.APEL fails to parse accounting records , affecting APEL 1.2.1 (released mid-August). Sites affected should move to 1.2.2. CVMFS upgrade to 2.1.19 almost done.
• Oracle: upgrade plans now available.
• T0: ARGUS latest version deployed. Looking at decommissioning AFS UI. A few users have already contacted CERN pointing out that they need SLC5 to build their software, as they haven't completed the porting to SLC6 yet - plan to push users to VMs on OpenStack.
• Confirmation wanted on the AFS UI tarball support.
• T1: No feedback.
• T2: No feedback.
• ALICE: steady production and analysis activities throughout the past weeks.
• ATLAS: No report.
• CMS: Finishing samples for CSA14; Computing Analysis Software challenge 2014 extended till mid-September. Users happy with AAA and miniAOD. Reminder for sites: Need to change xrootd redirectors, see this hn post; Need to adapt site-local-config.xml to include <phedex-node value=“Tx_CO_Site{_type}"/> (e.g. value=“T1_DE_KIT_Disk") in the <local-stage-out> section and the same format (but the PhEDEx name for the fallback endpoint) in <fallback-stage-out> NEW; Need to upgrade to CVMFS >= 2.1.19 immediately.
• LHCb: Low activity, mainly monte carlo simulation and user jobs. For SAM/Nagios in order to probe the ARC CEs at several UK sites, the probes are submitted now via a WMS instance from RAL-LCG2. The WMS instance was confirmed to be kept in production also for this purpose at least until 2015.
• Tracking tools: no report.
• FTS3: no report.
• glexec: no report.
• Machine/job features: Developer is leaving OSG.
• MW readiness: A new version of the WLCG Package Reporter has been released. A new BDII update 9 and Cream-ce 1.6.3 for CMS verification being deployed.
• Multicore: no report.
• SHA-2: Progress with new VOMS servers - compliance with the WLCG infrastructure being tested, ALICE results show CREAM/ARGUS config issues at some sites. Broadcast next week with hard deadline for 15th September. Sites that fail the SAM preprod tests by the end of Aug will be ticketed .
• WMS decommissioning: Condor validation - ATLAS and CMS ready. Deployment to production is planned on Wed 1st of October 2014.
• IPv6: Ewan ran tests on pure IPv6 EMI-3 UI. Mixed results.
• Squid monitoring/HTTP proxy: Reactivated Squid Monitoring TF to track its task list.
• Network and transfer metrics: Tasks/membership updated. perfSONAR Toolkit 3.4rc2 became available for testing, version 3.4 is a major milestone for the WG as it enables access via REST API and introduces several important performance improvements, therefore deployment campaign will follow once we get a stable release.

Tuesday 12th August

• The next meeting is on 21st August.

Tuesday 1st September

• There was a problem on Saturday (30th August) when a network switch failed. For reasons not yet understood some of the virtual machine infrastructure (supporting production services) had a problem despite not being on the network stack containing the failed switch. All services (except Castor) were declared down for around 5.5 hours.
• Both the FTS2 service and the software server used by the small VOs are being shutdown TODAY.

Monday 1st September

• FAX sites to update the C++ N2N rpms .
• There is interest regarding issues/performance when placing storage outside firewalls. JC will shortly start a (closed) discussion/survey.

Monday 11th August

• Pool nodes at RHUL have received test errors.

Tuesday 5th August

• The list of work Jens reviewed last Wednesday
  • WebFTS testing
  • Updating storage documentation (the wiki) and testing it
  • Upgrading DPM 1.8.7s?
  • GLUE 2.0 for storage revisited?
  • IPv6
  • WebDAV

Tuesday 2nd September

• Please check REBUS figures for your site.
• APEL status: Okay.

Tuesday 26th August

• Sheffield has stopped publishing.

See the worst KeyDocs list for documents needing review now and the names of the responsible people.

Tuesday 2nd September

• This work needs a kick-start! Reminders should now be being received.
• Tom/Andrew in discussion about options for main site - main considerations are Wordpress and Drupal.

Tuesday 26th August

• KeyDocs now working again. Several documents assigned to Jeremy for re-allocation. Owners need discussion.

Tuesday 12th August

• The keydocs php scripts are not yet working, so we cannot restart our review process....

Monday 8th September

• There is an EGI ops meeting today.

Tuesday 2nd September

• Monitoring consolation meeting last Friday

• • Validation of SAM2/3 results: https://twiki.cern.ch/twiki/bin/view/LCG/ValidationStatus
  • 4 UK sites (ECDF, Brunel, Durham, Oxford) had slight discrepancies: Looks either to be because more metrics are now being represented (blue = unknown, site not penalised), or older service still in vo feed (AGIS).
  • Next step is to compare availabilities for August for SAM2/3 and compare for sites
  • If sites see any discrepancies between http://dashb-atlas-sum.cern.ch/dashboard/request.py/historicalsmryview-sum and http://wlcg-sam-atlas.cern.ch/dashboard/request.py/historicalsmry for their site, please let me know

• Squid monitoring TF meeting last Thursday

• • Cosmin presented ALICE CVMFS proposal to revived TF
  • Notes from meeting: https://twiki.cern.ch/twiki/bin/view/LCG/SquidMonitoringTF20140828MeetingNotes

Tuesday 2nd September

• Sussex is back in business - kept closing their low availability alarm wrt the GGUS ticket.
• The UCL ticket is now finally receiving some attention.
• Ongoing problems at RAL.

Tuesday 26th August

• RAL : Nagios jobs staying in queue for long time - to be investigated.
• Sussex : Matt needs help probably from some SGE experts.
• UCL : No acknowledgement from the site (ticket escalated to second level).
• 100IT : There is an alarm from EGI federated cloud - this needs discussion.
• Durham : Availability alarms - require constant closing with some comments. Ticket with devs is open.

Tuesday 12th August

• Last week was quiet.
• Still one or to responses needed for next rota allocations.

Tuesday 26th August

• EMI3 WN tarball update needed soon (GGUS 107869)

Monday 28th July

• UMD v.3.8.0 was released on 24th July.

References

• Staged Rollout pages (now separated into EMI1 & 2), and the page listing the deployed versions is extractable from the bdii, so they should all be reasonably up-to-date:
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout.html
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout_emi2.html
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/state_of_the_nation.html

Monday 8th August

• There was a security team meeting last Wednesday.
• There was a CA TAG meeting also last Wednesday.

Monday 11th August

• Topics as mentioned during the last GridPP technical meeting.

• There is an issue at the moment in the evaluation of vulnerabilities causing everything rated 'High' by Pakiti to display as 'Critical' in the Dashboard.

• The EGI security dashboard.

- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).

Tuesday 2nd September

• Only a few of the RIPE probes went live last week - any issues at the other sites to be discussed?
• JANET is going to deploy a perfSONAR instance on one of the exchange points in London. They hope it will help raise awareness of issues with local systems affecting their transfer performance.

Tuesday 12th August

• A reminder to update site status information in the IPv6 pages.
• There is a new version (v3.4rc2) of perfSONAR being tested at QMUL [2]. Details here [3].
• We will shortly review issues being picked up by perfSONAR and the steps to take when investigating.

Monday 8th September 2014, 15.00 BST
25 Open UK tickets this week.

NO SITE IN PARTICULAR
108182(3/9)
As seen on TB-SUPPORT, the NGI has a ticket telling it to get sites to have the new voms servers configured for the switch over. Jeremy has kindly offered to field the ticket. I think we all have this in hand, but as I type this I realise I may have forgotten to set things up for the ops VO. I encourage everyone to double check their readiness ahead of next Monday's switchover. Assigned (8/9)

106615(2/7)
The RAL FTS2 service has been shutdown for nearly a week now, so I suspect this ticket tracking the switch off can be closed. In progress (3/9)

RALPP
108306(8/9)
CMS having trouble running a "locateall" AAA test at RALPP (TBH I don't know what that is) - Chris has let them know that this is due to their xrootd reverse proxy being down, and it should be up and running in a day or two after it's reinstalled. In progress (8/9)

OXFORD
107911(27/8)
As mentioned last week, Sno+ have been having trouble as they can't assign software tags on Arc CEs, and they use these tags to do stuff like black/white listing. There was some dicussion on this in the ticket, but it fizzled out- I suspect due to the topic moving offline. Can it have an update please? In progress (27/8)

BRISTOL
106554(29/6)
CMS transfer problems to Bristol. Winnie put an update, where she mentioned she has applied a fix to their Storm that might have fixed the problem. Maybe. She's asked if the problem still persists, as the monitoring links provided have all gone stale. Lukasz is on leave, can anyone CMS savvy help her? Waiting for reply (8/9)

106325(18/6)
CMS Pilots losing contact with home base. No progress since Winnie noticed that the problem only seems to affect one of the Bristol clusters, but none expected due to leave. On Hold (8/9)

EDINBURGH
108100(1/9)
Maarten ticketed ECDF about this CE's not having the new voms servers configured. Andy is working on it. There's a reminder that on top of adding the right configs services do need restarting. In progress (5/9)

95303(1/7/2013)
glexec tarball ticket. There's a bit more movement on getting this done, but it's all on me to get the tarball glexec working still - naught the Edinburgh chaps can do.

DURHAM
108273(5/9)
Duncan noticed some interesting goings on on the Durham perfsonar page. The Durham chaps are talking to their networking team to figure out what the flip is going on. In progress (8/9)

SHEFFIELD
107886(26/8)
Duncan's unwavering gaze also noticed a problem on Sheffield's perfsonar. Elena was tweaking it when it broke, and it looks like it's still broken, any luck fixing it Elena? In progress (26/8)

LIVERPOOL
108288(8/9)
Liverpool got a ROD ticket when their CREAM CE got poorly. Steve worked his magic and things were fixed, but Gareth asks about the persisting BDII tests still failing. Solved (8/9)

LANCASTER
100566(27/1)
My personal shame number 1. Lancaster's poor perfsonar performance. Despite a reinstall of the box and not showing any signs of a bottle neck in transfers or running manual tests we still have really poor perfsonar results. No problems with the network have been found. Duncan helped formulate a plan at GridPP, but I haven't had the time to test it out yet. On hold (8/9)

95299(1/7/13) My personal shame number 2 - Lancaster's glexec deployment ticket. Some news in that I have something I'd like to test now - I just need to find time to test it, then see if I can package it somehow. On hold (8/9)

UCL
95298(1/7/13)
UCL's glexec deployment ticket. This work was pushed back to the end of August - any news on it? On Hold (29/7)

107711(15/8)
A ROD ticket for UCL APEL publishing errors. The apel admins got involved and things are looking better now - although Gareth points out that there is some missing data in the Spring. In progress (8/9)

QMUL
107799(21/8)
Pointing VO_SNOPLUS_SNOLAB_CA_SW_DIR to /cvmfs/snoplus.gridpp.ac.uk. No news for a while on this after it was acknowledged - has the job fallen to the bottom of the stack? In progress (22/8)

108217(3/9)
Duncan ticketed QM about one of their pefsonar boxen - which Dan pointed out is their IPv6 perfsonar. So does that mean this ticket can be closed? In progress (4/9)

EFDA-JET
97485(21/9/13)
Longstanding LHCB ticket with JET. No movement on this, but none was expected. Still if anyone wants to heroically interject with some ideas I'm sure it would be appreciated. On hold (29/7)

TIER 1
107880(26/8)
As mentioned last week, Matt M of Sno+ fame has a user who only has access to srm tools and is having trouble accessing files at RAL. Brian has suggested using the webfts, but Matt doesn't think this will work for the user's limited abilities. Any thoughts? In progress (8/9)

107935(27/8)
Inconsistency between BDII and SRM reported storage capacity...hang on, haven't we been here before (105571)? It's not quite the same problem, but it's close. Brian has confirmed the mismatch, Maria has asked for an explanation for it (and how it only really effects ATLASHOTDISK). In progress (3/9)

105405(14/5)
Checking the site firewall configuration for RAL's Vidyo router. Last update was in July, is the dialogue between the Vidyo team and the RAL networking chaps ongoing? On hold (1/7)

106324(18/6)
The Tier 1's version of 106325 - CMS pilots losing contact. This was waiting on the firewall expert getting back from hols to compare the settings between the Tier 1 and Tier 2 (who don't see this issue). Are they back yet? On Hold (14/8)

Monday 14th July

Winnie reported on Saturday 12th July that most of the UK sites are failing nagios test. Problem started with unscheduled power cut at a Greek site hosting EGI Message broker (mq.afroditi.hellasgrid.gr) around 2PM on 11th July. Message broker was put in downtime but topbdii's continued to publish it for quite long time. Stephen Burke mentioned in TB support thread that now default caching time is 4 days. When I checked on Monday morning only Manchester was still publishing mq.afroditi and it went away after Alessandra manually restarted top bdii. It seams that Imperial is configured with much shorter cache time. Only Oxford and Imperial was almost not affected and the reason may be that Oxford WN's have Imperial top bdii as first option in BDII_LIST. Other NGI's have reported same problem and this outage is likely to be considered when calculating availability/reliability. All Nagios tests came back to normal now.

Emir reported this on tools-admin mailing list "We were planning to raise this issue at the next Operations meeting. In these extreme cases 24h cache rule in Top BDII has to be somehow circumvented."

Tuesday 1st July

• There was a monitoring problem on 26th June. All ARC CE's were using storage-monit.phyics.ox.ac.uk for replicating files as part of the nagios testing. storage-monit was updated but not re-yaimed until later. Storage-monit was broken for the morning leading to all ARC SRM tests failing.

Tuesday 24th June

• An update from Janusz on DIRAC:
• We had a stupid bug in Dirac which affected the gridpp VO and storage. Now it is fixed and I was able to successfully upload a test file to Liverpool and register the file with the DFC
• The async FTS is still under study, there some issues with this.
• I have a link to software to sync user database from a VOMS server, haven’t looked into this in detail yet.

Monday 11th August

• Steve J sent an email to hyperk on 7th regarding "software directory for Hyperk (CVMFS)" and entries in the VO ID card.

"Monday 14th July 2014"

• HyperK.org will initially use remote storage (irods at QMUL) - so CPU resources would be appreciated.

"Monday 30 June 2104"

• HyperK.org request for support from other sites
  • 2TB storage requested.
  • CVMFS required

• Cernatschool.org
  • WebDAV access to storage -world read works at QMUL.
  • ideally will configure federated access with DFC as LFC allows.

Monday 16 June 2014

• CVMFS
  • Snoplus almost ready to move to CVMFS - waiting on two sites. Will use symlinks in existing software

• VOMS server: Snoplus has problems with some of the VOMS servers - see ggus 106243 - may be related to update.

Tuesday 15th April

• Is there interest in an FTS3 web front end? (more details)

• Impact
  • Citation policy (https://www.gridpp.ac.uk/acknowledging.html)

Tuesday 20th May

• Various sites but notably Oxford have ARGUS problems. 100s of requests seen per minute. Performance issues have been noted after initial installation at RAL, QMUL and others.

Empty

Empty

Wednesday 3rd September 2014

• Operations report
• Tenders for this year's CPU & Disk purchases are underway.
• The FTS2 service was terminated on the 2nd September.
• The software server used by the smaller VOs has been turned off.
• Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday 23rd September.

Empty

Empty

Empty

Empty

Empty

• N/A