Difference between revisions of "Operations Bulletin Latest"

Tuesday 18th March

• The GridPP website was upgraded over the weekend (it is now SHA-2 ready). Please inform Andrew if you encounter any problems with it.
• Last week there was a pre-GDB on batch systems and a GDB. The March GDB meeting summary is now available. The GDB actions list has been updated.
• There was a GridPP Strategic Review yesterday. Recommendations will be shared with the PMB and CB in due course.
• The CERN VOMS service will move to new hosts whose host certificates are signed by the new (SHA-2) CERN CA during 2014. First VOMS-aware services in WLCG need to be aware of these hosts. See the EGI broadcast message which indicates a timeline of 6th May for services to be updated.
• The GridPP IPv6 site status table has been updated to provide an 'allocation' column. This follows discussion of allocation strategies across sites. This IETF document may be of interest.
• EGI invites entries to win a funded trip to the community forum in May.
• The next WLCG middleware readiness WG meeting takes place this afternoon at 13:30 UK time. There are pre-meeting updates in the wiki.
• Alarms (and then tickets) are now being raised against EMI-2 services at sites. Please respond to the tickets quickly and indicate your plans for removing the EMI-2 based node - replies are expected within 10 working days. If you believe alarms/tickets result from false-positives then please indicate this in the ticket... we are aware of some examples already.
• France is deploying an EGI wide Dirac instance for 'other communities' and EGI is considering to include this as part of a H2020 proposal.
• CVMFS v2.1.17 was recently released. Ian reported that RAL T1 had been using it stably for several weeks.
• The EGI availability/reliability figures for February have been added to the reports wiki page. The UK services show 100%. Ops for NGI_UK is 98%:98%. No sites were below the EGI 70% targets.

Monday 10th March

• EGI recently took part in a EC workshop on "Open Access in H2020: services and support for projects". EGI is starting a pilot distributed open data repository (interoperable with openAIRE) that may be of interest.
• A Vidyo test meeting room is still available for testing linked from this agenda. Headsets are recommended.
• Cristina Aiftimiei has joined the EGI.eu Operations team. She will continue to be employed by INFN but working full time to support the EGI Operations.
• The WLCG Tier-2 availability/reliability reports for February are now available. Please check your site for: ALICE; ATLAS; CMS and LHCb.(Reminder of Alessandra's suggestion to undertake a cloud view as in this FR cloud report).
• There was an outage of the GOCDB on 5th March for which an incident review has taken place. An initial database disk failure and no automated DNS switch from goc.egi.eu to goc.dl.ac.uk was complicated by a scheduled downtime for an OS update after which the primary DNS failed to resolve external addresses and a manual switch back to goc.egi.eu failed.
• EGI are moving their federated cloud sites to a production service (follow progress).
• This week there is a pre-GDB meeting on batch systems and a GDB.

Thursday 6th March

• There was a meeting today (agenda, minutes).
• Simone Campana was nominated ATLAS Distributed Computing coordinator and will step down as chair of WLCG Operations Coordination.

• Baselines: WMS fix for 512-bit keys, already applied at CERN.
• CERN would like to propose a deadline to switch FTS 2 off on the 1st of August.
• Review of baselines - main update is that fix for the 512-bit keys on WMSes is being applied.
• dCache is going to extend the security support for 2.2 until Enstore and dCache 2.6 are properly integrated. This will happen by summer.
• Tests ongoing with Oracle12
• dCache: is going to extend the security support for 2.2 until Enstore and dCache 2.6 are properly integrated.
• ALICE: Tier-1/2 workshop in Japan.
• ATLAS: in the middle of a disk crisis, many of the Tier1s are almost full of primary data. JEDI is under testing now. JEM activated (Job Evolution Monitor) for all the production resources. Rucio migration (Rucio as file catalog instead of LFC) in progress.
• CMS: Soon starting 13 TeV MC DIGI/RECO. Looking at ccess to high memory resources and multi-core jobs.
• LHCb: 2014 spring incremental stripping in full swing, 1/4 of the data has been processed (statistics).
• FTS3 deployment: Discussed with experiment DM developers how to integrate multiple FTS3 servers with experiment frameworks.
• glexec deployment: 79 tickets closed and verified, 16 still open
• Machine job features: no update
• Middleware readiness: Next meeting Tuesday 2014/03/18 @ 14:30h CET. See twiki.
• Multicore: Several meetings. Conclusions so far systems reviewed are capable of supporting multicore jobs however a tuning of each system is required to be able to absorb them (draining/reservation of resources) when running together with single core jobs.
• perfSONAR: perfSONAR 3.3.2 is now baseline. Deadline April 1, 2014 - all WLCG sites should have instances deployed, using the mesh - configuration and registered in OIM/GOCDB. Instructions in slides.
• SHA2: Many new users already registered OK with SHA-2 certificates. Host certs of CERN future VOMS servers are from the new SHA-2 CERN CA.
• Tracking tools: no update
• WMS decommissioning: no update
• xrootd deployment: no update

Tuesday 18th March

• The Tier1 moved to use the new site firewall yesterday (Monday 17th March). As announced FTSs (2 & 3) were drained/stopped around the change and Batch jobs were not started during it. There was one problem after the change that affected the LFC in particular and external access from many worker nodes but that was fixed in around an hour.
• The software server used by the small VOs will be withdrawn from service (aiming for June).
• A replacement MyProxy server has been put into production (to resolve the MyProxy issues raised in GGUS ticket 97025). This new service is called myproxy.gridpp.rl.ac.uk. Sites and VOs need to make appropriate reconfigurations to use this. We plan to turn the old one (lcgrbp01.gridpp.rl.ac.uk) off at the end of March.
• The Atlas file deletion rate has been improved. However, there is still a problem that needs to be understood.
• There have been problems with the CMS Castor instance at the end of last week and the start of this. The Castor /Database teams have some ideas for the cause of this which looks to be load related. Work is ongoing.
• EMI-3 WN has been rolled out to one batch of WNs which have behaved OK so far.
• There is a UPS/Generator load test tomorrow morning (Wednesday 19th March). A 'warning' has been announced in the GOC DB.

Tuesday 18th March

• Chris noticed some of Steve's tests failing. At IC this related to a full spacetoken. Bristol is not working as there is no SCRATCHDISK spacetoken. Durham fails with a no space left on device error message.

March 2014

• How would we move data between DiRAC and GridPP?

Tuesday 18th March

• A review of the HEPSPEC page reveals that most sites now have SL6 entries with the exceptions being: UCL-HEP; Manchester and RALPP. We will be checking HS06 figures in the 2014 Q1 quarterly reports so the deadline for action is 31st March 2014.
• APEL publishing appears up-to-date for all sites.

Tuesday 11th Febraury

• Another review of the HEPSPEC page shows no SL6 (or equivalent) entry for: UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; RALPP and RAL Tier-1. So... basically no change since last week. Tickets needed?
• The accounting pages show the following sites as not up-to-date with publishing accounting data: Glasgow (minor); RALPP and Sussex.
• Under publishing... sites are encouraged to run the Glue2 validator. There were problems observed for RAL-LCG2; UKI-LT2-IC-HEP; UKI-NORTHGRID-MAN-HEP; UKI-NORTHGRID-SHEF-HEP; UKI-SCOTGRID-GLASGOW and UKI-SOUTHGRID-RALPP.

Tuesday 4th February

• A review of the HEPSPEC page shows no SL6 (or equivalent) entry for: UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; RALPP and RAL Tier-1.
• The accounting pages show the following sites as not up-to-date with publishing accounting data: Lancaster (minor); RALPP and Sussex.

• Check publishing via: http://gstat2.grid.sinica.edu.tw/gstat/summary/Country/UK/

See the worst KeyDocs list for documents needing review now and the names of the responsible people.

Tuesday 18th March

• Keydocs action needed by: Mark M; Jens J; Rob H/Security T; Alessandra F; Wahid B; David C and Matt D.

Tuesday 25th February

• Keydocs owners need to take some action!

Tuesday 11th February

• Documents still need attention....

Tuesday 28th January

• Document assignments were reviewed at a core ops tasks meeting last week... so the dashboard for keydocs should improve soon!
• It has been noted that blog activity has dropped to a very low level. The blogs have been very useful to disseminate details of what is happening at sites, to share technical solutions and to make GridPP's work more visible. It also helps avoid duplication... so please give some thought to reviving your blogs!

Monday 10th March

• An EGI operations meeting took place today (agenda).

• URT recent or future planned releases
  • GridSite 2.2.2 (bug fix ) and dCache 2.6.20 for UMD-3

• SR updates
  • WMS 3.6.3 (today) - EMI3 WN tarball SR flagged

• New Nagios probes
  • emi-cream-nagios v. 1.1.1 - released with EMI 3 Update 14, released soon in SAM framework
  • org.sam.WN-SoftVer - new probes check the $EMI_TARBALL_BASE/etc/emi-version file
  • WN replication tests in emi-nagios are now distributed by the SAM-team, nagios-plugins-wn-rep
  • --wn-se-rep option as well as all the other previous --wn-* options will not be supported anymore by technology provider (see #88835 and #91683 )
  • NGIs requested to feedback how they feel about this option not being supported anymore.

• EMI-2 decommissioning
  • dCache extended the support for the 2.2.x versions until July 2014.
  • List of services failing given Services
  • Alarms to begin on Wednesday, so please check this list for errors ASAP.

• Cloud probes start raising alarms
  • 4 cloud sites have been certified in the last weeks, sites are currently monitored by cloudmon, but the errors are not raising alarms.

Tuesday 4th March

• Summary on HC functional tests
• Overview of feedback

Monday 24th February

• Next meeting this Friday, agenda looking at HammerCloud Functional tests

Tuesday 21 January

• Update from meeting on Friday (the 17th); the main item under discussion was the nagios probes (in particular, the Condorg and CREAM-CE).

• • One thing to pull out - new probes being written to remove use of WMS - ALICE/LHCb CREAM probes being rolled out Jan/Feb, ATLAS/CMS CondorG probes being rolled out after that
  • https://indico.cern.ch/getFile.py/access?contribId=2&resId=0&materialId=slides&confId=287653

Tuesday 18th March

• Many EMI-2 tickets created during the week. Some false positives add to confusion!
• Tickets outstanding (as on 15th March) for Brunel (2); Oxford - Although system in downtime until yesterday and Sheffield.

Tuesday 10th March

• Last week was "uneventful". Expect EMI-2 tickets this week.

Monday 3rd March

• A new dashboard is available for testing.
• The ROD rota has been extended to April
• Brunel sub-sites caused a problem leading to EMI-3 APEL alarms which is now fixed.

Tuesday 18th March

• The EMI-2 decommissioning task has started.
• The next WLCG middleware readiness WG meeting takes place this afternoon at 13:30 UK time.

Tuesday 11th February

• 31st May has been set as the deadline for EMI-2 decommissioning. There may be an issue for dCache (related to 3rd party/enstore component).

References

• Staged Rollout pages (now separated into EMI1 & 2), and the page listing the deployed versions is extractable from the bdii, so they should all be reasonably up-to-date:
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout.html
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout_emi2.html
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/state_of_the_nation.html

Tuesday 5th March

• Ready for more ARGUS testing
• SHA-2 looks ready for UK CA switch
• Looking at technologies

Monday 24th February

• Let Orlin know if you wish to try connections with the NGI ARGUS server. Tested working for EM including national banning. There is some setup documentation.

Tuesday 18th March

• A reminder of this perfSONAR overview for UK sites. Username is WLCGps. Currently shows a number of problems that need to be addressed at various sites.

Monday 10th March

• A reminder that the perfSONAR documentation is available here.
• Deadline for 3.3.2 is 1st April.

Tuesday 4th March

• The full UK perfSONAR view is given on this dashboard.
• When perfSONAR is performing in a stable fashion the site will appear on the main monitoring page.

Monday 17th March 2014, 14.00 GMT
47 Open UK tickets this week, a dozen of them are EMI2 retirement tickets so they'll get the lion's share of our attention.

NGI
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101502 (24/2)
The ILC software area move ticket. IC, RAL, QMUL, Cambridge and Liverpool have moved. Lancaster moved but is (hopefully was) broken for ILC (pardon my noise). Assuming that anyone not mentioned on the ticket hasn't migrated the ILC SW_DIR yet that leaves the following list of uk sites to migrate their ILC software area:
OXFORD
GLASGOW
BRISTOL
RALPP
BIRMINGHAM
BRISTOL
BRUNEL - Moved but hadn't updated the ticket
RHUL
DURHAM
MANCHESTER
(I might have missed some of you out, this list is from lcg-infosites and grepped using my admittedly poor eyeballs. A braver man would have crafted his own ldapsearch to glean this info).
If you don't want to make the change then removing support for ILC is a viable course of action, if you have made the change please update the ticket to let ILC know. In progress (17/3)

EMI ALARMS
Remember that you need to have *at least* your upgrade plans in these tickets within a fortnight of the ticket's submission - so by the 28th of March.

SUSSEX
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102212 (14/3)
BDII is the culprit here, ticket acknowledged but no other news (or plan). In Progress (17/3)

RALPP
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102207 (14/3)
ARC CE, a few CREAMs, site BDII and WNs - Chris reports that this is probably a false alarm, but is looking into it (in case the publishing is off). Chris has included a plan for the other components (if I'm reading right, is RALPP ditching all CREAMs?). In progress (14/3)

BRISTOL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102205 (14/3)
site BDII (seems to be a common one), some CEs and the WNs. Winnie has posted an assurance that the upgrade will be done in time, but I'm not sure if that'll count as a plan to the powers that be. In progress (17/3)

GLASGOW
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102202 (14/3)
Lots of services, but Dave has given a detailed upgrade battleplan. In progress (17/3)

EDINBURGH
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102201 (14/3)
I think site-BDII and WNs. Wahid has given assurances, but (sorry to be a pedantic patsy) not sure if that'll count as a "upgrade plan". In other news the testing for the SL6 WN tarball is going well. In progress (14/3)

DURHAM
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102199 (14/3)
The Durham DPM, CE and site BDII are on the list. Assigned (14/3)

SHEFFIELD
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102197 (14/3)
Some CEs and the APEL box (that's a guess). Elena has given a good plan, there's some hassle as their APEL and BDII box are shared. In progress (14/3)

UCL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102193 (14/3)
DPM, BDII, CE and WNs. Ben has said he will upgrade in the next few weeks. On hold (14/3)

RHUL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102189 (14/3)
A CREAM and the site BDII. Govind is planning his upgrade plans. In progress (14/3)

IMPERIAL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102185 (14/3)
Just some WNs. Engaged in testing and plan to upgrade the last cluster this week. In progress (14/3)

BRUNEL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102184 (14/3)
Just the DPM I think. Henry remarks that they're just about to embark on a physical server move and doesn't want to change anything significant before the move. I gave my recipe for the EMI3 dpm move in case it helps. In progress (14/3)

EFDA-JET
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102166 (14/3)
Again just the DPM. Acknowledged, but no plan. In progress (17/3)

I'm actually pretty sure Lancaster should have got a ticket as we still have one cluster on the EMI2 tarball. I'm not going to complain though.

NORMAL TICKETS

QMUL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101639 (26/2)
This ticket about jobs using RFC3820 style proxies not working at QM is in an odd state. The user seemed to be confused as to what feedback he should give. In progress (17/3)

https://ggus.eu/index.php?mode=ticket_info&ticket_id=101916 (8/3)
Sorry to be picking on QM, but this 444444 publishing jobs ticket is looking neglected, I suspect you've been frying bigger fish but can you please show it (or even better, the underlaying issue!) some love. Is this linked to your other information publishing problems? In progress (10/7)

UCL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101285 (16/2)
UCL Perfsonar ticket. After having his perfsonar box whacked by a power outage Ben is reinstalling, but is seeing some odd hardware issues. Has anyone else seen their R610s PERCs play up like this (only showing 3G partitions)? In progress (12/3)

As always please pipe up if I've missed anything or if there's any other ticket related issues you want to bring up.

Monday 17th March

• The GridPP DIRAC service is now able to submit jobs to VMs on Vac sites. Vac configuration for GridPP DIRAC explains how to configure a Vac site to run GridPP DIRAC jobs in VMs. More volunteer sites would be useful.
• EGI has published a new roadmap for operations tools.

Tuesday 26th November

• Regional Nagios updated to release 22. It is a glite to UMD update and it required a fresh installation.
• There have been some internal changes in SAM-Nagios. Test probes are now the responsibility of product team. Some test names have been changed as a result of this reorganization. For example the org.sam.CREAMCE-DirectJobSubmit test has become emi.cream.CREAMCE-DirectJobSubmit. This does not affect the operational activities.
• Please could all site admins look at services associated to their site and please mail Kashif if anything odd is noticed. Site admins can reschedule tests for their sites and it would be helpful if most functionalities are tested.
• Also, look at myegi which can be useful with links to the Dashboard, GSTAT, Accounting Portal and GGUS.

Monday 17 February 2014

• Proxy renewal
  • All RAL WMSs now renew proxies with 1024 bits. This looks like the end of this (at last).

Tuesday 11 February 2014

• Proxy renewal
  • lcgwms06 at RAL has been upgraded and works
  • Both Imperial's WMSs work
  • Glasgow's will still need to be upgraded (unless they have been since Friday).

• Impact
  • Citation policy (https://www.gridpp.ac.uk/acknowledging.html)

Actions

Empty

Empty

Wednesday 12th March

• Operations report
• All LHCb workflows now go through the ARC CEs OK.
• Network Changes: Move of Tier1 to use new site firewall will be on Monday 17th March. There will some interruption to services as seen from outside RAL. Internally services are expected to continue uninterrupted. Details in GOC DB.
• The Tier1 will be At Risk for an hour or so during the morning of Wedensday 19th March during a UPS/Generator load test.
• A new MyProxy server is in production (myproxy.gridpp.rl.ac.uk).
• Reminder: The software server usd by the small VOs will be withdrawn from service (aiming for June).

Empty

Empty

4th February 2014 With reference to the OMB on 30th January.

- UMD-2 will be decommissioned in the coming months. 30th April end of security support. 31st May all services to have been removed or upgraded.

- UK sites failing Glue2:

RAL-LCG2 UKI-LT2-IC-HEP UKI-NORTHGRID-MAN-HEP UKI-NORTHGRID-SHEF-HEP UKI-SCOTGRID-GLASGOW UKI-SOUTHGRID-RALPP

Check with the glue2 validator to see errors.

- A new Operations Dashboard will be in pre-production during February and moved into production in March.

- Availability/reliability targets for EGI are moving to 80%/85%.

- Some new ARC SAM tests are being introduced: org.nordugrid.ARC-CE-LFC-result; org.nordugrid.ARC-CE-LFC-submit; org.nordugrid.ARC-CE-SRM-result; org.nordugrid.ARC-CE-SRM-submit; org.nordugrid.ARC-CE-submit

- There is a summary page on how to publish from various middleware types: https://wiki.egi.eu/wiki/MAN09

- There was an overview of the French NGI adoption of iRODs.

- FedCloud to production: Management (GOCDB - ok); Monitoring (in progress); Accounting (in progress); Documentation (ok); Support (ok); Dashboard (in progress) and Security (in progress). - The SAM tests are: org.nagios.CloudBDII-Check; eu.egi.cloud.OCCI-VM and org.nagios.OCCI-TCP. Security checks not yet available.

Empty

Empty

Empty

• N/A