| General updates
|
Tuesday 13th November
- There is a GDB this week.
- An upgrade of the GridPP VOMS takes place tomorrow. New notifications will be introduced.
- Follow-up from last week's HEPSYSMAN?
- An EGI CSIRT process to handle unsupported gLite service end-points of unresponsive sites that failed to reply to COD tickets and to provide information about their upgrade plans has now been agreed. From today sites affected will be asked to put old endpoints into downtime and from 19th unresponsive sites will risk suspension. No UK site appears in the unresponsive list.
Tuesday 6th November
- VOMRS validator script led to confusing emails to LHC VO members (VOMRS checks user and CA DN and the old references had not been removed)
- Some progress is being made with the WN tarballs
- HEPSYSMAN is on Friday.
- Please take a look at and comment on the WLCG Software Life Cycle Process beyond EMI document.
- The current status of site upgrade plans is on Alessandra's summary page.
|
| Tier-1 - Status Page
|
Tuesday 13th November
- The main problem this week was the power outage that affected the RAL site, and for which the backup power via a diesel generator did not work. The power cut happened at around 11:30 on Wednesday 7th. All services were back early afternoon the next day. (A test of the UPS/diesel is scheduled for next Tuesday - 20th Nov).
- There was a problem that affected batch services on Saturday (10th) owing to a problem updating CERN CRLs.
- There was an outage of the Atlas SRM on Sunday caused by a database problem.
- This morning's planned intervention to replace a failing board in a router has been completed successfully.
- Investigating some asymmetric data rates seen to remote sites.
- A small start has been made in rolling out the over-commit of batch jobs to make use of hyperthreading.
- Upgrade to EMI-2 SL5 WNs in final testing / preparation.
- Test instance of FTS version 3 now available. Non-LHC VOs that use the existing service have been enabled on it and looking for one of the VOs to test.
|
| Storage & Data Management - Agendas/Minutes
|
Wednesday 10th October
- DPM EMI upgrades:
- 9 sites need to upgrade from gLite 3.2
- QMUL asking for FTS settings to be increased to fully test Network link.
- Initial discussion on how Brunel might upgrade it's SE and decommission is old SE
- Classic SE support , both for new SEs and plan to remove current publishing of classic SE endpoint
|
| Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06
|
Tuesday 30th October
- Storage availability in SL pages has been affected by a number of sites being asked by ATLAS to retire the ATLASGROUPDISK space token while the SUM tests were still testing it as critical. The availability will be corrected manually once the month ends. Sites affected in different degrees are RHUL, CAM, BHAM, SHEF and MAN.
Friday 28th September
- Tier-2 pledges to WLCG will be made shortly. The situation is fine unless there are significant equipment retirements coming up.
- See Steve Lloyd's GridPP29 talk for the latest on the GridPP accounting.
Wednesday 6th September
- Sites should check the atlas page reporting HS06 coefficient because according to the latest statement from Steve that is what it's going to be used Atlas Dashboard coefficients are averages over time.
I am going to suggest using the ATLAS production and analysis numbers given in hs06 directly rather
than use cpu secs and try and convert them ourselves as we have been doing. There doesn't seem to be
any robust way of doing it any more and so we may as well use ATLAS numbers which are the ones they are
checking against pledges etc anyway. If the conversion factors are wrong then we should get them fixed in our
BDIIs. No doubt there will be a lively debate at GridPP29!
|
| Interoperation - EGI ops agendas
|
Monday 5th November
- There was an EGI ops meeting today.
- UMD 2.3.0 in preparation. Release due 19 November, freeze date 12 November.
- EMI-2 updates: DPM/LFC and VOMS - bugfixes, and glue 2.0 in DPM.
- EGI have a list of sites considered unresponsive or having insufficient plans for the middleware migration. The one UK site mentioned has today updated their ticket again with further information.
- In general an upgrade plan cannot extend after the end of 2012.
- A dCache probe was being rolled into production yesterday, alarms should appear in the next 24 hours on the security dashboard
- CSIRT is taking over from COD on migration ticketing. By next Monday the NGIs with problematic sites will be asked to contact the sites, asking them to register a downtime for their unsupported services.
- Problems with WMS in EMI-2 (update 4) - WMS version 3.4.0. Basically, it can get proxy interaction with MyProxy a bit wrong. The detail is at GGUS 87802, and there exist a couple of workarounds.
Monday 8th October
- COD are about to launch monitoring tickets for 'out of support' services, (i.e. glite 3.2), for removal by the end of the month. (They seem to have missed some gLite 32 CREAM CE's, however - we need to make sure we don't).
- EMI updates. EMI-2, expected today or so.
DPM 1.8.4 (Yay! But let it filter through staged rollout a bit...)
LB and WMS 3.4 (Both with security updates).
UI and WN (including 32bit libs, and a few other dependancies).
- Tarballs were raised. Tiziana raised the need for a tarball (EMI-2) before the gLite 3.2 were retired.
- Staged Rollout. The ARC 2.0.0 clients are in the production repositories due to the emi-ui being in production. (Don't think that affects anyone in the UK).
- Released today: BDII Core and GFAL/lcgUtils.
- Products in staged rollout: WMS 3.3.8; CREAM 1.13.4 (due to a mismatch between EMI and UMD versions, this is 1.13.5 in UMD)
- It's been noted that there are a number of products without early adopters in EMI 2: EMIR, Pseudonymity, Wnodes, GridSAM and OGSA-DAI. These will not be included in UMD, unless there's an EA, and demand from NGI's. There's also a few with no EA in EMI-2, but there are in EMI-1, and these are expected to move to EMI-2 at some point: CLUSTER, CREAM-LSF. (VOMS was listed, but the EA was present, and pointed out they are on EMI-2).
- Unsupported services on 8th October EGI list.
|
| Monitoring - Links MyWLCG
|
Monday 2nd July
- DC has almost finished an initial ranking. This will be reviewed by AF/JC and discussed at 10th July ops meeting
Wednesday 6th June
- Ranking continues. Plan to have a meeting in July to discuss good approaches to the plethora of monitoring available.
- Glasgow dashboard now packaged and can be downloaded here.
|
| On-duty - Dashboard ROD rota
|
Monday 12th November
- Birmingham is in downtime till further notice as university is operating on emergency power.
- Durham has a open ticket.
- A lot of Nagios test failure because of power failure at Tier1 but now everything is back to normal.
Monday 5th November
Friday 19th October
- Many sites continue to have planned downtimes for EMI upgrades, with knock-on effects on other local services (eg SRM -> WN Rep alarms). Changeover back to Oxford GridPP Nagios midweek, with some caching weirdness (reloading the dashboard produced one of two different sets of results!) which quickly went away.
Friday 12th October
- There is a new ROD newsletter available from EGI.
- As of this week, John Walsh will no longer be contributing to the ROD work. Many thanks to John for his input over the years!
|
| Security - Incident Procedure Policies Rota
|
Monday 22nd October
- Last week's UK security activity was very much business as usual; there are a lot of alarms in the dashboard for UK sites, but for most of the week they only related to the gLite 3.2 retirement.
Friday 12th October
- The main activity over the last week has been due to new Nagios tests for obsoleted glite middleware and classic SE instances. Most UK sites have alerts against them in the security dashboard and the COD has ticketed sites as appropriate. Several problems have been fixed already, though it seems that the dashboard is slow to notice the fixes.
Tuesday 25th September
|
|
| Services - PerfSonar dashboard
|
Monday 5th November
- perfSONAR service types are now defined in GOCDB.
- Reminder that the gridpp VOMS will be upgraded next Wednesday.
Thursday 18th October
- VOMS sub-group meeting on Thursday with David Wallom to discuss the NGS VOs. Approximately 20 will be supported on the GridPP VOMS. The intention is to go live with the combined (upgrades VOMS) on 14th November.
- The Manchester-Oxford replication has been successfully tested. Imperial to test shortly.
|
| Tools - MyEGI Nagios
|
Tuesday 13th November
- Noticed two issues during tier1 powercut. SRM and direct cream submission uses top bdii defined in Nagios configuration to query about the resource. These tests started to fail because of RAL top BDII being not accessible. It doesn't use BDII_LIST so I can not define more than one BDII. I am looking into that how to make it more robust.
- Nagios web interface was not accessible to few users because of GOCDB being down. It is a bug in SAM-nagios and I have opened a ticket.
Availability of sites have not been affected due to this issue because Nagios sends a warning alert in case of not being able to find resource through BDII.
Wednesday 17th October
Monday 17th September
Monday 10th September
- Discusson needed on which Nagios instance is reporting for the WLCG (metrics) view
|
| VOs - GridPP VOMS VO IDs Approved VO table
|
Tuesday 23 October
- A local user is wanting to get on the grid and wants to set up his own UI. Do we have instructions?
Monday 15th October
- Sno+ jobs now work at Dresden https://ggus.eu/ws/ticket_info.php?ticket=86741, but there has got to be a better way.
- Discussion with SNO+ about their requirements - discussions started on the following topics:
- Robot certificates and hardware keys
- FCR
- Managing storage - how to avoid users filling up the space
Monday 8th October
- Sno+ had problems with EMI-2 WN and ganga - formatting changes in EMI-2 command output.
- Now fixed by Mark Slater (8 hours to install EMI2-WN and 20 mins to fix ganga.
- Snoplus jobs don't work at Dresden https://ggus.eu/ws/ticket_info.php?ticket=86741
- Draft e-mail to warning "non LHC VOs" about upcoming updates sent to ops list. Comments please.
|
| Site Updates
|
Monday 5th November
- SUSSEX: Site working on enabling of ATLAS jobs.
|
|
