RAL Tier1 Operations Report for 10th December 2014
| Review of Issues during the week 3rd to 10th December 2014.
|
- The problems reported last week that were triggered by CMS Castor being very full continued through until the night of Thursday/Friday (4/5 December). With Castor very full there were very few disk servers available with any space on to receive write requests. This coupled with large numbers of batch jobs running was the cause of the problem. CMS deleting files to make space and a reduction in the number of running batch jobs relieved thd strain.
- There was a problem with the SRM back-end daemon's for Atlas crashing on Sunday and into Monday (7/8 Dec) and an unscheduled outage was announced. This was found to be the same as a problem seen on the 'GEN' instance in the summer and was caused by the VO requesting filenames containing a double-slash (//). A workaround that had previously been applied to the GEN instance was applied to the Atlas stager database and the problem resolved on the Monday afternoon.
| Resolved Disk Server Issues
|
| Current operational status and issues
|
| Ongoing Disk Server Issues
|
| Notable Changes made this last week.
|
- CMS Castor headnodes were updated to SL6 on Tuesday 9th December and the Atlas ones this morning (Wednesday 10th Dec).
- Database trigger to replace a double-slash (//) with a single-slash in a filename applied to the Atas SRM database on Monday 7th December and the LHCb & CMS the following day.
- On Tuesday (9th) WebDAV download access to Castor was enabled for LHCb.
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- The rollout of the RIP protocol to the Tier1 routers still has to be completed. As part of udnerstanding the problem there will be an upgrade to the firmware on the Tier1 Routers on Tuesday 16th Dec, and a further one on Tuesday 6th January.
- First quarter 2015: Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room.(Provisional dates: Tue-Thu 13-15 January & Tue-Thu 20-22 January).
- Completing Castor headnode upgrades to SL6: Tuesday 6th Jan - GEN; Wednesday 7th Jan - Nameserver (transparent - at risk)
Listing by category:
- Databases:
- A new database (Oracle RAC) has been set-up to host the Atlas 3D database. This is updated from CERN via Oracle GoldenGate. This system is yet to be brought into use. (Currently Atlas 3D/Frontier still uses the OGMA datase system, although this was also changed to update from CERN using Oracle Golden Gate.)
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4
- Castor:
- Update Castor headnodes to SL6 (ongoing).
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Update Castor to 2.1-14-latest, this depends on SL6 being deployed.
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes finished; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
| Entries in GOC DB starting between the 3rd and 10th December 2014.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| srm-atlas
|
SCHEDULED
|
OUTAGE
|
10/12/2014 10:00
|
10/12/2014 14:00
|
4 hours
|
OS upgrade (SL6) on headnodes for Atlas Castor instance.
|
| srm-cms-disk, srm-cms
|
SCHEDULED
|
OUTAGE
|
09/12/2014 10:00
|
09/12/2014 14:00
|
4 hours
|
OS upgrade (SL6) on headnodes for CMS Castor instance.
|
| srm-atlas
|
UNSCHEDULED
|
WARNING
|
08/12/2014 16:35
|
09/12/2014 10:00
|
17 hours and 25 minutes
|
Warning on SRM Atlas following problems over last day.
|
| srm-atlas
|
UNSCHEDULED
|
OUTAGE
|
08/12/2014 12:00
|
08/12/2014 16:31
|
4 hours and 31 minutes
|
Ongoing outage on Atlas SRM due to instabilities in the service.
|
| srm-atlas
|
UNSCHEDULED
|
OUTAGE
|
07/12/2014 16:45
|
08/12/2014 12:00
|
19 hours and 15 minutes
|
Outage on Atlas SRM due to instabilities in the service.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 110605
|
Green
|
Less Urgent
|
In Progress
|
2014-12-08
|
2014-12-08
|
ops
|
[Rod Dashboard] Issues detected at RAL-LCG2 (srm-cms-disk.gridpp.rl.ac.uk)
|
| 110382
|
Green
|
Less Urgent
|
In Progress
|
2014-11-26
|
2014-12-10
|
N/A
|
RAL-LCG2: please reinstall your perfsonar hosts(s)
|
| 109712
|
Yellow
|
Urgent
|
In Progress
|
2014-10-29
|
2014-11-27
|
CMS
|
Glexec exited with status 203; ...
|
| 109694
|
Yellow
|
Urgent
|
On hold
|
2014-11-03
|
2014-12-03
|
SNO+
|
gfal-copy failing for files at RAL
|
| 108944
|
Red
|
Urgent
|
In Progress
|
2014-10-01
|
2014-12-09
|
CMS
|
AAA access test failing at T1_UK_RAL
|
| 107935
|
Red
|
Less Urgent
|
On Hold
|
2014-08-27
|
2014-11-03
|
Atlas
|
BDII vs SRM inconsistent storage capacity numbers
|
| 106324
|
Red
|
Urgent
|
On Hold
|
2014-06-18
|
2014-11-27
|
CMS
|
pilots losing network connections at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 03/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
47 |
|
| 04/12/14 |
100 |
100 |
100 |
100 |
100 |
83 |
n/a |
|
| 05/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
n/a |
|
| 06/12/14 |
100 |
100 |
100 |
100 |
95.8 |
100 |
n/a |
|
| 07/12/14 |
100 |
100 |
49.0 |
100 |
100 |
76 |
n/a |
Atlas SRM daemons crashing when // in filename.
|
| 08/12/14 |
100 |
100 |
74.0 |
100 |
100 |
100 |
n/a |
Atlas SRM daemons crashing when // in filename.
|
| 09/12/14 |
100 |
100 |
100 |
83.3 |
100 |
100 |
n/a |
Upgrade of CMS Castor headnodes to SL6.
|
