RAL Tier1 Operations Report for 17th December 2014
| Review of Issues during the three weeks 17th December 2014 to 7th January 2015.
|
- On Saturday (13th Dec) network problems severely affected Tier1 services. A network switch was found to have have problems coincident with the service issues. A member of staff came on site and resolved the switch probem. However, this turned out not to be the principle underlying cause of the service problems which were then traced to a DNS server that was not responding. Systems were re-configured not to use that as the primary DNS server. In parallel with a member of Networking staff attending on site to fix the DNS server. The problems lasted from around 07:00 to 22:00 that day.
| Resolved Disk Server Issues
|
- GDSS778 (LHCbDst D1T0) failed in the early hours of Monday 15th December. Tests revealed faulty RAM which was replaced. The server was returned to production around 09:15 this morning (Wed. 17th Dec).
| Current operational status and issues
|
| Ongoing Disk Server Issues
|
- GDSS757 (CMSDisk D1T0) failed on Saturday 3rd January. It is undergoing tests.
| Notable Changes made this last week.
|
- On Tuesday morning (16th Dec) the firmware in the Tier1 router pair was updated to the latest production version. This is ahead of a patch to be applied in the New Year that should fix the ongoing RIP protocol problem.
- Following a restriction on numbers of CMS batch jobs imposed during problems a week or so ago the CMS jobs limits on the farm have been progressively increased.
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| Whole site
|
SCHEDULED
|
WARNING
|
20/01/2015 08:30
|
22/01/2015 18:00
|
2 days, 9 hours and 30 minutes
|
Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
|
| Whole site.
|
SCHEDULED
|
WARNING
|
13/01/2015 18:30
|
15/01/2015 18:00
|
1 day, 23 hours and 30 minutes
|
Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
|
| Whole site.
|
SCHEDULED
|
WARNING
|
07/01/2015 10:00
|
07/01/2015 12:00
|
2 hours
|
Warning on site during quarterly UPS/Generator load test.
|
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- The rollout of the RIP protocol to the Tier1 routers still has to be completed. A software patch from the vendors will be applied to the Tier1 Routers on Tuesday 6th January.
- The next quarterly UPS/Generator load test will take place on Wednesday 7th January.
- Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room: Tue-Thu 13-15 January & Tue-Thu 20-22 January. There are some systms that need to be re-powered in preparation for this work.
- Completing Castor headnode upgrades to SL6: Tuesday 6th Jan - GEN; Wednesday 7th Jan - Nameserver (transparent - at risk)
Listing by category:
- Databases:
- A new database (Oracle RAC) has been set-up to host the Atlas 3D database. This is updated from CERN via Oracle GoldenGate. This system is yet to be brought into use. (Currently Atlas 3D/Frontier still uses the OGMA datase system, although this was also changed to update from CERN using Oracle Golden Gate.)
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4
- Castor:
- Update Castor headnodes to SL6 (ongoing).
- Update SRMs to new version (includes updating to SL6).
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Update Castor to 2.1-14-latest, this depends on SL6 being deployed.
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes finished; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during January.
| Entries in GOC DB starting between the 17th December 2014 and 7th January 2015.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| Whole site
|
SCHEDULED
|
WARNING
|
07/01/2015 10:00
|
07/01/2015 12:00
|
2 hours
|
Warning on site during quarterly UPS/Generator load test.
|
| Castor GEN instance
|
SCHEDULED
|
OUTAGE
|
06/01/2015 10:00
|
06/01/2015 12:57
|
2 hours and 57 minutes
|
OS upgrade (SL6) on headnodes for CMS GEN instance
|
| Whole site
|
UNSCHEDULED
|
OUTAGE
|
25/12/2014 21:30
|
26/12/2014 01:00
|
3 hours and 30 minutes
|
All Services unavailable due to network problem. (Retrospective entry).
|
| Whole site
|
UNSCHEDULED
|
OUTAGE
|
24/12/2014 11:30
|
24/12/2014 12:40
|
1 hour and 10 minutes
|
All Services off air due to network problem. (Retrospective entry).
|
| srm-atlas, srm-cms-disk, srm-cms, srm-lhcb-tape, srm-lhcb
|
SCHEDULED
|
WARNING
|
23/12/2014 10:00
|
23/12/2014 12:00
|
2 hours
|
Due to Kernel patching of EGI ADV 20141217, the Atlas, CMS and LHCb Castor headnodes will be rebooted. There may be short breaks in services
|
| arc-ce01, arc-ce02, arc-ce03, arc-ce04, arc-ce05, cream-ce01, cream-ce02
|
SCHEDULED
|
OUTAGE
|
23/12/2014 09:00
|
23/12/2014 13:00
|
4 hours
|
Due to Kernel patching of EGI ADV 20141217, the RAL tier1 batch farm worker nodes will need to be rebooted.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 110776
|
Green
|
Urgent
|
Waiting for Reply
|
2014-12-15
|
2014-12-17
|
CMS
|
Phedex Node Name Transition
|
| 110605
|
Green
|
Less Urgent
|
On Hold
|
2014-12-08
|
2014-12-12
|
ops
|
[Rod Dashboard] Issues detected at RAL-LCG2 (srm-cms-disk.gridpp.rl.ac.uk)
|
| 110382
|
Green
|
Less Urgent
|
In Progress
|
2014-11-26
|
2014-12-16
|
N/A
|
RAL-LCG2: please reinstall your perfsonar hosts(s)
|
| 109712
|
Red
|
Urgent
|
In Progress
|
2014-10-29
|
2014-11-27
|
CMS
|
Glexec exited with status 203; ...
|
| 109694
|
Red
|
Urgent
|
On hold
|
2014-11-03
|
2014-12-18
|
SNO+
|
gfal-copy failing for files at RAL
|
| 108944
|
Red
|
Urgent
|
Waiting for Reply
|
2014-10-01
|
2014-01-07
|
CMS
|
AAA access test failing at T1_UK_RAL
|
| 107935
|
Red
|
Less Urgent
|
On Hold
|
2014-08-27
|
2014-12-15
|
Atlas
|
BDII vs SRM inconsistent storage capacity numbers
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 17/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 18/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 19/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 20/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 21/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 22/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 23/12/14 |
83.3 |
83.3 |
83.3 |
83.3 |
83.3 |
97 |
100 |
Batch farm having kernel updates applied.
|
| 24/12/14 |
97.0 |
100 |
95.9 |
91.8 |
98.2 |
100 |
96 |
Problem with primary Tier1 Router.
|
| 25/12/14 |
90.7 |
100 |
90.1 |
92.3 |
88.9 |
98 |
N/A |
Problem with primary Tier1 Router. Late evening.
|
| 26/12/14 |
89.5 |
71.2 |
94.9 |
85.7 |
90.7 |
100 |
100 |
Problem with primary Tier1 Router. Continuation of yesterday's problem.
|
| 27/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
99 |
|
| 28/12/14 |
100 |
100 |
100 |
100 |
100 |
98 |
100 |
|
| 29/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
98 |
|
| 30/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 31/12/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 01/01/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 02/01/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 03/01/15 |
100 |
100 |
100 |
100 |
96.8 |
100 |
99 |
Single SRM test failure: Error listing file: No such file or directory.
|
| 04/01/15 |
100 |
100 |
100 |
100 |
99.0 |
100 |
100 |
Previous test was just before midnight.
|
| 05/01/15 |
100 |
100 |
100 |
100 |
100 |
100 |
? |
|
| 06/01/15 |
100 |
100 |
100 |
100 |
95.8 |
100 |
? |
Single SRM test failure: User timeout over
|
