RAL Tier1 Operations Report for 24th September 2014
| Review of Issues during the week 17th to 24th September 2014.
|
- On Saturday (13th Sep) there was a problem with the Atlas Castor instance that persisted into the beginning of Sunday. A number of measures were taken to improve it, although the root cause remains unknown.
- For the second half of last week there were problems with cream-ce02.
- This morning (Wednesday 17th Sep) there was a problem with some machines that run as VMs - the symptom was that their networking stopped. Restarting the network fixed the problem. This is similar to a problem seen on the 30th August. The configuration of the network interface on these systems has been changed to workaround this. One of the systems affected was the argus server and this caused a problem for batch job submissions for an hour or so.
| Resolved Disk Server Issues
|
- GDSS763 (AtlasDataDisk - D1T0) failed in the early morning of Thursday 18th Sep. It was restarted and tested but no fault found. The system was returned to service early the following afternoon (19th Sep).
- GDSS720 (AtlasDataDisk - D1T0) failed on Sunday afternoon, 21st Sep. It was restarted and tested but no fault found. The system was returned to service at the end of Monday afternoon (22nd Sep).
| Current operational status and issues
|
| Ongoing Disk Server Issues
|
| Notable Changes made this last week.
|
- VO Londongrid enabled on LFC.
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday
30th September.
- The Atlas Frontier service will be switched to use the new database system that updates from CERN using Oracle "GoldenGate" on 24th Sep.
Listing by category:
- Databases:
- Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC).
- A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
- Switch LFC/3D to new Database Infrastructure.
- Castor:
- Update Castor headnodes to SL6.
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
| Entries in GOC DB starting between the 17th and 24th September 2014.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| lcgfts3.gridpp.rl.ac.uk,
|
SCHEDULED
|
WARNING
|
23/09/2014 10:00
|
23/09/2014 12:00
|
2 hours
|
Service At Risk during upgrade to FTS3 version 3.2.27.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 108546
|
Green
|
Less Urgent
|
In Progress
|
2014-09-16
|
2014-09-22
|
Atlas
|
RAL-LCG2_HIMEM_SL6: production jobs failed
|
| 107935
|
Amber
|
Less Urgent
|
In Progress
|
2014-08-27
|
2014-09-02
|
Atlas
|
BDII vs SRM inconsistent storage capacity numbers
|
| 107880
|
Red
|
Less Urgent
|
In Progress
|
2014-08-26
|
2014-09-02
|
SNO+
|
srmcp failure
|
| 106324
|
Red
|
Urgent
|
In Progress
|
2014-06-18
|
2014-09-23
|
CMS
|
pilots losing network connections at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 17/09/14 |
100 |
100 |
100 |
100 |
100 |
98 |
96 |
|
| 18/09/14 |
100 |
100 |
100 |
100 |
100 |
99 |
99 |
|
| 19/09/14 |
100 |
100 |
100 |
100 |
100 |
100 |
98 |
|
| 20/09/14 |
100 |
100 |
100 |
100 |
100 |
96 |
97 |
|
| 21/09/14 |
100 |
100 |
100 |
100 |
100 |
97 |
97 |
|
| 22/09/14 |
100 |
100 |
100 |
100 |
100 |
95 |
98 |
|
| 23/09/14 |
100 |
100 |
100 |
100 |
100 |
98 |
98 |
|
