RAL Tier1 Operations Report for 27th August 2014
| Review of Issues during the fortnight 13th to 27th August 2014.
|
- Following some problems with disk server draining in Castor 2.1.14 a modified procedure has been tested on one disk server and been successful.
- While the farm was quiet around the 14th August the number of permitted Alice jobs was increased.
- Late evening on Monday 18th Aug there was an (Atlas) Oracle database crash due to a known (and reported) bug. The database failed over to another node in the Oracle RAC. There were some restarts of the (Atlas) SRM processes as the failover occured and again as the databse was manually returned to its 'correct' node in the RAC an hour later.
- There was load issues on the Atlas Scratch disk problems over the weekend (Sunday & Monday 24/25 August).
| Resolved Disk Server Issues
|
- GDSS648 (LHCbUser - D1T0) failed during the evening of Monday 25th August. The RAID controller crashed while recovering medium errors. It was returned to service this morning (Wednesday 27th Aug).
- GDSS612 (AtlasScratchDisk - D1T0) was taken out of service yesterday morning (26th Aug) as it was found to have two failed disk drives. It was returned to service in read-only mode at the end of the afternoon of that day while the first drive was rebuilding. The server was then put back in full production this morning (27th Aug).
| Current operational status and issues
|
- Discrepancies were found in some of the Castor database tables and columns. The Castor team are considering options with regard to fixing these. The issue has no operational impact.
- We are still investigating xroot access to CMS Castor following the upgrade on the 17th June. The service has improved but there may still be work to be done.
| Ongoing Disk Server Issues
|
| Notable Changes made this last week.
|
- Successful test of modified Castor diskserver draining procedure.
- 20/21st Aug: Deployed five disk servers (gdss633-gdss637) to AtlasTape cache.
- Systems (CEs etc) updated to Condor version 8.2.2.
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| lcgfts.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
02/09/2014 11:00
|
02/10/2014 11:00
|
30 days,
|
Service being decommissioned.
|
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- We are planning the termination of the FTS2 service (announced for 2nd September) now that almost all use is on FTS3.
- The removal of the (NFS) software server is scheduled for the 2nd September.
- We are planning stop access to the cream CEs - although possibly leaving them available to ALICE for some time. No date has yet been specified for this.
Listing by category:
- Databases:
- Apply latest Oracle patches (PSU) to the production database systems (Castor, Atlas3D).
- Switch LFC/FTS/3D to new Database Infrastructure.
- Castor:
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers. (Requires resolution of blocking issue).
- Fabric
- We are phasing out the use of the software server used by the small VOs.
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes; migration of GEN from 'A' to 'D' tapes.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during 2014.
| Entries in GOC DB starting between the 13th and 27th August 2014.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| lcgfts.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
02/09/2014 11:00
|
02/10/2014 11:00
|
30 days,
|
Service being decommissioned.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 107880
|
Green
|
Less Urgent
|
In Progress
|
2014-08-26
|
2014-08-27
|
SNO+
|
srmcp failure
|
| 107815
|
Green
|
Less Urgent
|
Waiting Reply
|
2014-08-22
|
2014-08-26
|
OPS
|
[Rod Dashboard] Issues detected at RAL-LCG2
|
| 107778
|
Green
|
Less Urgent
|
In Progress
|
2014-08-21
|
2014-08-22
|
Atlas
|
UK:RAL problems with staging
|
| 106324
|
Red
|
Urgent
|
On Hold
|
2014-06-18
|
2014-08-14
|
CMS
|
pilots losing network connections at T1_UK_RAL
|
| 105405
|
Red
|
Urgent
|
On Hold
|
2014-05-14
|
2014-07-29
|
|
Please check your Vidyo router firewall configuration
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 13/08/14 |
100 |
100 |
100 |
100 |
100 |
98 |
100 |
|
| 14/08/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 15/08/14 |
100 |
100 |
100 |
100 |
100 |
98 |
99 |
|
| 16/08/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 17/08/14 |
100 |
100 |
100 |
100 |
100 |
100 |
99 |
|
| 18/08/14 |
100 |
100 |
100 |
100 |
100 |
92 |
100 |
|
| 19/08/14 |
100 |
100 |
97.1 |
100 |
100 |
95 |
99 |
Load problems on Atlas scratch disk caused some SRM test failures.
|
| 20/08/14 |
100 |
100 |
98.2 |
100 |
100 |
98 |
100 |
As above
|
| 21/08/14 |
100 |
-100 |
95.8 |
100 |
100 |
77 |
92 |
As above
|
| 22/08/14 |
100 |
100 |
99.0 |
100 |
100 |
33 |
99 |
As above
|
| 23/08/14 |
100 |
100 |
96.4 |
100 |
100 |
31 |
100 |
As above
|
| 24/08/14 |
100 |
100 |
100 |
100 |
100 |
86 |
100 |
|
| 25/08/14 |
100 |
100 |
90.2 |
100 |
100 |
58 |
100 |
Some SRM test failures - one disk server unavaialble.
|
| 26/08/14 |
100 |
100 |
100 |
100 |
100 |
100 |
91 |
|
