Difference between revisions of "Tier1 Operations Report 2015-01-21"
From GridPP Wiki
(→) |
(→) |
||
Line 21: | Line 21: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues | ||
|} | |} | ||
− | * | + | * None |
<!-- ***********End Resolved Disk Server Issues*********** -----> | <!-- ***********End Resolved Disk Server Issues*********** -----> | ||
<!-- ***************************************************** -----> | <!-- ***************************************************** -----> |
Revision as of 11:55, 21 January 2015
RAL Tier1 Operations Report for 21st January 2015
Review of Issues during the week 14th to 21st January 2015. |
- The high load on the LHCb SRMs reported last week continued into this reporting period with intermittent SAM test failures. The problem has ben mitigated: LHCb are not loading teh SRMs as heavily and a third system has been added into the LHCb SRM alias.
- Electrical circuit testing in the machine room has continued into this week. Some problems have been encountered. No critical services have been affected although we have had some worker nodes stopped.
Resolved Disk Server Issues |
- None
Current operational status and issues |
- We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
- There is a problem with xroot access to the Castor GEN instance (not affecting ALICE).
Ongoing Disk Server Issues |
- None
Notable Changes made this last week. |
- In preparation for the electrical safety checks being carried out this week some "transfer switches" (enabling dual powering of some network switches and older systems) were installed during the second part of last week.
Declared in the GOC DB |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
Whole site | SCHEDULED | WARNING | 20/01/2015 08:30 | 22/01/2015 18:00 | 2 days, 9 hours and 30 minutes | Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day. |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
- Investigate problems on the primary Tier1 router.
Listing by category:
- Databases:
- A new database (Oracle RAC) has been set-up to host the Atlas 3D database. This is updated from CERN via Oracle GoldenGate. This system is yet to be brought into use. (Currently Atlas 3D/Frontier still uses the OGMA datase system, although this was also changed to update from CERN using Oracle Golden Gate.)
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4
- Castor:
- Update Castor headnodes to SL6 (Nameservers remain to be done).
- Update SRMs to new version (includes updating to SL6).
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Update Castor to 2.1-14-latest, this depends on SL6 being deployed.
- Networking:
- Resolve problems with primary Tier1 Router
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes finished; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during January.
Entries in GOC DB starting between the 14th and 21st January 2015. |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
Whole site | SCHEDULED | WARNING | 20/01/2015 08:30 | 22/01/2015 18:00 | 2 days, 9 hours and 30 minutes | Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day. |
All Castor (all SRM endpoints) | SCHEDULED | WARNING | 19/01/2015 09:15 | 19/01/2015 16:00 | 6 hours and 45 minutes | Warning during OS upgrade (SL6) on Castor Nameservers. |
Whole site | SCHEDULED | WARNING | 13/01/2015 18:30 | 15/01/2015 18:00 | 1 day, 23 hours and 30 minutes | Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day. |
Open GGUS Tickets (Snapshot during morning of meeting) |
GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
---|---|---|---|---|---|---|---|
111120 | Green | Less Urgent | In Progress | 2015-01-12 | 2015-01-15 | Atlas | large transfer errors from RAL-LCG2 to BNL-OSG2 |
110605 | Yellow | Less Urgent | In Progress | 2014-12-08 | 2015-01-20 | ops | [Rod Dashboard] Issues detected at RAL-LCG2 (srm-cms-disk.gridpp.rl.ac.uk) |
109712 | Red | Urgent | In Progress | 2014-10-29 | 2015-01-09 | CMS | Glexec exited with status 203; ... |
109694 | Red | Urgent | On hold | 2014-11-03 | 2015-01-20 | SNO+ | gfal-copy failing for files at RAL |
108944 | Red | Urgent | In Progress | 2014-10-01 | 2015-01-19 | CMS | AAA access test failing at T1_UK_RAL |
107935 | Red | Less Urgent | On Hold | 2014-08-27 | 2015-01-20 | Atlas | BDII vs SRM inconsistent storage capacity numbers |
Availability Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas HC | CMS HC | Comment |
---|---|---|---|---|---|---|---|---|
14/01/15 | 100 | 100 | 100 | 100 | 55.6 | 100 | 100 | Load (stripping campaign) affecting LHCb SRMs. User timeouts on tests. |
15/01/15 | 100 | 100 | 100 | 100 | 97.3 | 100 | 78 | Load. Single SRM test failure ([SRM_FILE_BUSY]). |
16/01/15 | 100 | 100 | 100 | 100 | 90.0 | 90 | 100 | Config problem as additional SRM set-up. |
17/01/15 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
18/01/15 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
19/01/15 | 100 | 100 | 100 | 100 | 100 | 100 | 99 | |
20/01/15 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |