Difference between revisions of "Tier1 Operations Report 2014-11-19"
From GridPP Wiki
(→) |
(→) |
||
Line 236: | Line 236: | ||
| 15/11/14 || 100 || 100 || 100 || 100 || 100 || 99 || 100 || | | 15/11/14 || 100 || 100 || 100 || 100 || 100 || 99 || 100 || | ||
|- | |- | ||
− | | 16/11/14 || 100 || 100 || style="background-color: lightgrey;" | 98.4 || 100 || 100 || 100 || 99 || | + | | 16/11/14 || 100 || 100 || style="background-color: lightgrey;" | 98.4 || 100 || 100 || 100 || 99 || Two SRM Put test failures (timeouts). |
|- | |- | ||
| 17/11/14 || 100 || 100 || 100 || 100 || 100 || 99 || 99 || | | 17/11/14 || 100 || 100 || 100 || 100 || 100 || 99 || 99 || | ||
|- | |- | ||
− | | 18/11/14 || 100 || | + | | 18/11/14 || 100 || style="background-color: lightgrey;" | 91.1 || style="background-color: lightgrey;" | 88.3 || style="background-color: lightgrey;" | 95.9 || 100 || 94 || 96 || Networking problems. Mainly affecting CE tests. |
|} | |} | ||
<!-- **********************End Availability Report************************** -----> | <!-- **********************End Availability Report************************** -----> | ||
<!-- *********************************************************************** -----> | <!-- *********************************************************************** -----> |
Revision as of 08:30, 19 November 2014
RAL Tier1 Operations Report for 19th November 2014=
Review of Issues during the week 12th to 19th November 2014. |
- During the morning of Tuesday 18th there were network problems - particularly affecting DNS lookups. This occured following a planned network change that was not expected to affect Tier1 operations.
Resolved Disk Server Issues |
- None
Current operational status and issues |
- Some problems on Atlas Castor instance. At various times in the last couple of weeks the Atlas workload has led to differing groups of disk servers spending a lot of time in a "wait i/o" state. This is triggered by the numbers of reads using xroot and has led to some SAM test failures.
Ongoing Disk Server Issues |
- GDSS673 (LhcbRawDst - D0T1) failed during the evening of Friday 14th November. The server is still undergoing checks. There are two files on the server that had not been migrated to tape.
Notable Changes made this last week. |
- Made 4 GB the default memory limit for LHCb jobs on arc-ce01.
- OS Errata rolled out to Castor Atals, CMS & LHCb instance (headnodes & disk servers) this morning (19th Nov) (to be confirmed)
Declared in the GOC DB |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
Atlas, CMS & LHCb Castor instances (srm-atlas, srm-cms, srm-lhcb) | SCHEDULED | WARNING | 19/11/2014 11:00 | 19/11/2014 12:00 | 1 hour | Warning on Castor Atlas, CMS and LHCb instance during application of OS errata updates. |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- First quarter 2015: Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room.(Provisional dates: week 12-16 January).
Listing by category:
- Databases:
- A new database (Oracle RAC) has been set-up to host the Atlas 3D database. This is updated from CERN via Oracle GoldenGate. This system is yet to be brought into use. (Currently Atlas 3D/Frontier still uses the OGMA datase system, although this was also changed to update from CERN using Oracle Golden Gate.)
- Switch LFC/3D to new Database Infrastructure.
- Castor:
- Update Castor headnodes to SL6.
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
Entries in GOC DB starting between the 12th and 19th November 2014. |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
Atlas, CMS & LHCb Castor instances (srm-atlas, srm-cms, srm-lhcb) | SCHEDULED | WARNING | 19/11/2014 11:00 | 19/11/2014 12:00 | 1 hour | Warning on Castor Atlas, CMS and LHCb instance during application of OS errata updates. |
srm-alice, srm-biomed, srm-dteam, srm-hone, srm-ilc, srm-mice, srm-minos, srm-na62, srm-pheno, srm-snoplus, srm-superb, srm-t2k. | SCHEDULED | WARNING | 12/11/2014 11:00 | 12/11/2014 12:00 | 1 hour | Warning on Castor GEN instance during application of OS errata updates. |
Open GGUS Tickets (Snapshot during morning of meeting) |
GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
---|---|---|---|---|---|---|---|
109712 | Green | Urgent | On Hold | 2014-10-29 | 2014-11-10 | CMS | Glexec exited with status 203; ... |
109694 | Green | Urgent | On hold | 2014-11-03 | 2014-11-06 | SNO+ | gfal-copy failing for files at RAL |
108944 | Amber | Urgent | In Progress | 2014-10-01 | 2014-11-13 | CMS | AAA access test failing at T1_UK_RAL |
107935 | Red | Less Urgent | On Hold | 2014-08-27 | 2014-11-03 | Atlas | BDII vs SRM inconsistent storage capacity numbers |
106324 | Red | Urgent | On Hold | 2014-06-18 | 2014-10-13 | CMS | pilots losing network connections at T1_UK_RAL |
Availability Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas HC | CMS HC | Comment |
---|---|---|---|---|---|---|---|---|
12/11/14 | 100 | 100 | 99.2 | 100 | 100 | 100 | 99 | Single SRM (Put) test failure. |
13/11/14 | 100 | 100 | 100 | 100 | 100 | 100 | 99 | |
14/11/14 | 100 | 100 | 100 | 100 | 100 | 98 | N/A | |
15/11/14 | 100 | 100 | 100 | 100 | 100 | 99 | 100 | |
16/11/14 | 100 | 100 | 98.4 | 100 | 100 | 100 | 99 | Two SRM Put test failures (timeouts). |
17/11/14 | 100 | 100 | 100 | 100 | 100 | 99 | 99 | |
18/11/14 | 100 | 91.1 | 88.3 | 95.9 | 100 | 94 | 96 | Networking problems. Mainly affecting CE tests. |