Difference between revisions of "Tier1 Operations Report 2017-10-11"
From GridPP Wiki
(→) |
(→) |
||
Line 224: | Line 224: | ||
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas Echo !! Comment | ! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas Echo !! Comment | ||
|- | |- | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
| 04/10/17 || 100 || 100 || 100 || 100 || 100 || 100 || | | 04/10/17 || 100 || 100 || 100 || 100 || 100 || 100 || | ||
|- | |- | ||
− | | 05/10/17 || 100 || 100 || 100 || | + | | 05/10/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 63 || style="background-color: lightgrey;" | 96 || 100 || CMS failed SRM tests with Disk full; LHCb: Single SRM test failure. |
|- | |- | ||
− | | 06/10/17 || 100 || 100 || 100 || | + | | 06/10/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 43 || 100 || 100 || CMS Disk full. |
|- | |- | ||
| 07/10/17 || 100 || 100 || 100 || 100 || 100 || 100 || | | 07/10/17 || 100 || 100 || 100 || 100 || 100 || 100 || | ||
Line 245: | Line 236: | ||
| 09/10/17 || 100 || 100 || 100 || 100 || 100 || 100 || | | 09/10/17 || 100 || 100 || 100 || 100 || 100 || 100 || | ||
|- | |- | ||
− | | 10/10/17 || 100 || 100 || 100 || | + | | 10/10/17 || 100 || 100 || 100 || 99 || 100 || 100 || Problem with monitoring system. |
− | + | ||
|} | |} | ||
<!-- **********************End Availability Report************************** -----> | <!-- **********************End Availability Report************************** -----> |
Revision as of 07:13, 11 October 2017
RAL Tier1 Operations Report for 11th October 2017
Review of Issues during the week 4th to 11th October 2017. |
- A quiet week operationally. Availabilities (from SAM tests) looking good now that the problem with the LHCb tests is resolved.
Resolved Disk Server Issues |
- GDSS733 (LHCbDst - D1T0) crashed during the early evening of Saturday (7th Oct). It was put back in service around midday the following day read-only. The RAID array verifivation found one faulty disk that was replaced. However, another disk drive failued during the rebuild of the array after the first one was changed. That second disk drive has also been swapped out.
Current operational status and issues |
- There is a problem on the site firewall which is causing problems for some specific data flows. Discussions have been taking place with the vendor. This is having an effect on our data that flows through the firewall (such as to/from worker nodes).
Ongoing Disk Server Issues |
- GDSS788 (AtlasDataDisk - D1T0) crashed during the early evening of this morning (Wednesday 11th Oct). Investigations are ongoing.
Limits on concurrent batch system jobs. |
- CMS Multicore 550
Notable Changes made since the last meeting. |
- Re-distribution of data in Echo onto the 2015 capacity hardware is ongoing.
- Services still using the older Quattor/SCDB system for configuration/deployment are being migrated to use Quattor/Aquilon.
Declared in the GOC DB |
- None
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
Pending - but not yet formally announced:
- Upgrade the FTS3 service to a version that will no longer support the SOAP interface. The SOAP interface was disabled on Monday (17th July).
- Re-distribute the data in Echo onto the 2015 capacity hardware. (Ongoing)
Listing by category:
- Castor:
- Move to generic Castor headnodes.
- Echo:
- Re-distribute the data in Echo onto the remaining 2015 capacity hardware.
- Networking
- Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, all squids and the CVMFS Stratum-1 servers).
- Services
- The production FTS will be updated now the requirement to support the deprecated SOAP interface has been removed.
Entries in GOC DB starting since the last report. |
- None
Open GGUS Tickets (Snapshot during morning of meeting) |
GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
---|---|---|---|---|---|---|---|
130883 | Green | Urgent | In Progress | 2017-10-02 | 2017-10-03 | CMS | T1_UK_RAL_Disk to T2_US_Florida transfers are failing due to missing file(s) |
130782 | Green | Top Priority | In Progress | 2017-09-27 | 2017-09-28 | LHCb | Please deploy new version of HEP_OSlibs |
130467 | Green | Urgent | In Progress | 2017-09-10 | 2017-09-25 | CMS | T1_UK_RAL has SAM3 CE and SRM critical > 2hours |
130207 | Yellow | Urgent | In Progress | 2017-08-24 | 2017-09-06 | MICE | Timeouts when copying MICE reco data to CASTOR |
130193 | Yellow | Urgent | Waiting for Reply | 2017-08-23 | 2017-09-29 | CMS | Staging from RAL tape systems |
128991 | Green | Less Urgent | In Progress | 2017-06-16 | 2017-09-18 | Solid | solidexperiment.org CASTOR tape support |
127597 | Red | Urgent | On Hold | 2017-04-07 | 2017-10-03 | CMS | Check networking and xrootd RAL-CERN performance |
124876 | Red | Less Urgent | On Hold | 2016-11-07 | 2017-01-01 | OPS | [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk |
117683 | Red | Less Urgent | On Hold | 2015-11-18 | 2017-07-06 | CASTOR at RAL not publishing GLUE 2. |
Availability Report |
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas Echo | Comment |
---|---|---|---|---|---|---|---|
04/10/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
05/10/17 | 100 | 100 | 100 | 63 | 96 | 100 | CMS failed SRM tests with Disk full; LHCb: Single SRM test failure. |
06/10/17 | 100 | 100 | 100 | 43 | 100 | 100 | CMS Disk full. |
07/10/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
08/10/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
09/10/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
10/10/17 | 100 | 100 | 100 | 99 | 100 | 100 | Problem with monitoring system. |
Hammercloud Test Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC Echo = Atlas Echo (Template 841);CMS HC = CMS HammerCloud
Day | Atlas HC | Atlas HC Echo | CMS HC | Comment |
---|---|---|---|---|
27/09/17 | 97 | 98 | 98 | |
28/09/17 | 93 | 97 | 100 | |
29/09/17 | 66 | 70 | 100 | Suspected Atlas problem as poor results at other Tier1 sites too. |
30/09/17 | 71 | 77 | 100 | Suspected Atlas problem as poor results at other Tier1 sites too. |
01/10/17 | 100 | 100 | 100 | |
02/10/17 | 99 | 97 | 100 | |
03/10/17 | 96 | 94 | 100 | |
04/10/17 | 100 | 100 | 100 | |
05/10/17 | 100 | 100 | 100 | |
06/10/17 | 100 | 100 | 100 | |
07/10/17 | 100 | 100 | 100 | |
08/10/17 | 100 | 100 | 100 | |
09/10/17 | 100 | 100 | 100 | |
10/10/17 | 100 | 100 | 100 |
Notes from Meeting. |
- MICE are taking data and storing it on Castor OK.
- Chris has cleaned up all bar around 50TB of CMS dark data oN Castor - an dthe remainder is understood.
- Chris has been commissioning links in CMS' PhEDEx to/from Echo. This will be completed by early next week.