Difference between revisions of "Tier1 Operations Report 2015-01-21"

From GridPP Wiki
Jump to: navigation, search
(→‎)
 
(19 intermediate revisions by one user not shown)
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 14th to 21st January 2015.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 14th to 21st January 2015.
 
|}
 
|}
−
* On the morning of Thursday 8th January there was a short (15 minute) network break in access to the Tier1 network. Diagnostic tests were being run on the faulty router – however after that the system restarted and took over as the master router of the pair (which was not anticipated). Some half hour later it failed again.
+
* The high load on the LHCb SRMs reported last week continued into this reporting period with intermittent SAM test failures. The problem has been mitigated: The particular LHCb workflow that loaded the SRMs heavily has eased and a third system has been added into the LHCb SRM alias.
−
* We have seen very high load on the LHCb SRMs through most of the week. Intermittent timeouts were seen on the tests. The number of LHCb batch jobs has been restricted to try and reduce the problem. In addition, during the day yesterday (Tuesday 13th) it was found that the SRMs were not responding and needed a restart.
+
* Electrical circuit testing in the machine room has continued into this week. Some further problems have been encountered and we have tripped off some (a few tens of) worker nodes. However, no critical services have been affected.
−
* Electrical circuit testing is underway. Two older batches of worker nodes were drained out beforehand. Some problems have been found. This has mainly affected some worker nodes to which access was lost for an hour or so. Also affected was access to ten disk servers which are currently in 'readonly' mode in AtlasDataDisk. A similar problem occurred during this morning (14th Jan). These were caused by a PDU becoming overloaded and tripping off some of it ports.
+
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 22: Line 21:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
|}
 
|}
−
* GDSS757 (CMSDisk D1T0) failed on Saturday 3rd January. It was returned to service during the afternoon of Wednesday 7th January following testing.
+
* None
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 34: Line 33:
 
|}
 
|}
 
* We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
 
* We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
−
* There is a problem with xroot access to the Castor GEN instance (not affecting ALICE).
 
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 56: Line 54:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
|}
 
|}
−
* In preparation for the electrical safety checks being carried out this week some "transfer switches" (enabling dual powering of some network switches and older systems) were installed during the second part of last week.
+
* The bulk of the safety checking of the electrical power circuits in the machine room has been completed. The whole task will finish tomorrow, Thursday 22nd January.
 +
* The problem with xroot access to the Castor GEN instance was fixed during the afternoon of Wednesday 14th Jan. (This problem did not affect ALICE).
 +
* A third SRM system was added to the LHCb alias (sm-lhcb.gridpp.rl.ac.uk) on Thursday 15th Jan.
 +
* The Castor Nameserver headnodes were updated to SL6 on Monday 19th Jan. This operation was transparent to Castor users.
 +
* The migration of all data off T10000A & B media is almost complete.
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 67: Line 69:
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
|}
 
|}
−
 
 
{| border=1 align=center
 
{| border=1 align=center
 
|- bgcolor="#7c8aaf"
 
|- bgcolor="#7c8aaf"
Line 78: Line 79:
 
! Reason
 
! Reason
 
|-
 
|-
−
| Whole site
+
| Whole site  
 
| SCHEDULED
 
| SCHEDULED
 
| WARNING
 
| WARNING
Line 84: Line 85:
 
| 22/01/2015 18:00
 
| 22/01/2015 18:00
 
| 2 days, 9 hours and 30 minutes
 
| 2 days, 9 hours and 30 minutes
−
| Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
 
−
|-
 
−
| All Castor (All SRM endpoints).
 
−
| SCHEDULED
 
−
| WARNING
 
−
| 19/01/2015 09:15
 
−
| 19/01/2015 16:00
 
−
| 6 hours and 45 minutes
 
−
| Warning during OS upgrade (SL6) on Castor Nameservers.
 
−
|-
 
−
| Whole site
 
−
| SCHEDULED
 
−
| WARNING
 
−
| 13/01/2015 18:30
 
−
| 15/01/2015 18:00
 
−
| 1 day, 23 hours and 30 minutes
 
 
| Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
 
| Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
 
|}
 
|}
Line 115: Line 100:
 
|}
 
|}
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
−
* Investigate problems on the primary Tier1 router.
+
* Investigate problems on the primary Tier1 router. We plan to run the primary (failing) router for a few hours tomorrow (Thursday 22nd Jan) in order to trap the problem. If there is a failure it will take a few minutes for a manual switch back to the secondary to be effected.
 +
* The circuit testing of the remaining (i.e. non-UPS) circuits in the machine room in ongoing and will be completed tomorrow (Thursday 22nd Jan).
 +
* A patch to FTS3 (to version 3.2.31) will be rolled out in the next days.
 +
* OS updates on disk servers (expected in week 26-30 Jan).
 +
* Move of connection for CERN Backup link.
 
'''Listing by category:'''
 
'''Listing by category:'''
 
* Databases:
 
* Databases:
 +
** Application of Oracle PSU patches to database systems.
 
** A new database (Oracle RAC) has been set-up to host the Atlas 3D database. This is updated from CERN via Oracle GoldenGate. This system is yet to be brought into use. (Currently Atlas 3D/Frontier still uses the OGMA datase system, although this was also changed to update from CERN using Oracle Golden Gate.)
 
** A new database (Oracle RAC) has been set-up to host the Atlas 3D database. This is updated from CERN via Oracle GoldenGate. This system is yet to be brought into use. (Currently Atlas 3D/Frontier still uses the OGMA datase system, although this was also changed to update from CERN using Oracle Golden Gate.)
 
** Switch LFC/3D to new Database Infrastructure.
 
** Switch LFC/3D to new Database Infrastructure.
 
** Update to Oracle 11.2.0.4
 
** Update to Oracle 11.2.0.4
 
* Castor:
 
* Castor:
−
** Update Castor headnodes to SL6 (Nameservers remain to be done).
 
 
** Update SRMs to new version (includes updating to SL6).
 
** Update SRMs to new version (includes updating to SL6).
 
** Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
 
** Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
Line 132: Line 121:
 
** Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
 
** Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
 
* Fabric
 
* Fabric
−
** Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes finished; migration of GEN from 'A' to 'D' tapes to follow.)
+
** Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes finished; migration of GEN from 'A' to 'D' almost complete.)
 
** Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
 
** Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
−
** There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during January.
 
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ************************************************************************** ----->
 
<!-- ************************************************************************** ----->
Line 154: Line 142:
 
! Duration
 
! Duration
 
! Reason
 
! Reason
 +
|-
 +
| Whole site
 +
| SCHEDULED
 +
| WARNING
 +
| 20/01/2015 08:30
 +
| 22/01/2015 18:00
 +
| 2 days, 9 hours and 30 minutes
 +
| Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
 +
|-
 +
| All Castor (all SRM endpoints)
 +
| SCHEDULED
 +
| WARNING
 +
| 19/01/2015 09:15
 +
| 19/01/2015 16:00
 +
| 6 hours and 45 minutes
 +
| Warning during OS upgrade (SL6) on Castor Nameservers.
 
|-
 
|-
 
| Whole site
 
| Whole site
Line 162: Line 166:
 
| 1 day, 23 hours and 30 minutes
 
| 1 day, 23 hours and 30 minutes
 
| Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
 
| Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
−
|-
 
−
| Whole site
 
−
| SCHEDULED
 
−
| WARNING
 
−
| 07/01/2015 10:00
 
−
| 07/01/2015 12:00
 
−
| 2 hours
 
−
| Warning on site during quarterly UPS/Generator load test.
 
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 191: Line 187:
 
| In Progress
 
| In Progress
 
| 2015-01-12
 
| 2015-01-12
−
| 2015-01-14
+
| 2015-01-15
 
| Atlas
 
| Atlas
 
| large transfer errors from RAL-LCG2 to BNL-OSG2
 
| large transfer errors from RAL-LCG2 to BNL-OSG2
 
|-
 
|-
 
| 110605
 
| 110605
−
| Green
+
| Yellow
 
| Less Urgent
 
| Less Urgent
 
| In Progress
 
| In Progress
 
| 2014-12-08
 
| 2014-12-08
−
| 2015-01-09
+
| 2015-01-20
 
| ops
 
| ops
 
| [Rod Dashboard] Issues detected at RAL-LCG2 (srm-cms-disk.gridpp.rl.ac.uk)
 
| [Rod Dashboard] Issues detected at RAL-LCG2 (srm-cms-disk.gridpp.rl.ac.uk)
−
|-
 
−
| 110382
 
−
| Yellow
 
−
| Less Urgent
 
−
| In Progress
 
−
| 2014-11-26
 
−
| 2015-01-07
 
−
| N/A
 
−
| RAL-LCG2: please reinstall your perfsonar hosts(s)
 
 
|-
 
|-
 
| 109712
 
| 109712
Line 227: Line 214:
 
| On hold
 
| On hold
 
| 2014-11-03
 
| 2014-11-03
−
| 2014-12-18
+
| 2015-01-20
 
| SNO+
 
| SNO+
 
| gfal-copy failing for files at RAL
 
| gfal-copy failing for files at RAL
Line 236: Line 223:
 
| In Progress
 
| In Progress
 
| 2014-10-01
 
| 2014-10-01
−
| 2015-01-07
+
| 2015-01-19
 
| CMS
 
| CMS
 
| AAA access test failing at T1_UK_RAL
 
| AAA access test failing at T1_UK_RAL
Line 245: Line 232:
 
| On Hold
 
| On Hold
 
| 2014-08-27
 
| 2014-08-27
−
| 2014-12-15
+
| 2015-01-20
 
| Atlas
 
| Atlas
 
| BDII vs SRM inconsistent storage capacity numbers
 
| BDII vs SRM inconsistent storage capacity numbers

Latest revision as of 17:14, 21 January 2015

RAL Tier1 Operations Report for 21st January 2015

Review of Issues during the week 14th to 21st January 2015.
  • The high load on the LHCb SRMs reported last week continued into this reporting period with intermittent SAM test failures. The problem has been mitigated: The particular LHCb workflow that loaded the SRMs heavily has eased and a third system has been added into the LHCb SRM alias.
  • Electrical circuit testing in the machine room has continued into this week. Some further problems have been encountered and we have tripped off some (a few tens of) worker nodes. However, no critical services have been affected.
Resolved Disk Server Issues
  • None
Current operational status and issues
  • We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
Ongoing Disk Server Issues
  • None
Notable Changes made this last week.
  • The bulk of the safety checking of the electrical power circuits in the machine room has been completed. The whole task will finish tomorrow, Thursday 22nd January.
  • The problem with xroot access to the Castor GEN instance was fixed during the afternoon of Wednesday 14th Jan. (This problem did not affect ALICE).
  • A third SRM system was added to the LHCb alias (sm-lhcb.gridpp.rl.ac.uk) on Thursday 15th Jan.
  • The Castor Nameserver headnodes were updated to SL6 on Monday 19th Jan. This operation was transparent to Castor users.
  • The migration of all data off T10000A & B media is almost complete.
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
Whole site SCHEDULED WARNING 20/01/2015 08:30 22/01/2015 18:00 2 days, 9 hours and 30 minutes Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • Investigate problems on the primary Tier1 router. We plan to run the primary (failing) router for a few hours tomorrow (Thursday 22nd Jan) in order to trap the problem. If there is a failure it will take a few minutes for a manual switch back to the secondary to be effected.
  • The circuit testing of the remaining (i.e. non-UPS) circuits in the machine room in ongoing and will be completed tomorrow (Thursday 22nd Jan).
  • A patch to FTS3 (to version 3.2.31) will be rolled out in the next days.
  • OS updates on disk servers (expected in week 26-30 Jan).
  • Move of connection for CERN Backup link.

Listing by category:

  • Databases:
    • Application of Oracle PSU patches to database systems.
    • A new database (Oracle RAC) has been set-up to host the Atlas 3D database. This is updated from CERN via Oracle GoldenGate. This system is yet to be brought into use. (Currently Atlas 3D/Frontier still uses the OGMA datase system, although this was also changed to update from CERN using Oracle Golden Gate.)
    • Switch LFC/3D to new Database Infrastructure.
    • Update to Oracle 11.2.0.4
  • Castor:
    • Update SRMs to new version (includes updating to SL6).
    • Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
    • Update Castor to 2.1-14-latest, this depends on SL6 being deployed.
  • Networking:
    • Resolve problems with primary Tier1 Router
    • Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
    • Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
  • Fabric
    • Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes finished; migration of GEN from 'A' to 'D' almost complete.)
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting between the 14th and 21st January 2015.
Service Scheduled? Outage/At Risk Start End Duration Reason
Whole site SCHEDULED WARNING 20/01/2015 08:30 22/01/2015 18:00 2 days, 9 hours and 30 minutes Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
All Castor (all SRM endpoints) SCHEDULED WARNING 19/01/2015 09:15 19/01/2015 16:00 6 hours and 45 minutes Warning during OS upgrade (SL6) on Castor Nameservers.
Whole site SCHEDULED WARNING 13/01/2015 18:30 15/01/2015 18:00 1 day, 23 hours and 30 minutes Warning during safety checks on power circuits in machine room. Testing carried out during working hours on each day.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
111120 Green Less Urgent In Progress 2015-01-12 2015-01-15 Atlas large transfer errors from RAL-LCG2 to BNL-OSG2
110605 Yellow Less Urgent In Progress 2014-12-08 2015-01-20 ops [Rod Dashboard] Issues detected at RAL-LCG2 (srm-cms-disk.gridpp.rl.ac.uk)
109712 Red Urgent In Progress 2014-10-29 2015-01-09 CMS Glexec exited with status 203; ...
109694 Red Urgent On hold 2014-11-03 2015-01-20 SNO+ gfal-copy failing for files at RAL
108944 Red Urgent In Progress 2014-10-01 2015-01-19 CMS AAA access test failing at T1_UK_RAL
107935 Red Less Urgent On Hold 2014-08-27 2015-01-20 Atlas BDII vs SRM inconsistent storage capacity numbers
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
14/01/15 100 100 100 100 55.6 100 100 Load (stripping campaign) affecting LHCb SRMs. User timeouts on tests.
15/01/15 100 100 100 100 97.3 100 78 Load. Single SRM test failure ([SRM_FILE_BUSY]).
16/01/15 100 100 100 100 90.0 90 100 Config problem as additional SRM set-up.
17/01/15 100 100 100 100 100 100 100
18/01/15 100 100 100 100 100 100 100
19/01/15 100 100 100 100 100 100 99
20/01/15 100 100 100 100 100 100 100
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2015-01-21&oldid=7268"