Difference between revisions of "Tier1 Operations Report 2018-10-02"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(19 intermediate revisions by one user not shown)
Line 29: Line 29:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
−
* Tier-1 encountered another IPv6 issue 17/9/18 - 23:00), however Digital Solutions were able to resolve this issue within 2 hours
+
*  
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 94: Line 94:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Limits on concurrent batch system jobs.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Limits on concurrent batch system jobs.
 
|}
 
|}
−
* GROUP_CMS_LIMIT = 4000
+
*
−
* GROUP_ATLAS_LIMIT = 8000
+
 
<!-- ******************End Limits On Batch System Jobs***************** ----->
 
<!-- ******************End Limits On Batch System Jobs***************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 129: Line 128:
 
! Reason
 
! Reason
 
|-
 
|-
−
| LFC
 
−
| 26064
 
−
| No
 
−
| Outage
 
−
| 20-Sep-18 17:00:00
 
−
| 24-Sep-18 11:00:02
 
 
| -
 
| -
−
| investigating problems with the LFC - possibly in the back end database.
+
| -
 +
| -
 +
| -
 +
| -
 +
| -
 +
| -
 +
| -
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 188: Line 187:
 
   
 
   
 
* Castor:  
 
* Castor:  
 +
** Disk only storage will end in Castor. A new single tape-only Castor instance (called WLCGTape) is being tested. This is using generic Catsor headnodes on SL7 configured by Quattor/Aquilon with a slightly newer Castor version.
 
** Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
 
** Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
−
** Move to generic Castor headnodes.
 
 
* Internal
 
* Internal
 
** DNS servers will be rolled out within the Tier1 network.
 
** DNS servers will be rolled out within the Tier1 network.
 +
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ************************************************************************** ----->
 
<!-- ************************************************************************** ----->
Line 214: Line 214:
 
! Subject
 
! Subject
 
! Scope
 
! Scope
−
|-
 
−
| style="background-color: green;" | 137398
 
−
| cms
 
−
| in progress
 
−
| urgent
 
−
| 26/09/2018
 
−
| 26/09/2018
 
−
| CMS_Data Transfers
 
−
| Transfers failing from SPRACE to RAL - No data available
 
−
| WLCG
 
 
|-
 
|-
 
| style="background-color: green;" | 137391
 
| style="background-color: green;" | 137391
Line 230: Line 220:
 
| urgent
 
| urgent
 
| 25/09/2018
 
| 25/09/2018
−
| 26/09/2018
+
| 03/10/2018
 
| Network problem
 
| Network problem
 
| UK RAL-LCG2 transfer errors with Communication error on send
 
| UK RAL-LCG2 transfer errors with Communication error on send
 
| WLCG
 
| WLCG
−
|-
 
−
| style="background-color: green;" | 137254
 
−
| ops
 
−
| in progress
 
−
| less urgent
 
−
| 18/09/2018
 
−
| 25/09/2018
 
−
| Operations
 
−
| [Rod Dashboard] Issue detected : ch.cern.LFC-Write-ops@lfc.gridpp.rl.ac.uk
 
−
| EGI
 
 
|-
 
|-
 
| style="background-color: green;" | 137195
 
| style="background-color: green;" | 137195
Line 250: Line 230:
 
| less urgent
 
| less urgent
 
| 14/09/2018
 
| 14/09/2018
−
| 24/09/2018
+
| 26/09/2018
 
| Operations
 
| Operations
 
| [Rod Dashboard] Issues detected at RAL-LCG2
 
| [Rod Dashboard] Issues detected at RAL-LCG2
Line 263: Line 243:
 
| Data Management - generic
 
| Data Management - generic
 
| LFC entry has file size 0, preventsw registering of additional replicas
 
| LFC entry has file size 0, preventsw registering of additional replicas
−
| EGI
 
−
|-
 
−
| style="background-color: yellow;" | 136840
 
−
| snoplus.snolab.ca
 
−
| in progress
 
−
| very urgent
 
−
| 23/08/2018
 
−
| 25/09/2018
 
−
| Other
 
−
| Cannot upload files to LFN from Storage node
 
 
| EGI
 
| EGI
 
|-
 
|-
Line 280: Line 250:
 
| very urgent
 
| very urgent
 
| 14/08/2018
 
| 14/08/2018
−
| 26/09/2018
+
| 03/10/2018
 
| File Transfer
 
| File Transfer
 
| background of transfer errors
 
| background of transfer errors
Line 287: Line 257:
 
| style="background-color: red;" | 136199
 
| style="background-color: red;" | 136199
 
| lhcb
 
| lhcb
−
| in progress
+
| on hold
 
| very urgent
 
| very urgent
 
| 18/07/2018
 
| 18/07/2018
−
| 07/08/2018
+
| 01/10/2018
 
| File Transfer
 
| File Transfer
 
| Lots of submitted transfers on RAL FTS
 
| Lots of submitted transfers on RAL FTS
−
| WLCG
 
−
|-
 
−
| style="background-color: red;" | 136028
 
−
| cms
 
−
| in progress
 
−
| top priority
 
−
| 10/07/2018
 
−
| 21/09/2018
 
−
| CMS_AAA WAN Access
 
−
| Issues reading files at T1_UK_RAL_Disk
 
 
| WLCG
 
| WLCG
 
|-
 
|-
Line 339: Line 299:
 
! Scope
 
! Scope
 
|-
 
|-
−
| 137294
+
| 137434
−
| snoplus.snolab.ca
+
| cms
−
| solved
+
−
| very urgent
+
−
| 20/09/2018
+
−
| 21/09/2018
+
−
| File Access
+
−
| No copy on tape error on Castor at RAL
+
−
| EGI
+
−
|-
+
−
| 137267
+
−
| atlas
+
 
| solved
 
| solved
 
| urgent
 
| urgent
−
| 19/09/2018
+
| 26/09/2018
−
| 19/09/2018
+
| 27/09/2018
−
| Other
+
| CMS_AAA WAN Access
−
| RAL-LCG2 transfer fail with "Communication error on send"
+
| xrootd host not reachable at T1_UK_RAL
 
| WLCG
 
| WLCG
 +
|-
 +
| 137169
 +
| none
 +
| closed
 +
| less urgent
 +
| 13/09/2018
 +
| 28/09/2018
 +
| Monitoring
 +
| enable the nagios notifications - CVMFS
 +
| EGI
 
|-
 
|-
 
| 137151
 
| 137151
Line 369: Line 329:
 
| EGI
 
| EGI
 
|-
 
|-
−
| 137136
+
| 137128
−
| lhcb
+
| snoplus.snolab.ca
−
| verified
+
| closed
−
| very urgent
+
| urgent
−
| 12/09/2018
+
| 11/09/2018
−
| 18/09/2018
+
| 28/09/2018
 
| File Access
 
| File Access
−
| Data transfers problem at RAL-LCG2
+
| RAL remote host name change?
−
| WLCG
+
| EGI
 
|-
 
|-
−
| 137047
+
| 136884
 
| t2k.org
 
| t2k.org
 +
| verified
 +
| top priority
 +
| 27/08/2018
 +
| 27/09/2018
 +
| Data Management - generic
 +
| lcg-cr not working for t2k vo
 +
| EGI
 +
|-
 +
| 136757
 +
| mice
 
| solved
 
| solved
 
| less urgent
 
| less urgent
−
| 06/09/2018
+
| 17/08/2018
−
| 20/09/2018
+
| 25/09/2018
−
| File Access
+
| Other
−
| Connection to RAL tape storage fails
+
| Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ?
 
| EGI
 
| EGI
 
|-
 
|-
−
| 136967
+
| 136366
−
| cms
+
| mice
−
| closed
+
| verified
−
| urgent
+
−
| 31/08/2018
+
−
| 18/09/2018
+
−
| CMS_Data Transfers
+
−
| Phedex transfers failing from RAL to FNAL
+
−
| WLCG
+
−
|-
+
−
| 136942
+
−
| t2k.org
+
−
| closed
+
 
| less urgent
 
| less urgent
−
| 29/08/2018
+
| 25/07/2018
−
| 20/09/2018
+
| 28/09/2018
−
| File Transfer
+
| Local Batch System
−
| Copying ONLINE_AND_NEARLINE files from RAL tape storage times out
+
| Remove MICE Queue from RAL T1 Batch
 
| EGI
 
| EGI
 
|-
 
|-
−
| 136884
+
| 136028
−
| t2k.org
+
| cms
 
| solved
 
| solved
 
| top priority
 
| top priority
−
| 27/08/2018
+
| 10/07/2018
−
| 24/09/2018
+
| 28/09/2018
−
| Data Management - generic
+
| CMS_AAA WAN Access
−
| lcg-cr not working for t2k vo
+
| Issues reading files at T1_UK_RAL_Disk
−
| EGI
+
| WLCG
−
|}<!-- **********************End Availability Report************************** ----->
+
|}
 +
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- **********************End GGUS Tickets************************** ----->
 
<!-- **********************End GGUS Tickets************************** ----->
Line 442: Line 403:
 
! Comments
 
! Comments
 
|-
 
|-
−
| 2018-09-17
+
| 2018-09-24
 +
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: orange;" | 96
 
 
| 100
 
| 100
 
| 100
 
| 100
Line 451: Line 412:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-18
+
| 2018-09-25
 
| 100
 
| 100
 
| 100
 
| 100
Line 460: Line 421:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-19
+
| 2018-09-26
 
| 100
 
| 100
 
| 100
 
| 100
Line 469: Line 430:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-20
+
| 2018-09-27
 
| 100
 
| 100
 
| 100
 
| 100
Line 478: Line 439:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-21
+
| 2018-09-28
 +
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| 97
 
 
| 100
 
| 100
 
| 100
 
| 100
Line 487: Line 448:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-22
+
| 2018-09-29
 +
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: orange;" | 96
 
 
| 100
 
| 100
 
| 100
 
| 100
Line 496: Line 457:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-23
+
| 2018-09-30
 
| 100
 
| 100
 
| 100
 
| 100
Line 505: Line 466:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-24
+
| 2018-10-01
−
| 100
+
| style="background-color: red;" | 53
 
| 100
 
| 100
 
| 100
 
| 100
Line 531: Line 492:
 
! Day !! Atlas HC !! CMS HC !! Comment
 
! Day !! Atlas HC !! CMS HC !! Comment
 
|-
 
|-
−
| 2018-08-10 || 100 || 98 ||   
+
| 2018-08-24 || 100 || 99 ||   
 
|-
 
|-
−
| 2018-08-11 || 100 || 98 ||  
+
| 2018-08-25 || 100 || 99 ||  
 
|-
 
|-
−
| 2018-08-12 || 98  || 99 ||  
+
| 2018-08-26 || 100 || 100 ||  
 
|-
 
|-
−
| 2018-08-13 || style="background-color: orange;" | 95  || 98 ||  
+
| 2018-08-27 || 100 || 100 ||  
 
|-
 
|-
−
| 2018-08-14 || 100 || 99||  
+
| 2018-08-28 || 100 || 100 ||  
 
|-
 
|-
−
| 2018-08-15 || style="background-color: red;" | 79 || 99 ||  
+
| 2018-08-29 || 100 || 99 ||  
 
|-
 
|-
−
| 2018-08-16 || 100 || 99 ||  
+
| 2018-08-30 || 100 || 99 ||  
 
|-
 
|-
−
| 2018-08-17 || - || - ||  
+
| 2018-08-01 || 100 || 99 ||  
 
|-
 
|-
 
|}  
 
|}  

Latest revision as of 14:01, 3 October 2018

RAL Tier1 Operations Report for 2 October 2018

Review of Issues during the week 24th September 2018 to the 2nd October 2018.
  • The on-going LFC issue appeared to have had been resolved early in the week (w/e 30th), as such the service was returned to Production.
  • First breakage of database = 19th August Fixed 21st September Second breakage of database = Tuesday 25th September 1pm.
  • Currently we have reasonable confidence that LFC issues are now understood and there is a fix:
    • In conjunction with Fabrizio Furano (former LFC code developer at CERN), we have identified a flaw in the Oracle to MySQL migration process.
    • An Oracle SEQUENCE had not been migrated into a MySQL table therefore it was only possible to register 6820 new records in the new MYSQL backend each time before a crash.
    • This is explanation also accounts for the fact that rolling back the database to an earlier backup also worked
    • Fabrizio provided a simple solution to fix the problem.
  • We are now planning to implement the fix and restore the LFC to a full production service by close of business 02/10/2018.
Current operational status and issues
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
- - - - -
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss747 Atlas atlasStripInput d1t0 Currently in intervention.
gdss753 Atlas atlasStripInput d1t0 Currently in intervention.
Limits on concurrent batch system jobs.
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Disk only storage will end in Castor. A new single tape-only Castor instance (called WLCGTape) is being tested. This is using generic Catsor headnodes on SL7 configured by Quattor/Aquilon with a slightly newer Castor version.
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
  • Internal
    • DNS servers will be rolled out within the Tier1 network.


Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137391 atlas in progress urgent 25/09/2018 03/10/2018 Network problem UK RAL-LCG2 transfer errors with Communication error on send WLCG
137195 ops in progress less urgent 14/09/2018 26/09/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI
137153 t2k.org in progress urgent 12/09/2018 25/09/2018 Data Management - generic LFC entry has file size 0, preventsw registering of additional replicas EGI
136701 lhcb waiting for reply very urgent 14/08/2018 03/10/2018 File Transfer background of transfer errors WLCG
136199 lhcb on hold very urgent 18/07/2018 01/10/2018 File Transfer Lots of submitted transfers on RAL FTS WLCG
124876 ops in progress less urgent 07/11/2016 23/07/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137434 cms solved urgent 26/09/2018 27/09/2018 CMS_AAA WAN Access xrootd host not reachable at T1_UK_RAL WLCG
137169 none closed less urgent 13/09/2018 28/09/2018 Monitoring enable the nagios notifications - CVMFS EGI
137151 t2k.org solved urgent 12/09/2018 24/09/2018 File Transfer Cannot upload files to grid an register replica EGI
137128 snoplus.snolab.ca closed urgent 11/09/2018 28/09/2018 File Access RAL remote host name change? EGI
136884 t2k.org verified top priority 27/08/2018 27/09/2018 Data Management - generic lcg-cr not working for t2k vo EGI
136757 mice solved less urgent 17/08/2018 25/09/2018 Other Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ? EGI
136366 mice verified less urgent 25/07/2018 28/09/2018 Local Batch System Remove MICE Queue from RAL T1 Batch EGI
136028 cms solved top priority 10/07/2018 28/09/2018 CMS_AAA WAN Access Issues reading files at T1_UK_RAL_Disk WLCG

Availability Report

Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-09-24 100 100 100 100 100 100
2018-09-25 100 100 100 100 100 100
2018-09-26 100 100 100 100 100 100
2018-09-27 100 100 100 100 100 100
2018-09-28 100 100 100 100 100 100
2018-09-29 100 100 100 100 100 100
2018-09-30 100 100 100 100 100 100
2018-10-01 53 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018-08-24 100 99
2018-08-25 100 99
2018-08-26 100 100
2018-08-27 100 100
2018-08-28 100 100
2018-08-29 100 99
2018-08-30 100 99
2018-08-01 100 99

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-10-02&oldid=18642"