Difference between revisions of "Tier1 Operations Report 2018-05-21"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(14 intermediate revisions by one user not shown)
Line 11: Line 11:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 14th May to the 21st May 2018.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 14th May to the 21st May 2018.
 
|}
 
|}
−
* The RAL networking team have finally resolved their failing, fail-over firewall issues. Primary and secondary router IPv4 and IPv6 fail-over tests are being conducted this week (09:00 15/05/18 and 16/05/18), but are not anticipated to impact production services.
+
* No incidents(major or minor), have been flagged during this reporting period.
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 33: Line 33:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues
 
|}
 
|}
−
* gdss700 (lhcbDst- D1T0) - Back in production RW
+
* gdss732 (lhcbDst- D1T0) - Back in production after completion of rebuilding of the replacement drive.
−
* gdss711 (lhcbDst- D1T0) - Back in production RO
+
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
  
Line 44: Line 43:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues
 
|}
 
|}
−
* gdss732 (lhcbDst- D1T0) - Currently in intervention
+
* None
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ************************************************************* ----->
 
<!-- ************************************************************* ----->
Line 122: Line 121:
 
{| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;"
 
{| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;"
 
|-
 
|-
−
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Open GGUS Tickets (Snapshot during morning of the report)
+
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Open  
 +
GGUS Tickets (Snapshot during morning of the report).  The latest ticket snapshot can be found here[https://ggus.eu/index.php?mode=ticket_search&show_columns_check%5B%5D=TICKET_TYPE&show_columns_check%5B%5D=AFFECTED_VO&show_columns_check%5B%5D=AFFECTED_SITE&show_columns_check%5B%5D=PRIORITY&show_columns_check%5B%5D=RESPONSIBLE_UNIT&show_columns_check%5B%5D=STATUS&show_columns_check%5B%5D=DATE_OF_CHANGE&show_columns_check%5B%5D=SHORT_DESCRIPTION&show_columns_check%5B%5D=SCOPE&ticket_id=&supportunit=&su_hierarchy=0&former_su=&vo=&user=&keyword=&involvedsupporter=&assignedto=&affectedsite=RAL-LCG2&specattrib=none&status=open&priority=&typeofproblem=all&ticket_category=all&mouarea=&date_type=creation+date&tf_radio=1&timeframe=any&from_date=14+May+2018&to_date=22+May+2018&untouched_date=&scope=&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO%21].
 +
 
 
|}
 
|}
 
{| border=1 align=center
 
{| border=1 align=center
Line 135: Line 136:
 
! Subject
 
! Subject
 
! Scope
 
! Scope
 +
|-
 +
| style="background-color: green;" | 135133
 +
| cms
 +
| in progress
 +
| urgent
 +
| 15/05/2018
 +
| 17/05/2018
 +
| CMS_Data Transfers
 +
| Likely corrupted File at T1_UK_RAL
 +
| WLCG
 
|-
 
|-
 
| style="background-color: green;" | 134703
 
| style="background-color: green;" | 134703
Line 141: Line 152:
 
| urgent
 
| urgent
 
| 23/04/2018
 
| 23/04/2018
−
| 11/05/2018
+
| 18/05/2018
 
| CMS_Data Transfers
 
| CMS_Data Transfers
 
| Transfer failing from RAL_Disk
 
| Transfer failing from RAL_Disk
Line 156: Line 167:
 
| EGI
 
| EGI
 
|-
 
|-
−
| style="background-color: orange;" | 134468
+
| style="background-color: red;" | 134468
 
| cms
 
| cms
 
| waiting for reply
 
| waiting for reply
 
| top priority
 
| top priority
 
| 09/04/2018
 
| 09/04/2018
−
| 11/05/2018
+
| 18/05/2018
 
| CMS_AAA WAN Access
 
| CMS_AAA WAN Access
 
| Xrootd redirector not seeing some files in ECHO
 
| Xrootd redirector not seeing some files in ECHO
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: yellow;" | 133992
+
| style="background-color: orange;" | 133992
 
| atlas
 
| atlas
 
| in progress
 
| in progress
Line 230: Line 241:
 
! Scope
 
! Scope
 
|-
 
|-
−
| 135001
+
| 135164
−
| cms
+
−
| solved
+
−
| urgent
+
−
| 09/05/2018
+
−
| 10/05/2018
+
−
| CMS_Data Transfers
+
−
| Fts-client needs to be updated
+
−
| WLCG
+
−
|-
+
−
| 134880
+
 
| none
 
| none
 
| verified
 
| verified
 
| top priority
 
| top priority
−
| 02/05/2018
+
| 16/05/2018
−
| 09/05/2018
+
| 16/05/2018
−
| Middleware
+
| Other
−
| please upgrade Singularity
+
| This TEST ALARM has been raised for testing GGUS alarm work flow after a new GGUS release.
−
| EGI
+
−
|-
+
−
| 134769
+
−
| cms
+
−
| solved
+
−
| urgent
+
−
| 26/04/2018
+
−
| 08/05/2018
+
−
| CMS_Data Transfers
+
−
| Transfers from RAL_Disk to Florida are failing
+
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 134744
+
| 134853
−
| cms
+
−
| solved
+
−
| top priority
+
−
| 25/04/2018
+
−
| 08/05/2018
+
−
| File Transfer
+
−
| Zero Phedex Transfers - via RAL FTS service on certain links
+
−
| EGI
+
−
|-
+
−
| 134627
+
 
| cms
 
| cms
 
| closed
 
| closed
 
| urgent
 
| urgent
−
| 19/04/2018
+
| 02/05/2018
−
| 09/05/2018
+
| 16/05/2018
−
| CMS_Data Transfers
+
| CMS_Facilities
−
| Transfer backlogs
+
| T1_UK_RAL HammerCloud failures
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 134619
+
| 134839
−
| cms
+
−
| solved
+
−
| urgent
+
−
| 19/04/2018
+
−
| 08/05/2018
+
−
| CMS_SAM tests
+
−
| Problems reading data from ECHO
+
−
| WLCG
+
−
|-
+
−
| 134599
+
−
| t2k.org
+
−
| closed
+
−
| less urgent
+
−
| 18/04/2018
+
−
| 09/05/2018
+
−
| File Access
+
−
| Unable to reach 25 files at RAL which are ONLINE AND NEARLINE
+
−
| EGI
+
−
|-
+
−
| 134488
+
 
| snoplus.snolab.ca
 
| snoplus.snolab.ca
 
| closed
 
| closed
 
| urgent
 
| urgent
−
| 10/04/2018
+
| 30/04/2018
−
| 09/05/2018
+
| 16/05/2018
 
| File Transfer
 
| File Transfer
−
| FTS failures to RAL
+
| Data Transfer Failure
 
| EGI
 
| EGI
 
|-
 
|-
−
| 134342
+
| 134737
 
| cms
 
| cms
 
| closed
 
| closed
 
| urgent
 
| urgent
−
| 29/03/2018
+
| 25/04/2018
−
| 10/05/2018
+
| 14/05/2018
−
| CMS_Data Transfers
+
| CMS_SAM tests
−
| CERN->MSS rate test
+
| SAM CE test failing at T1_UK_RAL
 +
| WLCG
 +
|-
 +
| 134494
 +
| atlas
 +
| closed
 +
| urgent
 +
| 11/04/2018
 +
| 17/05/2018
 +
| Storage Systems
 +
| json space reporting not updated
 
| WLCG
 
| WLCG
 
|}
 
|}
Line 345: Line 316:
 
! LHCB
 
! LHCB
 
! Alice
 
! Alice
 +
! OPS
 +
! Comments
 
|-
 
|-
−
| 2018-05-07
+
| 2018-05-14
 
| 100
 
| 100
 
| 100
 
| 100
Line 352: Line 325:
 
| 100
 
| 100
 
| 100
 
| 100
−
|-
 
−
| 2018-05-08
 
−
| style="background-color: red;" | 85
 
−
| style="background-color: red;" | 85
 
−
| style="background-color: red;" | 86
 
−
| style="background-color: orange;" | 94
 
 
| 100
 
| 100
 +
|
 
|-
 
|-
−
| 2018-05-09
+
| 2018-05-15
−
| style="background-color: orange;" | 91
+
| 100
−
| style="background-color: orange;" | 91
+
| 100
 
| 99
 
| 99
 +
| 100
 +
| style="background-color: orange;" | 96
 +
| 100
 +
|
 +
|-
 +
| 2018-05-16
 +
| 100
 +
| 100
 +
| 100
 +
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 +
|
 
|-
 
|-
−
| 2018-05-10
+
| 2018-05-17
 
| 100
 
| 100
 
| 100
 
| 100
 +
| 98
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 +
|
 
|-
 
|-
−
| 2018-05-11
+
| 2018-05-18
 
| 100
 
| 100
 
| 100
 
| 100
 +
| 98
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 +
|
 
|-
 
|-
−
| 2018-05-12
+
| 2018-05-19
 
| 100
 
| 100
 
| 100
 
| 100
 +
| style="background-color: orange;" | 96
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 +
|
 
|-
 
|-
−
| 2018-05-13
+
| 2018-05-20
 +
| 98
 
| 100
 
| 100
 
| 100
 
| 100
Line 394: Line 380:
 
| 100
 
| 100
 
| 100
 
| 100
 +
|
 
|}
 
|}
  
Line 413: Line 400:
 
! Day !! Atlas HC !! CMS HC !! Comment
 
! Day !! Atlas HC !! CMS HC !! Comment
 
|-
 
|-
−
| 2018/05/08 || 98|| style="background-color: red;" | 81 ||   
+
| 2018/05/22 || 98|| 100 ||   
 
|-
 
|-
−
| 2018/05/09 || 97 || 99 ||  
+
| 2018/05/23 || 98 || 98 ||  
 
|-
 
|-
−
| 2018/05/10 || 99 || 99 ||  
+
| 2018/05/24 || 97 || 99 ||  
 
|-
 
|-
−
| 2018/05/11 || style="background-color: orange;" | 95 || 100 ||  
+
| 2018/05/25 || style="background-color: orange;" | 96 || 99 ||  
 
|-
 
|-
−
| 2018/05/12 || 100 || - ||  
+
| 2018/05/26 || 98 || style="background-color: red;" | 56 ||  
 
|-
 
|-
−
| 2018/05/13 || style="background-color: orange;" | 95 || - ||  
+
| 2018/05/27 || 100 || style="background-color: red;" | 60 ||  
 
|-
 
|-
−
| 2018/05/14 || - || - ||  
+
| 2018/05/28 || style="background-color: orange;" | 93 || 100 ||  
 
|-
 
|-
 
|}  
 
|}  

Latest revision as of 08:52, 30 May 2018

RAL Tier1 Operations Report for 21st May 2018

Review of Issues during the week 14th May to the 21st May 2018.
  • No incidents(major or minor), have been flagged during this reporting period.
Current operational status and issues
  • None
Resolved Castor Disk Server Issues
  • gdss732 (lhcbDst- D1T0) - Back in production after completion of rebuilding of the replacement drive.
Ongoing Castor Disk Server Issues
  • None
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
  • None
Declared in the GOC DB
  • None
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Networking
    • Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
  • Infrastructure
    • Testing of power distribution boards in the R89 machine room is being scheduled for some time late July / Early August. The effect of this on our services is being discussed.
Open

GGUS Tickets (Snapshot during morning of the report). The latest ticket snapshot can be found here[1].

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
135133 cms in progress urgent 15/05/2018 17/05/2018 CMS_Data Transfers Likely corrupted File at T1_UK_RAL WLCG
134703 cms in progress urgent 23/04/2018 18/05/2018 CMS_Data Transfers Transfer failing from RAL_Disk WLCG
134685 dteam in progress less urgent 23/04/2018 02/05/2018 Middleware please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7 EGI
134468 cms waiting for reply top priority 09/04/2018 18/05/2018 CMS_AAA WAN Access Xrootd redirector not seeing some files in ECHO WLCG
133992 atlas in progress less urgent 12/03/2018 19/04/2018 File Transfer RAL-LCG2-ECHO: No such file or directory EGI
127597 cms on hold urgent 07/04/2017 30/04/2018 File Transfer Check networking and xrootd RAL-CERN performance EGI
124876 ops on hold less urgent 07/11/2016 13/11/2017 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
117683 none on hold less urgent 18/11/2015 09/05/2018 Information System CASTOR at RAL not publishing GLUE 2 EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
135164 none verified top priority 16/05/2018 16/05/2018 Other This TEST ALARM has been raised for testing GGUS alarm work flow after a new GGUS release. WLCG
134853 cms closed urgent 02/05/2018 16/05/2018 CMS_Facilities T1_UK_RAL HammerCloud failures WLCG
134839 snoplus.snolab.ca closed urgent 30/04/2018 16/05/2018 File Transfer Data Transfer Failure EGI
134737 cms closed urgent 25/04/2018 14/05/2018 CMS_SAM tests SAM CE test failing at T1_UK_RAL WLCG
134494 atlas closed urgent 11/04/2018 17/05/2018 Storage Systems json space reporting not updated WLCG
Availability Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-05-14 100 100 100 100 100 100
2018-05-15 100 100 99 100 96 100
2018-05-16 100 100 100 100 100 100
2018-05-17 100 100 98 100 100 100
2018-05-18 100 100 98 100 100 100
2018-05-19 100 100 96 100 100 100
2018-05-20 98 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Day Atlas HC CMS HC Comment
2018/05/22 98 100
2018/05/23 98 98
2018/05/24 97 99
2018/05/25 96 99
2018/05/26 98 56
2018/05/27 100 60
2018/05/28 93 100
Notes from Meeting.
  • None yet
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-05-21&oldid=17710"