Difference between revisions of "Tier1 Operations Report 2014-08-06"
From GridPP Wiki
(→) |
(→) |
||
| (13 intermediate revisions by one user not shown) | |||
| Line 1: | Line 1: | ||
| − | = | + | =RAL Tier1 Operations Report for 6th August 2014= |
__NOTOC__ | __NOTOC__ | ||
====== ====== | ====== ====== | ||
| Line 7: | Line 7: | ||
{| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;" | {| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;" | ||
|- | |- | ||
| − | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week | + | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 31st to 6th August 2014. |
|} | |} | ||
| − | + | * Discrepancies were found in some of the Castor database tables and columns. The Castor team are considering options with regard to fixing these. The issue has no operational impact. | |
| − | * Discrepancies were found in some of the Castor database tables and columns. The Castor team are considering options with regard to fixing these. | + | |
* The problem with the dteam SRM regional nagios tests has been fixed with a CIP update. | * The problem with the dteam SRM regional nagios tests has been fixed with a CIP update. | ||
| − | * The fix to the problems with the RIP on the tier1 routers was rolled out on Thursday 31st July as announced. RIP did not start on the routers and this is being investigated. | + | * The fix to the problems with the RIP on the tier1 routers was rolled out on Thursday 31st July as announced. RIP did not start on the routers and this is being investigated by the networking team. |
* At about 22:15 on Friday 1st Aug, a network fault on site caused the Tier1 routers to flip the active routing system from rtr-x670v-1 to rtr-x670v-2. This is expected behaviour when a link dies. The active route was changed back on Tuesday 5th Aug at 10:15. | * At about 22:15 on Friday 1st Aug, a network fault on site caused the Tier1 routers to flip the active routing system from rtr-x670v-1 to rtr-x670v-2. This is expected behaviour when a link dies. The active route was changed back on Tuesday 5th Aug at 10:15. | ||
<!-- ***********End Review of Issues during last week*********** -----> | <!-- ***********End Review of Issues during last week*********** -----> | ||
| Line 35: | Line 34: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | ||
|} | |} | ||
| − | * We are still investigating xroot access to CMS Castor following the upgrade on the 17th June. The | + | * There are problems with disk server draining for Atlas in Castor 2.1.4. This is under investigation. |
| + | * We are still investigating xroot access to CMS Castor following the upgrade on the 17th June. The service has improved but there may still be work to be done. | ||
<!-- ***********End Current operational status and issues*********** -----> | <!-- ***********End Current operational status and issues*********** -----> | ||
<!-- *************************************************************** -----> | <!-- *************************************************************** -----> | ||
| Line 99: | Line 99: | ||
|} | |} | ||
<!-- ******* still to be formally scheduled and/or announced ******* -----> | <!-- ******* still to be formally scheduled and/or announced ******* -----> | ||
| − | * | + | * The rollout of the RIP protocol to the Tier1 routers still has to be completed. |
* We are planning the termination of the FTS2 service (announced for 2nd September) now that almost all use is on FTS3. | * We are planning the termination of the FTS2 service (announced for 2nd September) now that almost all use is on FTS3. | ||
* The removal of the (NFS) software server is scheduled for the 2nd September. | * The removal of the (NFS) software server is scheduled for the 2nd September. | ||
| Line 158: | Line 158: | ||
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject | ! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject | ||
|- | |- | ||
| − | | | + | | 107416 |
| Yellow | | Yellow | ||
| Less Urgent | | Less Urgent | ||
| − | | | + | | waiting for reply |
| − | | 2014-07- | + | | 2014-07-31 |
| − | | 2014- | + | | 2014-08-04 |
| − | | | + | | Atlas |
| − | | | + | | RAL FTS parameters for communication with US MWT2 |
|- | |- | ||
| 106324 | | 106324 | ||
| Line 172: | Line 172: | ||
| In Progress | | In Progress | ||
| 2014-06-18 | | 2014-06-18 | ||
| − | | 2014- | + | | 2014-08-05 |
| CMS | | CMS | ||
| pilots losing network connections at T1_UK_RAL | | pilots losing network connections at T1_UK_RAL | ||
| Line 181: | Line 181: | ||
| On Hold | | On Hold | ||
| 2014-05-14 | | 2014-05-14 | ||
| − | | 2014-07- | + | | 2014-07-29 |
| | | | ||
| please check your Vidyo router firewall configuration | | please check your Vidyo router firewall configuration | ||
| Line 203: | Line 203: | ||
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! CMS HC !! Comment | ! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! CMS HC !! Comment | ||
|- | |- | ||
| − | | | + | | 30/07/14 || 100 || 100 || style="background-color: lightgrey;" | 97.2 || 100 || 100 || 100 || 99 || Atlas errors were 'file not copied to SRM' |
|- | |- | ||
| − | | | + | | 31/07/14 || 100 || 100 || style="background-color: lightgrey;" | 99.22 || 100 || 100 || 94 || 99 || Atlas errors were 'Invalid argument ... File not copied to SRM' |
|- | |- | ||
| − | | | + | | 01/08/14 || 100 || 100 || 100 || 100 || 100 || 100 || 99 || |
|- | |- | ||
| − | | | + | | 02/08/14 || 100 || 100 || 100 || 100 || style="background-color: lightgrey;" | 95.85 || 94 || n/a || LHCb: 'No such file or directory' |
|- | |- | ||
| − | | | + | | 03/08/14 || 100 || 100 || 100 || 100 || 100 || 98 || n/a || |
|- | |- | ||
| − | | | + | | 04/08/14 || 100 || 100 || 100 || 100 || 100 || 89 || 100 || |
|- | |- | ||
| − | | | + | | 05/08/14 || 100 || 100 || 100 || 100 || 100 || 89 || 99 || |
|} | |} | ||
<!-- **********************End Availability Report************************** -----> | <!-- **********************End Availability Report************************** -----> | ||
<!-- *********************************************************************** -----> | <!-- *********************************************************************** -----> | ||
Latest revision as of 12:25, 6 August 2014
RAL Tier1 Operations Report for 6th August 2014
| Review of Issues during the week 31st to 6th August 2014. |
- Discrepancies were found in some of the Castor database tables and columns. The Castor team are considering options with regard to fixing these. The issue has no operational impact.
- The problem with the dteam SRM regional nagios tests has been fixed with a CIP update.
- The fix to the problems with the RIP on the tier1 routers was rolled out on Thursday 31st July as announced. RIP did not start on the routers and this is being investigated by the networking team.
- At about 22:15 on Friday 1st Aug, a network fault on site caused the Tier1 routers to flip the active routing system from rtr-x670v-1 to rtr-x670v-2. This is expected behaviour when a link dies. The active route was changed back on Tuesday 5th Aug at 10:15.
| Resolved Disk Server Issues |
- Gdss680 was returned to production on thursday 31st July.
| Current operational status and issues |
- There are problems with disk server draining for Atlas in Castor 2.1.4. This is under investigation.
- We are still investigating xroot access to CMS Castor following the upgrade on the 17th June. The service has improved but there may still be work to be done.
| Ongoing Disk Server Issues |
- None
| Notable Changes made this last week. |
- 10 disk servers deployed into lhcbDst
| Declared in the GOC DB |
| Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
|---|---|---|---|---|---|---|
| perfsonar-ps01.gridpp.rl.ac.uk, perfsonar-ps02.gridpp.rl.ac.uk, | SCHEDULED | OUTAGE | 14/07/2014 11:00 | 14/08/2014 11:00 | 31 days, | Systems being decommissioned. They have been replaced by lcgps01.gridpp.rl.ac.uk and lcgps02.gridpp.rl.ac.uk |
| Advanced warning for other interventions |
| The following items are being discussed and are still to be formally scheduled and announced. |
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- We are planning the termination of the FTS2 service (announced for 2nd September) now that almost all use is on FTS3.
- The removal of the (NFS) software server is scheduled for the 2nd September.
- We are planning stop access to the cream CEs - although possibly leaving them available to ALICE for some time. No date has yet been specified for this.
Listing by category:
- Databases:
- Switch LFC/FTS/3D to new Database Infrastructure.
- Castor:
- None.
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Fabric
- We are phasing out the use of the software server used by the small VOs.
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during 2014.
| Entries in GOC DB starting between the 23rd and 30th July 2014. |
| Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
|---|---|---|---|---|---|---|
| perfsonar-ps01.gridpp.rl.ac.uk, perfsonar-ps02.gridpp.rl.ac.uk, | SCHEDULED | OUTAGE | 14/07/2014 11:00 | 14/08/2014 11:00 | 31 days, | Systems being decommissioned. They have been replaced by lcgps01.gridpp.rl.ac.uk and lcgps02.gridpp.rl.ac.uk |
| Open GGUS Tickets (Snapshot during morning of meeting) |
| GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
|---|---|---|---|---|---|---|---|
| 107416 | Yellow | Less Urgent | waiting for reply | 2014-07-31 | 2014-08-04 | Atlas | RAL FTS parameters for communication with US MWT2 |
| 106324 | Red | Urgent | In Progress | 2014-06-18 | 2014-08-05 | CMS | pilots losing network connections at T1_UK_RAL |
| 105405 | Red | Urgent | On Hold | 2014-05-14 | 2014-07-29 | please check your Vidyo router firewall configuration |
| Availability Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day | OPS | Alice | Atlas | CMS | LHCb | Atlas HC | CMS HC | Comment |
|---|---|---|---|---|---|---|---|---|
| 30/07/14 | 100 | 100 | 97.2 | 100 | 100 | 100 | 99 | Atlas errors were 'file not copied to SRM' |
| 31/07/14 | 100 | 100 | 99.22 | 100 | 100 | 94 | 99 | Atlas errors were 'Invalid argument ... File not copied to SRM' |
| 01/08/14 | 100 | 100 | 100 | 100 | 100 | 100 | 99 | |
| 02/08/14 | 100 | 100 | 100 | 100 | 95.85 | 94 | n/a | LHCb: 'No such file or directory' |
| 03/08/14 | 100 | 100 | 100 | 100 | 100 | 98 | n/a | |
| 04/08/14 | 100 | 100 | 100 | 100 | 100 | 89 | 100 | |
| 05/08/14 | 100 | 100 | 100 | 100 | 100 | 89 | 99 |
