Difference between revisions of "RAL Tier1 weekly operations castor 14/09/2018"
(Created page with "== Standing agenda == 1. Problems encountered this week 2. Upgrades/improvements made this week 3. What are we planning to do next week? 4. Long-term project updates (if n...") |
(→Long-term projects) |
||
(18 intermediate revisions by one user not shown) | |||
Line 31: | Line 31: | ||
== Operation problems == | == Operation problems == | ||
+ | Issue with CMS migrations: CASTOR marked all cmsr18rec tapes available are in a BUSY state, but not mounted. After Tim | ||
+ | cleared the BUSY state, migrations started again. Need to find why CASTOR marked the tapes as BUSY. | ||
+ | |||
+ | "Probably need a Nagios check for tapes that are BUSY and are not mounted. Won’t be easy as there is a 5 min (max) window where the tape is marked BUSY, and the tapegatewayd has not triggered to get the tape mounted" (ticket on the support queue) | ||
+ | |||
+ | |||
+ | LHCb issue again with failed downloads from CASTOR to WN [https://ggus.eu/index.php?mode=ticket_info&ticket_id=137136 GGUS137136] | ||
== Operation news == | == Operation news == | ||
− | * Completed Oracle patching of | + | * Completed Oracle patching of preprod and vcert |
− | * | + | |
+ | * Preprod instance is back up and running with the new host names | ||
+ | |||
+ | * OPS tests are coming in for the srm-wlcgtape endpoint of the new CASTOR instance | ||
== Plans for next few weeks == | == Plans for next few weeks == | ||
Line 45: | Line 55: | ||
== Long-term projects == | == Long-term projects == | ||
− | gsi authentication for xrootd: 1) Return preprod to a working state 2) Ask Raja and Tim to test the gsi config on preprod | + | gsi authentication for xrootd: 1) Return preprod to a working state (done) 2) Ask Raja and Tim to test the gsi config on preprod |
3) Perform a stress test | 3) Perform a stress test | ||
− | Things need doing: log rotation, xrootd for ALICE | + | New CASTOR WLCGTape instance: Upgraded stagers and ns nodes to 2.1.17-35. Need to upgrade tape servers to make migrations/recalls |
+ | to work. Because of CASTOR version depedencies, tape servers cannot all be upgraded and allocated to the new instance at the same | ||
+ | time. Rob suggested we do 5 tape servers for every VO migration to the new CASTOR instance. Also need to negotiate with TimA how | ||
+ | many tape drives ATLAS would be happy to use for the tape "carouselle" test. | ||
+ | |||
+ | Things need doing: log rotation([https://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=212634 RT212634]), xrootd for ALICE | ||
== Actions == | == Actions == | ||
+ | |||
+ | GP to add ldif entry for the new srm-wlcgtape endpoint | ||
JK to investigate why ralreplicas fails and submit a "bug" report | JK to investigate why ralreplicas fails and submit a "bug" report | ||
Line 60: | Line 77: | ||
== Staffing == | == Staffing == | ||
− | + | RA on call | |
− | + |
Latest revision as of 11:27, 14 September 2018
Contents
Standing agenda
1. Problems encountered this week
2. Upgrades/improvements made this week
3. What are we planning to do next week?
4. Long-term project updates (if not already covered)
1. SL5 elimination from CIP and tape verification server 2. CASTOR stress test improvement 3. Generic CASTOR headnode setup 4. Aquilonised headnodes
5. Special topics
6. Actions
7. Review Fabric tasks
1. Link
8. AoTechnicalB
9. Availability for next week
10. On-Call
11. AoOtherB
Operation problems
Issue with CMS migrations: CASTOR marked all cmsr18rec tapes available are in a BUSY state, but not mounted. After Tim cleared the BUSY state, migrations started again. Need to find why CASTOR marked the tapes as BUSY.
"Probably need a Nagios check for tapes that are BUSY and are not mounted. Won’t be easy as there is a 5 min (max) window where the tape is marked BUSY, and the tapegatewayd has not triggered to get the tape mounted" (ticket on the support queue)
LHCb issue again with failed downloads from CASTOR to WN GGUS137136
Operation news
* Completed Oracle patching of preprod and vcert
* Preprod instance is back up and running with the new host names
* OPS tests are coming in for the srm-wlcgtape endpoint of the new CASTOR instance
Plans for next few weeks
* Push gsi authentication for xrootd in production
* Decommission gdss747
Long-term projects
gsi authentication for xrootd: 1) Return preprod to a working state (done) 2) Ask Raja and Tim to test the gsi config on preprod 3) Perform a stress test
New CASTOR WLCGTape instance: Upgraded stagers and ns nodes to 2.1.17-35. Need to upgrade tape servers to make migrations/recalls to work. Because of CASTOR version depedencies, tape servers cannot all be upgraded and allocated to the new instance at the same time. Rob suggested we do 5 tape servers for every VO migration to the new CASTOR instance. Also need to negotiate with TimA how many tape drives ATLAS would be happy to use for the tape "carouselle" test.
Things need doing: log rotation(RT212634), xrootd for ALICE
Actions
GP to add ldif entry for the new srm-wlcgtape endpoint
JK to investigate why ralreplicas fails and submit a "bug" report
RA/BD: Run GFAL unit tests against CASTOR. Get them here: https://gitlab.cern.ch/dmc/gfal2/tree/develop/test/
GP/RA to write a Nagios test to check for large number of requests on the NewRequests DB table that remain for a long time - the check will run every xxx min (to be aggreed with Production) and issue an alarm if the number exceeds 1,000 for an extended period of time (15 min)
Staffing
RA on call