Difference between revisions of "RAL Tier1 weekly operations castor 20/04/2018"

From GridPP Wiki
Jump to: navigation, search
(Created page with "== Draft agenda == 1. Problems encountered this week 2. Upgrades/improvements made this week 3. What are we planning to do next week? 4. Long-term project updates (if not ...")
 
Line 31: Line 31:
 
== Operation problems ==
 
== Operation problems ==
  
   * Load issue on genTape - possible fault with the recently deployed 2013 ds?
+
   * Unroutable files to tape from CMS - need to anticipate this
  * Failed snopuls transfers; FTS point : either a issue with gdss766 or FTS/SRM mis-scheduled the transfers [https://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=206651 RT206651]
+
  * fdsdss21 crashed and removed from production - it is now back in production 
+
  
   * ChrisP reports that a functional test that used to run on lcgcadm05 doesn't work anymore (probably because the machine has been turned off)
+
   * Load issue on genTape - possible fault with the recently deployed 2013 ds? Increased slot count
      * Should have been ported to castor-functional-test1 - maybe just needs repointing?
+
    on 2013 boxes. Will set the 2011 servers to RO
  
 
== Operation news ==
 
== Operation news ==
  
   * GridPP40 meeting
+
   * Rob has retired fdsadm01 which is the old Fac WebDAV
 +
  * lcgcadm03 and lcgcadm04 have been patched against Spectre/Meltdown
  
 
== Plans for next few weeks ==
 
== Plans for next few weeks ==
  
  * Incease the slot count on the 2013 genTape disk servers and see if it makes a difference on performance
 
  
 
== Long-term projects ==
 
== Long-term projects ==
Line 60: Line 58:
 
== Actions ==  
 
== Actions ==  
  
Investigate hte history of the failed files on gdss766
+
RA/GP to talk to Chris Brew about alerting for new CMS file classes and tape families
  
Tell whether a Facilities functional test which was running on on lcgcadm05 is running on castor-functional-test1
+
RA to remove cedaRetrieve service class from cheksum validation
 +
 
 +
Investigate the history of the failed files on gdss766 (include RT tickets!!)
  
 
JK to investigate why ralreplicas fails and submit a "bug" report
 
JK to investigate why ralreplicas fails and submit a "bug" report
 
JK to investigate castor functional test on facilities
 
  
 
RA/BD:  Run GFAL unit tests against CASTOR. Get them here: https://gitlab.cern.ch/dmc/gfal2/tree/develop/test/
 
RA/BD:  Run GFAL unit tests against CASTOR. Get them here: https://gitlab.cern.ch/dmc/gfal2/tree/develop/test/
 
RA to check if there is any outstanding action for Fabric
 
  
 
GP/RA to write a Nagios test to check for large number of requests on the NewRequests DB table that remain for a long time
 
GP/RA to write a Nagios test to check for large number of requests on the NewRequests DB table that remain for a long time
Line 76: Line 72:
 
== Staffing ==
 
== Staffing ==
  
RA on call
+
GP on call

Revision as of 10:11, 20 April 2018

Draft agenda

1. Problems encountered this week

2. Upgrades/improvements made this week

3. What are we planning to do next week?

4. Long-term project updates (if not already covered)

  1. SL5 elimination from CIP and tape verification server
  2. CASTOR stress test improvement
  3. Generic CASTOR headnode setup
  4. Aquilonised headnodes

5. Special topics

6. Actions

7. Review Fabric tasks

  1.   Link

8. AoTechnicalB

9. Availability for next week

10. On-Call

11. AoOtherB

Operation problems

  * Unroutable files to tape from CMS - need to anticipate this
  * Load issue on genTape - possible fault with the recently deployed 2013 ds? Increased slot count 
    on 2013 boxes. Will set the 2011 servers to RO

Operation news

  * Rob has retired fdsadm01 which is the old Fac WebDAV
  * lcgcadm03 and lcgcadm04 have been patched against Spectre/Meltdown

Plans for next few weeks

Long-term projects

Headnode migration to Aquilon - Stager, scheduler, utility and nameserver configuration mainly complete. Stager, scheduler, utility tested seperately and all together on preprod. Combined Stager, scheduler, utility and SRM features in one node which passes functional tests

No point to move Atlas and CMS to the macroheadnode setup since their d1t0 data are moving to Echo over the next two months

HA-proxyfication of the CASTOR SRMs: HA proxy set up has been tested on the SRMs

Target: Combined headnodes running on SL7/Aquilon - implement CERN-style 'Macro' headnodes.

Actions

RA/GP to talk to Chris Brew about alerting for new CMS file classes and tape families

RA to remove cedaRetrieve service class from cheksum validation

Investigate the history of the failed files on gdss766 (include RT tickets!!)

JK to investigate why ralreplicas fails and submit a "bug" report

RA/BD: Run GFAL unit tests against CASTOR. Get them here: https://gitlab.cern.ch/dmc/gfal2/tree/develop/test/

GP/RA to write a Nagios test to check for large number of requests on the NewRequests DB table that remain for a long time

Staffing

GP on call