Difference between revisions of "RAL Tier1 weekly operations castor 27/04/2018"

From GridPP Wiki
Jump to: navigation, search
(Created page with "== Draft agenda == 1. Problems encountered this week 2. Upgrades/improvements made this week 3. What are we planning to do next week? 4. Long-term project updates (if not ...")
 
Line 31: Line 31:
 
== Operation problems ==
 
== Operation problems ==
  
   * Unroutable files to tape from CMS - need to anticipate this
+
   * Firewall update on 25/04 caused transient problems with CASTOR services [https://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=207369 ]
  
   * Load issue on genTape - possible fault with the recently deployed 2013 ds? Increased slot count
+
   * Network interface of gdss766 was misconfigured as eth2 causing problems with access - now fixed
    on 2013 boxes. Will set the 2011 servers to RO
+
 
 +
  * Transfers from RAL_Disk to Florida are failing [ GGUS ticket]
  
 
== Operation news ==
 
== Operation news ==
 +
 +
  *
  
 
   * Rob has retired fdsadm01 which is the old Fac WebDAV
 
   * Rob has retired fdsadm01 which is the old Fac WebDAV
Line 46: Line 49:
 
== Long-term projects ==
 
== Long-term projects ==
  
Headnode migration to Aquilon - Stager, scheduler, utility and nameserver configuration mainly complete.  
+
Headnode migration to Aquilon - Trying to configure former vCert headnodes as generic CASTOR headnodes. Almost done
Stager, scheduler, utility tested seperately and all together on preprod. Combined  Stager, scheduler,
+
but have some problems with SRM
utility and SRM features in one node which passes functional tests
+
  
 
No point to move Atlas and CMS to the macroheadnode setup since their d1t0 data are moving to Echo over the next two months
 
No point to move Atlas and CMS to the macroheadnode setup since their d1t0 data are moving to Echo over the next two months
Line 54: Line 56:
 
HA-proxyfication of the CASTOR SRMs: HA proxy set up has been tested on the SRMs
 
HA-proxyfication of the CASTOR SRMs: HA proxy set up has been tested on the SRMs
  
Target: Combined headnodes running on SL7/Aquilon - implement CERN-style 'Macro' headnodes.
+
RA writing a new accounting system to match WLCG archival requirements
  
 
== Actions ==  
 
== Actions ==  
  
 
RA/GP to talk to Chris Brew about alerts for new CMS file classes and tape families
 
RA/GP to talk to Chris Brew about alerts for new CMS file classes and tape families
 
RA to remove cedaRetrieve service class from cheksum validation
 
 
Investigate the history of the failed files on gdss766 [https://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=206651 RT ticket]
 
  
 
JK to investigate why ralreplicas fails and submit a "bug" report
 
JK to investigate why ralreplicas fails and submit a "bug" report
Line 72: Line 70:
 
== Staffing ==
 
== Staffing ==
  
GP on call
+
RA on call

Revision as of 10:00, 27 April 2018

Draft agenda

1. Problems encountered this week

2. Upgrades/improvements made this week

3. What are we planning to do next week?

4. Long-term project updates (if not already covered)

  1. SL5 elimination from CIP and tape verification server
  2. CASTOR stress test improvement
  3. Generic CASTOR headnode setup
  4. Aquilonised headnodes

5. Special topics

6. Actions

7. Review Fabric tasks

  1.   Link

8. AoTechnicalB

9. Availability for next week

10. On-Call

11. AoOtherB

Operation problems

  * Firewall update on 25/04 caused transient problems with CASTOR services [1]
  * Network interface of gdss766 was misconfigured as eth2 causing problems with access - now fixed
  * Transfers from RAL_Disk to Florida are failing [ GGUS ticket]

Operation news

  * 
  * Rob has retired fdsadm01 which is the old Fac WebDAV
  * lcgcadm03 and lcgcadm04 have been patched against Spectre/Meltdown

Plans for next few weeks

Long-term projects

Headnode migration to Aquilon - Trying to configure former vCert headnodes as generic CASTOR headnodes. Almost done but have some problems with SRM

No point to move Atlas and CMS to the macroheadnode setup since their d1t0 data are moving to Echo over the next two months

HA-proxyfication of the CASTOR SRMs: HA proxy set up has been tested on the SRMs

RA writing a new accounting system to match WLCG archival requirements

Actions

RA/GP to talk to Chris Brew about alerts for new CMS file classes and tape families

JK to investigate why ralreplicas fails and submit a "bug" report

RA/BD: Run GFAL unit tests against CASTOR. Get them here: https://gitlab.cern.ch/dmc/gfal2/tree/develop/test/

GP/RA to write a Nagios test to check for large number of requests on the NewRequests DB table that remain for a long time

Staffing

RA on call