Difference between revisions of "RAL Tier1 weekly operations castor 22/06/2018"

From GridPP Wiki
Jump to: navigation, search
(Operation problems)
 
Line 31: Line 31:
 
== Operation problems ==
 
== Operation problems ==
  
   * gdss685 and gdss699 and gdss746 failed and went out of production. gdss699 was set to readonly but failed again under heavy load
+
   * gdss685 and gdss699, gdss746 and gdss776 failed and went out of production. gdss699 was set to readonly but failed again under heavy load. gdss776 is still out of prod
 
+
  * Network glitch on 11/06. Affected mainly Facilities machines and DB systems
+
  
 
== Operation news ==
 
== Operation news ==
  
   * Central services schemas on preprod were upgraded to the latest version
+
   * Mgration of the tape-verification server to Aquilon completed
  
 
== Plans for next few weeks ==
 
== Plans for next few weeks ==
  
   * Finalise the migration of the tape-verification server to Aquilon
+
   * Decommission the CIP assuming staic file in bdii works
  
 
== Long-term projects ==
 
== Long-term projects ==
  
All VMware CASTOR headnodes passed functional tests [https://wiki.e-science.cclrc.ac.uk/web1/bin/view/EScienceInternal/CreationOfTheNewCASTORTier1TapeInstance CASTOR plan page]
+
Things need doing: log rotation, preparation in aquilon for new instance,...
  
 
RA writing a new accounting system to match WLCG archival requirements
 
RA writing a new accounting system to match WLCG archival requirements
Line 59: Line 57:
 
== Staffing ==
 
== Staffing ==
  
GP on call for next week
+
RA on call for next week

Latest revision as of 10:08, 22 June 2018

Draft agenda

1. Problems encountered this week

2. Upgrades/improvements made this week

3. What are we planning to do next week?

4. Long-term project updates (if not already covered)

  1. SL5 elimination from CIP and tape verification server
  2. CASTOR stress test improvement
  3. Generic CASTOR headnode setup
  4. Aquilonised headnodes

5. Special topics

6. Actions

7. Review Fabric tasks

  1.   Link

8. AoTechnicalB

9. Availability for next week

10. On-Call

11. AoOtherB

Operation problems

  * gdss685 and gdss699, gdss746 and gdss776 failed and went out of production. gdss699 was set to readonly but failed again under heavy load. gdss776 is still out of prod

Operation news

 * Mgration of the tape-verification server to Aquilon completed

Plans for next few weeks

  * Decommission the CIP assuming staic file in bdii works

Long-term projects

Things need doing: log rotation, preparation in aquilon for new instance,...

RA writing a new accounting system to match WLCG archival requirements

Actions

JK to investigate why ralreplicas fails and submit a "bug" report

RA/BD: Run GFAL unit tests against CASTOR. Get them here: https://gitlab.cern.ch/dmc/gfal2/tree/develop/test/

GP/RA to write a Nagios test to check for large number of requests on the NewRequests DB table that remain for a long time - the check will run every xxx min (to be aggreed with Production) and issue an alarm if the number exceeds 1,000 for an extended period of time (15 min)

Staffing

RA on call for next week