Tier1 Operations Report 2010-03-31

RAL Tier1 Operations Report for 31st March 2010.

We should note the success at CERN in having the first 7TeV collisions at the LHC yesterday, Tuesday 30th March.

Review of Issues during week 24th to 31st March 2010.

• Over the weekend there were problems with migration to tape. These were traced to a single tape drive that has failed. Removing this drive from operation on Monday resolved the operational problem.
• Work carried out in the last week includes the clean-up of the LFC schema, updating LSF (Castor scheduler) license keys and updating the microcode in half the tape drives.

Current operational status and issues.

• The rolling upgrade to the batch nodes to SL 5.4 is well under way.

Declared in the GOC DB:

• None

Advanced warning:

The following items remain to be scheduled:

• Castor Oracle Database infrastructure. One change, the removal of unstable node from Oracle RAC and its replacement by another node, remains to be done.
• Kernel and glibc updates will need to be done on LFC, FTS & 3D Oracle database (RAC) nodes. Awaiting change control review.
• Various updates (Oracle, kernels) on Castor database nodes.
• Update microcode on remaining half of tape drives.

Entries in GOC DB starting between 24th and 31st March 2010.

There was one unscheduled entry in the GOC DB for this last week.

Service	Scheduled?	Outage/At Risk	Start	End	Duration	Reason
All Castor	SCHEDULED	AT_RISK	29/03/2010 09:30	29/03/2010 10:30	1 hour	At Risk on Castor. Second part of updating licence keys for the LSF scheduler within castor.
All Castor	UNSCHEDULED	AT_RISK	26/03/2010 12:00	26/03/2010 12:19	19 minutes	At Risk on Castor while license key for LSF (the scheduler within Castor) is updated.
lfc.gridpp.rl.ac.uk,	SCHEDULED	OUTAGE	25/03/2010 09:00	25/03/2010 12:00	3 hours	Outage while database schema cleaned to remove redundant information. This will NOT affect the other LFC instances based at the RAL Tier1 (lfc-atlas.gridpp.rl.ac.uk and lhcb-lfc.gridpp.rl.ac.uk)