https://www.gridpp.ac.uk/w/api.php?action=feedcontributions&user=James+adams&feedformat=atom
GridPP Wiki - User contributions [en]
2024-03-29T11:03:27Z
User contributions
MediaWiki 1.22.0
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Grid_20120723
RAL Tier1 weekly operations Grid 20120723
2012-07-23T14:57:24Z
<p>James adams: </p>
<hr />
<div>== Operational Issues ==<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Severity<br />
! Status<br />
|-<br />
| <br />
| <br />
| <br />
| <br />
| <br />
| <br />
|}<br />
<br />
== Downtimes ==<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Description<br />
! Hosts<br />
! Type<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
|-<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|-<br />
|}<br />
<br />
== Blocking Issues ==<br />
<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Description<br />
! Requested Date<br />
! Required By Date<br />
! Priority<br />
! Status<br />
|-<br />
| <br />
| <br />
|<br />
|<br />
| <br />
|}<br />
<br />
== Developments/Plans ==<br />
<br />
=== Highlights for Tier-1 Ops Meeting ===<br />
<br />
=== Highlights for Tier-1 VO Liaison Meeting ===<br />
<br />
=== Detailed Individual Reports ===<br />
<br />
==== Andrew ====<br />
* Last week:<br />
** A/L<br />
* Coming week:<br />
** Add SNO+ to FTS; move more FTS channel agents to VMs<br />
** Starting looking into WN power consumption project<br />
** Kernel/errata updates<br />
** CMS computing workshop Wed-Fri<br />
** CMS processing<br />
<br />
==== Catalin ====<br />
* A/L<br />
<br />
====Ian====<br />
* Last week:<br />
** Planning CV05/SL06 decommissioning<br />
** Investigating gridppnagios failures<br />
** Recruitment work<br />
<br />
* Coming week:<br />
** WMS security update<br />
** Getting Martin started with hyper-v & PDU survey<br />
** Look at quattor aii issues<br />
** Errata updates<br />
<br />
====James====<br />
* Installing and configuring storage test-bed nodes.<br />
<br />
==== Orlin ====<br />
* Create & Test Quattor templates for glite-CLUSTER nodetype in virtual environment. [done]<br />
* Test in virtual environment if EMI CREAMCE and glite-CLUSTER are publishing successfully their GLUE records. [done]<br />
* Update the documentation for copying munge keys and ssh configuration on CREAMCEs and WNs. [done]<br />
* Implement centralized banning policy Argus PAP server/client and test it on WNs. [ongoing]<br />
* Check how EMI-2 SL-6 WNs can be implemented at RAL. [to do]<br />
<br />
== VO Reports ==<br />
<br />
=== ALICE ===<br />
<br />
=== ATLAS ===<br />
<br />
=== CMS ===<br />
<br />
=== LHCb ===<br />
<br />
== OnCall/AoD Cover ==<br />
[https://wiki.e-science.cclrc.ac.uk/web1/bin/view/EScienceInternal/TierOneOncallRota OnCall Rota]<br />
* Grid OnCall: Andrew<br />
<br />
[[Category:RAL Tier1]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Grid_20120514
RAL Tier1 weekly operations Grid 20120514
2012-05-14T12:56:13Z
<p>James adams: </p>
<hr />
<div>== Operational Issues ==<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Severity<br />
! Status<br />
|-<br />
| <br />
| <br />
| <br />
| <br />
| <br />
| <br />
|}<br />
<br />
== Downtimes ==<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Description<br />
! Hosts<br />
! Type<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
|-<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|-<br />
|}<br />
<br />
== Blocking Issues ==<br />
<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Description<br />
! Requested Date<br />
! Required By Date<br />
! Priority<br />
! Status<br />
|-<br />
| <br />
| <br />
|<br />
|<br />
| <br />
|}<br />
<br />
== Developments/Plans ==<br />
<br />
=== Highlights for Tier-1 Ops Meeting ===<br />
<br />
=== Highlights for Tier-1 VO Liaison Meeting ===<br />
<br />
=== Detailed Individual Reports ===<br />
<br />
==== Andrew ====<br />
* Setup UMD APEL on a VM, now running in parallel to production system [Done]<br />
* Applied 2 FTS patches [Done]<br />
* Completed APR [Done]<br />
* Investigating batch system problems; investigating why FTS proxy problem disappeared<br />
* Prepared config updates to allow NA62 to run jobs & SNO+ close SE publishing issue<br />
* Setup CMS tape families for Run2012B [Done]<br />
* CMS reprocessing [Ongoing]<br />
* This week<br />
** Job plan<br />
** Compare VM APEL with production system, write change control document<br />
** Review all scripts which query batch system<br />
** Start working on updates to capacity planning system as discussed at previous capacity signoff meeting<br />
<br />
==== Catalin ====<br />
* catching up<br />
* prepare the gLite LFC v1.8.2 update<br />
* look into EMI CREAM publishing<br />
<br />
====Ian====<br />
* Last week - strauslab installations, Hepsysman<br />
* Jobplan<br />
* Helping Frazer (SCT) get started with StratusLab & Quattor<br />
* Work on QWG/YAIM integration with Dimitris & Andrew<br />
* Assisting Vasilij with Hyper-V clustering<br />
<br />
====James====<br />
* Job plan<br />
* Storage research<br />
* Tidying up after people in Quattor<br />
* Some work on Aquilon<br />
<br />
==== Orlin ====<br />
* Quattorise NFS server and integrate it in a test virtual machine environment. [ongoing]<br />
* Test ARGUS EMI Server in production environment with shared NFS gridmapdir accounts. [to do]<br />
* Submit Change Control for ARGUS EMI Server. [to do]<br />
* Test EMI/UMD Worker Nodes lcg1060 lcg1062 in production with the latest kernel/errata/cvmfs. [ongoing]<br />
* Create checklist and wiki-page for EMI/UMD Worker Node services . [ongoing]<br />
* GGUS ticket #81606 - MyProxy renewal for T2K.org on lcgWMS 02/03 [done]<br />
<br />
== VO Reports ==<br />
<br />
=== ALICE ===<br />
<br />
=== ATLAS ===<br />
<br />
=== CMS ===<br />
<br />
=== LHCb ===<br />
<br />
== OnCall/AoD Cover ==<br />
[https://wiki.e-science.cclrc.ac.uk/web1/bin/view/EScienceInternal/TierOneOncallRota OnCall Rota]<br />
* Grid OnCall: Catalin (Mon-Sun)<br />
[[Category:RAL Tier1]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_T1_weekly_ops_Farbic_20110801
RAL T1 weekly ops Farbic 20110801
2011-07-25T12:55:41Z
<p>James adams: </p>
<hr />
<div>#REDIRECT [[RAL T1 weekly ops Fabric 20110801]]<br />
</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_T1_weekly_ops_Farbic_20110718
RAL T1 weekly ops Farbic 20110718
2011-07-25T12:55:25Z
<p>James adams: </p>
<hr />
<div>#REDIRECT [[RAL T1 weekly ops Fabric 20110718]]<br />
</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_T1_weekly_ops_Farbic_20110711
RAL T1 weekly ops Farbic 20110711
2011-07-25T12:55:14Z
<p>James adams: </p>
<hr />
<div>#REDIRECT [[RAL T1 weekly ops Fabric 20110711]]<br />
</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_T1_weekly_ops_Farbic_20110704
RAL T1 weekly ops Farbic 20110704
2011-07-25T12:53:41Z
<p>James adams: </p>
<hr />
<div>#REDIRECT [[RAL T1 weekly ops Fabric 20110704]]<br />
</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_T1_weekly_ops_Farbic_20110725
RAL T1 weekly ops Farbic 20110725
2011-07-25T12:53:21Z
<p>James adams: </p>
<hr />
<div>#REDIRECT [[RAL T1 weekly ops Fabric 20110725]]<br />
</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Fabric_20110117
RAL Tier1 weekly operations Fabric 20110117
2011-01-24T14:36:08Z
<p>James adams: </p>
<hr />
<div>Editing RAL Tier1 weekly operations Fabric 20110110<br />
<br />
== Developments ==<br />
* All:<br />
<br />
* Martin:<br />
** <br />
<br />
* Ian:<br />
** Work on virtualisation<br />
** Preparatory work on cluster groups in Quattor<br />
** Setting up acceptance testing on new db nodes<br />
** Prep for CERN visit<br />
<br />
* Tim:<br />
** <br />
<br />
* James A:<br />
** Working through issues with ClusterVision WNs with Dell.<br />
** Preparation for next Atlas power off.<br />
** Benchmarking.<br />
** Annual Leave on Wednesday.<br />
<br />
<br />
* James T<br />
** Prep for ATLAS SL5 x86_64 upgrade<br />
** RAID controller summary to Sam<br />
** AFS L&D<br />
<br />
* Cheney<br />
** amanda performance testing<br />
** clear down ancient database backups<br />
** design model for disk server analysis<br />
** investigate DMF DR<br />
** investigate DMF rsync<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Decommissioning old batch systems.(R 27)<br />
** gdss380 received from Streamline and moved into rack.<br />
** gdss606 fixed for testing.<br />
** gdss496 Scsi errors. (Intervention)<br />
** gdss305 and gdss327 given back to Castor team.<br />
** Fabric Hardware failure metrics.<br />
** Streamline/areca disk servers crashed due to single faulty drive. (ongoing)<br />
** gdss576 and gdss577 not in testing. (Informed James T)<br />
** gdss337 Kernel panic (Faulty memory)<br />
** gdss283 crashed with File system problem.(Intervention)<br />
** gdss68 ready for decommission.<br />
** SL 2010 and Viglen 2010 disk servers in testing.<br />
** SL 2009 Auto rebuild on hotspare fails. <br />
<br />
<br />
=== Operational Issues and Incidents ===<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Index<br />
! Description<br />
! Start<br />
! End<br />
! Severity<br />
! Affected VO(s)<br />
|-<br />
|}<br />
<br />
== Summary of plans for week ahead ==<br />
<br />
=== Scheduled and Cancelled Down Times ===<br />
<br />
Type=Down/At Risk/Cancelled entries in/planned to go to GOCDB<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Component<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Type<br />
|-<br />
|}<br />
<br />
=== Development priorities ===<br />
<br />
* All<br />
<br />
* Martin:<br />
**<br />
<br />
* Ian:<br />
** Visiting CERN<br />
** Hepix virtualisation working group meeting<br />
** Meeting with cvmfs developers<br />
<br />
<br />
* Tim:<br />
** <br />
<br />
* Cheney<br />
** DMF DR<br />
** DMF rsync<br />
** Prep for TDG talk<br />
<br />
<br />
* James T:<br />
** ATLAS SL5 x86_64 upgrade<br />
** First aid course Wednesday and Thursday<br />
** iSCSI and AFS L&D<br />
<br />
* James A:<br />
** Working through issues with ClusterVision WNs with Dell.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** SL 2009 Auto rebuild on hotspare fails.<br />
** Hardware failure metrics continue.<br />
** SL08 testing.<br />
** Continuous decommissioning old batch systems.(R 27)<br />
<br />
=== Absences ===<br />
* Ian out Tuesday-Monday - A/L Friday 21st and Monday 24th<br />
* JRHA out Wednesday (Annual Leave)<br />
<br />
=== Fabric On-Call ===<br />
<br />
* Monday - Sunday - Kashif<br />
<br />
=== Advanced Warning of Requirements and Blocking issues ===<br />
<br />
<br />
=== Services Issues ===<br />
<br />
<br />
----<br />
[[RAL Tier1 weekly operations fabric]]<br />
<br />
[[:Category:RAL_Tier1]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Fabric_20101129
RAL Tier1 weekly operations Fabric 20101129
2010-11-29T15:48:44Z
<p>James adams: </p>
<hr />
<div>== Developments ==<br />
* All:<br />
<br />
* Martin:<br />
** <br />
<br />
* Ian:<br />
** Project plan for preprod Quattorised SRMs<br />
** Plans with LHCb for cvmfs tests<br />
** Ongoing virtualisation<br />
** Job plan reviews<br />
<br />
* Tim:<br />
** <br />
<br />
* Jonathan:<br />
** <br />
<br />
* James A:<br />
** Experimented with OpenVPN access to management network.<br />
** Rewrote part of ncm-download and extended it to support minimum file age.<br />
** Worked on CASTOR facilities Quattor templates.<br />
<br />
* James T<br />
** <br />
<br />
* Cheney<br />
** <br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Decommissioning old batch systems.(R 27)<br />
** gdss380 still with Streamline for fix.(Crashed with single faulty drive)<br />
** gdss417 acceptance testing. (Crashed with single faulty drive)<br />
** gdss280 crashed again with replacement raid card borrowed from gdss338. (Testing) <br />
** gdss117 failed during test. (Probably raid card)<br />
** Hardware failure stats/graphs.<br />
** Streamline/areca disk servers crashed due to single faulty drive. (ongoing)<br />
** 4 days linux course (Tuesday - Friday)<br />
<br />
<br />
=== Operational Issues and Incidents ===<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Index<br />
! Description<br />
! Start<br />
! End<br />
! Severity<br />
! Affected VO(s)<br />
|-<br />
|}<br />
<br />
== Summary of plans for week ahead ==<br />
<br />
=== Scheduled and Cancelled Down Times ===<br />
<br />
Type=Down/At Risk/Cancelled entries in/planned to go to GOCDB<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Component<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Type<br />
|-<br />
|}<br />
<br />
=== Development priorities ===<br />
<br />
* All<br />
<br />
* Martin:<br />
**<br />
<br />
* Ian:<br />
** Project plan for preprod Quattorised SRMs<br />
** Public wiki page for cvmfs testing/setup<br />
** Test lates version of cvmfs<br />
** Ongoing virtualisation testing<br />
** Job plan reviews<br />
<br />
* Tim:<br />
** <br />
<br />
* Cheney<br />
** <br />
<br />
* Jonathan:<br />
** <br />
<br />
* James T:<br />
** <br />
<br />
* James A:<br />
** Work on CASTOR facilities upgrade with Chris.<br />
** Prepare zipwire for grid map file use and work on generation mechanism.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Continuous decommissioning old batch systems.(R 27)<br />
<br />
=== Absences ===<br />
* <br />
* Jonathan on partial retirement (not in on Monday and Friday)<br />
* Cheney - changed date for being off - now Nov 24th - early warning -likely to be off most of december - date subject to change - <br />
<br />
=== Fabric On-Call ===<br />
<br />
* Ian Collier<br />
<br />
=== Advanced Warning of Requirements and Blocking issues ===<br />
<br />
<br />
=== Services Issues ===<br />
<br />
<br />
----<br />
[[RAL Tier1 weekly operations fabric]]<br />
<br />
[[:Category:RAL_Tier1]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Fabric_20100405
RAL Tier1 weekly operations Fabric 20100405
2010-04-12T13:45:45Z
<p>James adams: </p>
<hr />
<div>=== Developments ===<br />
* All:<br />
<br />
* Martin:<br />
<br />
* Ian:<br />
<br />
* Tim:<br />
** sort out ADS mess after weekend<br />
** get dougal back into production after engineer intervention<br />
** Rhubarb - get new disk into operation<br />
<br />
* Cheney:<br />
** Building new robot controller<br />
** sort out atlasbackup space problem<br />
** cleaning the machine room<br />
** updates to docco on wiki<br />
** move cdbd03 to new chassis with proc and memory upgrade<br />
** replace psu kevin's array (preprod)<br />
** replace drive vtl<br />
* James T:<br />
** Tier1 tours for open day and OPB events.<br />
** Viglen09 testing.<br />
** A/L Wednesday and Thursday<br />
<br />
* Jonathan:<br />
** fixed atlasbackup problems on some nodes<br />
** new version of tier1-nagios-plugins RPM<br />
** killed many lftp processes on install02<br />
** Nagios configuration updates<br />
<br />
* James A:<br />
** Monday-Wednesday: Tier1 OPB and Open-Day Tours<br />
** Continued SL54 upgrade.<br />
** Some work on BMS-ARTEMIS-NAGIOS integration with Operations.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Decommissioning old batch systems.(R 27)<br />
** install01 faulty heatsink fan.(Intervention)<br />
<br />
=== Absences ===<br />
* Jonathan on partial retirement (not in on Monday and Friday); also 1 day Annual Leave (Wed 31/3)<br />
<br />
=== Operational Issues and Incidents ===<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Index<br />
! Description<br />
! Start<br />
! End<br />
! Severity<br />
! Affected VO(s)<br />
|-<br />
|}<br />
<br />
== Summary of plans for week ahead ==<br />
<br />
=== Scheduled and Cancelled Down Times ===<br />
<br />
Type=Down/At Risk/Cancelled entries in/planned to go to GOCDB<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Component<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Type<br />
|-<br />
|}<br />
<br />
=== Development priorities ===<br />
* All<br />
<br />
* Martin:<br />
<br />
* Ian:<br />
<br />
* Tim:<br />
** new ASD hardware into production<br />
** DMF disk areas newwd enlarging<br />
** T10K on castor - testing finalised<br />
** T10K on castor - New tape servers installed<br />
<br />
* Cheney:<br />
** job plan<br />
* James T:<br />
** Viglen09 testing<br />
** Chase up Streamline09 disk<br />
** TOASTER prep<br />
** SL5 disk server build<br />
<br />
* Jonathan:<br />
** Nagios configuration updates<br />
** continue reconfiguration of nagios06<br />
** continue work on disposal of old kit from A1 Upper machine room<br />
<br />
* James A:<br />
** Finish SL54 Upgrade on WNs.<br />
** Benchmark and calculate scaling factors for Viglen 2009 WNs so they can go into production.<br />
** Provide network cables for new ADS racks.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Continuous decommissioning old batch systems.(R 27)<br />
<br />
=== Absences ===<br />
<br />
* Everyone away on Bank Holiday Monday<br />
* Jonathan on partial retirement (not in on Monday and Friday)<br />
* James T on Annual Leave Tuesday.<br />
<br />
=== Fabric On-Call ===<br />
<br />
<br />
=== Advanced Warning of Requirements and Blocking issues ===<br />
<br />
=== Services Issues ===<br />
<br />
<br />
----<br />
[[RAL Tier1 weekly operations fabric]]<br />
<br />
[[:Category:RAL_Tier1]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Fabric_20100208
RAL Tier1 weekly operations Fabric 20100208
2010-02-08T14:50:02Z
<p>James adams: </p>
<hr />
<div>== Summary of week gone ==<br />
<br />
=== Developments ===<br />
* All:<br />
<br />
* Martin:<br />
** Minor procurements<br />
<br />
* Ian:<br />
** Carried out first CIP upgrade and prepared second.<br />
** Tested WN minor upgrade to SL5.4<br />
** Reviewed and updated some local Quattor docs<br />
** Made first contribution to SCDB Quattor docs at LAL<br />
** Reinstituted fabric automation steering group<br />
<br />
* James T:<br />
** Viglen '08 disk servers<br />
*** 5 installed with production config<br />
*** 22 finished testing over the weekend.<br />
** Quattorisation of disk servers<br />
*** OPN routing<br />
*** SSH lockdown<br />
*** rc.local tuning<br />
*** Script to import disk servers from hardware database<br />
<br />
* Jonathan:<br />
** Administrator on Duty (Wednesday)<br />
** restarted password cracker on enigma; updated iptables configuration to stop logging dropped packets<br />
** sorted out atlasbackup problems on various nodes<br />
** fixed ntpd process on lcgfts0423<br />
** NIS configuration changes<br />
** installed local NRPE and Ganglia configurations on ccse01<br />
** fixed access to Bfactory disk servers for userid bbdatsrv<br />
** Nagios configuration of updates<br />
** new versions of RPMs tier1-nagios-plugins, tier1-nrpe-config and tier1-sudo-config<br />
** worked on Quattor configuration of Nagios slave servers; reinstalled new slave server (found Quattor bug)<br />
<br />
* James A:<br />
** Prepared and shipped equipment for integration at supplier's premises in preparation for delivery.<br />
** QUATTORising various pieces of hand-configured functionality on quattor01 in order to be able to integrate SINDES.<br />
** Network cabling for CASTOR team.<br />
** Kickstart and Quattor trouble-shooting for various people.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Decommissioning old batch systems.(R 27)<br />
** gdss211 completed 7 days acceptance test.<br />
** gdss150 and gdss226 given back to castor. (Fixed)<br />
** gdss77 no display. (Found faulty memory) - Intervention<br />
** nc21 (lcg0280) found faulty memory. - Intervention<br />
** lcglb01 replaced drive with hotswap.<br />
** lcgvo-alice offline sectors started long smart test. (offline mode)<br />
** Moved streamline switches and other parts to (R56)logistics.<br />
** Replaced 9 faulty drives in Viglen 2008 disk servers with Viglen engineer.<br />
** Working on 2008 Disk servers and working nodes.<br />
** Working on gdss77, 282 and 364.<br />
<br />
=== Absences ===<br />
<br />
=== Operational Issues and Incidents ===<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Index<br />
! Description<br />
! Start<br />
! End<br />
! Severity<br />
! Affected VO(s)<br />
|-<br />
| <br />
| EMC arrays serving 3D/LFC/FTS databases made unstable by attempts to stabilise the Castor EMC arrays<br />
| Tuesday 6/0ct am<br />
| UPS issues to be fixed<br />
| Catastrophic<br />
| All<br />
|-<br />
|}<br />
== Summary of plans for week ahead ==<br />
<br />
=== Scheduled and Cancelled Down Times ===<br />
<br />
Type=Down/At Risk/Cancelled entries in/planned to go to GOCDB<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Component<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Type<br />
|-<br />
|}<br />
<br />
=== Development priorities ===<br />
* All<br />
<br />
* Martin:<br />
** Minor procurements<br />
<br />
* Ian:<br />
** Further research into virtualization platforms<br />
** Plan for rolling upgrade of WNs to SL5.4<br />
** Further work on integration of Castor fabric management into Fabric team<br />
<br />
* James T:<br />
** Quattorisation of disk servers.<br />
*** Get CASTOR info directly from Overwatch<br />
*** Testing<br />
** Re-install latest tranche (22) of Viglen '08 disk servers.<br />
** Writing nagios checks<br />
<br />
* Jonathan:<br />
** Administrator on Duty (Wednesday)<br />
** implement cron job with checks to run daily test restores of home filesystem<br />
** complete work on installing Nagios slave server via Quattor<br />
** Nagios configuration updates<br />
<br />
* James A:<br />
** Continue with SINDES integration where possible.<br />
** Spend some time developing the Hardware Database with Kash.<br />
** Prepare machine room to accept deliveries of new hardware.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** lcgce07 drive replacement. (Hot swap)<br />
** gdss77 and gdss86 replace 4x 1gb memory. (recently bought by Martin)<br />
** Continuous work (memory replacement) with Cheney.<br />
** Viglen 2006 (8) disk servers for decommissioning/prepod. (Label and configure) <br />
** Continuous decommissioning old batch systems.(R 27)<br />
** Continuous working on 2008 disk servers and working nodes.<br />
** Continuous working on gdss77, 130, 282 and 364.<br />
<br />
=== Absences ===<br />
* Jonathan - as from this week changing work pattern to 3 days per week (normally Tuesday, Wednesday, Thursday)<br />
<br />
=== Fabric On-Call ===<br />
<br />
Ian Mon-Sun<br />
<br />
=== Advanced Warning of Requirements and Blocking issues ===<br />
<br />
* Unable to proceed with Atlas TAG migration to 64bit due to arrays being used for 3D systems while EMC kit is flakey.<br />
<br />
=== Services Issues ===<br />
<br />
* Various requests for hardware.<br />
** Working on hardware provision for Services team testbeds.<br />
<br />
[[:Category:RAL_Tier1]]<br />
<br />
[[RAL Tier1 weekly operations fabric]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Fabric_20100201
RAL Tier1 weekly operations Fabric 20100201
2010-02-01T13:31:01Z
<p>James adams: </p>
<hr />
<div>== Summary of week gone ==<br />
<br />
=== Developments ===<br />
* All:<br />
** Strategy meeting<br />
<br />
* Martin:<br />
** Minor procurements<br />
<br />
* Ian:<br />
** Upgraded CIp filesystem layouts<br />
** Upgraded batch server binaries<br />
** Upgraded kernels on SL5 WNs<br />
** Planning for handover of fabric management for Castor systems<br />
<br />
* James T:<br />
** "Mega intervention" preparation/documentation<br />
** Mega Intervention<br />
** Fisrt Viglen '08 disk servers out of testing.<br />
** Ongoing quattorisation of disk servers.<br />
** Primary on call<br />
<br />
* Jonathan:<br />
** added new NIS groups and create new pool accounts<br />
** checked SSH problem on lcgdb05; removed special userids oracle, lsfadmin, stage and corresponding groups oinstall, lsfadmin, st from NIS (NIS entries sometimes take precedence over local entries whatever the setting of /etc/nsswitch,conf; this can cause system problems)<br />
** updated RPMs on core systems and rebooted where required<br />
** reconfigured and restarted ntpd on lcgvo0425 (updating ntp RPM can sometimes loose the local NTP configuration)<br />
** Nagios configuration updates<br />
** reinstalled and reconfigured nagios04 after disk replacement<br />
<br />
* James A:<br />
** Networking preparations ahead of mega-intervention.<br />
** Added snapshotting feature to cacti weather-map.<br />
** Finished cabling IPMI ports in castor racks B&E.<br />
** Updated certificate on t1pg0373.<br />
** Fixed bug in check_spma for handling rotated logs.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Decommissioning old batch systems.(R 27)<br />
** gdss211 running 7 days acceptance test.<br />
** gdss70 given back to castor. (Fixed)<br />
** gdss77 no display. (Found faulty memory) - Intervention<br />
** gdss87 given back to castor for testing.<br />
** nagios04 replaced drive.<br />
** gdss170 given back to castor.<br />
** Moved switches and cables from R27 with James A.<br />
** Working on 2008 Disk servers and working nodes.<br />
** Working on gdss77, 282 and 364.<br />
<br />
=== Absences ===<br />
* Jonathan (1/2 day, domestic reasons)<br />
<br />
=== Operational Issues and Incidents ===<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Index<br />
! Description<br />
! Start<br />
! End<br />
! Severity<br />
! Affected VO(s)<br />
|-<br />
| <br />
| EMC arrays serving 3D/LFC/FTS databases made unstable by attempts to stabilise the Castor EMC arrays<br />
| Tuesday 6/0ct am<br />
| UPS issues to be fixed<br />
| Catastrophic<br />
| All<br />
|-<br />
|}<br />
== Summary of plans for week ahead ==<br />
<br />
=== Scheduled and Cancelled Down Times ===<br />
<br />
Type=Down/At Risk/Cancelled entries in/planned to go to GOCDB<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Component<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Type<br />
|-<br />
|}<br />
<br />
=== Development priorities ===<br />
* All<br />
<br />
* Martin:<br />
** Minor procurements<br />
<br />
* Ian:<br />
** Upgrading and reconfiguring CIPs<br />
** Work with Catalin on Quattorising further grid services nodes<br />
** Quattor documentation<br />
** (Re-)Instituting steering group for Fabric automation project<br />
** Researching Virtualisation platform options<br />
<br />
* James T:<br />
** Ongoing quattorisation of disk servers.<br />
** Install first Viglen '08 disk servers.<br />
** Writing nagios checks<br />
** Apply WAN tuning<br />
<br />
* Jonathan:<br />
** implement cron job with checks to run daily test restores of home filesystem<br />
** complete work on installing Nagios slave server via Quattor<br />
** Nagios configuration updates<br />
<br />
* James A:<br />
** Two days of SINDES integration.<br />
** Connect uplinks to CASTOR IPMI switches.<br />
** Ensure IPMI on CASTOR boxes comes up.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** lcglb01 drive replacement. (Hot swap)<br />
** Continuous work (memory replacement) with Cheney.<br />
** Continuous decommissioning old batch systems.(R 27)<br />
** Continuous working on 2008 disk servers and working nodes.<br />
** Continuous working on gdss77, 282 and 364.<br />
<br />
=== Absences ===<br />
* Jonathan - as from week beginning 8th February, changing work pattern to 3 days per week (normally Tuesday, Wednesday, Thursday)<br />
<br />
=== Fabric On-Call ===<br />
<br />
Ian Primary on call<br />
<br />
=== Advanced Warning of Requirements and Blocking issues ===<br />
<br />
* Unable to proceed with Atlas TAG migration to 64bit due to arrays being used for 3D systems while EMC kit is flakey.<br />
<br />
=== Services Issues ===<br />
<br />
* Various requests for hardware.<br />
** Working on hardware provision for Services team testbeds.<br />
<br />
[[:Category:RAL_Tier1]]<br />
<br />
[[RAL Tier1 weekly operations fabric]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Fabric_20100118
RAL Tier1 weekly operations Fabric 20100118
2010-01-18T14:30:46Z
<p>James adams: </p>
<hr />
<div>== Summary of week gone ==<br />
<br />
=== Developments ===<br />
* All:<br />
<br />
* Martin:<br />
** Procurements<br />
** GridPP4<br />
** Networking plans for capacity procurements<br />
** Intervention on lcgdb14<br />
<br />
* Ian:<br />
** Work on Quattor config of vobox<br />
** Planning for batch server upgrade and other interventions<br />
** Planning update of Quattor server<br />
<br />
* James T:<br />
** Fixed two problems with Ganglia<br />
*** The data sources for the Miscellaneous cluster had been decommissioned.<br />
*** Workers_SL5 graphs were fluctuating wildly due to wrongly configured Workers_SL4.<br />
** Quattorisation of disk servers<br />
*** fsprobe added<br />
*** puppet added<br />
*** Work on processing errata<br />
** Various system updates<br />
** Dry run of procedures prior to "Mega Intervention".<br />
** Progress meeting with Viglen on disk testing. All machines now in testing, complete mid-February.<br />
<br />
* Jonathan:<br />
** updated CSFadduser script (in /usr/local/sbin on wyatt) for new Tier1 home directory and added new userids for Castor evaluation<br />
** corrected backup problems on several nodes<br />
** followed up chkrootkit problem on afs2<br />
** updated RPMs on several nodes<br />
** investigated Callout problems on several nodes<br />
** Nagios configuration updates<br />
** 2 days out (home emergency)<br />
<br />
* James A:<br />
** Finalised plan for SINDES implementation.<br />
** Worked on new user contact database.<br />
** Moved castoradm2 and castoradm3 from A1 upper to A5 lower.<br />
** Begun last of CASTOR rack IPMI cabling.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Decommissioning old batch systems.(R 27)<br />
** lcgdb14 replaced memory and motherboard by Engineer. (Fixed)<br />
** gdss134 given back to castor.<br />
** Produce graphs of hardware failures.<br />
** gdss105 and 171 given back to castor.<br />
** Working on 2008 Disk servers and working nodes.<br />
** Working on gdss66, 70, 282, 364 and 380.<br />
<br />
=== Absences ===<br />
* Jonathan (2 days - home emergency)<br />
<br />
=== Operational Issues and Incidents ===<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Index<br />
! Description<br />
! Start<br />
! End<br />
! Severity<br />
! Affected VO(s)<br />
|-<br />
| <br />
| EMC arrays serving 3D/LFC/FTS databases made unstable by attempts to stabilise the Castor EMC arrays<br />
| Tuesday 6/0ct am<br />
| UPS issues to be fixed<br />
| Catastrophic<br />
| All<br />
|-<br />
|}<br />
<br />
== Summary of plans for week ahead ==<br />
<br />
=== Scheduled and Cancelled Down Times ===<br />
<br />
Type=Down/At Risk/Cancelled entries in/planned to go to GOCDB<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Component<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Type<br />
|-<br />
|}<br />
<br />
=== Development priorities ===<br />
* All<br />
<br />
* Martin:<br />
** Minor procurements<br />
** planning and chanage control activities for pre-datataking period<br />
<br />
* Ian:<br />
** Update Quattor server<br />
** Work with James A to deploy and test Sindes on Quattor server<br />
** Implement CIP config update on Thursday<br />
** Virtualisation platform planning<br />
<br />
* James T:<br />
** Document procedure for "mega intervention". <br />
** Ongoing quattorisation of disk servers.<br />
** CRISTAL2 support group.<br />
** ATLAS WAN tuning for Brian.<br />
** Progress meeting with Viglen.<br />
** Updates to some systems.<br />
<br />
* Jonathan:<br />
** work on test restore of home filesystem subdirectory<br />
** final checks of change to restrict SSH login on disk servers<br />
** complete work on installing Nagios slave server via Quattor<br />
** update RPMs on various servers<br />
** Nagios configuration updates<br />
<br />
* James A:<br />
** Rolling out SINDES.<br />
** Working on user contact database.<br />
** Finishing IPMI cabling.<br />
** Working on forwarding BMS alerts to Nagios.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** gdss380 given back to castor.<br />
** afs2 drive failure.<br />
** Continuous decommissioning old batch systems.(R 27)<br />
** Continuous working on 2008 disk servers and working nodes.<br />
** Continuous working on gdss66, 70, 282, 364 and 380<br />
<br />
=== Absences ===<br />
* Kashif (Thursday - A/L)<br />
<br />
=== Fabric On-Call ===<br />
<br />
JamesT Monday-Thursday<br />
Ian Friday-Sunday<br />
<br />
=== Advanced Warning of Requirements and Blocking issues ===<br />
<br />
* Unable to proceed with Atlas TAG migration to 64bit due to arrays being used for 3D systems while EMC kit is flakey.<br />
<br />
=== Services Issues ===<br />
<br />
* Various requests for hardware.<br />
** Working on hardware provision for Services team testbeds.<br />
<br />
[[:Category:RAL_Tier1]]<br />
<br />
[[RAL Tier1 weekly operations fabric]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Fabric_20091207
RAL Tier1 weekly operations Fabric 20091207
2009-12-07T15:14:07Z
<p>James adams: </p>
<hr />
<div>== Summary of week gone ==<br />
<br />
=== Developments ===<br />
* All:<br />
<br />
* Martin:<br />
<br />
* Ian:<br />
** Finalised new flex license servers for LSF<br />
** Further Quattor tutorial for Cheney & Matt<br />
** Provided physical hardware for CIP<br />
***Went through various configuration options<br />
<br />
* James T:<br />
** Quattorisation of disk servers<br />
** Primary on call Mon - Thurs<br />
** Viglen disk swap out support<br />
** [[RAL_Tier1_Incident_20091130|Post-mortem]] on gdss138<br />
<br />
* Jonathan:<br />
** maintained NIS netgroup<br />
** corrected atlasbackup problems for a few hosts<br />
** Administrator on Duty (Wednesday)<br />
** unmounted /home/csf from lcg0617/618<br />
** Nagios configuration updates<br />
** system tuning of nagger to try to reduce scheduling queue<br />
** installed RPM mrtg on nagger and added configuration to collect performance statistics from Nagios (see http://nagger.gridpp.rl.ac.uk/mrtg/nagios-[a-n].html at present)<br />
<br />
* James A:<br />
** Achieved a working preliminary SINDES server.<br />
** Upgraded Cacti on thor.gridpp.rl.ac.uk<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** gdss138 double disks failure. (Intervention)<br />
** gdss149, 162, 163 and 367 given back to castor.<br />
** gdss77 kernel panic possibly faulty memory. (Intervention)<br />
** gdss139 given back to castor.<br />
** Moved 3 batch systems from R27 to HPD room (CV 2005 rack) with MJB.<br />
** Working on 2008 Disk servers and working nodes.<br />
** Working on gdss77, 138 and 282.<br />
<br />
=== Absences ===<br />
<br />
* Jonathan: S/L Monday<br />
* Jonathan: A/L Thursday am<br />
<br />
=== Operational Issues and Incidents ===<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Index<br />
! Description<br />
! Start<br />
! End<br />
! Severity<br />
! Affected VO(s)<br />
|-<br />
| <br />
| EMC arrays serving 3D/LFC/FTS databases made unstable by attempts to stabilise the Castor EMC arrays<br />
| Tuesday 6/0ct am<br />
| UPS issues to be fixed<br />
| Catastrophic<br />
| All<br />
|-<br />
|<br />
| Gdss138 double disk failure: two drives failed in quick succession (30 minutes)<br />
| Monday 0530-0600<br />
| Ongoing<br />
| Severe<br />
| LHCb Dst data. Data loss confirmed<br />
|}<br />
<br />
== Summary of plans for week ahead ==<br />
<br />
=== Scheduled and Cancelled Down Times ===<br />
<br />
Type=Down/At Risk/Cancelled entries in/planned to go to GOCDB<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Component<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Type<br />
|-<br />
| Cacti (http://cacti.gridpp.rl.ac.uk)<br />
| Upgrade Cacti software. Subject to team manager's approval of plan.<br />
| Tuesday 2009-12-08 13:00<br />
| Tuesday 2009-12-08 17:00<br />
| none<br />
| At Risk<br />
|}<br />
<br />
=== Development priorities ===<br />
* All<br />
** Work on evacuating A1 Upper (Castor LSF/FlexLM triplet)<br />
<br />
* Martin:<br />
<br />
* Ian:<br />
** Reconfigure physical CIP again<br />
** Implement second CIP with Quattor (T2K)<br />
** Start work on Quattor managed glite 3.2 vobox with Catalin<br />
** Assist with new disk servers as required<br />
** Incorporation of latest QWG template updates<br />
<br />
* James T:<br />
** Quattorisation of disk servers<br />
** Remove nincom as Ganglia data source for Services_Monitoring<br />
** Script to compare Overwatch with real CASTOR status<br />
** TOASTER preparation<br />
<br />
* Jonathan:<br />
** Quattor implementation for Nagios slave<br />
** security updates to disk servers to prevent general user logins<br />
** Nagios configuration updates<br />
<br />
* James A:<br />
** Continue with SINDES.<br />
** Upgrade Cacti on cacti.gridpp.rl.ac.uk, install plugins and apply internal patches.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** gdss339 kernel panic.(Intervention)<br />
** gdss138 double disk failure. (Intervention)<br />
** Decommissioning old batch systems with Production Team.<br />
** Continuous working on 2008 disk servers and working nodes.<br />
** Continuous working on gdss77, 138, 282 and 339.<br />
<br />
=== Absences ===<br />
<br />
* Kashif: A/L Wednesday<br />
<br />
=== Fabric On-Call ===<br />
<br />
* Mon-Sun: Ian Primary on call<br />
<br />
=== Advanced Warning of Requirements and Blocking issues ===<br />
<br />
* Unable to proceed with Atlas TAG migration to 64bit due to arrays being used for 3D systems while EMC kit is flakey.<br />
<br />
=== Services Issues ===<br />
<br />
* Various requests for hardware.<br />
** Working on various hardware requests for Services team.<br />
<br />
[[:Category:RAL_Tier1]]<br />
<br />
[[RAL Tier1 weekly operations fabric]]</div>
James adams
https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_Fabric_20091012
RAL Tier1 weekly operations Fabric 20091012
2009-10-12T15:04:15Z
<p>James adams: </p>
<hr />
<div>== Summary of week gone ==<br />
<br />
=== Developments ===<br />
* All<br />
<br />
* Martin:<br />
** Disk procurement ITT evaluation<br />
** Depolyment of 3D databases onto old hardware due to power feed problems making the EMC arrays unstable<br />
** Meeting with Seagate about disk problems<br />
<br />
* Ian:<br />
<br />
* James T:<br />
** Viglen testing:<br />
*** Meeting with <br />
*** Drives swapped for a different batch in 10 machines (220 drives).<br />
*** Logs captured on 2 October by Seagate showed further issues so they issued another updated firmware.<br />
*** More logs captured from timed-out drives on Thursday 8th.<br />
*** Tested racks with the functional earth removed - same problems.<br />
** ''user_xattr'' mount option rolled out to all CASTOR disk servers.<br />
** Created ''Storage_CASTOR_Gen'' ganglia cluster for Brian (former CASTOR team blocking issue).<br />
** Cleaned up some fabric tickets.<br />
** DNS request for repack server.<br />
** HEPSYSMAN on Wednesday 7th (talked about Tier1 storage).<br />
<br />
<br />
* Jonathan:<br />
** configured nagios@nagger.gridpp.rl.ac.uk as PBS operator <br />
** worked on migration of user home filesystems to new server<br />
** updated RPMs on core servers and rebooted where required<br />
** updated wiki documentation referring to change Nagios master server to nagger<br />
** added new users to Tier1 and AFS<br />
** added new top directory superb for Babar (RT #52070)<br />
** Nagios configuration updates on servers and clients<br />
<br />
* James A:<br />
** Lots of work on BatchWorkers in QUATTOR.<br />
** Brought SL5 farm to 90% of KSI2K Capacity.<br />
** Shrunk SL4 farm respectively.<br />
** Made some minor progress with SINDES.<br />
** Some changes to ARTEMIS for UPS room.<br />
** Removed AtlasBackup from base machine template in QUATTOR<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** gdss354 fixed and back in production.<br />
** gdss218 wrong way round backplane cables. (Fixed)<br />
** gdss126 double disks failure. Completed verifying array.<br />
** Seagate 220 drives dispatched, given to Seagate Engineer.<br />
** Completed adding additional raid cards in v06 (Castor disk servers).<br />
** Working on 2008 Disk servers and working nodes.<br />
** Working on gdss67, 86, 126 and 170.<br />
<br />
=== Operational Issues and Incidents ===<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Index<br />
! Description<br />
! Start<br />
! End<br />
! Severity<br />
! Affected VO(s)<br />
|-<br />
| <br />
| EMC arrays serving 3D/LFC/FTS databases made unstable by attempts to stabilise the Castor EMC arrays<br />
| Tuesday am<br />
| not in site<br />
| Catastrophic<br />
| All<br />
|-<br />
|}<br />
<br />
== Summary of plans for week ahead ==<br />
<br />
=== Scheduled and Cancelled Down Times ===<br />
<br />
Type=Down/At Risk/Cancelled entries in/planned to go to GOCDB<br />
<br />
{| border=1 align=center<br />
|- bgcolor="#7c8aaf"<br />
! Component<br />
! Description<br />
! Start<br />
! End<br />
! Affected VO(s)<br />
! Type<br />
|-<br />
|}<br />
<br />
=== Development priorities ===<br />
* All<br />
<br />
* Martin:<br />
** Disk procurement ITT evaluation<br />
** CPU procurement ITT clarifications<br />
<br />
* Ian:<br />
<br />
* James T:<br />
** Assign machines for deployment.<br />
** Send out requests for people to complete CRISTAL 2 feedback forms.<br />
** Viglen testing:<br />
*** Continue testing latest firmware.<br />
*** Prepare to hand over to someone else.<br />
<br />
* Jonathan:<br />
** work on migration of Tier1 home filesystem to new server<br />
** work on installing Nagios slave servers using Quattor<br />
** Nagios configuration updates as required<br />
<br />
* James A:<br />
** Continue pushing forward with SINDES.<br />
** Take over disk issues from James T.<br />
** Integrate of BMS alerts into ARTEMIS data stream.<br />
<br />
* Kash:<br />
** Drive replacement.<br />
** Fixing broken WNs.<br />
** Continuous working on 2008 disk servers and working nodes.<br />
** Continuous Working on gdss67, 86, 126 and 170.<br />
<br />
=== Absences ===<br />
<br />
* James T<br />
** James T on A/L from Thursday 15th until Monday November 2nd.<br />
<br />
=== Fabric On-Call ===<br />
<br />
* Mon-Fri: <br />
<br />
=== Advanced Warning of Requirements and Blocking issues ===<br />
<br />
<br />
=== Services Issues ===<br />
<br />
* Various requests for hardware.<br />
<br />
[[:Category:RAL_Tier1]]<br />
<br />
[[RAL Tier1 weekly operations fabric]]</div>
James adams
https://www.gridpp.ac.uk/wiki/Farm_Shutdown
Farm Shutdown
2007-11-14T11:12:42Z
<p>James adams: </p>
<hr />
<div>#REDIRECT [[RAL Tier1 Farm Shutdown]]<br />
</div>
James adams