https://www.gridpp.ac.uk/w/index.php?title=RAL_Tier1_weekly_operations_castor_12/07/2019&feed=atom&action=historyRAL Tier1 weekly operations castor 12/07/2019 - Revision history2024-03-28T17:58:55ZRevision history for this page on the wikiMediaWiki 1.22.0https://www.gridpp.ac.uk/w/index.php?title=RAL_Tier1_weekly_operations_castor_12/07/2019&diff=20193&oldid=prevRob Appleyard 7f7797b74a at 10:39, 12 July 20192019-07-12T10:39:00Z<p></p>
<table class='diff diff-contentalign-left'>
<col class='diff-marker' />
<col class='diff-content' />
<col class='diff-marker' />
<col class='diff-content' />
<tr style='vertical-align: top;'>
<td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td>
<td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 10:39, 12 July 2019</td>
</tr><tr><td colspan="2" class="diff-lineno">Line 37:</td>
<td colspan="2" class="diff-lineno">Line 37:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>** Some problems with ET.</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>** Some problems with ET.</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* Configured DUNE to work with WLCG, tests pass.</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* Configured DUNE to work with WLCG, tests pass.</div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">* Decommissioned a bunch of old HyperV VMs.</ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>== Operation problems ==</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>== Operation problems ==</div></td></tr>
</table>Rob Appleyard 7f7797b74ahttps://www.gridpp.ac.uk/w/index.php?title=RAL_Tier1_weekly_operations_castor_12/07/2019&diff=20192&oldid=prevRob Appleyard 7f7797b74a: Created page with "[https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_castor Parent article] == Standing agenda == 1. Achievements this week 2. Problems encountered this week 3. What..."2019-07-12T10:18:00Z<p>Created page with "[https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_castor Parent article] == Standing agenda == 1. Achievements this week 2. Problems encountered this week 3. What..."</p>
<p><b>New page</b></p><div>[https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_castor Parent article]<br />
<br />
== Standing agenda ==<br />
<br />
1. Achievements this week<br />
<br />
2. Problems encountered this week<br />
<br />
3. What are we planning to do next week?<br />
<br />
4. Long-term project updates (if not already covered)<br />
<br />
5. Special topics<br />
<br />
6. Actions<br />
<br />
7. Review Fabric tasks<br />
1. [https://wiki.e-science.cclrc.ac.uk/web1/bin/view/EScienceInternal/FabricTasksFromDataServices Link]<br />
<br />
8. AoTechnicalB<br />
<br />
9. Availability for next week<br />
<br />
10. On-Call<br />
<br />
11. AoOtherB<br />
<br />
== Achievements this week ==<br />
<br />
* Cleanup of LHCb data from lhcbDst ongoing.<br />
* Sorting out personal proxy being used to support CASTOR xrootd functional test.<br />
** Test is currently failing, as the proxy ran out.<br />
** GP to work with CC to figure this out.<br />
*** Action on Rob and Brian to understand the callout system, what it is supposed to do, and develop a plan of what it should do.<br />
** Not completed, but expected soon.<br />
* New Facilities headnodes on VMWare have been tested in VCert and work for Diamond<br />
** Some problems with ET.<br />
* Configured DUNE to work with WLCG, tests pass.<br />
<br />
== Operation problems ==<br />
<br />
* Facilities tape drives went down for about an hour for a handbot replacement on Thursday morning.<br />
<br />
== Plans for next few weeks ==<br />
<br />
* Decommission lhcbDst hardware.<br />
* Brian C is currently testing StorageD/ET on the new robot<br />
* Replace Facilities headnodes with VMs.<br />
** Waiting until Kevin is back from holiday.<br />
** Scheduled for the 30th July.<br />
<br />
== Long-term projects ==<br />
<br />
* New CASTOR disk servers currently with Martin.<br />
* Migration of name server to VMs on 2.1.17-xx is waiting until aliceDisk is decommissioned.<br />
* CASTOR disk server migration to Aquilon.<br />
** Agreed a testing plan with Fabric<br />
* Facilties headnode replacement:<br />
** SL7 VM headnodes are being tested<br />
* Turn VCert into a facilities test instance.<br />
<br />
== Actions ==<br />
<br />
* AD wants us to make sure that experiments cannot write to that part of namespace that was used for d1t0 data: namespace cleanup/deletion of empty dirs. <br />
** Some discussion about what exactly is required and how this can be actually implemented.<br />
** CASTOR team proposal is either:<br />
*** to switch all of these directories to a fileclass with a requirement for a tape copy but no migration route; this will cause an error whenever any writes are attempted.<br />
*** to run a recursive nschmod on all the unneeded directories to make them read only.<br />
<br />
== Staffing ==<br />
<br />
* Everybody in<br />
<br />
== AoB ==<br />
<br />
* Discussion over how to do the upgrade of Facilities<br />
** Idea 1: As planned. Upgrade CASTOR DB schema from 2.1.16 to 2.1.17 and bring in new headnodes as one intervention<br />
*** Pro: Smallest number of operations<br />
*** Con: Never upgraded 2.1.16 to 2.1.17<br />
** Idea 2: Create a new DB 2.1.17 stager schema on Bellona. Repoint CASTOR stager to use that.<br />
*** Pro: This is what we did on Tier 1 (But that was because of necessity rather than choice)<br />
*** Con: Complexity of two schemas.<br />
*** Con: More disruptive to users.<br />
*** Con: CASTOR team would need to add all the config entries and configure everything from scratch.<br />
** Idea 3: Split the interventions. Move to new headnodes, running 2.1.16, then upgrade them to 2.1.17<br />
*** Pro: Split into discrete steps, easy to debug issues<br />
*** Con: Possible to end up debugging issues specific to 2.1.16/SL7 which is a config we do not expect to use long term.<br />
** Meeting concluded on idea 1, with the need to do a dress rehearsal upgrade from 2.1.16 to 2.1.17.<br />
<br />
== On Call ==<br />
<br />
RA on Call</div>Rob Appleyard 7f7797b74a