Operations Bulletin 090913

From GridPP Wiki
Revision as of 10:16, 9 September 2013 by Jeremy coles (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Bulletin archive


Week commencing 2nd September 2013
Task Areas
General updates

Tuesday 3rd September

  • There is a call (deadline this Thursday) for expressions of interest to attend the WLCG workshop in November.
  • We are joined by 2 new year in industry students: Sam Worley and Mohit Mittal. Please be patient with them as they learn the how to direct tickets!
  • Progress with the HEPiX puppet working group was expected via an announcement on hepix-users last week (Bob Jones).
  • The GOCDB v5 release, which was provisionally scheduled for the 2nd of September, was delayed while some newly encountered issues are resolved (compatibility problems and hardware issues experienced by the server hosting the [https://gocdb-test.esc.rl.ac.uk/v5 GOCDB test

instance]). The delay will be of order 3 weeks.

  • The agenda for next week's GDB is now online.


Tuesday 27th August

  • An EGI task force has started (see kickoff agenda) to develop the use of CVMFS by non-LHC VOs.Task force progress is being tracked in the associated CVMFS wiki page. Webinar at 10am on 5th September.
  • There is an EGI OMB today (agenda). Remember the SHA2 deadline is 1st October and tickets are tracking upgrade plans.

Tuesday 20th August

  • An EMI-2 SL5 tarball WN has been put into CVMFS thanks to Matt and Jakob (see GGUS:96030 for further information).
  • SL6 s/w areas have been discussed several times - the conclusion is that it is too late for a single approach to be adopted.
  • The WLCG T2 availability/reliability figures are now final.
  • A provisional date of 2nd September has been scheduled for the GOCDB v5release. This is a major release which involves migration of the data to a new domain model that supports different DBs. The change involves the use of new primary keys. For more (testing) information see here.
WLCG Operations Coordination - Agendas

Tuesday 2nd September

  • Middleware
    • New BDII release in the latest EMI-2/3 update, including better GLUE-2 support and security fixes. Sites should update all their BDII instances
    • New CVMFS version released for a security fix. Sites should upgrade or at least apply the hot fix in the above twiki
    • perfSONAR: sites should upgrade to the latest version, fixing many deployment problems
    • The end of support for dCache 1.9.12 has been postponed to September 30 due to a delay in releasing the SHA-2 compliant version in the dCache 2.2 series.
    • Consult https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions
  • SHA-2
    • Discussion mostly dedicated to the experiments testing status. Atlas and LHCb have tested the services but not job submission yet. All experiments have been encouraged to test this.
  • SL6
    • T2 Done: 49/129 (Alice 11/39, Atlas 28/89, CMS 22/65, LHCb 13/45) -> 80/129 still to be done.
    • HS06: Reminder that sites are requested to run HS06 benchmark and update the value in the BDII. Increased values might be discussed at the WLCG MB.
    • EMI-3: voms-clients have been fixed and the latest version is in the PT repository but not in EMI-3 yet. Both CMS and Atlas work on DPM/dcache sites with this patch. (QMUL might want to give an update on Storm when they upgrade)
    • UK status: Liverpool to be finished soon, Bham in downtime to upgrade this week, Bristol and Sussex should be done by the 15/9/2013, RALPP 20/09/2013 and QMUL, Lancaster, UCL 30/09/2013
  • glexec
    • 55 sites still to respond they have attached the installation to SL6 upgrade.

Monday 12th August

  • There have been no recent meetings. The next is on 29th August.
      • Sites monitoring requirements: SUM tests not representing the real experiment status for example.
Tier-1 - Status Page

Tuesday 3rd September

  • During the day last Wednesday (28th August) there were problems with the Top-BDIIs.
  • Testing of alternative batch system (Condor/ARC CEs/Sl6) proceeding.
Storage & Data Management - Agendas/Minutes

Tuesday 15th July

  • Three sites are now run all UK FTS traffic via 'FTS3' service as a test. Mostly successful; (small issue with a few US sites to be resolved before taking tests further.)

Tuesday 28th May

  • The 'Big Data' agenda is being compiled here. There is also now a suggestion for a cross disciplinary clouds and virtualisation workshop in July - the idea is 'in progress' but no more detail is yet available.


Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06

Tuesday 13th August

Tuesday 23rd July

  • Sites moving to SL6 are reminded of the need to re-benchmark their WNs. Some sites have updated the wiki already and provide an idea of the performance change.
  • There is an ongoing PMB discussion about the timeline for the next Tier-2 hardware tranche. Please let Pete or Jeremy know if your site will benefit from a spend this financial year.

Tuesday 30th April

  • A discussion is starting about how to account/reward disk that is reallocated to LHCb. By way of background, LHCb is changing its computing model to use more of Tier-2 sites. They plan to start with a small number of big/good T2 sites in the first instance, and commission them as T2-Ds with disk. Ideally such sites will provide >300TB but for now may allocate 100TB and build it up over time. Andrew McNab is coordinating the activity for LHCb. (Note the PMB is already aware that funding was not previously allocated for LHCb disk at T2s).

Tuesday 12th March

  • APEL publishing stopped for Lancaster, QMUL and ECDF

Tuesday 12th February

  • SL HS06 page shows some odd ratios. Steve says he now takes "HS06 cpu numbers direct from ATLAS" and his page does get stuck every now and then.
  • An update of the metrics page has been requested.
Documentation - KeyDocs

See the worst KeyDocs list for documents needing review now and the names of the responsible people.

Tuesday 3 September 2013

  • Proposal for "Instant UI", with the aim to produce a suite of documentation and software that will enable a new user to set up a UI and join the grid with the minimum of hassle. Doc will show for to admin a UI that can be used to submit jobs and retrieve output for a given set of users belonging to a given set of VOs. "Instant UI" is currently in consulation phase with GridPP admin community.

Tuesday 27th August

  • The document status was reviews at a core ops meeting last Thursday. Several documents are to be reassigned and some removed.
  • Looking at VO lifecycle.

KeyDocs monitoring status: Grid Storage(7/0) Documentation(3/0) On-duty coordination(3/0) Staged rollout(3/0) Ticket follow-up(3/0) Regional tools(3/0) Security(3/0) Monitoring(3/0) Accounting(3/0) Core Grid services(3/0) Wider VO issues(3/0) Grid interoperation(3/0) Cluster Management(1/0) (brackets show total/missing)

Interoperation - EGI ops agendas

Monday 2nd September

  • Yesterday's agenda. Attended by David and Raul.

1. Middleware releases and staged rollout 1.1 News from URT StoRM 1.11.2 still not released The Product Team found an issue (fixed) during the testing and the fix is currently under testing (within days if everything goes well).

Gridsite 2.1.3(UMD-3) and 1.7.28(UMD-2) To be released during this week by PT Not in EPEL yet (gridsite PT is not releasing yet in EPEL) Single fix for a problem affecting delegation

DPM v 1.8.7 Need to remove packages from the EMI repositories before releasing in EPEL

dCache 2.2.15 released Not the SHA-2 compliant version for UMD-2 2.2.16 will include sha-2 support and will be released tomorrow (Tuesday)

Globus 3.2.2 -Bug fix release from IGE -GridWay -GSI-SSHTerm -BES-GRAM -Globus Info Provider Service


1.2 VOMS: A question was asked, "has voms been tested with IPv6 now?" - the answer was not yet, as there was no IPv6 machines in the testbed - this was to be followed up.

1.3 UMD-3 to be released ~next week, waiting a couple of days to allow for the possibility of StoRM update to be tested.

2.2 Host name in the host certificates

As reported in several GGUS tickets (Rules for issuing certificates for hosts with an alias, Problems related to a myproxy service). Host certificates must have the hostname used to reach the service in either the CN or DNS fields (Common Name and Alternate Subject Name). Please make sure to provide all the needed information to your CA when requesting a new host certificate.

The only service which this affects in the UK is lcgrbp01.gridpp.rl.ac.uk ; Peter noted that he would issue tickets shortly (if not by now).

2.A. Peter Solagna reported on the Central emergency suspension project, - he requested that NGIs report back to him about whether each has a NGI ARGUS instance for use by the framework, so that the central ARGUS server can have a limited set of servers talking to it.

As a reminder, register ARGUS servers in GOCDB.

2.3 UserDN publication in the accounting records Reminder: follow with sites generating alarms for non published User DN. Currently only few sites have 'critical' failures on that check.

2.4 CVMFS webinar They trailed Catalin's webinar as discussed on TB-SUPPORT

2.4 AAI workshop at TF Madrid (Authentication Authorization Infrastructure) AAI workshop on Tuesday September 17th at 11:00

AOB Monday 16th September presentation by Product Teams during Technical Forum.

3.2 Next meeting September 23rd, h14:00 Amsterdam time - might be modified because of Friday 27th Meeting OMB meeting

No meeting in 2 weeks because of TF.


gLite support calendar.


Monitoring - Links MyWLCG

Tuesday 23rd July

Tuesday 18th June

  • David C is taking feedback on the Graphite implementation presented at the HEPSYSMAN meeting. Also considering integrating Site Nagios.
  • Glasgow dashboard now packaged and can be downloaded here.
On-duty - Dashboard ROD rota

Monday 2nd September

  • A lot of intermittent Nagios failures due to tier1 BDII issue. Imperial WMS also had some issue on Thursday and lot of nagios jobs stayed in waiting because of that.
  • One ticket escalated to the NGI.
  • A few SHA2 tickets remain open. Bristol is testing an ARC CE and that CE was not in production but monitored. It was generating alarms in Dashboard. Luke agreed to put it in downtime until fixed.
  • Several of the SHA-2 tickets have gone red as they've reached 30 days of being open. We need to decide how to handle them.

Tuesday 27th August

  • A COD ticket was raised due to overdue SHA2 tickets: https://ggus.eu/ws/ticket_info.php?ticket=96765
  • QMUL has an odd alarm for a non-production machine: eu.egi.MPI-GOCDB-Check. The machine appears to be declared correctly in the GOCDB.
  • Sussex is quite far behind in its APEL publishing.
  • Durham has several issues.
Rollout Status WLCG Baseline

Tuesday 9th July

  • New EMI2 and EMI3 release yesterday. No staged rollout requests yet. Imperial upgraded their WMS and they have been somewhat shaky ever since.

Tuesday 18th June

  • New EMI3 CE coming into SR. Liverpool will test.
  • A lot of EMI3 testing done at Brunel.
  • EMI-3 testing page contains all issues I am aware off. It's a Wiki though, so if you find an issue, please put it in the appropriate category.

Tuesday 14th May

  • A reminder. Please could sites fill out the EMI-3 testing contributions page. This is for all testing not just SR sites as we want to know which sites have experience with each component.

References


Security - Incident Procedure Policies Rota

Tuesday 3rd September

  • Security contacts and system staff of the partners of EGI, EUDAT and PRACE are invited for a joint security event from Monday, October 7 - Wednesday, October 9 in Linköping, Sweden. Monday and Tuesday will be a training event which should be of interest for all staff managing the systems which are part of our infrastructures. The second part of the meeting will consist of a discussion of security policies and the collaboration among the different infrastructures.
  • Rob H has moved to the Tier-1. Looking at options for security team.
  • At the next team meeting we should review our approach to 100percentit (just moved to uncertified).

Tuesday 20th August

  • Several sites showing up in pakiti this week.
  • Update on workaround discussed last week.


Services - PerfSonar dashboard | GridPP VOMS

Tuesday 27th August

  • PerfSONAR: version 3.3.1 was released on 21st Aug. This update should fix problems with the WLCG mesh (which can now be included in the config file). Indeed traceroute and pingER are now working at Cambridge and Imperial but not, yet, at Oxford or Lancaster. Please could sites upgrade to this or at a minimum check their existing perfsonar host(s) are working OK here dashboard.


Tickets

Monday 2nd September 2013, 14.00 BST</br> Ye gads, it's the start of another month already. So that means we're going over all the UK tickets again.

40 Open UK tickets this month.

CAMBRIDGE, BRISTOL, BIRMINGHAM, EDINBURGH, DURHAM, SHEFFIELD, MANCHESTER, LANCASTER, UCL, RHUL, QMUL and EFDA-JET all have glexec tickets. A lot of these could do with some updating, as a change of tactic I rolled them into each site's ticket roundup below.

NGI</br> https://ggus.eu/ws/ticket_info.php?ticket=95469 (5/7)</br> One of the remaining "Unresponsive VO" tickets, belonging to Supernemo. It's pretty much decided that the VO is defunct, we just need to decide what to do next. On hold (20/8)

https://ggus.eu/ws/ticket_info.php?ticket=95472 (5/7)</br> The other "Unresponsive VO" ticket, this one for minos. Pretty much the same situation as Supernemo. On hold (27/8)

https://ggus.eu/ws/ticket_info.php?ticket=95442 (4/7)</br> The "Unresponsive VOs" master ticket, just the above tickets left. On hold (12/8)

https://ggus.eu/ws/ticket_info.php?ticket=94780 (11/6)</br> Ye olde 100IT cloud site creation ticket. It's chugging along, the company has signed the OLA (ticket 96634). In progress (27/8)

https://ggus.eu/ws/ticket_info.php?ticket=95833 (17/7)</br> Decommissioning ral-ngs2. The last decommissioning ticket that I can see, was this guy forgotten about? On hold (25/7).

SUSSEX</br> https://ggus.eu/ws/ticket_info.php?ticket=95165 (28/6)</br> Sussex's perfsonar box playing up. Have you got round to reinstalling it yet? On hold (14/8)

CAMBRIDGE</br> https://ggus.eu/ws/ticket_info.php?ticket=95306 (1/7)</br> gLexec deployment ticket. Late summer is approaching, how goes things? On hold (9/7)

BRISTOL</br> https://ggus.eu/ws/ticket_info.php?ticket=96990 (2/9)</br> CMS seeing some transfer problems, it's being looked at. 'nuff said. In progress (2/9)

https://ggus.eu/ws/ticket_info.php?ticket=96261 (30/7)</br> After several quiet weeks the submitter of this ticket got back to Bristol, with the permissions problem still existing. He included some more information, maybe some bad mapping is going on. In progress (27/8)

https://ggus.eu/ws/ticket_info.php?ticket=95305 (1/7)</br> Glexec deployment ticket. On hold pending a laundry list of other tasks being completed. How goes it? On hold (11/7)

BIRMINGHAM</br> https://ggus.eu/ws/ticket_info.php?ticket=95304 (1/7)</br> glExec deployment ticket. It's September now, how goes it? On hold (17/7)

GLASGOW</br> https://ggus.eu/ws/ticket_info.php?ticket=96231 (29/7)</br> Sno+ had some WMS issues at Glasgow, which were probably caused by a temporary hiccup. Waiting on word back from Sno+ for a while. Waiting for reply (8/8)

https://ggus.eu/ws/ticket_info.php?ticket=96234 (29/7)</br> HyperK support on the Glasgow WMS. Glasgow were wanting to hold off until HyperK are in the Ops portal- which they still aren't. On hold (2/8)

EDINBURGH</br> https://ggus.eu/ws/ticket_info.php?ticket=96002 (22/7)</br> SHA-2 ticket. The ticket needs soothing, even if you don't solve it (it's *just* an update, although I successfully killed my backend database when I upgraded from my far too crusty cream version). On hold (20/8)

https://ggus.eu/ws/ticket_info.php?ticket=95303 (1/7)</br> GLexec ticket. Presumably waiting on me to get the tarball sorted (https://ggus.eu/ws/ticket_info.php?ticket=95832). On hold (27/8)

DURHAM</br> https://ggus.eu/ws/ticket_info.php?ticket=96628 (15/8)</br> Durham failing APEL pub tests. This ticket has been extended a lot. Are you in the middle of an EMI3 APEL move? Needs looking at, I think it's been extended as much as it can without an explanation. In progress (16/8)

https://ggus.eu/ws/ticket_info.php?ticket=96001 (22/7)</br> SHA-2 ticket. Will be completed this month. On hold (2/9)

https://ggus.eu/ws/ticket_info.php?ticket=96758 (21/8)</br> Another nagios error, this time SRM-Put failing. Again in urgent need of some input or else the ROD will have to escalate. They don't like doing that. In progress (21/8)

https://ggus.eu/ws/ticket_info.php?ticket=95302 (1/7)</br> glExec deployment. On hold until mid-September due to leave after some progress. On hold (2/9)

https://ggus.eu/ws/ticket_info.php?ticket=96791 (22/8)</br> Biomed noticed transfer failures to/from Durham Maybe related to 96758? Could do with at least On-holding. In progress (23/8)

SHEFFIELD</br> https://ggus.eu/ws/ticket_info.php?ticket=95301 (1/7)</br> GLExec deployment. Elena said last week that she aims to finish it off soon. How goes it? On hold (21/8)

MANCHESTER</br> https://ggus.eu/ws/ticket_info.php?ticket=95300 (1/7)</br> glexEC deployment ticket. Manchester will deploy this on the 14th of October - which if the tests become critical on the 1st of October might cause some hassle to them (but this was only a tentative date). On hold (1/7)

https://ggus.eu/ws/ticket_info.php?ticket=96837 (24/8)</br> Atlas transfers failing with "INVALID_PATH" errors. A disk server decided to take the bank holiday off and one of the switches is still playing up - waiting for things to calm down before closing. On hold (2/9)

https://ggus.eu/ws/ticket_info.php?ticket=96081 (23/7)</br> SHA-2 ticket. Planning to upgrade this mid-October during Manchester's big push. On hold (23/7)

LIVERPOOL</br> https://ggus.eu/ws/ticket_info.php?ticket=96940 (30/8)</br> SHA-2 failures, although you guys are late to the party on this one. Perhaps something whacked your ldifs and you're publishing old and stale information? In progress (2/9)

LANCASTER</br> https://ggus.eu/ws/ticket_info.php?ticket=96589 (12/8)</br> Failing APEL-Pub tests. An upgrade to EMI3 apel hasn't gone smoothly, trying to work out why (as nothing seems to broken). See ticket 95365 for details- that's where all the work is happening. On hold (13/8)

https://ggus.eu/ws/ticket_info.php?ticket=95299 (1/7)</br> GleXEC deployment. Still waiting on the tarball, leave and more pressing site issues have caused there to be no progress in the last month. On hold (17/7)

UCL</br> https://ggus.eu/ws/ticket_info.php?ticket=95298 (1/7)</br> gleXec deployment. Hope to get done after Sl6 deployment, by the end of September. On hold (29/8)

RHUL</br> https://ggus.eu/ws/ticket_info.php?ticket=95297 (1/7)</br> GlEXeC deployment. Plans to roll it out before the end of August went awry due to SL6 delays. Hope to do the two together. On hold (2/9)

QMUL</br> https://ggus.eu/ws/ticket_info.php?ticket=96618 (14/8)</br> Hone find that jobs are failing with not enough disk errors on cn633. Chris took the node offline and put the ticket on hold awaiting a permanent solution. On hold (19/8)

https://ggus.eu/ws/ticket_info.php?ticket=96856 (26/8)</br> I believe this is the same as above (but for a different node), although most jobs do not have a problem. In progress (27/8)

https://ggus.eu/ws/ticket_info.php?ticket=95296 (1/7)</br> gLeXeC deployment. It appears to be working for the SL6 nodes QM have deployed, just need to roll out more SL6 nodes. On hold (12/8)

EFDA-JET</br> https://ggus.eu/ws/ticket_info.php?ticket=95295 (1/7)</br> GlexeC deployment. Any ideas what the plan is with JET? On hold (19/7)

TIER 1</br> https://ggus.eu/ws/ticket_info.php?ticket=96235 (29/7)</br> Enabling HyperK on the RAL LFC. This is done, and Catalin has requested some tests (but IIRC Chris is on leave). Waiting for reply (22/8)

https://ggus.eu/ws/ticket_info.php?ticket=96233 (29/7)</br> WMS support for hyperk.org. Again this has been done, and just needs some testing. Waiting for reply (28/8)

https://ggus.eu/ws/ticket_info.php?ticket=96321 (2/8)</br> Sno+ SRM tests failing at the Tier-1, primarily caused by Castor not speaking Vomese. Kashif has suggested we ignore this test until such time as Castor learns how to read extensions. It needs some action though. In progress (23/8)

https://ggus.eu/ws/ticket_info.php?ticket=96968 (31/8)</br> CMS spotted a black hole node at RAL, with cvmfs problems. Offlined. In progress (31/8)

https://ggus.eu/ws/ticket_info.php?ticket=95996 (22/7)</br> SHA-2 ticket. One of the WMS is upgraded, but the CEs (and the other two WMS) need upgrading too. Confident that it will be done by the deadline though. In progress (23/8)

https://ggus.eu/ws/ticket_info.php?ticket=91658 (20/2)</br> The LFC WEbdav support ticket. Between leave and higher priority work I don't think there's been much progress on this over the last month though. In Progress (9/8)

https://ggus.eu/ws/ticket_info.php?ticket=86152 (17/9/12)</br> The oldest, and most neglected ticket - "correlated packet-loss on perfsonar host". Needs some sprucing up. On hold (17/6)

TICKETS FROM THE UK</br> My will to live is fading, but an argus ticket from Dan at QM was mentioned on TB-SUPPORT, which might affect some other UK sites:</br> https://ggus.eu/ws/ticket_info.php?ticket=96228

Tools - MyEGI Nagios

Monday 2nd September

  • Intermittent Nagios errors -> Imperial WMS and all the jobs going through it were failing with ‘no compatible error’. Some reports of ongoing issues. What is the direct impact?
  • MyEGI and gstat were also down last week.
  • Jens is testing SHA-2 compliance of components. The version of gridsite on the GridPP website is not compliant but SHA-2 will be supported with a move to a new server (when?).

Monday 12 Aug

  • VO-Nagios

t2wlcgnagios.physics.ox.ac.uk monitors few UK VO and it uses Robot Certificate. Due to some confusion Robot Certificate got expired. I applied for extension of Robot Certificate beforehand but Cert Wizard doesn't understand Robot Certificate and I thought that it has been extended. Finally Jens stepped in sorted it out. Now VO Nagios is working.

  • SHA2 Certificates

I have been issued a SHA2 certificate by Jens. I tested few CE's and some interesting results came out. Gridpp VOMS server is SHA2 compatible so SHA2 proxies can be created for VO's hosted at voms.gridpp.ac.uk. None of CERN voms server are sha2 compatible but there is workaround to add a secondary SHA2 certificate. Details are here https://twiki.cern.ch/twiki/bin/view/LCG/SHA2readinessTesting#SHA_2_VOMS_server I have added my SHA2 certificate but it is not approved yet as most of the people are on holiday. Interestingly when I submitted few jobs using ngs.ac.uk with SHA2 certificate, it finished successfully on the CE's which are not suppose to be SHA2 complaint. I will test again with OPS vo to confirm it.

Tuesday 23rd July

  • In a campaign to update VO ID card details it turns out that a few of our supported VOs are obsolete: babar, possibly supernemo and ngs.ac.uk. The first of these can be safely removed but we need to confirm our announcement process.
VOs - GridPP VOMS VO IDs Approved VO table

Monday 2nd September

  • The next quarterly Tier 1 allocation/resourcing meeting is scheduled for Wednesday 18th September (after the weekly T1 meeting) the hardware requirements and fair-shares for the period October-December 2013 will be reviewed. It looks ahead over the next 12 month timeframe. Can all experiments/projects please let Pete G have any updates or requests to these numbers by Friday 13th September please?

Monday 19 August

  • EPIC
    • Support requested at Tier-1
    • Any other sites prepared to support them?
  • Catalogue synchronisation - Biomed working on it.


Monday 12 August

  • HyperK.org
    • VOMS servers set up (Manchester, Oxford, Imperial)
    • VOID card - stalled on a homepage.
    • WMS set up (Imperial) - awaiting Glasgow, Ral
    • Site set up (QMUL)
    • LFC - in progress
    • CVMFS - considering
  • SNO+
    • Dirac set up for some CEs
  • Epic
    • Doing stuff
  • ngs.ac.uk VO - any reason to keep it?
  • Software areas for SL6
    • Are we keeping the same areas as sl5?
    • What about the software tags?
    • Push CVMFS?
Site Updates

Actions


Meeting Summaries
Project Management Board - MembersMinutes Quarterly Reports

Empty

GridPP ops meeting - Agendas Actions Core Tasks

Empty


RAL Tier-1 Experiment Liaison Meeting (Wednesday 13:30) Agenda EVO meeting

Wednesday 4th September

  • Operations report
  • A quiet week so far. We continue to work with Atlas on the testing of FTS3.
WLCG Grid Deployment Board - Agendas MB agendas

Empty



NGI UK - Homepage CA

Empty

Events

Empty

UK ATLAS - Shifter view News & Links

Empty

UK CMS

Empty

UK LHCb

Empty

UK OTHER
  • N/A
To note

  • N/A