Difference between revisions of "Operations Bulletin Latest"
(→) |
|||
| Line 443: | Line 443: | ||
===== ===== | ===== ===== | ||
<!-- ******************Edit start********************* -----> | <!-- ******************Edit start********************* -----> | ||
| − | + | '''Monday 3rd September 2018, 14.30 BST.'''<br /> | |
| + | 47 Open UK Tickets this month. | ||
| + | |||
| + | '''SUSSEX'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131617 (3/11/17)<br /> | ||
| + | The site's IPv6 ticket. Good news from Leo today, with all external services dual-stacked (the perfsonar just needs adding to the mesh). Nice one! Waiting for reply (3/9) | ||
| + | |||
| + | '''RALPP'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136958 (30/8)<br /> | ||
| + | t2k had their replications to ralpp timing out, but Chris noticed that they've run out of space. He's kindly trying to free them up a bit more room. In progress (31/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136927 (29/8)<br /> | ||
| + | CMS transfer failures, which look to be due to a bad file. It's been invalidated and a proper replica moved to the site. This should be just about done with? In progress (30/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131616 (3/11/17)<br /> | ||
| + | RALPP v6 ticket. Any updates? In progress (16/1) | ||
| + | |||
| + | '''OXFORD'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131615 (3/11/17)<br /> | ||
| + | Oxford's v6 ticket. Last update was July, I suspect there's been no progress over the summer. On hold (13/7) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136687 (13/8)<br /> | ||
| + | A bit of an odd LHCB FTS ticket as it's intended to mirror the issure for a site rather then track an issue at a site. From my understanding these are known issues with third party http transfers? In progress (17/8) | ||
| + | |||
| + | '''CAMBRIDGE'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131614 (3/11/17)<br /> | ||
| + | Cambridge's v6 ticket. Last update was back in June, any more news on the move to a new address block? On hold (5/6) | ||
| + | |||
| + | '''BRISTOL'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131613 (3/11/17)<br /> | ||
| + | And Bristol's v6 ticket. Winnie's kept us appraised of the situation back in July. On hold (16/7) | ||
| + | |||
| + | '''BIRMINGHAM'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=129930 (4/8/17)<br /> | ||
| + | The http ticket that kind of tracks the move to EOS now. I do wonder if it's worth keeping this ticket around much anymore, as there's an epic Jira ticket charting the migration. On hold (14/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131612 (3/11/17)<br /> | ||
| + | The Birmingham v6 ticket. Mark gave some not great news last month. Andrea has asked to turn up the pressure. Maybe we can help? On hold (27/8) | ||
| + | |||
| + | '''GLASGOW'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=134689 (23/4)<br /> | ||
| + | Request to upgrade perfsonar to C7. Perfsonar 4.1 is out now if you feel like revisiting this. On hold (14/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131611 (3/11/17)<br /> | ||
| + | Glasgow's v6 ticket. Any new (aspirational) plans that need to go into the ticket? On hold (26/2) | ||
| + | |||
| + | '''ECDF'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131610 (3/11/17)<br /> | ||
| + | Just the old v6 ticket at ECDF. Any news since your misadventures back in May? On hold (28/5) | ||
| + | |||
| + | '''DURHAM'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136909 (28/8)<br /> | ||
| + | Atlas deletion error ticket. The Durham guys are just back in the office and poking the ticket. In progress (3/9) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=134687 (23/4)<br /> | ||
| + | The other outstanding request to upgrade the perfsonar host. In progress (14/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131609 (3/11/17)<br /> | ||
| + | Durham's v6 ticket. From the feel of the July update I'm going to hazard a guess that there's not been any recent progress. In progress (should be On Hold?) (10/7) | ||
| + | |||
| + | '''SHEFFIELD'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136014 (10/7)<br /> | ||
| + | Atlas transfer errors, evolved to problems with the error message "job has been canceled because it stayed in the queue for too long". As this an FTS side error? Perhaps the SE is simply being overworked? In progress (30/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131608 (3/11/17)<br /> | ||
| + | Sheffield's v6 ticket. How goes the work that was supposed to resume in July? On hold (10/7) | ||
| + | |||
| + | '''MANCHESTER'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136976 (1/9)<br /> | ||
| + | A fresh atlas transfer error ticket. Robert found that their DPM's mysql database wasn't responding, but a restart should have fixed it (and indeed a peek at the monitoring shows this to be the case). In progress (1/9) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131607 (3/11/17)<br /> | ||
| + | Manchester's v6 ticket. Any news on this since the April update? On hold (24/4) | ||
| + | |||
| + | '''LIVERPOOL'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131606 (3/11/17)<br /> | ||
| + | Just the v6 ticket at Liverpool. Any news since the June update? In progress (4/6) | ||
| + | |||
| + | '''LANCASTER'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136793 (20/8)<br /> | ||
| + | A ticket from snoplus as the Lancaster SE wasn't in the cern bdii. It looks like none of the Lancaster resources are in the cern bdii, even though other top bdiis know about this. A bit of a head scratcher. Has anyone else been "censored" (or censured) by the CERN bdii? In progress (3/9) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136635 (9/8)<br /> | ||
| + | A low availability ticket, not too far off being able to close it. On hold (9/8) | ||
| + | |||
| + | '''UCL'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=134686 (23/4)<br /> | ||
| + | Request to upgrade perfsonar. No news since the ticket was acknowledged. On hold (23/4) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131604 (3/11/17)<br /> | ||
| + | UCL's v6 ticket. There was a re-poking of the network team back in May but no news since. On hold (4/5) | ||
| + | |||
| + | '''RHUL'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=131603 (3/11/17)<br /> | ||
| + | Just the v6 ticket at Royal Holloway. How's it going? It looks like lack of v6 DNS was the problem here again. In progress (perhaps should be On Hold?) (6/2) | ||
| + | |||
| + | '''QMUL'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136719 (15/8)<br /> | ||
| + | LHCB having file access problems (again?). Daniel thought it might be the SE misbehaving under load that's causing the problems. There was some testing, but I'm not sure of the conclusion. Waiting for reply (23/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136550 (4/8)<br /> | ||
| + | t2k having file access problems, with the root cause being the top bdii they were using being broken (sounds similar to what Lancaster has been seeing). It looks like the problem has gone away here though, so I think this ticket can be closed. In progress (14/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136714 (14/8)<br /> | ||
| + | The same t2k user having problems reliably copying files, but again this issue seems fixed. In progress (15/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136918 (28/8)<br /> | ||
| + | t2k not noticing downtime notices. It looks like this ticket can be closed too. In progress (3/9) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136178 (17/7)<br /> | ||
| + | It's a seemingly solved t2k ticket at Queen Mary that hasn't been closed by the user. You don't see many of those around. In progress (14/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136712 (14/8)<br /> | ||
| + | LHCB noticed they weren't running (many) jobs at QM. Dan explained why (all very reasonable). It looks to me this ticket is resolved. In progress (14/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136576 (6/8)<br /> | ||
| + | A low-availability ROD ticket after the cooling troubles. The A/R numbers are almost up to par. On hold (6/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=132929 (18/1)<br /> | ||
| + | APEL accounting for slurm ticket. Dan has been working on this, and has spread the new accounting scripts around his CEs and APEL box. In progress (3/9) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=134573 (17/4)<br /> | ||
| + | Request from CMS to install singularity. Dan has it on his to do list, is the move to C7 still planned for the end of the Summer (i.e. soonish)? On hold (17/4) | ||
| + | |||
| + | '''BRUNEL'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136806 (21/8)<br /> | ||
| + | CMS jobs having problems at Brunel. There was an interesting case where WNs lost v4 connectivty whilst maintaining v6 and this were still able to get jobs, but the root cause looks to be problems with the xroot fallback mechanism. I think this might be above the site's metaphorical paygrade. In progress (31/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=133956 (9/3)<br /> | ||
| + | A CMS ticket to reconfigure the site's xrootd configs. Postponed due to waiting on a move to C7/DOME. Have you made any progress with this? FYI we're planning on turning DOME on at Lancaster soonish. In progress (3/7) | ||
| + | |||
| + | '''TIER 1'''<br /> | ||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136884 (27/8)<br /> | ||
| + | lcg-cr not working for t2k, an lfc ticket that's been ported over to RAL as it looks like their database is corrupted. In progress (29/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136840 (23/8)<br /> | ||
| + | A Sno+ ticket, which looks to be related to the LFC issues. In progress (29/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136942 (29/8)<br /> | ||
| + | t2k noticing timeouts copying ONLINE_AND_NEARLINE files at RAL. After investigation it lead to the RALPP ticket above, and this ticket was left unclosed. In progress (can be closed) (30/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136701 (14/8)<br /> | ||
| + | LHCB would like to investigate the high background failure rate of jobs transferring their data out at RAL. A lot of back and forth on the ticket. Waiting for reply (3/9) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136967 (31/8)<br /> | ||
| + | CMS Phedex transfers from RAL to FNAL failing. Checking on it has been passed to the ECHO team. In progress (31/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136366 (25/7)<br /> | ||
| + | Removing MICE from the batch queues. It looks like submission has been successfully disabled. In progress (20/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136757 (17/8)<br /> | ||
| + | MICE VO voms configs missing from the LFC. This looks to be fixed (although the suspected database problems might interfere with stuff). In progress (21/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136028 (10/7)<br /> | ||
| + | CMS have issues reading files on ECHO, which looks to be a xroot problem (I couldn't follow the ticket). Chris B has put a lot of effort into this, and Brian Bockelman is roped into the ticket now. In progress (29/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=136199 (18/7)<br /> | ||
| + | An LHCB ticket to the FTS team, progress on the ticket stalled nearly a month ago (have people been on holiday?). In progress (7/8) | ||
| + | |||
| + | https://ggus.eu/?mode=ticket_info&ticket_id=124876 (17/11/16)<br /> | ||
| + | Getting ECHO gridftp ROD tests working. Things were looking quite good, but it looks like the ticket is waiting on a WN config change to be rolled out at RAL still? The tests are still broken. In progress (23/7) | ||
<!-- ******************Edit stop********************* -----> | <!-- ******************Edit stop********************* -----> | ||
Revision as of 16:16, 3 September 2018
| Week commencing Monday 3rd September 2018 |
| Task Areas |
|
