Difference between revisions of "Operations Bulletin Latest"
(→) |
(→) |
||
Line 388: | Line 388: | ||
32 Open UK tickets this week. | 32 Open UK tickets this week. | ||
− | No site in particular. | + | '''No site in particular.'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101502 (24/2) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101502 (24/2)<br /> |
The ILC cvmfs rollout ticket. Glasgow, Oxford, Durham and Bristol were missing at last head count - although Glasgow are mid-rollout and should be fully deployed any day now (if not already). I think Oxford are in a similar boat? As JK points out, we've got to the point where probably need to on hold the ticket whilst I harass the last few stragglers. In progress (3/4) | The ILC cvmfs rollout ticket. Glasgow, Oxford, Durham and Bristol were missing at last head count - although Glasgow are mid-rollout and should be fully deployed any day now (if not already). I think Oxford are in a similar boat? As JK points out, we've got to the point where probably need to on hold the ticket whilst I harass the last few stragglers. In progress (3/4) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=103043 (7/4) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=103043 (7/4)<br /> |
Squire Whyntie has asked for cern@school registration on the Imperial Dirac. Janusz has done so and Tom confirmed it works and cab be solved. If only all things were solved so quickly! Assigned (7/4) | Squire Whyntie has asked for cern@school registration on the Imperial Dirac. Janusz has done so and Tom confirmed it works and cab be solved. If only all things were solved so quickly! Assigned (7/4) | ||
− | SUSSEX | + | '''SUSSEX'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102810 (28/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102810 (28/3)<br /> |
The new Sussex EMI2 upgrade ticket. Matt RB copied the Sussex plan over from the original ticket. Daniela cleared up the mystery of what happened to the original ticket (dashboard shenanigans) and posted some useful instructions for the BDII upgrade. In progress (1/4) | The new Sussex EMI2 upgrade ticket. Matt RB copied the Sussex plan over from the original ticket. Daniela cleared up the mystery of what happened to the original ticket (dashboard shenanigans) and posted some useful instructions for the BDII upgrade. In progress (1/4) | ||
− | RALPP | + | '''RALPP'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102990 (3/4) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102990 (3/4)<br /> |
Duncan's unending perfsonar vigilance discovered a a problem with the RALPP latency box. Ian reports firewall problems that have been solved, so it looks like this one can be closed (if all is well). In progress (can be closed) (4/4) | Duncan's unending perfsonar vigilance discovered a a problem with the RALPP latency box. Ian reports firewall problems that have been solved, so it looks like this one can be closed (if all is well). In progress (can be closed) (4/4) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102953 (24/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102953 (24/3)<br /> |
CMS glidein hammercloud jobs not running at the site (specifically their defunct cream CEs)- Chris points out another ticket (https://ggus.eu/index.php?mode=ticket_info&ticket_id=102915) essentially detailing the same problem (just for different job types). Probably worth on holding this one whilst waiting on the other, as it looks like the problems are CMS side. In progress (2/4) | CMS glidein hammercloud jobs not running at the site (specifically their defunct cream CEs)- Chris points out another ticket (https://ggus.eu/index.php?mode=ticket_info&ticket_id=102915) essentially detailing the same problem (just for different job types). Probably worth on holding this one whilst waiting on the other, as it looks like the problems are CMS side. In progress (2/4) | ||
− | OXFORD | + | '''OXFORD'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=103027 (5/4) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=103027 (5/4)<br /> |
LHCB pilots aborting, Kashif asks if the problem persists, the ticket fairy set the ticket to Waiting for Reply (5/4) | LHCB pilots aborting, Kashif asks if the problem persists, the ticket fairy set the ticket to Waiting for Reply (5/4) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102469 (19/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102469 (19/3)<br /> |
cvmfs for t2k. I think this has fallen through some cracks, no word for a while. In progress (21/3) | cvmfs for t2k. I think this has fallen through some cracks, no word for a while. In progress (21/3) | ||
− | BRISTOL | + | '''BRISTOL'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102205 (14/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102205 (14/3)<br /> |
Bristol's EMI2 upgrade ticket. Not much news, although there was a positive update from Winnie that looks like the April deadline will be made. In progress (4/4) | Bristol's EMI2 upgrade ticket. Not much news, although there was a positive update from Winnie that looks like the April deadline will be made. In progress (4/4) | ||
− | GLASGOW | + | '''GLASGOW'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102914 (1/4) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102914 (1/4)<br /> |
An atlas ticket, detailing some odd transfer behaviour for some files, likely attributed to some off tcp window settings on a disk server. There was a similar looking (although possibly not identical) problem at RHUL (https://ggus.eu/index.php?mode=ticket_info&ticket_id=102311). Some interesting stuff. In progress (4/4) | An atlas ticket, detailing some odd transfer behaviour for some files, likely attributed to some off tcp window settings on a disk server. There was a similar looking (although possibly not identical) problem at RHUL (https://ggus.eu/index.php?mode=ticket_info&ticket_id=102311). Some interesting stuff. In progress (4/4) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102202 (14/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102202 (14/3)<br /> |
Not as interesting, Glasgow's EMI upgrade ticket. Chugging along, last word was from David a little while back about having watching some atlas canary jobs running on the EMI3 worker nodes. How did these pan out? In progress (27/3) | Not as interesting, Glasgow's EMI upgrade ticket. Chugging along, last word was from David a little while back about having watching some atlas canary jobs running on the EMI3 worker nodes. How did these pan out? In progress (27/3) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101565 (26/2) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101565 (26/2)<br /> |
LHCB asked Glasgow to publish their max CPU time. Not wanting to be made liars of, Sam pointed out why they didn't (shouldn't) do this. This has seemed to send LHCB back to the drawing board, so the ticket is on hold. On Hold (12/3) | LHCB asked Glasgow to publish their max CPU time. Not wanting to be made liars of, Sam pointed out why they didn't (shouldn't) do this. This has seemed to send LHCB back to the drawing board, so the ticket is on hold. On Hold (12/3) | ||
− | EDINBURGH | + | '''EDINBURGH'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102201 (14/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102201 (14/3)<br /> |
The ECDF EMI upgrade ticket. Not much to report here, although the apel box through a wobbly as well, Andy's on it. In progress (2/4) | The ECDF EMI upgrade ticket. Not much to report here, although the apel box through a wobbly as well, Andy's on it. In progress (2/4) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=95303 (1/7/13) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=95303 (1/7/13)<br /> |
glexec ticket. Word on that later. On Hold (27/1) | glexec ticket. Word on that later. On Hold (27/1) | ||
− | DURHAM | + | '''DURHAM'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102199 (14/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102199 (14/3)<br /> |
Another EMI upgrade deadline ticket. A plan is in place and the work is underway. On Hold (24/3) | Another EMI upgrade deadline ticket. A plan is in place and the work is underway. On Hold (24/3) | ||
− | SHEFFIELD | + | '''SHEFFIELD'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=100037 (3/1) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=100037 (3/1)<br /> |
Sheffield's perfsonar having trouble. Elena upgraded and got the Sheffield IT guys to open port 8086 - it looks like she's nailed the problem and has asked for confirmation. Waiting for reply (7/4) (And before I even finished the review, the ticket was solved). | Sheffield's perfsonar having trouble. Elena upgraded and got the Sheffield IT guys to open port 8086 - it looks like she's nailed the problem and has asked for confirmation. Waiting for reply (7/4) (And before I even finished the review, the ticket was solved). | ||
− | LANCASTER | + | '''LANCASTER'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=95299 (1/7) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=95299 (1/7)<br /> |
GLEXEC ticket. The tarball glexec isn't going well (no thanks to EMI3 taking up the last 6 weeks of tarball time). I might have to admit defeat (but will ask the devs for help before I do). On hold (4/4) | GLEXEC ticket. The tarball glexec isn't going well (no thanks to EMI3 taking up the last 6 weeks of tarball time). I might have to admit defeat (but will ask the devs for help before I do). On hold (4/4) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=100566 (27/1/13) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=100566 (27/1/13)<br /> |
Lancaster's Poo Perfsonar Performance (I said I wouldn't use that alliteration again, I lied). Using "normal" iperf to probe the boxes I see no 1Gb bottlenecks in my network, could be problem be software? On hold (7/4) | Lancaster's Poo Perfsonar Performance (I said I wouldn't use that alliteration again, I lied). Using "normal" iperf to probe the boxes I see no 1Gb bottlenecks in my network, could be problem be software? On hold (7/4) | ||
− | UCL | + | '''UCL'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101285 (16/2) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101285 (16/2)<br /> |
UCL's perfsonar also having difficulty, although their difficulty is caused by the hardware going kaput on them. Ben is chasing up Dell for new bits. In progress (3/4) | UCL's perfsonar also having difficulty, although their difficulty is caused by the hardware going kaput on them. Ben is chasing up Dell for new bits. In progress (3/4) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102193 (14/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102193 (14/3)<br /> |
EMI upgrade ticket. Ben put in a brief plan, but the reminder date has passed. How goes it? The bdii and DPM are fairly straightforward to upgrade. On hold (14/3) | EMI upgrade ticket. Ben put in a brief plan, but the reminder date has passed. How goes it? The bdii and DPM are fairly straightforward to upgrade. On hold (14/3) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=95298 (1/7/13) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=95298 (1/7/13)<br /> |
GlexeC ticket. No news for a while, is this work to be rolled into the EMI3 upgrade? On hold (27/1) | GlexeC ticket. No news for a while, is this work to be rolled into the EMI3 upgrade? On hold (27/1) | ||
− | RHUL | + | '''RHUL'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102189 (14/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102189 (14/3)<br /> |
RHUL's EMI upgrade ticket. Not much news here. On hold (21/3) | RHUL's EMI upgrade ticket. Not much news here. On hold (21/3) | ||
− | QMUL | + | '''QMUL'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=103028 (6/4) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=103028 (6/4)<br /> |
Atlas seeing production jobs failing due to pilot errors. Chris asked if production job options have changed recently? The ticket fairy struck again, setting the ticket to Waiting for reply (although he's less sure if that was the intention of Chris' reply). In progress (7/4) | Atlas seeing production jobs failing due to pilot errors. Chris asked if production job options have changed recently? The ticket fairy struck again, setting the ticket to Waiting for reply (although he's less sure if that was the intention of Chris' reply). In progress (7/4) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101639 (26/2) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101639 (26/2)<br /> |
RFC3820 proxy problems at QM (and elsewhere). JK has asked the submitter for his ticket intentions. Set to Waiting for reply by our friend, the ticket fairy. (1/4) | RFC3820 proxy problems at QM (and elsewhere). JK has asked the submitter for his ticket intentions. Set to Waiting for reply by our friend, the ticket fairy. (1/4) | ||
(Please remember to set your tickets to Waiting for Reply after asking a question to the submitter. Don't make me spend yet another Monday afternoon referring to myself as the ticket fairy.) | (Please remember to set your tickets to Waiting for Reply after asking a question to the submitter. Don't make me spend yet another Monday afternoon referring to myself as the ticket fairy.) | ||
− | IMPERIAL | + | '''IMPERIAL'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102888 (1/4) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102888 (1/4)<br /> |
Biomed asked for access to their cvmfs repo to be rolled out at IC. Daniela has said fine but asked that they completely migrate to it within 3 months (nfs or cvmfs). Daniela has completed the rollout and asked biomed to test. Waiting for reply (7/4) | Biomed asked for access to their cvmfs repo to be rolled out at IC. Daniela has said fine but asked that they completely migrate to it within 3 months (nfs or cvmfs). Daniela has completed the rollout and asked biomed to test. Waiting for reply (7/4) | ||
− | EFDA-JET | + | '''EFDA-JET'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102166 (14/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102166 (14/3)<br /> |
The JET EMI upgrade ticket. There was a hope to upgrade before the end of April. On hold (24/3) | The JET EMI upgrade ticket. There was a hope to upgrade before the end of April. On hold (24/3) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=97485 (21/9/13) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=97485 (21/9/13)<br /> |
SSL type errors for LHCB at JET. No progress on this for a while, the problem somehow survived the move to SL6/EMI3. On hold (11/2) | SSL type errors for LHCB at JET. No progress on this for a while, the problem somehow survived the move to SL6/EMI3. On hold (11/2) | ||
− | TIER 1 | + | '''TIER 1'''<br /> |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102611 (24/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=102611 (24/3)<br /> |
The Tier 1 EMI upgrade ticket. There seem to be some false positives on the list, which could do with clarification which these are (especially due to the dashboard noise on the ticket). In progress (27/3) | The Tier 1 EMI upgrade ticket. There seem to be some false positives on the list, which could do with clarification which these are (especially due to the dashboard noise on the ticket). In progress (27/3) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=98249 (21/10/13) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=98249 (21/10/13)<br /> |
CVMFS for SNO+. Matt reported that the collaboration has given permission to have their software on cvmfs, and hoped to have tarballs ready for last week. Has there been any progress offline? In progress (26/3) | CVMFS for SNO+. Matt reported that the collaboration has given permission to have their software on cvmfs, and hoped to have tarballs ready for last week. Has there been any progress offline? In progress (26/3) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101079 (9/2) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101079 (9/2)<br /> |
− | ARC CEs publishing the wrong DefaultSE. Andrew has hacking this on his todo list, but bumped this issue down the | + | ARC CEs publishing the wrong DefaultSE. Andrew has hacking this on his todo list, but bumped this issue down the list (which is fine, as it's low priority) . In progress (1/4) |
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=99556 (6/12/13) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=99556 (6/12/13)<br /> |
The NGI argus ticket. I'm pretty sure that this can be closed, as argusngi.gridpp.rl.ac.uk is setup and tested by several sites- so all looks well here. On Hold (21/3) | The NGI argus ticket. I'm pretty sure that this can be closed, as argusngi.gridpp.rl.ac.uk is setup and tested by several sites- so all looks well here. On Hold (21/3) | ||
− | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101968 (11/3) | + | https://ggus.eu/index.php?mode=ticket_info&ticket_id=101968 (11/3)<br /> |
Atlas deletion errors at the Tier 1. The problem is known, but not well understood, and sadly persists (last set of errors reported on the 4th). Alastair has put in a good explanation of the symptoms. On hold (4/4) | Atlas deletion errors at the Tier 1. The problem is known, but not well understood, and sadly persists (last set of errors reported on the 4th). Alastair has put in a good explanation of the symptoms. On hold (4/4) | ||
Revision as of 15:01, 7 April 2014
Week commencing 7th April 2014 |
Task Areas |
|
|
Meeting Summaries |
|
|