Operations Bulletin 090712

Bulletin archive

Week commencing 2nd July 2012

Task Areas

General updates Monday 2nd July Tier-2 quarterly reports requested by Wednesday 18th July There is now an LHCb CVMFS sites map Monday 25th June The batch systems APEL are planning to support are: PBS / Torque; LSF; Grid Engine and SLURM (to be confirmed). Do we have any other needs/requests? For those interested in the GridPP review of the Tier-1 (meeting agenda). Good GridPP presence at the JANET6 meeting last Thursday. Q2 2012 finishes this week. Tier-2 quarterly reports due by 13th July. Changing IP addresses & DNS (DNS updates) ATLAS are considering increasing the limit of their ESD file size from the current 10GB to 15GB (or 12GB at least). Would this create any problem at sites? ATLAS job recovery now appears to be working well - allowing jobs to finish even if the SE is not up/unstable when the job finishes (see ATLAS subsection). A recent SL5 update (to python and its libraries), now in fnal and cern repositories, broke running ATLAS jobs last week. Tier-1 - Status Page Tuesday 3rd July The Castor databases were successfully updated to Oracle 11 on Wednesday 27th June. The FTS database was also back to its correct location. Storage & Data Management - Agendas/Minutes Considering input to a community support model for DPM and possible alternatives. Wednesday 20th June snoplus needs/plans on agenda last week. The collaboration will depend on the RAL LFC and are looking to increase Tier-2 usage – current needs of 10TB/site will increase to 20TB/site. Data taking will start in the autumn and continue for 6 months. Accounting - UK Grid Metrics HEPSPEC06 Wednesday 6th June - Core-ops Request sites to publish HS06 figures from new kit to this page. Please would all sites check the HS06 numbers they publish. Will review in detail on 26th June. Friday 11th May - HEPSYSMAN Discussion on HS06 reminding sites to publish using results from 32-bit mode benchmarking. A reminder for new kit results to be posted to the HS06 wiki page. See also the blog article by Pete Gronbech. The HEPiX guidelines for running the benchmark tests are at this link. Check publishing via: http://gstat2.grid.sinica.edu.tw/gstat/summary/Country/UK/ Documentation - KeyDocs Tuesday 3rd July Started a page on stale documents. Please update this page if you find documents or pages that need attention. Wednesday, 6th June Released a document, hep.ph.liv.ac.uk/~sjones/VomsSnooper.odt, that describes how to Maintain site VOMS info document for the approved VOs Check a site's VOMS records correspond exactly with CIC portal Create new site VOMS records direct from CIC portal, without manual transcription Note: I'm accepting tips from GridPP core task members etc. about other use cases for these processes. This will be converted to wiki formatted and made available in the normal way. Next jobs: review logical/sequence of VOMS admin process, document it if it works, fix it if it doesn't. create standard baseline for proxy renewal process, and write it up in wiki. Note: I'm accepting tips from other Gridpp core team members etc. for document priorities. Please think about where the problems lie (i.e. what costs us yet is easy to fix) and get back to me. Tuesday, 29th May VOMS Records in GridPP Approved VO list now up to date with CICs Portal XML. This can be used by Site Admins to ensure their site-info.def/vo.d directories are up to date. A tool, SidFormatter, will be released this week to facilitate comparison with the benchmark. A process has been devised to ensure that GridPP Approved VO is kept up to date to within a week of CIC Portal changes. Consultation to be made about further fields that we may wish to advertise in this manner. Friday 27th April Appeal for a volunteer to enhance "Grid User Crash Course" (https://www.gridpp.ac.uk/wiki/Grid_user_crash_course) with simple use case for dependable proxy renewal for long jobs, as this is a recurrent requirement that has caused multiple queries on TB_SUPPORT. Interoperation - EGI ops agendas Tuesday 3rd July There was an EGI meeting yesterday. Minutes have been uploaded. Monitoring - Links MyWLCG Monday 2nd July DC has almost finished an initial ranking. This will be reviewed by AF/JC and discussed at 10th July ops meeting Wednesday 6th June Ranking continues. Plan to have a meeting in July to discuss good approaches to the plethora of monitoring available. Current priority is ranking the tools available. Glasgow dashboard now packaged and can be downloaded here. On-duty - Dashboard ROD Rota Monday 2nd July - AM Good week overall. No major regionwide problems. Tickets opened towards the end for job failures at Durham and Lancaster, and for WMS at Glasgow (which is in downtime now.) QMUL continues to have machines that are not in production raising alarms outside of downtimes, and clearing these counts against the ROD quality Rollout Status WLCG Baseline Monday 11th June EMI2 is released but not in Staged Rollout yet. Buyers beware. Thursday 10th May The cream ce and the WMS which were released at the end of April have finally gone into Staged Rollout Call for more sites to take part in EMI-2 rollout tests. The overall SR contributions are in this table. Friday 27th April Updated version information on rollout page WN scan indicates some sites not keen on OS updates to those nodes. Security - Incident Procedure Policies Monday 25th June Rota availability responses slow Is anyone following up on SSC5/6? Stratuslab VM (ex UK) gridftp

Services - PerfSonar dashboard Monday 2nd July Lancaster added to dashboard. Are any other sites ready? Still seeking clarification of GridMon boxes (no reply). Tickets Monday 2nd of July, 14:00 BST</br> 23 Open UK tickets this week.</br> TIER 1</br> https://ggus.eu/ws/ticket_info.php?ticket=83672</br> Might be of interest to others, GGUS will be updating the certificate it uses to sign alerts next Monday (9th). OXFORD</br> https://ggus.eu/ws/ticket_info.php?ticket=83711</br> Hone asked if something can be done to increase job throughput at Oxford, which was kindly obliged. There's been a few sites that I noticed ticketed by hone, who don't seem to want more then for jobs to spend <24 hours queuing. Do people find these requests reasonable. RHUL/BIRMINGHAM</br> https://ggus.eu/ws/ticket_info.php?ticket=83627</br> https://ggus.eu/ws/ticket_info.php?ticket=83628</br> Biomed are experiencing "negative space" values for lcg-infosite queries at these two sites. QMUL/SNO+</br> https://ggus.eu/ws/ticket_info.php?ticket=83587</br> SNO+ have asked for git to be installed on the QMUL clusters, which could be a bit of a blimmer to implement. This request could find its way to other sites. T2K.org</br> ref: https://ggus.eu/ws/ticket_info.php?ticket=83209</br> Having a poke around I don't think that t2k.org have ticketed ggus about getting a VO entry. IC</br> https://ggus.eu/ws/ticket_info.php?ticket=82946</br> This old ticket concerning some odd cvmfs behaviour experienced by atlas has been updated to include some interesting information about running the SW tests "by hand".</br> NEW THIS MORNING</br> LANCASTER</br> https://ggus.eu/ws/ticket_info.php?ticket=83812</br> The dpm crashing all the time bug has struck us again, and struck hard. We're on glite 1.8.2-3. No "Solved Cases" stand out this week, nor due any Tickets from the UK. YVMV of course! Tools - MyEGI Nagios Monday 2nd July Wwitched on backup Nagios at Lancaster and stopped Nagios instance at Oxford. Stopping Nagios instance at Oxford means that it is not sending results to the dashboard and central DB. Keeping a close eye on it and will revert it back to original position if any problems encountered. Saturday 23rd June Email alerts from Nagios appear to have stopped. Reported to developers. Tuesday 12th June Lancaster backup Nagios now available (link). A reminder as to who can see the data: all members of ops and dteam and any one who is registered in GOCDB as site-admin, regional manager etc. it is also possible to add any one who has a PKI certificate. VOs - GridPP VOMS VO IDs Approved Wednesday 6th July Cross-checking VOs enabled vs VO table. Surveying VO-admins for problems faced in their VOs. SNO+ have asked about using git to deploy software - what are the options? Site Updates Monday 25th June N/A

Meeting Summaries

Project Management Board - MembersMinutes Quarterly Reports Monday 2nd July No meeting. Next PMB on Monday 9th. GridPP ops meeting - Agendas Actions Core Tasks Tuesday 8th May - link Agenda Minutes ATLAS DATADISK now being used for production input files Deploy perfsonar-ps (for LHC compatibility), but run a Perfsonar-MDM portal to collate the information Target date for perfsonar-ps at sites is the end of July KeyDocs still not in place for all areas Tuesday 1st May 2012 - Agenda Minutes Trying the present bulletin as a conduit for information 7 sites below 90% in ATLAS monitoring! Instructions for small VO space-token setup/usage requested EMI WN tarball now available - comment in ticket Not every site running latest SL5 OS RAL Tier-1 Experiment Liaison Meeting (Wednesday 13:30) Agenda EVO meeting Wednesday 27th June The move of the Castor databases to Oracle 11 this morning has been completed successfully. There were useful discussions with MICE representatives present about their resource and other requirements for reconstruction of data at the Tier1. Operations report WLCG Grid Deployment Board - Agendas MB agendas June meeting Wednesday 13th June WLCG meeting notes Welcome [MJ] August meeting is cancelled. October meeting is in Annecy. EGI Technical Forum 17-21st September: http://tf2012.egi.eu/ HEPiX Fall – 15th -19th October. Post-TEG Working Groups [Ian Bird] Large number of WGs proposed. DM&S: Benchmarking. Federation. Networking WLM: Extensions of CE (multi-core; whole node; pilot support). Information System. Security: Proposals coming Database: share experiences. Operations: m/w sw process. Monitoring. Teams approaches: Operations coordination team. Sharing experiences/tech watch (pre-GDB discussion) Possibly Missing? Cloud. SRM (but to be more generic in title!). Bartch systems. Storage Accounting (John Gordon) StAR Plan to publish to APEL but in EMI-3 for May 2013 Interim possibility to use gstat Noted that information that is published is not precise. Information System Status and Evolution (Maria Alandes Pradillo) Caching BDII. In since February. Documentation improved. Use variable BDII_DELETE_DELAY. (https://tomtools.cern.ch/confluence/display/IS Failover. LCG_GFAL_INFOSYS Use 1& 2 BDII in region and 3 as CERN. On data quality - glue-validator (in EMI-1 ans 2) - glue 2 still to be deployed widely - Future work (EMIR; ginfo and IS monitoring/metdata). Question if OSG fully engaged? AAI on WN update (Romain Wartel) Security controls – central banning body required ARGUS locally needed (to pull banning lists from central ARGUS) Ownership of traceability. VO-site collaboration needed to cover all cases Recommendations to fulfill logging and traceability policy on WN. Not current possible to use clouds (VMs) in a way that conforms with WLCG security policies. Critical proxy extension (ALICE less limited) Proxy lifetime - reduce back to 24hrs? Balanced compromise between complexity and risk. Proxy credentials can not be revoked. Pool account recycling – recycle only after 6 months. EMI update (Cristina Aiftimiei) EMI-1 at update 15 (23.04.2012) EMI-1 Full support & maintenance until 28.02.2012. Updates till 31.10.2012. EMI-2 released 21.05.2012. Supports SL5 and SL6. Some Debian6. New products: CANL, EMIR, EMI-Nagios, Pseudonymity, WNoDeS. Hydra and WMS not released yet. Some backward incompatibilities due to existing EPEL package names. UI/WN tarballs in the next update. Globus SW support at OSG Discussions including use of Cream/Glue2; this to be investigated as it impacts use of the WMS EMI Sustainability Plans (Alberto Di Meglio) The end of EMI is the end of the coordination between product teams – not the end of those product teams. Ian Bird: the outcome of the above WLCG-EMI-EGI meeting needs to be how do we manage software in the future, also to discuss: how do we do certification, staged rollout and deployment in general. Communicating Machine Features to Batch Jobs (Tony Cass) Jeff will share a script for PBS to test implementation using /etc/machinefeatures. MUPJ – gLexec update (Maarten Litmaath) gLExec flag needs to be set in GOCDB Status at http://cern.ch/go/PX7 Deployment guide https://twiki.cern.ch/twiki/bin/view/LCG/GlexecDeployment CMS pushing because of July security challenge. Federated Identity Vision (Romain Wartel) Document presented at last GDB. Approved by MB on 5th June. Pilot project for WLCG - any volunteers to be involved?

NGI UK - Homepage CA Monday 2nd July Next meeting is on 9th July. Events WLCG workshop - 19th-20th May (NY) Information CHEP 2012 - 21st-25th May (NY) Agenda UK ATLAS - Shifter view News & Links Thursday 21st June Over the last few months ATLAS have been testing their job recovery mechanism at RAL and a few other sites. This is something that was 'implemented' before but never really worked properly. It now appears to be working well and saving allowing jobs to finish even if the SE is not up/unstable when the job finishes. Job recovery works by writing the output of the job to a directory on the WN should it fail when writing the output to the SE. Subsequent pilots will check this directory and try again for a period of 3 hours. If you would like to have job recovery activated at your site you need to create a directory which (atlas) jobs can write too. I would also suggest that this directory has some form of tmp watch enabled on it which clears up files and directories older than 48 hours. Evidence from RAL suggest that its normally only 1 or 2 jobs that are ever written to the space at a time and the space is normally less than a GB. I have not observed more than 10GB being used. Once you have created this space if you can email atlas-support-cloud-uk at cern.ch with the directory (and your site!) and we can add it to the ATLAS configurations. We can switch off job recovery at any time if it does cause a problem at your site. Job recovery would only be used for production jobs as users complain if they have to wait a few hours for things to retry (even if it would save them time overall...) UK CMS Tuesday 24th April Brunel will be trialling CVMFS this week, will be interesting. RALPP doing OK with it. UK LHCb Tuesday 24th April Things are running smoothly. We are going to run a few small scale tests of new codes. This will also run at T2, one UK T2 involved. Then we will soon launch new reprocessing of all data from this year. CVMFS update from last week; fixes cache corruption on WNs. UK OTHER Thursday 21st June - JANET6 JANET6 meeting in London (agenda) Spend of order £24M for strategic rather than operational needs. Recommendations to BIS shortly Requirements: bandwidth, flexibility, agility, cost, service delivery - reliability & resilience Core presently 100Gb/s backbone. Looking to 400Gb/s and later 1Tb/s. Reliability limited by funding not ops so need smart provisioning to reduce costs Expecting a 'data deluge' (ITER; EBI; EVLBI; JASMIN) Goal of dynamic provisioning Looking at ubiquitous connectivity via ISPs Contracts were 10yrs wrt connection and 5yrs transmission equipment. Current native capacity 80 channels of 100Gb/s per channel Fibre procurement for next phase underway (standard players) - 6400km fibre Transmission equipment also at tender stage Industry engagement - Glaxo case study. Extra requiements: software coding, security, domain knowledge. Expect genome data usage to explode in 3-5yrs. Licensing is a clear issue To note Tuesday 26th June On Tuesday 31st July 2012 the GOCDB read-write portal at https://gocdb4.esc.rl.ac.uk/portal will be decommissioned and replaced by a single read-write version at https://goc.egi.eu/portal. This will consolidate the service (including the PI) under the same URL. All GOCDB client maintainers are requested to ensure their PI configuration URLs point to https://goc.egi.eu/portal.