Operations Bulletin 020712

From GridPP Wiki
Jump to: navigation, search

Bulletin archive


Week commencing 25th June 2012
Task Areas
General updates

Monday 25th June

  • The batch systems APEL are planning to support are: PBS / Torque; LSF; Grid Engine and SLURM (to be confirmed). Do we have any other needs/requests?
  • For those interested in the GridPP review of the Tier-1 (meeting agenda).
  • Good GridPP presence at the JANET6 meeting last Thursday.
  • Q2 2012 finishes this week. Tier-2 quarterly reports due by 13th July.
  • Changing IP addresses & DNS (DNS updates)
  • ATLAS are considering increasing the limit of their ESD file size from the current 10GB to 15GB (or 12GB at least). Would this create any problem at sites?
  • ATLAS job recovery now appears to be working well - allowing jobs to finish even if the SE is not up/unstable when the job finishes (see ATLAS subsection).
  • A recent SL5 update (to python and its libraries), now in fnal and cern repositories, broke running ATLAS jobs last week.

Monday 18th June

Tuesday 12th June

  • There is a WLCG pre-GDB meeting on WN security - the agenda is here. Vidyo connection available.
Tier-1 - Status Page

Tuesday 26th June

  • Problem found & fixed with packets to the RAL Tier1 not being routed over the OPN link (13 - 21 June).
  • The major site networking upgrade successfully took place last Tuesday morning (19th June).
  • Problem with CMS SRMs over this last weekend affected SUM tests and FTS transfers.
  • Castor databases will be updated to Oracle 11 on Wednesday 27th June.. (Declared in GOC DB) Will also move the FTS database back to its correct location. Minimising interruptions to batch services during this time at VOs request.
Storage & Data Management - Agendas/Minutes

Considering input to a community support model for DPM and possible alternatives.

Wednesday 20th June

  • snoplus needs/plans on agenda last week.
  • The collaboration will depend on the RAL LFC and are looking to increase Tier-2 usage – current needs of 10TB/site will increase to 20TB/site.
  • Data taking will start in the autumn and continue for 6 months.

Wednesday 6 June 2012 - we are still digesting CHEP information, see also blog, plus a few of the usual operational upgradional stuff. Hoping to find a few spare clock cycles for some slightly more experimental stuff.

Wednesday 23 May 2012 - lots of exciting stuff at CHEP, we have about five things in, some posters, some oral.


Accounting - UK Grid Metrics HEPSPEC06

Wednesday 6th June - Core-ops

  • Request sites to publish HS06 figures from new kit to this page.
  • Please would all sites check the HS06 numbers they publish. Will review in detail on 26th June.

Friday 11th May - HEPSYSMAN

  • Discussion on HS06 reminding sites to publish using results from 32-bit mode benchmarking. A reminder for new kit results to be posted to the HS06 wiki page. See also the blog article by Pete Gronbech. The HEPiX guidelines for running the benchmark tests are at this link.


Documentation - KeyDocs

Wednesday, 6th June

Released a document, hep.ph.liv.ac.uk/~sjones/VomsSnooper.odt, that describes how to

  • Maintain site VOMS info document for the approved VOs
  • Check a site's VOMS records correspond exactly with CIC portal
  • Create new site VOMS records direct from CIC portal, without manual transcription

Note: I'm accepting tips from GridPP core task members etc. about other use cases for these processes. This will be converted to wiki formatted and made available in the normal way. Next jobs:

  • review logical/sequence of VOMS admin process, document it if it works, fix it if it doesn't.
  • create standard baseline for proxy renewal process, and write it up in wiki.

Note: I'm accepting tips from other Gridpp core team members etc. for document priorities. Please think about where the problems lie (i.e. what costs us yet is easy to fix) and get back to me.


Tuesday, 29th May

  • VOMS Records in GridPP Approved VO list now up to date with CICs Portal XML. This can be used by Site Admins to ensure their site-info.def/vo.d directories are up to date. A tool, SidFormatter, will be released this week to facilitate comparison with the benchmark. A process has been devised to ensure that GridPP Approved VO is kept up to date to within a week of CIC Portal changes. Consultation to be made about further fields that we may wish to advertise in this manner.

Friday 27th April

  • Appeal for a volunteer to enhance "Grid User Crash Course" (https://www.gridpp.ac.uk/wiki/Grid_user_crash_course) with simple use case for dependable proxy renewal for long jobs, as this is a recurrent requirement that has caused multiple queries on TB_SUPPORT.
Interoperation - EGI ops agendas

Monday 18th June

The EMI 1 updates are just minor revisions: Top BDII, BLAH and Storm. A repackage for GFAL/lcg-utils to handle the globus lib dependancy problems. Further EMI-2 updates, probably of interest only for those doing EA of them.

  • Staged rollout: Lot's of EMI-2 packages, working their way through the verification/SR process. The software that is just a repackage from EMI-1 to EMI-2 are skipping SR on SL5 - the SL6 versions will be tested. Most of the products in SL5, support upgrade and reconfiguration from the EMI1 versions.
  • Note that CREAM is one of the products that can't to an inplace update - new DB schema, so needs a drain/wipe/re-install.
  • Question from Tiziana - anyone using CREAM in Cluster Mode? Any feedback on that?


Monitoring - Links MyWLCG

Wednesday 6th June

  • Ranking continues. Plan to have a meeting in July to discuss good approaches to the plethora of monitoring available.
  • Glasgow dashboard now packaged and can be downloaded here.
On-duty - Dashboard ROD Rota

Monday 25th June - JW

  • Only one issue of note this week, namely I could not close the Cambridge ticket despite it being solved in GGUS. After filling in all the form data correctly, adding in the solution, setting the ticket to "solved" in the dashboard, the dashboard still complained that I had not filled in "something" with correct data. I extended the ticket by 1 day, and was able to close it the following day by (having entered in the same data). Anyhow, it is an unrepeatable bizarre and annoying problem.
Rollout Status WLCG Baseline

Monday 11th June

  • EMI2 is released but not in Staged Rollout yet. Buyers beware.

Thursday 10th May

  • The cream ce and the WMS which were released at the end of April have finally gone into Staged Rollout
  • Call for more sites to take part in EMI-2 rollout tests.
  • The overall SR contributions are in this table.

Friday 27th April

  • Updated version information on rollout page
  • WN scan indicates some sites not keen on OS updates to those nodes.
Security - Incident Procedure Policies

Monday 25th June

  • Rota availability responses slow
  • Is anyone following up on SSC5/6?
  • Stratuslab VM (ex UK)
  • gridftp



Services - PerfSonar dashboard

Tuesday 26th June

  • Cambridge added to UK dashboard
  • Currently clarifying the ownership of the GridMon boxes (assumption was ownership was transferred to sites but they may still be in a RAL asset database).


Tuesday 19th June

  • Some of the volunteer sites may not have perfsonar by end of June. Which other sites are close?
  • GridPP will resume running VOMS. Current plan is for the master to remain at Manchester and to host backups at Oxford/Imperial.


Tickets

Monday 25th of June, 14:00 BST</br> 17 open UK tickets this week, nothing very exciting going down.</br> Java has broken on my machine so I might not be able to join the meeting in time, but there's not too much to report.

DURHAM</br> https://ggus.eu/ws/ticket_info.php?ticket=83473</br> Yet another in the trend of lhcb cvmfs-related tickets, along with these two open tickets from last week:</br> https://ggus.eu/ws/ticket_info.php?ticket=82946 (IC, also atlas rather then lhcb)</br> https://ggus.eu/ws/ticket_info.php?ticket=83283 (GLASGOW)</br> NEW https://ggus.eu/ws/ticket_info.php?ticket=83577 (Brunel)


NGI</br> https://ggus.eu/ws/ticket_info.php?ticket=82492</br> Chris' ticket to the VOMS admins, possibly moot now (something to take into consideration for the new GridPP voms servers?).</br>

SOLVED CASES</br> https://ggus.eu/ws/ticket_info.php?ticket=83376</br> Oxford also saw some lhcb cvmfs troubles, although they were swiftly fixed.</br>

FROM THE UK</br> https://ggus.eu/ws/ticket_info.php?ticket=83133</br> na62.gridpp.ac.uk should (finally) have an fts instance up and running at CERN.</br>

NEW https://ggus.eu/ws/ticket_info.php?ticket=83562</br> Chris W has ticketed the APEL team concerning the default SGE apel parser memory settings being too low. This is a known problem, the APEL team are on it.

Tools - MyEGI Nagios

Saturday 23rd June

  • Email alerts from Nagios appear to have stopped. Reported to developers.

Tuesday 12th June

  • Lancaster backup Nagios now available (link).
  • A reminder as to who can see the data: all members of ops and dteam and any one who is registered in GOCDB as site-admin, regional manager etc. it is also possible to add any one who has a PKI certificate.


VOs - GridPP VOMS VO IDs Approved

Wednesday 6th June

  • Cross-checking VOs enabled vs VO table.
  • Surveying VO-admins for problems faced in their VOs.


Site Updates

Monday 25th June

  • N/A



Meeting Summaries
Project Management Board - MembersMinutes Quarterly Reports

Monday 25th June

  • Update on VOMS (based on ops discussion last week)
  • Summary of Tier-1 review (agenda)
  • Comments on JANET 6 meeting
  • Standing items
GridPP ops meeting - Agendas Actions Core Tasks

Tuesday 8th May - link Agenda Minutes

  • ATLAS DATADISK now being used for production input files
  • Deploy perfsonar-ps (for LHC compatibility), but run a Perfsonar-MDM portal to collate the information
  • Target date for perfsonar-ps at sites is the end of July
  • KeyDocs still not in place for all areas

Tuesday 1st May 2012 - Agenda Minutes

  • Trying the present bulletin as a conduit for information
  • 7 sites below 90% in ATLAS monitoring!
  • Instructions for small VO space-token setup/usage requested
  • EMI WN tarball now available - comment in ticket
  • Not every site running latest SL5 OS
RAL Tier-1 Experiment Liaison Meeting (Wednesday 13:30) Agenda EVO meeting

Wednesday 27th June

  • The move of the Castor databases to Oracle 11 this morning has been completed successfully.
  • There were useful discussions with MICE representatives present about their resource and other requirements for reconstruction of data at the Tier1.
  • Operations report
WLCG Grid Deployment Board - Agendas MB agendas

June meeting Wednesday 13th June

WLCG meeting notes

Welcome [MJ]

  • August meeting is cancelled. October meeting is in Annecy.
  • EGI Technical Forum 17-21st September: http://tf2012.egi.eu/
  • HEPiX Fall – 15th -19th October.

Post-TEG Working Groups [Ian Bird]

  • Large number of WGs proposed.
  • DM&S: Benchmarking. Federation. Networking
  • WLM: Extensions of CE (multi-core; whole node; pilot support). Information System.
  • Security: Proposals coming
  • Database: share experiences.
  • Operations: m/w sw process. Monitoring.
  • Teams approaches: Operations coordination team. Sharing experiences/tech watch (pre-GDB discussion)
  • Possibly Missing? Cloud. SRM (but to be more generic in title!). Bartch systems.

Storage Accounting (John Gordon)

  • StAR
  • Plan to publish to APEL but in EMI-3 for May 2013
  • Interim possibility to use gstat
  • Noted that information that is published is not precise.

Information System Status and Evolution (Maria Alandes Pradillo)

- glue-validator (in EMI-1 ans 2) - glue 2 still to be deployed widely - Future work (EMIR; ginfo and IS monitoring/metdata). Question if OSG fully engaged?

AAI on WN update (Romain Wartel)

  • Security controls – central banning body required
  • ARGUS locally needed (to pull banning lists from central ARGUS)
  • Ownership of traceability. VO-site collaboration needed to cover all cases
  • Recommendations to fulfill logging and traceability policy on WN.
  • Not current possible to use clouds (VMs) in a way that conforms with WLCG security policies.
  • Critical proxy extension (ALICE less limited)
  • Proxy lifetime - reduce back to 24hrs? Balanced compromise between complexity and risk. Proxy credentials can not be revoked.
  • Pool account recycling – recycle only after 6 months.

EMI update (Cristina Aiftimiei)

  • EMI-1 at update 15 (23.04.2012)
  • EMI-1 Full support & maintenance until 28.02.2012. Updates till 31.10.2012.
  • EMI-2 released 21.05.2012. Supports SL5 and SL6. Some Debian6.
  • New products: CANL, EMIR, EMI-Nagios, Pseudonymity, WNoDeS.
  • Hydra and WMS not released yet.
  • Some backward incompatibilities due to existing EPEL package names.
  • UI/WN tarballs in the next update.


Globus SW support at OSG

  • Discussions including use of Cream/Glue2; this to be investigated as it impacts use of the WMS

EMI Sustainability Plans (Alberto Di Meglio)

  • The end of EMI is the end of the coordination between product teams – not the end of those product teams.
  • Ian Bird: the outcome of the above WLCG-EMI-EGI meeting needs to be how do we manage software in the future, also to discuss: how do we do certification, staged rollout and deployment in general.


Communicating Machine Features to Batch Jobs (Tony Cass)

  • Jeff will share a script for PBS to test implementation using /etc/machinefeatures.

MUPJ – gLexec update (Maarten Litmaath)

Federated Identity Vision (Romain Wartel)

  • Document presented at last GDB. Approved by MB on 5th June.
  • Pilot project for WLCG - any volunteers to be involved?



NGI UK - Homepage CA

Monday 11th June

  • Main discussion was around NGS funding and the impacts on GridPP.
Events

WLCG workshop - 19th-20th May (NY) Information

CHEP 2012 - 21st-25th May (NY) Agenda

UK ATLAS - Shifter view News & Links

Thursday 21st June

  • Over the last few months ATLAS have been testing their job recovery mechanism at RAL and a few other sites. This is something that was 'implemented' before but never really worked properly. It now appears to be working well and saving allowing jobs to finish even if the SE is not up/unstable when the job finishes.
  • Job recovery works by writing the output of the job to a directory on the WN should it fail when writing the output to the SE. Subsequent pilots will check this directory and try again for a period of 3 hours. If you would like to have job recovery activated at your site you need to create a directory which (atlas) jobs can write too. I would also suggest that this directory has some form of tmp watch enabled on it which clears up files and directories older than 48 hours. Evidence from RAL suggest that its normally only 1 or 2 jobs that are ever written to the space at a time and the space is normally less than a GB. I have not observed more than 10GB being used. Once you have created this space if you can email atlas-support-cloud-uk at cern.ch with the directory (and your site!) and we can add it to the ATLAS configurations. We can switch off job recovery at any time if it does cause a problem at your site. Job recovery would only be used for production jobs as users complain if they have to wait a few hours for things to retry (even if it would save them time overall...)
UK CMS

Tuesday 24th April

  • Brunel will be trialling CVMFS this week, will be interesting. RALPP doing OK with it.
UK LHCb

Tuesday 24th April

  • Things are running smoothly. We are going to run a few small scale tests of new codes. This will also run at T2, one UK T2 involved. Then we will soon launch new reprocessing of all data from this year. CVMFS update from last week; fixes cache corruption on WNs.
UK OTHER

Thursday 21st June - JANET6

  • JANET6 meeting in London (agenda)
  • Spend of order £24M for strategic rather than operational needs.
  • Recommendations to BIS shortly
  • Requirements: bandwidth, flexibility, agility, cost, service delivery - reliability & resilience
  • Core presently 100Gb/s backbone. Looking to 400Gb/s and later 1Tb/s.
  • Reliability limited by funding not ops so need smart provisioning to reduce costs
  • Expecting a 'data deluge' (ITER; EBI; EVLBI; JASMIN)
  • Goal of dynamic provisioning
  • Looking at ubiquitous connectivity via ISPs
  • Contracts were 10yrs wrt connection and 5yrs transmission equipment.
  • Current native capacity 80 channels of 100Gb/s per channel
  • Fibre procurement for next phase underway (standard players) - 6400km fibre
  • Transmission equipment also at tender stage
  • Industry engagement - Glaxo case study.
  • Extra requiements: software coding, security, domain knowledge.
  • Expect genome data usage to explode in 3-5yrs.
  • Licensing is a clear issue
To note

Tuesday 26th June

  • There is an upcoming transition of JISCMail - the host of many of our email lists. There has been confirmation of the date and time for the transition to be Friday 29th June at 5:00 p.m. (evening) for the switch-over from STFC to the JISC Advance in-house service. This means that all support responsibilities, helpdesk responses and associated service turn-around targets will no longer fall to STFC but to JISC Advance. Any issues around JISCMail service availability will be for JISC Advance to resolve.