After the recent GridPP meeting in Manchester it is a good time to take stock of how GridPP is doing in an international context. It is looking great for the UK with 17% of the total time provided by the grid in Europe being done on GridPP machines. This amounts to almost 30 million more hours than Germany (in 2nd place) and puts France and Italy in 3rd and 4th places respectively.

Graph showing percentage of CPU hours provided by NGIs in Europe to all VOs (click to enlarge).

The UK has been involved since the very beginning of grid computing in Europe, even being one of the first countries to actually form the original testbed over a decade ago. Since then GridPP has been funded to provide support to the LHC with hardware and money allocated based on the depth of the UK’s involvement in a particular experiment. Despite the relatively low UK participation in ALICE and LHCb the collaboration processes over a million jobs a year for both these projects. “Efficiency is key to allowing us to perform at the level we do. I firmly believe it is our knowledge, expertise and experience that allows us to consistently be one of the top resource providers in the world. You can buy as much kit as you want, or think you need, but without the people and knowledge that 10 years brings you, you can’t run an infrastructure ” explains Pete Gronbech, GridPP’s Project Manager “We are delighted to support all of the LHC experiments and at the heart of this success is our operations team. The recent meeting in Manchester really demonstrated that the people we have are world class and I’m not surprised by these numbers”.

Graph showing percentage of CPU hours provided by all countires to all VOs (click to enlarge).

Sadly being number one in the world is not as easy but the UK comes close in 2nd place., the only bigger country worldwide is the United States of America. However there is one silver lining, the UK is a bigger supporter of non-LHC experiments with over 25 million hours compared to the USA’s 7 thousand hours.

“The U.S. is a much larger country than the UK and their support of the LHC experiments is exceptional but I still think we compare favourably” says David Britton, GridPP’s Project Leader “Especially when you look at our support of the other project’s using the grid. We are committed to making sure that the LHC researchers in Britain have the resources they need. However we are also aware that what we have built over the last 10 years can be used by all academics, regardless of their discipline. We are always looking for new collaborations and hopefully our work with NGS and EGI can help expand the use of grid into new areas”.

The theme for the 2 days was “UK operations” and after a welcome from Prof Steve Watts, the head of Manchester’s School for Physics and Astronomy and long-time associate of GridPP, the first day got under way. To start were 2 overviews on high level aspects of GridPP from Dave Britton, the project leader, and Pete Gronbech, the project manager and a great talk from Prof. Chris Parkes, which gave an overview of LHC physics, at a level that was appreciated by many of the audience. After lunch there was a session of micro-talks on various aspects of GridPP operations, from monitoring and accounting to documentation and interoperability. The day ended with an open ended discussion session based on any issues raised by the micro-talks.

GridPP on the display of the Baby replica (Click image to be taken to a video of Baby in action)

For dinner, the entire meeting decamped to the Museum of Science and Industry (MOSI) in the centre of town to a fantastic spread sponsored by Dell and Viglen alongside a working replica of Baby, the world’s first stored-program electronic digital computer. The night didn’t end there and the local hostelries benefited from the custom of many members of the collaboration and their discussions.

GridPP members during a coffee break at GridPP28

However there was still one more day to go and bright and early on Wednesday morning it was a chance for the experiments GridPP supports to have their say. All 4 LHC experiments were presented and Chris Walker and representatives from some of the smaller experiments talked about the other VOs and the work GridPP has done with them. After a quick break for tea and coffee it was back to very in-depth nuts and bolts talks looking at networking, storage and optimising “many core” resources. The final session after lunch focussed on security but not before a status update from the Tier1 at RAL from Andrew Sansum.

Almost too soon it was then time to head home but after 2 days there was plenty to think about and a lot of things had been added to everyone’s to-do list. David Britton was especially happy with how the meeting went “It was great to have such a professionally hosted meeting at Manchester and I must thank all those who were involved in making it such a success. I think we made good progress in understanding the absolutely central role that the ops-team needs to continue to play in GridPP. The ops-team really helps demonstrates the value-added by having a GridPP project”.

The agenda can be seen here and all the talks from the 2 days can be downloaded from there.

GridPP29 will be held from the 25th to the 27th of September in Oxford.

Each year the high energy physics community gather to present their latest work and hear about the latest developments from other researchers in the field. This year the meeting was held at Queen Mary, University of London and with it being the centenary year of the discovery of cosmic rays it was the perfect time to share the event with the astro particle physics (APP) community.

As usual the first day included a town meeting with the main funding body for the HEPP group, the Science and Technology Facilities Council (STFC). The recently appointed Chief Executive Officer John Womersley opened that session with plenty of discussion from the attendees during it. However the highlight of the 1st day was Prof. Don Perkins talking about the early days of cosmic ray research and his experiences in the field in 2nd half of the 20th century.

Question Time at the STFC Town Meeting.

The rest of the week was filled with networking, cutting edge physics, conference dinners and fruitful discussions. A lot of the research presented during the week would not be possible without the resources provided by GridPP, and its international partners wLCG and EGI. GridPP itself supports the LHC experiments and other high energy particle physics experiments. In the same way the European APP community are starting to make greater use of the European grid through experiments like The Pierre Auger Observatory, the MAGIC telescope, LOFAR and the Cherenkov Telescope array among many others. However UK involvement is low so GridPP had a stand in the exhibition area selling the benefits of grid to that community.

The meeting website can be found at http://pprc.qmul.ac.uk/research/iop-2012 with the timetable on Indico http://indico.cern.ch/conferenceDisplay.py?confId=175304.

The Large Hadron Collider (LHC) smashes together protons at close to the speed of light and is the world’s most powerful machine. Nothing has ever been built on this scale before. Prof. Clarke’s talk was a vivid examination of the implications that this has not only on physics but also how the community copes with crunching the data being generated. Starting from the very beginning of time Prof. Clarke laid out what we currently understand about the universe from its state just after the big bang to the unaccountable 95% of its contents. With the LHC taking us to within 10-12 seconds of the big bang he discussed the massive engineering challenges this brings, super conducting magnets, 5 storey high detectors and the attendant data deluge. He also discussed the latest physics results, with data on the Higgs boson that was less than two weeks old. Once he had taken the audience on this tour around the world’s largest experiment he took questions from the floor covering black holes, super symmetry, matter/anti-matter imbalance and even the carbon footprint of the LHC.

Prof. Peter Clarke at the Kelvin Lecture

“I was very honoured to be asked by the IET to give this year’s Kelvin lecture, and delighted I could combine the physics, the engineering and the computing sides of my interests in the LHC” says Prof. Clarke “The crowd were great, I wasn’t expecting such a turn out and the questions afterwards showed they really engaged with the talk”.

Hosted by the Institute of Engineering and Technology (IET), the lecture is one of the most prestigious in the UK. Set up to honour Lord Kelvin, the first one was in April 1908, only 4 months after the death of the giant of modern science. Prof. Clarke joins a very illustrious list of previous speakers including Ernest Rutherford (1922), J J Thomson (1926), Fred Hoyle (1970), Harry Kroto (1998) and Simon Singh (2007).

You can watch the recorded lecture on IET TV.

The masterclass programme has been running since 1997 and is an opportunity for students still at school to learn about cutting-edge science and find out what it means to do research for a living. The great thing about the idea is that each university decides how to put together their own day, want to focus on the LHC? Fine. More interested in talking about particle physics as a discipline? That works too. This year GridPP has been invited by some of the organisers to include some content about grid computing. The grid is a major tool in modern research, especially particle physics and the UK is a major partner in the global project managing it. Computing already plays a big part in some of the masterclasses. One of the popular activities uses data from particle physics experiments to create event displays and the students are asked to identify the particles present. While this is just a simulation, it is in the spirit of the kind work done on the grid by researchers.

Student’s attending UCL’s masterclass being shown around their computing cluster

In the last 15 years the masterclasses have grown in the UK but since 2005 they have had an international dimension. The International Particle Physics Outreach Group has used the template created in the UK to run the International Masterclasses connecting schools and universities around the world. Using the results discovered by the teams at each university during the computing exercises they organise a video conference for discussion and combination of the results between two or more schools in Europe. This helps really explain the virtual and distributed nature of the discipline, highlighting the excitement of international collaboration and the benefits of working together on these global projects.

This year the Daresbury Laboratory (8th and 9th of March), University of Oxford (12th and 13th of March), UCL (19th of March), Queen Mary, University of London (21st of March), and Bristol University (27th and 28th of March) will all have talks about GridPP included in their programme. If you have an idea for an event that GridPP could be involved in or want to work with us on a similar event just contact us on info@gridpp.ac.uk.

Maintaining up-to-date software at every site for each experiment is technically difficult, costly and current solutions are prone to problems. The software used by the LHC experiments is constantly evolving, as people’s needs change it changes. This has led to different choices being made by various experiments and institutes about what software they run and how they interact with the computational resources on offer.

Jakob Blomer, CernVM-FS Developer

This means that there are frequent changes to the code and the applications are gigabytes in size. So shipping a copy of the software with every job is impractical but ensuring every site has a local up to date copy is almost impossible. Developers at CERN however saw some important characteristics in the software that they could exploit to solve the problem. “CERN Virtual Machine File System (CernVM-FS) actually started life as a part of a larger CERN project, the CERN Virtual Machine, designed to create an easy install software environment that would be up-to-date as long as it had internet access” explains Jakob Blomer, who is developing CernVM-FS “This last part appealed to people working on the grid and we started testing CernVM-FS as a stand alone service”.

The first realisation was that the software mirrored the experiments themselves. They look like one single entity but when you dig deeper you see that individuals are only interested in certain analyses and certain parts of the whole. This means while everyone had things in common they are unlikely to need everything the software could do. The next was that different parts of the software used duplicate files, linking these would mean that file could be downloaded once but used by various aspects of the applications. Also updates to a piece of software usually only meant small changes so that differences between different releases were relatively minor. The final characteristic was that if the software looks for a file that is not there it doesn’t remember that it failed to find it and will look for it again if asked to. These “unsuccessful lookups” happen as often as successful, reducing them would reduce the load on the machine it is running on. Using these things as a starting point the team developed a system for providing up-to-date software on the fly, CernVM-FS.

CernVM-FS at its heart is quite a simple concept and boils down to various computers remembering what components it has used or seen recently. The experiment’s software developers keep their part of the code up-to-date on a server at CERN. CernVM-FS then treats the various pieces of software as if they are individual files The service then tags all these files with what is called a hash that is generated based on its details so that identical components have identical hashes. An institute’s grid cluster stays in contact with the CERN server via a local machine called a squid, this means everything transferred over CernVM-FS passes through a single machine and it can track what has been downloaded. Therefore when a piece of software running on a computer needs the next step in the application it looks to see if it is already on that computer, if not it asks the squid if it has been used elsewhere on the cluster and finally if necessary gets the squid to download it from CERN. This means the correct component is asked for (thanks to the hash), that it is up to date and downloaded only when necessary, reducing load on the computers, bandwidth and the sys admin.

One of the first sites outside CERN to test and use CernVM-FS was the Rutherford Appleton Laboratory (RAL), in Oxfordshire. Ian Collier was a member of the team working on that “We first heard about CernVM-FS in the summer of 2010. We’d been experiencing acute performance problems with (one of) our experiment software servers, affecting both job performance and site availability. I saw the potential to solve both these problems, at the same time reducing overhead for both the site and the experiments. This seemed almost too good to be true. After a few months of testing with ATLAS and then LHCb we migrated to using CernVM-FS for all ATLAS and LHCb jobs early in 2011”.

As the usefulness of the software became more apparent the decision was made to get it up and running across GridPP. “We were very happy with the results coming from RAL” says Alessandra Forti, who has led the effort in the UK to get GridPP using CernVM-FS “so we started testing it on other sites. It has not always been simple but installing the software was not intrusive so sites did not have to go offline at any stage. It only took us 3 months to move it into a production service across the entire collaboration, that is close to a site a week. Now we are happy to say that we are providing the best service we can to our users”. GridPP also hosts a replica of the CernVM-FS repository at RAL, allowing sites in the UK to download the what they need from a server in their own country.

Currently CernVM-FS is being used by the ATLAS and LHCb experiments but it is software agnostic, it can be used by anyone. Chris Walker from Queen Mary, University of London is involved in supporting new and small VOs on GridPP “We were the first site in the UK to put CernVM-FS into production. I believe that if we can support CernVM-FS for the LHC experiments there is no reason why we couldn’t support the same thing from any experiment. This makes it easier for a site to support multiple disciplines any make the decision to support a new VO a simple one. But most importantly it makes it easy for a community to maintain its software and push updates out across the grid so they can do there work quickly and efficiently”.

In 1964 three ground breaking papers were published in Physical Review Letters that are widely recognized as milestones in modern physics and gave birth to the theory of the Higgs Boson. Currently it is the missing piece of the Standard Model, a theory that explains how the building blocks of the universe interact and behave. Put very simply without the Higgs there is no way of explaining how anything has mass, but that is not the whole story. Without the Higgs we simply don’t know how the universe works. The announcement today is one step closer to finishing the puzzle. “The results are tantalising, but not yet conclusive.” says Roger Jones, head of ATLAS Computing UK “The full picture will be known by next Summer, and accelerator, detectors and computing around the world will be working hard to add even more data to settle the issue until then.”

Simulated production of a Higgs event in ATLAS.

Peter Higgs Visiting the CMS expriment

Named after Prof. Peter Higgs of the University of Edinburgh, the eponymous boson has been searched for in numerous experiments including the Tevatron, which just recently finished running and the LHC’s predecessor at CERN, LEP. There have even been other experiments that have operated at the right energy level with the possibility of creating the Higgs. The difference with the LHC? The volume of data. The experiment is simply running with enough power that if the Higgs exists it is creating it in a far greater abundance than any other experiment. And this is important when you work with statistics, there is no single eureka moment, but there are eureka plots.

The scientists looking for the Higgs know what it could look like, and they have looked for how they expect the Higgs to interact with the detectors. This is where the grid comes in. The UK has a leading involvement in ATLAS and CMS but also is one of the largest resource providers for the LHC Computing Grid. As Roger explains “The fact we can present sophisticated results based on the 1000s of terabytes of data that were collected up to only three weeks ago is an astonishing tribute to the power of our grid computing systems.” Using the grid to analyse the collisions at CERN resulted in a peak in the data at around 125GeV. This is a result that agrees with theory and suggests that the Higgs is about 120 times heavier than a proton. While the hint is currently no more than a spike on a graph of data points, more importantly it is a spike on two independent graphs from different experiments that give physicists hope that with more data they will announce the discovery of the Higgs in 2012.

The idea of a computational grid as envisaged by Foster and Kesselman in their book “Grid: Blueprint for a New Computing Infrastructure” is on the face of it very simple. A grid should enable access to distributed remote computational resources when needed by the user. Back in 2001, however, it was proving to not be as easy as it sounded. Various projects and institutes had started the work of creating a system which would allow users from across the world access to the computing resources they needed without the hassle of negotiating deals with individual providers or even choosing what resources to use. One of these was the European DataGrid project (EDG), which would become the Enabling Grids for EsciencE projects and ultimately evolve into the European Grid Infrastructure. GridPP had become heavily involved in EDG from the start and by mid-2001 the project had a series of independent sites that users could submit work to by hand to test some of the tools developed by the partners.

Entire Manchester cluster (Foreground) circa 2001

“We had actually done a lot in a very short period of time”, David Britton at the time GridPP’s Project Manger, “GridPP had gone from a plan for a proposal in December 2000 to a fully funded infrastructure project, within less than a year. And we were about to make the first big step towards a proper grid”.

That step was pretty big though; they needed two very important components, an information system and a resource broker. By monitoring the state of the resources on the infrastructure, and taking into account a user’s requirements, these would allow a user to submit jobs “into the grid” and have them automatically routed to a suitable site. By early December the first versions of these tools were available and over the weekend of the 8th and 9th of December CERN, Manchester University, the CNAF site at Bologna and the Rutherford Appleton Laboratory, in Oxfordshire had installed the software. By the Monday morning they had created the first international grid infrastructure and on the Tuesday it was officially announced at an EDG meeting that the testbed was up and running with the four sites. It wasn’t long before they had company with CCIN2P3 in Lyon and NIKHEF in Amsterdam joining them within a week.

They were pretty humble beginnings, the entire system only consisted of 14 machine, Manchester’s entire cluster was a single worker node. Andrew McNab from Manchester was part of the integration team that was involved in the work, “It was a pretty exciting time as we were motivated by the idea of connecting hundreds of sites to do computing in a new way for the LHC. The Integration Team had come to CERN for a fortnight from all across Europe to make it work, and I found out that the CERN and CNAF/Bologna sites had gone live on the Sunday afternoon. I put the final pieces in place on the Manchester site and then late on Sunday night successfully submitted a job to the Broker service at CERN the job was then run at Manchester. It was like seeing the World Wide Web in the early 1990s: it wasn’t very big and things were very rough round the edges, but you knew the technology would scale up enormously and change the way everyone worked.”.

In the intervening decade the grid has become indispensable to many communities, the LHC experiments would be lost without the 200,000 machines they can access to do their analysis. However the grid is now able to look forward to the next ten years, and beyond, supporting researchers from across the globe and in numerous disciplines.

LHCb is one of the 4 main experiments using the Large Hadron Collider accelerator at CERN. They are looking into an area called B physics, investigating various aspects of physics theory including the matter/anti-matter imbalance in the universe. In particular they are interested in bottom quarks and their creation during the collisions in the LHC. Running alongside the other 3 experiments LHCb has been collecting and processing data since 2008 using almost 400 million CPU hours on the grid. However this year was the first time they have accumulated an inverse femtobarn of data in one calendar year. This equates to about 70 billion b-quark pairs produced by the collisions which occured in the LHCb detector. The analysis of this data is going ahead with a lot of exiciting results in the pipeline for various conferences next year.

Pie Chart of data reprocessing for LHCb by country

“The LHC has been extremely successful in producing an amazing amount of data” explained Dr. Marco Cattaneo, the LHCb Computing Co-ordinator. “Processing all this data ahead of schedule would not have been possible without the grid and has proved that the grid is needed by our community. The UK contribution has been extremely useful in supporting this.”