Sections

Deployment

Users

Administrators

Security

Grid Basics

This page attempts to give a quick overview of basic Grid concepts, and in particular those relevant to the LCG/EGEE Grid, from the point of view of a UK HEP user. Unfortunately there are a lot of new concepts (and acronyms) and the learning curve is rather steep, so this can only be an introduction. There are many other introductory descriptions around; see the web sites listed in the GridPP introduction or in the list of documentation links. Also see the GridPP acronym list, the iSGTW glossary, the EGEE Glossary and the Italian Grid Dictionary. Some of these are quite old, but much of the information is still valid.

What is a Grid?

There are many definitions of Grids, and this description is oriented to the HEP view of things. The picture of Grid systems that you might get from Oracle or IBM, or even from other academic disciplines, is likely to be somewhat different.

Twenty years ago, when people wanted a powerful computer system they generally bought a single large machine (a mainframe). However, during the 1990s the cost of PCs fell dramatically, and the free Linux operating system became widespread, so the cheapest way to get a given amount of computing power was to buy a large number of PCs running Linux. This was particularly suited to HEP applications because they usually involve the processing of large numbers of discrete events, which can easily be broken up and split across a large number of machines. Also people in HEP experiments were used to developing their own systems rather than relying on off-the-shelf solutions from vendors.

Initially such systems were operated as "farms" of a few hundred machines at the major laboratories, with each farm being operated and used independently. However, in the last few years both the size of individual farms and the number of different sites operating farms has grown dramatically; in the UK many HEP groups in Universities now have their own major facilities.

The Grid is a general term for a set of technologies (known as middleware) which allow a large number of such farms, distributed over the whole world, to be used as though they were a single system. Conceptually this seems like a fairly simple idea, but the reality is extremely complex and involves attention to a vast number of details. Grid systems in HEP have been in development since about 2001, and are now large and work reasonably well. However, the technology is still very much in development, so users should be prepared to find many rough edges, and to see quite significant changes over the next few years.

There are many technologies used to create Grids, and many functioning Grid systems around the world. However, for the purposes of GridPP the unqualified term "the Grid" generally refers to the production system run jointly by the LCG and EGEE projects (and associated projects like GridPP), and/or the set of software used within it.

Grid Projects

There are many Grid projects, and the relationships between them can often be confusing. The ones mentioned below are the most relevant for GridPP users.

The earliest Grid tools were developed by the Globus project. These consisted of the basic Grid security infrastructure (based on digital certificates), a secure way to send computing tasks (jobs) to a remote site, a file transfer tool and a way of distributing information about Grid resources. In some form these are still the foundation of the current Grid.

The Condor project developed a system similar to the familiar screensaver programs (e.g. SETI@home) which allowed the use of otherwise idle time on machines used for other purposes. This has since been integrated into the Grid world and is now used in various aspects of job submission systems.

The Global Grid Forum (GGF), recently renamed as the Open Grid Forum (OGF) after a merger with another organisation, is a worldwide body aiming to standardise the many services and protocols in use in the Grid world. Most middleware developers participate in the OGF to try to ensure convergence and interoperability between services developed by different projects. However, at present the technology is still fairly immature so this is not always achieved in practice.

The European DataGrid (EDG) was an EU-funded project which ran from 2001-04, with the aim of developing a more extensive set of middleware to enable a useful, widespread Grid to be built. The project was based at CERN and was very strongly connected with the HEP community, but also involved users from biomedical and earth-science communities.

Enabling Grids for E-sciencE (EGEE) is the somewhat strangely named successor to EDG. However, the project is much larger, and has the aim of developing a general Grid infrastructure for scientific research in Europe, and also has links to projects in many other countries. It was initially approved for two years 2004-06, but was subsequently extended to 2008 and then 2010, and is likely to be succeeded by a permanent organisation to operate the Grid it develops.

The LHC Computing Grid (LCG) project is run from CERN, and has the task of developing all the computing infrastructure needed for the LHC experiments, which are due to start taking data in 2008. The vast data and computing requirements of the LHC have been one of the major stimuli for the development of Grids.

There is a strong, complex and often confusing relationship between LCG and EGEE. Both projects are based at CERN, the main Grid infrastructure and middleware are essentially the same for both, and many people have roles in both. However, their goals are somewhat different. In some areas the scope of LCG goes beyond EGEE, e.g. the development of HEP-specific software and the use of Grid infrastructures which are not part of EGEE. Conversely, EGEE is intended to serve the entire scientific community in Europe and not just HEP. Members of HEP experiments not based at CERN are formally clients of EGEE and not LCG, but the distinction can sometimes get blurred.

The Open Science Grid (OSG) is in some respects the US equivalent of EGEE, and provides most of the US computing resources for LCG. It runs much of the same middleware as EGEE, and many things are similar, but there are also substantial differences. There is currently a lot of work in progress to make the two systems interoperable in a transparent way.

Last but not least, GridPP is the UK Particle Physics Grid project. In most respects it can be considered as the UK component of LCG and EGEE, and is one of the largest contributors to them. More information about GridPP can be found elsewhere on this site.

The LCG/EGEE Grid

For most purposes the LCG and EGEE Grids are the same, even though particular sites may be affiliated to one or the other (all GridPP sites are part of both EGEE and LCG). For the big picture have a look at the Grid Map. You can see that most Grid sites are in Europe, but others are scattered around the world. The resources at the various sites vary from a few machines to a few thousand, and from data storage on single hard disks up to multi-petabyte tape stores.

In theory, the Grid concept is that all sites are equal, everything exists in a "cloud" from which the Grid middleware should select the best resources for whatever you want to do. However, for practical reasons LCG and EGEE impose some structure on the system, although they do it in somewhat different ways.

LCG has the concept of "tiers" of sites. Tier 0 is CERN, the source of all the LHC data. The tier 1 sites are major computing centres around the world; in the UK the tier 1 centre is at the Rutherford Appleton Laboratory (RAL) near Oxford. Each tier 1 site then has some satellite tier 2 sites. In the UK we have four "virtual" tier 2 centres, each of which consists of several sites. In principle LCG also considers tier 3 (local computing facilities for local people) and tier 4 (the machine on your desk), but these are not really part of the general Grid infrastructure.

EGEE organises by region, with each region being substantially autonomous. GridPP is part of the UK-Ireland (UKI) region, which also includes the Grid Ireland project and the UK e-science National Grid Service (NGS). The regions are co-ordinated by a Regional Operations Centre (ROC), which for UKI is again at RAL. ROCs are often co-located with tier 1 sites as they both tend to be at major centres, but this is not essential.

EGEE is developing a model for operating the Grid as a production-quality service. In such a large system problems of all kinds are constantly arising - if a single machine would fail once in ten years, 100,000 machines will see a failure once an hour! A range of monitoring tools has been developed, and people in the ROCs take it in turns to monitor the whole Grid for a week at a time, and follow up on any problems they find. Mechanisms are also being developed to screen out any sites with problems so that the impact on users is reduced.

Virtual Organisations

The term Virtual Organisation (VO) is widely used in the Grid world, but the underlying meaning can vary somewhat. Broadly speaking it refers to a group of users, and sometimes of hardware resources, who may belong to various different real-world organisations but who want to work together on some common task. As far as HEP users are concerned a VO is generally the same as an experiment; for example, the CMS experiment has a VO called cms.

To do anything in the Grid you have to belong to at least one VO - most people will only belong to one. Each VO has administrators who maintain lists of members, having checked that they are entitled to belong. Each VO will have its own policies about what its users are allowed to do on the Grid. Mechanisms are being developed to assign users to groups and roles within the VO which may have differing privileges and priorities.

Each site decides which VOs it will allow to use its services, and what level of resources they can get. Use of resources by each VO is accounted over the entire Grid. VOs may also run services of their own which interact with the Grid.

Job Submission

The most basic Grid function is job submission, i.e. performing some computing task on a remote machine. Jobs are specified using a Job Definition Language (JDL), and this, together with various files (e.g. a job script or the source code for a program) are sent to a Resource Broker (RB). This decides on the best place to run the job, sends it there and retrieves the output once it finishes. To some extent it can also deal with errors, e.g. by re-running a failed job at a different site.

Each site accepts jobs from the RB through a machine known as a Computing Element (CE) or a gatekeeper. The jobs are then passed to something known as a Local Resource Management System (LRMS) or more simply as a batch system. There are various such systems in use, of which the most common are variants of an open-source package called the Portable Batch System (PBS). This puts jobs into queues, and then dispatches them to execute on a suitable free machine, known as a Worker Node (WN).

Data Management

Small files can be sent and retrieved directly through the job submission system, but Grid jobs often need to deal with very large files. Dedicated Grid storage and data movement facilities are therefore provided.

Grid data is stored on systems known as Storage Elements (SEs). The storage behind them can vary from single hard disks, to large disk arrays, to robotic systems managing very large volumes of tape storage, but the Grid interface is kept uniform as far as possible. The current state of the art is a protocol called Storage Resource Manager (SRM) which has been developed by a group of HEP laboratories, including RAL. This provides mechanisms for storing, retrieving and copying files, and obtaining information about the underlying storage system. Data storage is a complex area and this protocol is still under development, and is likely to be so for several years as more features are added.

Grid files may be stored on different SEs around the world, and potentially there may be many replicas of the same file in different places. Even with fast networks it may take a long time to move large data volumes from one place to another. There is therefore a substantial need for facilities to manage files on a Grid-wide basis, and this is an active area of development; the middleware in use is currently in a state of flux. Some of the basic components are:

  • File catalogues assign Grid-wide names to files (or collections of files) and store their current location(s).
  • Metadata catalogues associate so-called metadata with files, i.e. information about the kind of file and its content.
  • A reliable file transfer service manages the movement of files between SEs, allocating bandwidth according to priorities and making sure that the files are transferred successfully.
  • A data scheduler manages requests for file movements over the entire Grid to make the optimum use of bandwidth and storage.

Information Systems

In a large distributed system, getting information about the state of the system is an important task. Broadly speaking this divides into two areas: resource discovery tells you what resources are out there and what their properties are, and monitoring tells you the current state. It is also useful for users or VOs to be able to transmit information of their own across the Grid.

To be useful the information must be published in a common format (known as a schema) across the entire Grid. It is also useful if different Grid infrastructures can use the same schema where possible. Most information is currently published using something known (for somewhat obscure reasons) as the Glue schema, which is defined in a joint project between several of the major Grids.

Information must also be transported across the Grid, in a way which is reasonably fast and reliable. There are currently two general technologies in use for this. One is based on a standard protocol known as LDAP (Lightweight Directory Access Protocol). This presents information as key-value pairs organised in a hierarchical tree. Information from across the Grid is collected in a server known as a BDII (Berkeley Database Information Index), by issuing queries to similar servers at each site.

A second service known as R-GMA (Relational Grid Monitoring Architecture) has been developed within EDG/EGEE. As the name suggests this uses a relational data model, i.e. the data are organised in tables, with relations between the tables via common (key) fields. It is much easier to define new tables in R-GMA than to change the LDAP schema, and publication of data into R-GMA is also much easier, so this is more suitable for user information. Data may be collected from across the Grid in an archiver in a similar way to the BDII, or it may be read directly from each site.

Last modified Tue  3 March 2009 . View page history