Summary of HEPCAL-I for Metadata GroupSteven Hanlon, May 2004 |
| Use Cases for Metadata | Metadata Working Group | |
Introduction
HEPCAL-I seeks to identify implementation-independent use cases and requirements for computing Grids, common to the LHC experiments. It distinguishes between two classes of job - organised and chaotic. Organised jobs have well-defined input which is accessed sequentially and output which is typically some transform of the input. In general the focus for these jobs is on total throughput, rather than response time. Production and Reconstruction are examples of this class of job. In contrast, Analysis tends to be more chaotic. HEPCAL-I focuses on organised jobs, leaving the problem of analysis to HEPCAL-II.
Common Ground - Definitions and Requirements
The concept of a dataset is introduced. A dataset is any collection of information, such as raw or processed events or sets of histograms. A dataset may be composed of other datasets. A software package can also be considered a dataset, if it is managed by the Grid. Datasets are registered to the Grid with a unique logical dataset name (LDN). Once registered, a dataset is fixed.
Assumed common characteristics are described. These are
- Data Structures - raw data, Monte Carlo data, event summary data (reconstructed information + enough to redo reconstruction), analysis object data (physics objects), event tags (brief information for first pass selection).
- Event Identifiers - there exists a method of unambiguously identifying an event.
- Mapping events to datasets - there will be a method of getting from an event to the datasets containing information on it.
- Identifying Datasets - datasets have a unique, user-chosen LDN. There will be a catalogue of metadata on each dataset in the form of keyword-value pairs.
- Event Metadata - considered application dependent, dealt with by the experiments.
- Conditions Data - there will be a database of running conditions and calibration data.
- Event Independence - events can be processed in any order.
- Access Permissions - there will be a method for restricting rights to datasets.
- Job Information - basic job identifiers, composite job identifiers (e.g. process chains, split jobs), production job identifiers. Should be a job catalogue, containing keyword-value pairs.
Use Cases
A set of actors which will take part in computing activity are defined. For each of these, a set of high-level activities are defined, forming an outline of the use cases. Examples of these activities are:
- User - run algorithms on data, build private collections of data, fill histograms, make selections, run simulations.
- Production Manager - process all events of a certain type to a new data sample, data quality checks, submit and monitor job progress.
- Database Manager - publish new detector/beam conditions data, check conditions data, publish Event Summary Data and Analysis Object Data.
- Experiment Resource Manager - allocate resources (disk, CPU) to groups/individuals.
- Manager - Physics Coordinator approving production and monitoring production progress, Computing Coordinator supervising and altering resource allocations,
- Software Developer - test software on Grid.
- Software Distributor - release and register new package version.
Use cases are presented, organised by purpose. Not all use cases are listed here, just the main ones.
General
- Typically low level e.g. Grid log on.
- Browse Grid Resources
Data Management
- Deal with user access to datasets. Users access datasets by requesting an LDN and receiving a physical file identifier. Data management should track access patterns in order to replicate intelligently.
- Dataset metadata update/access
- Dataset upload
- Dataset download
- Dataset delete
- Dataset browsing
- Dataset conditions browse
- User-defined catalogue create/delete
- Dataset access cost evaluation
- Additional use cases are lower level e.g. dataset replication, verification, registration to the Grid.
Job Management
- Job catalogue update
- Job catalogue query
- Job submission
- Job output access/retrieval
- Error recovery
- Job Control - suspend, cancel, resume etc.
- Job Monitoring
- Job resource estimation
- Additional lower level use cases e.g. job splitting, environment changes, resource estimation.
Virtual Organisation Management
- Sketchier use cases are provided for this area. Deals with the configuration of groups e.g. privileges, quotas, job catalogue options, adding/removing users, software publishing.
Future Work
Many issues are not addressed. The most important areas for future work are:
- Catalogue requirements, which are not complete;
- Interactive Grid work;
- Job splitting.
s.hanlon@physics.gla.ac.uk
Last modified Thu 24 June 2004 . View page history
Switch to HTTPS . Website Help . Print View . Built with GridSite 1.4.3