Difference between revisions of "RAL Tier1 Echo"
(→Usage examples) |
(→Echo) |
||
| Line 1: | Line 1: | ||
==Echo== | ==Echo== | ||
| − | Echo is the name of the disk storage service provided by the RAL Tier-1. | + | Echo is the name of the disk storage service provided by the RAL Tier-1. Access is provided via Cloud based APIs described below. |
===WebDAV=== | ===WebDAV=== | ||
| Line 9: | Line 9: | ||
===S3 / Swift=== | ===S3 / Swift=== | ||
| − | S3 is the name of Amazon Web Services storage service. It is also the name given to the API to access it. Swift is the OpenStack equivalent | + | S3 is the name of Amazon Web Services storage service. It is also the name given to the API to access it. Swift is the OpenStack equivalent and functions in a very similar way. An S3 account provides an access key and a secret key, however these should not be considered like a regular username and password, rather they can be used to programmatically produce what is known as a presigned URL. Presigned URLs create a temporary link that can be used to share an object publicly, or to provide a temporary target for sending a PUT request without needing to provide authentication information. |
| + | In S3/Swift data (known as objects) is stored in "buckets". Within these buckets there is a flat namespace (i.e. no file system structure). Buckets can be configured to delete data after a certain amount of time. | ||
===DynaFed=== | ===DynaFed=== | ||
| − | DynaFed is a service that provides an authentication and authorization layer on top of S3. | + | DynaFed is a service that provides an authentication and authorization layer on top of S3. DynaFed can be configured to use multiple different authentication methods although it is currently setup to use X509 certificates. When trying to perform an action via DynaFed, it will check to see if the X509 certificate is present in the Grid-mapfile and if it is whether the attempted actions is allowed. If it is allowed a presigned URL will be generated and the user automatically directed to the backend storage. |
| + | DynaFed can be accessed via a [https://dynafed.stfc.ac.uk/gridpp web browser]. In order for the user to view data they will need to import their X509 certificate into their browser. | ||
| + | |||
| + | DynaFed also provides a basic file system like structure for each bucket. If an object name contains a "/", then this will be considered a directory. Note that there is no need to actually create or delete directories. | ||
==Procedures== | ==Procedures== | ||
Latest revision as of 00:00, 12 June 2018
Contents
Echo
Echo is the name of the disk storage service provided by the RAL Tier-1. Access is provided via Cloud based APIs described below.
WebDAV
Web Distributed Authoring and Versioning (WebDAV) is an extension of the Hypertext Transfer Protocol (HTTP) that allows clients to perform remote Web content authoring operations. Or more simply, it allows you to do more than just read webpages.
Davix is an implementation of WebDAV developed by CERN. It has been optimized for high-performance remote I/O and data management of large collections of files. Davix currently supports WebDAV, Amazon S3, Microsoft Azure, and HTTP protocols.
S3 / Swift
S3 is the name of Amazon Web Services storage service. It is also the name given to the API to access it. Swift is the OpenStack equivalent and functions in a very similar way. An S3 account provides an access key and a secret key, however these should not be considered like a regular username and password, rather they can be used to programmatically produce what is known as a presigned URL. Presigned URLs create a temporary link that can be used to share an object publicly, or to provide a temporary target for sending a PUT request without needing to provide authentication information.
In S3/Swift data (known as objects) is stored in "buckets". Within these buckets there is a flat namespace (i.e. no file system structure). Buckets can be configured to delete data after a certain amount of time.
DynaFed
DynaFed is a service that provides an authentication and authorization layer on top of S3. DynaFed can be configured to use multiple different authentication methods although it is currently setup to use X509 certificates. When trying to perform an action via DynaFed, it will check to see if the X509 certificate is present in the Grid-mapfile and if it is whether the attempted actions is allowed. If it is allowed a presigned URL will be generated and the user automatically directed to the backend storage.
DynaFed can be accessed via a web browser. In order for the user to view data they will need to import their X509 certificate into their browser.
DynaFed also provides a basic file system like structure for each bucket. If an object name contains a "/", then this will be considered a directory. Note that there is no need to actually create or delete directories.
Procedures
Requesting Access
Usage examples
- Setup
$ voms-proxy-init --voms [vo]
It is recommended to use the gfal commands
- Copying a file
$ gfal-copy file:///tmp/testfile davs://dynafed.stfc.ac.uk/gridpp/atlas/test/testfile
Note, that gfal-copy will perform puts, gets and third party transfers depending on the source and destination endpoints.
- Deleting a file:
$ gfal-rm davs://dynafed.stfc.ac.uk/gridpp/atlas/test/testfile
- Listing a "directory":
$ gfal-ls davs://dynafed.stfc.ac.uk/gridpp/atlas
To use Davix directly.
- Copying a file in to Echo:
$ davix-put -P grid testfile davs://dynafed.stfc.ac.uk/gridpp/atlas/test/testfile
- Copying a file out of Echo:
$ davix-get -P grid davs://dynafed.stfc.ac.uk/gridpp/atlas/test/testfile /tmp/testfile
- Deleting a file:
$ davix-rm -P grid davs://dynafed.stfc.ac.uk/gridpp/atlas/test/testfile
- Listing a "directory":
$ davix-ls -P grid davs://dynafed.stfc.ac.uk/gridpp/atlas/test/
