Difference between revisions of "MultiVO Userguide"
(→Uploading Data) |
(→Uploading Data) |
||
(2 intermediate revisions by one user not shown) | |||
Line 61: | Line 61: | ||
`~/.globus/usercert.pem` and `~/.globus/userkey.pem` for your certificate in | `~/.globus/usercert.pem` and `~/.globus/userkey.pem` for your certificate in | ||
your home folder, then run: | your home folder, then run: | ||
− | + | ||
$ grid-proxy-init | $ grid-proxy-init | ||
$ export X509_USER_PROXY=/tmp/x509up_u$(id -u) | $ export X509_USER_PROXY=/tmp/x509up_u$(id -u) | ||
$ rucio ping | $ rucio ping | ||
1.23.4 | 1.23.4 | ||
− | |||
==== Creating a client node ==== | ==== Creating a client node ==== | ||
Line 275: | Line 274: | ||
In Rucio files and their replicas are represented by Data IDentifiers | In Rucio files and their replicas are represented by Data IDentifiers | ||
− | ([https://rucio.readthedocs.io/en/latest/overview_File_Dataset_Container.html DIDs] | + | ([https://rucio.readthedocs.io/en/latest/overview_File_Dataset_Container.html DIDs]), |
which are composed of a scope and name. Furthermore, multiple files can be | which are composed of a scope and name. Furthermore, multiple files can be | ||
attached to a dataset, which in turn can be attached to a container (which can | attached to a dataset, which in turn can be attached to a container (which can | ||
Line 371: | Line 370: | ||
Once a DID exists within the Rucio catalogue, replicas of that file, dataset or | Once a DID exists within the Rucio catalogue, replicas of that file, dataset or | ||
collection are created and maintained by | collection are created and maintained by | ||
− | [ | + | [https://rucio.readthedocs.io/en/latest/overview_Replica_management.html Replication rules]. |
By uploading a file to a particular RSE, a replication rule is created for that | By uploading a file to a particular RSE, a replication rule is created for that | ||
file, however rules can also be added for existing DIDs. As a minimum an RSE | file, however rules can also be added for existing DIDs. As a minimum an RSE |
Latest revision as of 13:51, 19 April 2021
Overview
This is a Conversion of the markdown file to wiki templating, any errors please either edit or let me know
This guide covers how to get started as a new VO on the Rucio instance at RAL. For more in depth documentation on using Rucio, see the main documentation, in particular:
Setup
In order to get started, first contact ian.johnson@stfc.ac.uk with details of your situation (Ian should also be the point of contact for any other queries about the instance):
- Whether a Rucio instance is currently used to manage file replication (in which case data can be transferred to the m-VO instance). - Estimate of the number and size of files to be managed by Rucio. For reference, between ten thousand and ten million files with a total size between 10TB and 10PB would be typical. - A description and three character identifier for your VO, along with an email address to use as a contact for the VO. The three character identifier must be unique, so this may need to be re-chosen if already in use by another VO.
In addition to these multi-VO considerations there are the normal steps required for setting up a Rucio instance, detailed below. If you use an existing Rucio database, it should be possible to start using the m-VO instance without any of the following configuration steps once the database has been transferred.
Client Access
The Rucio functions can be accessed at a number of levels. At the surface, Rucio has a CLI, which calls one of several Python clients. In turn, these send requests to the Rucio server via a RESTful API. These can be accessed for the RAL instance by ssh-ing to rucio-bastion.gridpp.rl.ac.uk. This node is used by multiple VOs to access Rucio. Alternatively, you can set up your own client and for exclusive use by your VO.
Bastion
In order to get access contact Ian. The Rucio commands are installed in a virtual environment, which should first be activated. Then, optionally export your vo identifier and Rucio account name. If not exported then these will have to provided to all CLI commands and Python Client constructors.
$ source /opt/rucio-env/bin/activate $ export RUCIO_VO=abc $ export RUCIO_ACCOUNT=root
Further details specific to your authentication method will also be required with your first Rucio command of the session (afterwards a token is saved locally so credentials are not needed). For example, if using X509 proxies (the default method for the bastion node), first create `~/.globus/usercert.pem` and `~/.globus/userkey.pem` for your certificate in your home folder, then run:
$ grid-proxy-init $ export X509_USER_PROXY=/tmp/x509up_u$(id -u) $ rucio ping
1.23.4
Creating a client node
Whether use of the CLI or the Python clients is preferred, both are installed by following these instructions. By default Rucio runs in single-VO mode, so to ensure your client is compatible with the multi-VO server and database the config file (`etc/rucio.cfg`) should have the following options set (if for example the VO identifier was `abc`), along with the address of the RAL server along with the other options not affected by m-VO operation:
[common] multi_vo = True
[client]
rucio_host = https://rucio-server.gridpp.rl.ac.uk:443
auth_host = https://rucio-server.gridpp.rl.ac.uk:443
vo = abc
...
If you are using a Docker container and set the config options using
environment variables then these should be set in the same manner.
Rucio Identities
Once a client is running, the next step is to setup Rucio accounts and identities for users at your VO, starting with `root` (the main admin account). This account will be created when your VO is added to the instance along with a few default identities to allow for setup. These should be removed once the actual identities are configured. Accounts and identities can be set up to allow multiple people to access the same account; a more in depth look at Rucio accounts can be found here.
Identity types supported by Rucio include:
- Username/password
- X509 certificates
- OIDC/OAuth2
- SSH
- GSS
- SAML
First, add the new identity (or identities) and check that you can authenticate successfully with them. The identity used to authenticate when running Rucio commands can be set in several places (with the following order of priority):
- Optional arguments to CLI commands or any of the Python `Client()` functions
- Environment variables `RUCIO_AUTH_TYPE` and `X509_USER_PROXY`
- The `[client]` section of `etc/rucio.cfg`
As well as changing these to reflect the new identity, any client tokens created by Rucio using the old identity should be removed before testing the new one. Unless another path has been explicitly set, these are saved at `<tmp>/.rucio_<account>@<vo>/auth_token_<account>` where the tmp directory is chosen based on the following:
- TMP environment variable
- TMPDIR environment variable
- TEMP environment variable
- `/tmp`
Once the new identity is added and the old tokens removed, check that commands can be run using it. Finally, any old default identities can be removed. The list of identities associated with an account can also be checked at any time.
CLI Example
$ rucio-admin identity add --account root --type USERPASS --id username --email root@email.com --password password
Added new identity to account: username-root
Remove old client tokens (and alter config file if appropriate).
$ rucio -u username -pwd password whoami
status : ACTIVE account : root account_type : SERVICE created_at : 2020-08-07T08:27:29 updated_at : 2020-08-07T08:27:29 suspended_at : None deleted_at : None email : N/A
$ rucio-admin identity delete --account root --type USERPASS --id root@abc
Deleted identity: root@abc
$ rucio-admin account list-identities root
Identity: username, type: USERPASS
Python Client Example
$ python
>>> from rucio.client import Client >>> CLIENT = Client() >>> CLIENT.add_identity('root', 'username', 'USERPASS', 'root@email.com', default=False, password='password') True
Remove old client tokens (and alter config file if appropriate).
$ python
>>> CLIENT = Client(creds={'username': 'username', 'password': 'password'}) >>> CLIENT.del_account('root', 'root@abc', 'USERPASS') True >>> list(CLIENT.list_identities('root') [{u'type': u'USERPASS', u'email': u'root@email.com', u'identity': u'username'}
Creating Accounts
For other users, the account will need to be created before adding identities using the same methods as above (though without the need to remove any old identities). The account to use when running a command is determined in the same places as for identities (optional CLI arguments, etc.) and accounts will have different permissions and access (such as how much data they can store on a particular RSE).
CLI Example
$ rucio-admin account add --type USER --email jdoe@email.com jdoe
Added new account: jdoe
$ rucio-admin identity add --account jdoe --type USER --id userjdoe --email jdoe@email.com --password jdoepass
Added new identity to account: userjdoe-jdoe
Python Client Example
$ python >>> from rucio.client import Client >>> CLIENT = Client() >>> CLIENT.add_account('jdoe', 'USER', 'jdoe@email.com') True >>> CLIENT.add_identity('jdoe', 'USER', 'jdoe@email.com') True
Creating RSEs
Rucio Storage Elements (RSEs) are how Rucio represents the physical storage available to your VO. As with many aspects of Rucio there are a lot of optional attributes that can be set for an RSE, but as a minimum a protocol for transfers need to be added before it can be used.
CLI Example
$ rucio-admin rse add NEW_RSE
Added new deterministic RSE: NEW_RSE
$ rucio-admin rse add-protocol --hostname test.org --scheme gsiftp --prefix '/filepath/rucio/' --port 8443 NEW_RSE --domain-json '{"wan": {"read": 1, "write": 1, "third_party_copy": 0, "delete": 1}, "lan": {"read": 1, "write": 1,"third_party_copy": 0, "delete": 1}}'
Python Client Example
$ python >>> from rucio.client import Client >>> CLIENT = Client() >>> CLIENT.add_rse('NEW_RSE') True >>> CLIENT.add_protocol('NEW_RSE', {'hostname': 'test.org', 'scheme': 'gsiftp', 'prefix': '/filepath/rucio/', 'port': 8443, 'impl': 'rucio.rse.protocols.gfalv2.Default', 'domain': {"wan": {"read": 1, "write": 1, "third_party_copy": 0, "delete": 1}, "lan": {"read": 1, "write": 1, "third_party_copy": 0, "delete": 1}}}) True
Updating RSE Protocols
On occasion, it may be necessary to change or update an RSE protocol. Unlike settings (`rucio-admin rse update`) or attributes (`rucio-admin rse set-attribute`), there isn't a direct CLI function for changing a protocol. It would therefore be necessary to remove (`rucio-admin rse delete-protocol`) and then add (`rucio-admin rse add-protocol`) it again using different information. Alternatively, the Python client has additional functions which can directly update or swap the priority of RSE protocols. For example to update the `impl` without changing anything else (the `data` argument is used to update the protocol, with the other settings used to specify the protocol to change):
$ python >>> from rucio.client import Client >>> CLIENT = Client() >>> CLIENT.update_protocols(rse='NEW_RSE', scheme='gsiftp', data={'impl': 'rucio.rse.protocols.gfal.Default'}, hostname='test.org', port=8433) True
To swap the priority of two protocols for the third party copy operation:
$ python >>> from rucio.client import Client >>> CLIENT = Client() >>> CLIENT.swap_protocols(rse='NEW_RSE', domain='wan', operation='third_party_copy', scheme_a='gsiftp', scheme_b='root') True
It's also worth noting that when an RSE is deleted using `rucio-admin rse delete`, the entry remains in the database. This "soft" deletion means that attempting to add a new RSE with the same name as a deleted RSE will fail due to it not having a unique name/VO combination. In practice it is therefore better to update a badly configured RSE rather than attempting to delete and re-add it. However, if the latter method is preferred, it is possible manually rename the deleted RSE in the database (as there are no foreign key constraints on its name, just the ID and VO) so that the old name can be re-used.
Basic Usage
This section covers some of the basic Rucio functions that can be run once the VO has accounts and RSEs set up. As with the setup, there are many options that won't be covered here. For more information refer to either the main documentation or the help for the function in question.
Daemons
Most operations in Rucio (such as transfers, deletions, rule evaluation) require one or more of the daemons to be running in order to take effect. For a multi-VO instance, these should be running for all VOs already. However if it seems like nothing is happening following a command, it may be worth checking with Ian that the daemons are running as intended.
Uploading Data
In Rucio files and their replicas are represented by Data IDentifiers (DIDs), which are composed of a scope and name. Furthermore, multiple files can be attached to a dataset, which in turn can be attached to a container (which can be attached to another container and so on). Datasets and containers are also represented by DIDs.
Scopes are always associated with a particular Rucio account, and must be added to Rucio using an admin account. If no scope is provided when uploading, Rucio will default to `user.<account>`, but this still needs to have been added by an admin.
Once a file has been uploaded via the CLI or Python client, it can then be attached to a dataset. It's worth noting that by default, some Rucio commands will not list files, only datasets.
CLI Example
Assuming the file `test.txt` exists locally:
$ rucio-admin scope add --account root --scope user.root
Added new scope to account: user.root-root
$ rucio upload --rse NEW_RSE test.txt
2020-08-14 15:28:15,059 INFO Preparing upload for file test.txt 2020-08-14 15:28:15,235 INFO Successfully added replica in Rucio catalogue at NEW_RSE 2020-08-14 15:28:15,334 INFO Successfully added replication rule at NEW_RSE 2020-08-14 15:28:15,579 INFO Trying upload with mock to NEW_RSE 2020-08-14 15:28:15,579 295 INFO Trying upload with mock to NEW_RSE 2020-08-14 15:28:15,580 INFO Successful upload of temporary file. mock://test.org:123/filepath/rucio/user/root/46/6b/test.txt.rucio.upload 2020-08-14 15:28:15,580 295 INFO Successful upload of temporary file. mock://test.org:123/filepath/rucio/user/root/46/6b/test.txt.rucio.upload 2020-08-14 15:28:15,580 INFO Successfully uploaded file test.txt 2020-08-14 15:28:15,580 295 INFO Successfully uploaded file test.txt 2020-08-14 15:28:15,583 295 DEBUG Starting new HTTPS connection (1): rucio:443 2020-08-14 15:28:15,598 295 DEBUG https://rucio:443 "POST /traces/ HTTP/1.1" 201 7 2020-08-14 15:28:15,662 295 DEBUG https://rucio:443 "PUT /replicas HTTP/1.1" 200 0
$ rucio list-dids user.root:test.txt --filter type=ALL +--------------------+--------------+ | SCOPE:NAME | [DID TYPE] | |--------------------+--------------| | user.root:test.txt | FILE | +--------------------+--------------+ $ rucio add-dataset user.root:test_dataset Added user.root:test_dataset
$ rucio attach user.root:test_dataset user.root:test.txt
DIDs successfully attached to user.root:test_dataset
$ rucio list-content user.root:test_dataset +--------------------+--------------+ | SCOPE:NAME | [DID TYPE] | |--------------------+--------------| | user.root:test.txt | FILE | +--------------------+--------------+
Python Client Example
Assuming the file `test.txt` exists locally:
python >>> from rucio.client import Client >>> CLIENT = Client() >>> CLIENT.add_scope('root', 'user.root')
True
>>> from rucio.client.uploadclient import UploadClient >>> UPLOAD_CLIENT = UploadClient() >>> UPLOAD_CLIENT.upload([{'path': 'test.txt', 'rse': 'NEW_RSE'}])
2020-08-14 14:47:31,147 8431 DEBUG Starting new HTTPS connection (1): rucio:443 2020-08-14 14:47:31,166 8431 DEBUG https://rucio:443 "POST /traces/ HTTP/1.1" 201 7 2020-08-14 14:47:31,224 8431 DEBUG https://rucio:443 "PUT /replicas HTTP/1.1" 200 None 0
>>> list(CLIENT.list_dids('user.root', {}, type='all'))
[u'test.txt'] >>> CLIENT.add_dataset('user.root', 'test_dataset')
True
>>> CLIENT.attach_dids('user.root', 'test_dataset', [{'scope': 'user.root', 'name': 'test.txt'}]) >>> list(CLIENT.list_content('user.root', 'test_dataset'))
[{u'adler32': u'00000001', u'name': u'test.txt', u'bytes': 0, u'scope': u'user.root', u'type': u'FILE', u'md5': u'd41d8cd98f00b204e9800998ecf8427e'}]
Replication Rules
Once a DID exists within the Rucio catalogue, replicas of that file, dataset or collection are created and maintained by Replication rules. By uploading a file to a particular RSE, a replication rule is created for that file, however rules can also be added for existing DIDs. As a minimum an RSE and number of copies must be specified, but further options such as lifetime of the rule and selecting RSEs based on user set attributes are also possible.
CLI Example
$ rucio list-rules --account root
ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC) -------------------------------- --------- ------------------ ---------------------- ---------------- -------- --------------- ------------------- 991f9ace7ed74cad989efde90b6a23c5 root user.root:test.txt OK[1/0/0] NEW_RSE 1 2020-08-14 15:28:15 $ rucio add-rule user.root:test_dataset 1 NEW_RSE bd51b767ef524878bb3cc68db16d2374
$ rucio list-rules --account root
ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC) -------------------------------- --------- ---------------------- ---------------------- ---------------- -------- --------------- ------------------- 991f9ace7ed74cad989efde90b6a23c5 root user.root:test.txt OK[1/0/0] NEW_RSE 1 2020-08-14 15:28:15 bd51b767ef524878bb3cc68db16d2374 root user.root:test_dataset OK[1/0/0] NEW_RSE 1 2020-08-14 15:47:15
Python Client Example
$ python >>> from rucio.client import Client >>> CLIENT = Client() >>> list(CLIENT.list_account_rules('root'))
[{u'locks_ok_cnt': 1, u'source_replica_expression': None, u'locks_stuck_cnt': 0, u'purge_replicas': False, u'rse_expression': u'NEW_RSE', u'updated_at': datetime.datetime(2020, 8, 14, 15, 28, 15), u'meta': None, u'child_rule_id': None, u'id': u'991f9ace7ed74cad989efde90b6a23c5', u'ignore_account_limit': False, u'error': None, u'weight': None, u'locks_replicating_cnt': 0, u'notification': u'NO', u'copies': 1, u'comments': None, u'split_container': False, u'priority': 3, u'state': u'OK', u'scope': u'user.root', u'subscription_id': None, u'stuck_at': None, u'ignore_availability': False, u'eol_at': None, u'expires_at': None, u'did_type': u'FILE', u'account': u'root', u'locked': False, u'name': u'test.txt', u'created_at': datetime.datetime(2020, 8, 14, 15, 28, 15), u'activity': u'User Subscriptions', u'grouping': u'DATASET'}]
>>> CLIENT.add_replication_rule([{'scope': 'user.root', 'name': 'test_dataset'}], 1, 'NEW_RSE')
[u'76b262b45dca4e769221224e1ccf5c7a']
>>> list(CLIENT.list_account_rules('root'))
[{u'locks_ok_cnt': 1, u'source_replica_expression': None, u'locks_stuck_cnt': 0, u'purge_replicas': False, u'rse_expression': u'NEW_RSE', u'updated_at': datetime.datetime(2020, 8, 14, 15, 28, 15), u'meta': None, u'child_rule_id': None, u'id': u'991f9ace7ed74cad989efde90b6a23c5', u'ignore_account_limit': False, u'error': None, u'weight': None, u'locks_replicating_cnt': 0, u'notification': u'NO', u'copies': 1, u'comments': None, u'split_container': False, u'priority': 3, u'state': u'OK', u'scope': u'user.root', u'subscription_id': None, u'stuck_at': None, u'ignore_availability': False, u'eol_at': None, u'expires_at': None, u'did_type': u'FILE', u'account': u'root', u'locked': False, u'name': u'test.txt', u'created_at': datetime.datetime(2020, 8, 14, 15, 28, 15), u'activity': u'User Subscriptions', u'grouping': u'DATASET'}, {u'locks_ok_cnt': 1, u'source_replica_expression': None, u'locks_stuck_cnt': 0, u'purge_replicas': False, u'rse_expression': u'NEW_RSE', u'updated_at': datetime.datetime(2020, 8, 14, 15, 47, 15), u'meta': None, u'child_rule_id': None, u'id': u'bd51b767ef524878bb3cc68db16d2374', u'ignore_account_limit': False, u'error': None, u'weight': None, u'locks_replicating_cnt': 0, u'notification': u'NO', u'copies': 1, u'comments': None, u'split_container': False, u'priority': 3, u'state': u'OK', u'scope': u'user.root', u'subscription_id': None, u'stuck_at': None, u'ignore_availability': False, u'eol_at': None, u'expires_at': None, u'did_type': u'DATASET', u'account': u'root', u'locked': False, u'name': u'test_dataset', u'created_at': datetime.datetime(2020, 8, 14, 15, 47, 15), u'activity': u'User Subscriptions', u'grouping': u'DATASET'}]
Multi-VO Features
From a users perspective, whether the instance is multi or single VO should not affect the majority of functions and VO does not need to be provided. There are however some occasions when an optional argument for the VO can be given in a multi-VO instance.
Swapping VOs
Just like how an identity can be associated with (and used to authenticate against) multiple accounts, the same identity can be used for accounts at more than one VO. Account and identity can be retrieved from the config file if present, and the VO set there will be used (unless the environment variable `RUCIO_VO` is also set, in which case the latter takes precedent). Both will be ignored however if the VO is passed as an optional argument in the CLI or Python client. Using this optional argument allows a user to quickly run commands on a different VO they have access to. On the Bastion node, this is also an alternative method to setting the environment variable for VO (note that it will need to be passed for every command however).
CLI Example
$ rucio whoami
status : ACTIVE account : jdoes_abc_account account_type : USER created_at : 2020-08-07T08:27:29 updated_at : 2020-08-07T08:27:29 suspended_at : None deleted_at : None email : N/A
$ rucio --vo xyz whoami
status : ACTIVE account : jdoes_xyz_account account_type : USER created_at : 2020-08-11T12:13:58 updated_at : 2020-08-11T12:13:58 suspended_at : None deleted_at : None email : N/A
Python Client Example
$ python
>>> from rucio.client import Client >>> CLIENT = Client() >>> CLIENT.whoami()
{u'status': u'ACTIVE', u'account': u'jdoes_abc_account', u'account_type': u'USER', u'created_at': u'2020-08-07T08:27:29', u'updated_at': u'2020-08-07T08:27:29', u'suspended_at': None, u'deleted_at': None, u'email': u'N/A'} >>> CLIENT_XYX = Client(vo='xyz') >>> CLIENT_XYZ.whoami()
{u'status': u'ACTIVE', u'account': u'jdoes_xyz_account', u'account_type': u'USER', u'created_at': u'2020-08-11T12:13:58', u'updated_at': u'2020-08-11T12:13:58', u'suspended_at': None, u'deleted_at': None, u'email': u'N/A'}
Monitoring
While it's possible to observe the status of the Rucio instance using CLI and client functions, both the Rucio WebUI and RAL's dashboard can be easier methods of monitoring.
Kibana Dashboard
- Link when we have it - Login if we have it - VO structuring - Description of panels in the dashboard
The Kibana dashboards for the RAL instance can be found at https://rucio-mon.gridpp.rl.ac.uk:5601/app/kibana#/dashboards, and accessed using username and password. Note that this is unrelated from the Rucio identities, and only allows access to the Kibana monitoring. The details should have been created and provided as part of the process for adding your VO, if you don't have access get in touch with Ian. You can then select your VO's dashboard from the list.
The dashboard has a number of panels providing information for transfers, deletions, and other Rucio events for the whole VO, and broken down by the RSE(s) involved. If there's something you'd like adding to the dashboard get in touch with Ian.
WebUI
Access to the WebUI for the RAL instance uses the same address as the server, rucio-server.gridpp.rl.ac.uk. At login, the VO of the user will be determined from the credentials provided if possible. If there is not a unique mapping of identity to VO, then the possible VOs will be given in a drop down list and one must be selected before continuing. Once logged into the WebUI with an account associated with one VO, all of the rules and RSEs displayed will be for that VO. In order to access a different VO then the account used must be changed.