Difference between revisions of "RucioDocker"

From GridPP Wiki
Jump to: navigation, search
(Created page with " = Testing RUCIO deployment using Docker = == Resources == This is a collection of resources used as reference in throwing up a RUCIO service using containers: [https://gi...")
 
(Testing RUCIO deployment using Docker)
 
Line 1: Line 1:
  
 
= Testing RUCIO deployment using Docker =
 
= Testing RUCIO deployment using Docker =
 +
 +
== Docker ==
 +
 +
I've always found it more reliable/safer/faster/better to just install the packages from the docker projects' own repos.
 +
 +
(stable repos from)
 +
https://docs.docker.com/engine/install/centos/
 +
(and docker compose from)
 +
https://docs.docker.com/compose/install/
 +
 +
These tend to have the latest greatest features etc and are widely used in production inside and outside of HEP.
  
 
== Resources ==
 
== Resources ==

Latest revision as of 16:15, 8 June 2021

Testing RUCIO deployment using Docker

Docker

I've always found it more reliable/safer/faster/better to just install the packages from the docker projects' own repos.

(stable repos from) https://docs.docker.com/engine/install/centos/ (and docker compose from) https://docs.docker.com/compose/install/

These tend to have the latest greatest features etc and are widely used in production inside and outside of HEP.

Resources

This is a collection of resources used as reference in throwing up a RUCIO service using containers:


[1] SKA RUCIO prototype project [2] RUCIO ReadTheDocs [3] Talk on RUCIO in containers

Running basic commands in test RUCIO

Checkout RUCIO

cd $WORKDIR
git clone https://github.com/rucio/rucio.git
cd rucio/etc/docker/dev

Launch containers needed for test

# Update the docker containers on your system
docker-compose --file docker-compose-storage-alldb.yml pull

# Launch the containers
docker-compose --file docker-compose-storage-alldb.yml up -d

Run dev tests

# Launch interactive shell on RUCIO main image (Akin to bastion server)
docker exec -it dev_rucio_1 /bin/bash

# Setup simple configs, setup databases and add 3 RSEs which speak XRootD
[root@bb88b6f88320 rucio]# tools/run_tests_docker.sh -ir

# To run the full test suite if you've already initialised the DB
[root@bb88b6f88320 rucio]# tools/run_tests_docker.sh -a

# This will go away and run the full suite of dev tests for RUCIO and takes some time but can be informative

Remove containers after finished

# Destroy the containers and network
docker-compose --file docker-compose-storage-alldb.yml down

Manually play with placing files and such in RSEs

docker exec -ti dev_rucio_1 bash

# Setup simple configs, setup databases and add 3 RSEs which speak XRootD
[root@bb88b6f88320 rucio]# tools/run_tests_docker.sh -ir

# NB it's only really possible to run this script once depite what the docs suggest otherwise it will put at least some components into a funny state
# If you want to start again it's best to remove the container instances and start again

[root@bb88b6f88320 rucio]# rucio add-dataset test:mynewdataset
Added test:mynewdataset
[root@bb88b6f88320 rucio]# rucio attach test:mynewdataset test:file1 test:file2 test:file3 test:file4
DIDs successfully attached to test:mynewdataset

# Shamelessly borrowed from https://rucio.readthedocs.io/en/latest/demo_environment.html#start-the-daemons

# You now have 4 files in your dataset. E.g:

[root@bb88b6f88320 rucio]# rucio list-file-replicas test:mynewdataset
+---------+--------+------------+-----------+------------------------------------------------+
| SCOPE   | NAME   | FILESIZE   | ADLER32   | RSE: REPLICA                                   |
|---------+--------+------------+-----------+------------------------------------------------|
| test    | file1  | 10.486 MB  | d4ed8299  | XRD1: root://xrd1:1094//rucio/test/80/25/file1 |
| test    | file2  | 10.486 MB  | 7822b5da  | XRD1: root://xrd1:1094//rucio/test/f3/14/file2 |
| test    | file3  | 10.486 MB  | 2bac1f0e  | XRD2: root://xrd2:1095//rucio/test/a9/23/file3 |
| test    | file4  | 10.486 MB  | f6af16a9  | XRD2: root://xrd2:1095//rucio/test/2b/c2/file4 |
+---------+--------+------------+-----------+------------------------------------------------+

# To check this file exists, exit the current container and go into a RSE container with a file:

[root@gridpp01 dev]# docker exec -ti dev_xrd1_1 bash

[root@bb88b6f88320 /]# ls /rucio/test/80/25/file1 
/rucio/test/80/25/file1

# NB these files will be lost when the container is destroyed by construction as of writing based on the container configs