BaBar Data Management

From GridPP Wiki
Jump to: navigation, search

BaBar Data Management

[move03] /home/csf/bbdatsrv > shimp
=========== ======== =========== ========== ======== =======
1P          SP       signal      full              8     5.9
=========== ======== =========== ========== ======== =======
Totals                                             8     5.9
8 rows returned from bbkr18 at ral

or add dfile_modified for mark date.

ral-import-status    # when total goes to 0TB, import stops
shimp2 --file_status=3   # shows files to sweep

shimp == shimp2 --file_status='1*'

shjobs   # lists processes run via "job" command for current user
shjobs -l # just list logfile names

Thus, to see which jobs are stuck

cd ~bbdatsrv/logs
ls -lt `shjobs -l`

If any logfiles haven't been updated for some time, they will show at the bottom of the list with an old time (less than an hour or two should be OK).

To start/restart import jobs, use


(with -h shows possibilities), but restart-r18 can only stop jobs that are still running normally and notice the signal (~/BbkImport.stop.$$ file).

Freeing disk space

shimp2 --file_status=0S    # data to delete
shimp2 --file_status=0 --dse_type=PR,SP --set file_status=0S --dbuser=bfactory # change status to 0S
cd ~bbdatsrv/lists
ads-files -z | sort -k2,2 > ads.lis
BbkUser --dbname=bbkr18 --file_status=0S file bytes file_status -q \
| sort -k1,1 \
| join -11 -22 - ads.lis \
| ads-size-check \
> 070125a-del.lis
files-in-catalog < 070125a-del.lis | sort -u > 070125a-del.pfn
sed 's=^.*/store/=/store/=' 070125a-del.pfn \
| sort | diff - 070125a-del.lis | wc # to check that just 1 copy and all there
shimp2 -i file=070125a-del.lis --set file_status=0T --dbuser=bfactory  # mark as being on tape only
cd ~bbdatsrv
job -NQ cat lists/070125a-del.pfn \| xargs -ti /home/csf/bbdatsrv/mps/mps_Xeq rm '{}'

This sends a message when the job is complete. Check logfile for errors.

Allocating datastore tapes for user archiving

Users can archive their data to the datastore, using tape names with a 3-character prefix, allocated, on request, by us. To allocate a prefix for a user, chose a prefix like "UX1", where "U" is used for all user tapes, "X" is the first character of their username, and "1" is chosen for a unique prefix for each user who asks.

Use the command

datastore rule query babar

to list all of our existing Datastore access settings. The prefixes should be listed in entries like

Address         Username        Group           Name            Mode
%ATLAS          MOHANTY         *               UM1*            RWCPAL

Pick a new prefix and add permissions like this

datastore rule add babar %ATLAS TLATHAM + UT1+ RWCPAL

("+" can be used in place of "*", but isn't expanded by the shell). Check it is in the list with a new

lcgui0360> datastore rule query babar
Rule BABAR owned by BABAR with components:-
Address         Username        Group           Name            Mode
%ATLAS          MOHANTY         *               UM1*            RWCPAL
%ATLAS          TLATHAM         *               UT1*            RWCPAL

This allows user tlatham to create and write to tapes named UT1001 etc. Other users have read access.