Difference between revisions of "BaBar Data Management"

Latest revision as of 11:39, 28 March 2007

BaBar Data Management

[move03] /home/csf/bbdatsrv > shimp
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
=========== ======== =========== ========== ======== =======
1P          SP       signal      full              8     5.9
=========== ======== =========== ========== ======== =======
Totals                                             8     5.9
8 rows returned from bbkr18 at ral

or add dfile_modified for mark date.

ral-disk-status
ral-import-status    # when total goes to 0TB, import stops

shimp2 --file_status=3   # shows files to sweep

shimp == shimp2 --file_status='1*'

shjobs   # lists processes run via "job" command for current user
shjobs -l # just list logfile names

Thus, to see which jobs are stuck

cd ~bbdatsrv/logs
ls -lt `shjobs -l`

If any logfiles haven't been updated for some time, they will show at the bottom of the list with an old time (less than an hour or two should be OK).

To start/restart import jobs, use

restart-r18

(with -h shows possibilities), but restart-r18 can only stop jobs that are still running normally and notice the signal (~/BbkImport.stop.$$ file).

Freeing disk space

shimp2 --file_status=0S    # data to delete
shimp2 --file_status=0 --dse_type=PR,SP --set file_status=0S --dbuser=bfactory # change status to 0S

cd ~bbdatsrv/lists
ads-files -z | sort -k2,2 > ads.lis

BbkUser --dbname=bbkr18 --file_status=0S file bytes file_status -q \
| sort -k1,1 \
| join -11 -22 - ads.lis \
| ads-size-check \
> 070125a-del.lis

files-in-catalog < 070125a-del.lis | sort -u > 070125a-del.pfn

sed 's=^.*/store/=/store/=' 070125a-del.pfn \
| sort | diff - 070125a-del.lis | wc # to check that just 1 copy and all there
shimp2 -i file=070125a-del.lis --set file_status=0T --dbuser=bfactory  # mark as being on tape only

cd ~bbdatsrv
job -NQ cat lists/070125a-del.pfn \| xargs -ti /home/csf/bbdatsrv/mps/mps_Xeq rm '{}'

This sends a message when the job is complete. Check logfile for errors.

Allocating datastore tapes for user archiving

Users can archive their data to the datastore, using tape names with a 3-character prefix, allocated, on request, by us. To allocate a prefix for a user, chose a prefix like "UX1", where "U" is used for all user tapes, "X" is the first character of their username, and "1" is chosen for a unique prefix for each user who asks.

Use the command

datastore rule query babar

to list all of our existing Datastore access settings. The prefixes should be listed in entries like

Address         Username        Group           Name            Mode
%ATLAS          MOHANTY         *               UM1*            RWCPAL

Pick a new prefix and add permissions like this

datastore rule add babar %ATLAS TLATHAM + UT1+ RWCPAL

("+" can be used in place of "*", but isn't expanded by the shell). Check it is in the list with a new

lcgui0360> datastore rule query babar
Rule BABAR owned by BABAR with components:-
Address         Username        Group           Name            Mode
%ATLAS          MOHANTY         *               UM1*            RWCPAL
%ATLAS          TLATHAM         *               UT1*            RWCPAL
...

This allows user tlatham to create and write to tapes named UT1001 etc. Other users have read access.