Difference between revisions of "BaBar Data Changelog"
(No difference)
|
Latest revision as of 22:54, 24 May 2007
Contents
- 1 24/05/07 part 3
- 2 24/05/07 part 2
- 3 24/05/07 part 1
- 4 10/05/07
- 5 09/05/07
- 6 03/04/07
- 7 20/03/07 part 2
- 8 20/03/07 part 1
- 9 09/02/07 part 2
- 10 09/02/07
- 11 06/02/07 part 2
- 12 06/02/07
- 13 25/01/07
- 14 16/11/06
- 15 02/11/06
- 16 10/10/06
- 17 04/10/06
- 18 25/08/06
- 19 31/07/06
- 20 28/07/06
- 21 14/07/06
- 22 27/06/06
- 23 26/06/06
- 24 24/06/06
- 25 21/06/06
- 26 19/06/06
- 27 16/06/06
- 28 09/06/06
- 29 26/05/06
- 30 23/05/06
- 31 11/05/06
- 32 02/05/06
- 33 21/04/06
- 34 19/04/06
- 35 05/04/06
- 36 04/04/06
- 37 27/03/06
- 38 14/03/06
- 39 23/02/06
- 40 14/02/06
- 41 12/02/06
- 42 08/02/06
- 43 04/02/06
- 44 30/01/06
- 45 13/01/06
- 46 12/01/06
- 47 23/12/05
- 48 17/12/05
24/05/07 part 3
Removed 31050 files (11549.2 GB, or 11.3 TB) from disk (includes 138 Nonevent files (2.0 GB) duplicated across two disks), listed in ~bbdatsrv/lists/070524e-del.lis. These were all the files previously marked 0 in bbkr14 or 0S in bbkr18.
DB FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =========== ======== =========== ========== ======== ======= bbkr14 0T PR full 514 160.7 bbkr14 0T PRskims full 47 49.7 bbkr14 0T PRskims pointer 2 2.9 bbkr14 0T SP signal full 108 53.4 bbkr14 0T SPskims generic full 41 26.2 bbkr18 0T Nonevent pointer 3321 149.5 bbkr18 0T PR full 78 64.3 bbkr18 0T PRskims full 17 18.1 bbkr18 0T PRskims pointer 3 2.9 bbkr18 0T SP generic full 2203 1654.6 bbkr18 0T SP signal full 17268 4729.1 bbkr18 0T SPskims generic full 6385 3955.1 bbkr18 0T SPskims generic pointer 2 0.1 bbkr18 0T SPskims signal full 923 680.4 ====== =========== ======== =========== ========== ======== ======= Totals 30912 11547.2
24/05/07 part 2
Marked 7738 SP files for removal from disk.
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0S SP generic full 98 75.9 0S SP signal full 7640 2911.8 =========== ======== =========== ========== ======== ======= Totals 7738 2987.7
Also marked 5048 files for removal from disk. These all had dse_status=2 and weren't in any tags.
DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
========== =========== ======== =========== ========== ======== =======
2 0S SPskims generic full 4191 2355.0
2 0S SPskims signal full 857 641.0
========== =========== ======== =========== ========== ======== =======
Totals 5048 2996.0
Also marked 3647 old Nonevent files for removal from disk. These all had dse_status=2 and weren't used in $BFROOT/kanga/config.
DSE_STATUS DS_NAME FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
========== ================================ =========== ======== =========== ========== ======== =======
2 Nonevent-CDB-SP9-full 0S Nonevent pointer 798 57.1
2 Nonevent-CDB-cond18boot-betamini 0S Nonevent pointer 1563 24.3
2 Nonevent-CDB-cond22boot-full 0S Nonevent pointer 1267 91.0
2 Nonevent-CfgDB 0S Nonevent pointer 19 0.2
========== ================================ =========== ======== =========== ========== ======== =======
Totals 3647 172.7
24/05/07 part 1
Reimported /store/SPskims/R12/14.4.3d/Tau1N/00/000998/200302/Tau1N_000998_1397.02HBCAT.root, which had been removed from tape because it was bad, but was not on disk.
Everything with dfile_modified before 1 May looks good, except for the 6 files in ~bbdatsrv/lists/070524c-bad.lis (one is the file mentioned above). They still have path catalogue entries, but no tapes (4 have zero-size ADS entries).
Everything after 1 May looks good, though a few files still need to be migrated (Manny paused migration 14-24 May due to heavy load on ADS).
10/05/07
Removed /stage/bdata-data63/kanga/store/SP/R18/002269/run4offres/18.6.1a/SP_002269_044137.01.root from disk, since it has the wrong size. I have set it to reimport. I will ask Manny to remove the tape (YL8492) and FLFSYS catalogue entry, since the tape was (re)migrated from the bad disk copy.
09/05/07
Found 4 files that were missing from disk (those still missing from the previous problem-file list, ~bbdatsrv/lists/070416a-misstape.lis), and had not been archived (imported 13 March onto /stage/bdata-data18), listed in 070509a-misstape.lis.
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0S SP signal full 4 1.6
Found 18 more files completely missing from disk (not even cache symlinks), listed in 070509b-misstape.lis. They had been removed from tape because the tape copy was bad (part of the previous list 070416a-corrtape.lis).
DB DFILE_MODIFIED DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =================== ========== =========== ======== =========== ========== ======== ======= bbrk14 2005/09/30 18:12:39 1 0T PRskims pointer 1 1.5 bbrk14 2005/11/15 20:04:26 1 0T SPskims generic full 1 1.0 bbrk14 2005/11/15 20:04:26 1 0T SPskims generic pointer 1 1.1 bbrk14 2005/12/23 17:26:37 1 0T PRskims full 1 1.6 bbrk14 2005/12/23 17:26:37 1 0T SPskims generic full 4 5.3 bbrk14 2005/12/23 17:58:10 1 0T SPskims generic full 1 1.6 bbrk14 2006/01/13 20:26:47 1 0T SP signal full 2 0.2 bbrk14 2006/02/04 15:36:27 1 0T PRskims full 1 1.6 bbrk18 2006/01/13 20:28:11 2 0T SP signal full 1 0.9 bbrk18 2006/02/12 18:46:30 2 0T SP signal full 1 0.2 bbrk18 2006/03/14 22:23:36 1 0T SP signal full 1 0.8 bbrk18 2006/03/14 22:23:36 2 0T SP signal full 3 1.3 ====== =================== ========== =========== ======== =========== ========== ======== ======= Totals 18 17.2
Marked them all for reimport.
Update 07/05/10: The file /store/PRskims/R12/14.3.2/A0/00/A0_0034.02HBCA.root in the above list no longer seems accessible at SLAC (despite being in the A0-Run1-OnPeak-R14 dataset). I have marked it with file_status=1Z and is_local=0. In the unlikely event of anyone needing this file again, at least this means they will receive a warning from BbkDatasetTcl.
03/04/07
Found 15 BToCXXDCP signal skim files missing from both disk and tape (~bbdatsrv/lists/070330a-R18-misstape.lis). All are marked bad (dse_status=2), but 8 of them were in the F2KBug dataset tag. Mark them for reimport. The remaining 7 missing files aren't needed at all, so marked with file_status '2'.
SKIM TAG FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ========= ====== =========== ======== =========== ========== ======== ======= BToCXXDCP F2KBug 1B SPskims signal full 8 1.0 BToCXXDCP 2 SPskims signal full 7 2.4 ========= ====== =========== ======== =========== ========== ======== ======= Totals 15 3.4
Removed files in ~bbdatsrv/lists/070330a-R18-badfiles.pfn, which had the wrong size (both disk and tape). Marked them for reimport.
SKIM FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =========== ======== =========== ========== ======== ======= BToCXX 1B SPskims generic full 1 1.6 SP 1B SP generic full 30 27.8 ====== =========== ======== =========== ========== ======== ======= Totals 31 29.4
20/03/07 part 2
/stage/bdata-data18 had RAID problems and had to be reinitialised. The data that is still needed on disk will be staged from tape or reimported (just one file). The rest was marked '0T' or '2' depending on whether the tape copy was OK. 3 of the '2' files were then automatically flagged for re-import (status '1P'), but the rest aren't needed even on tape (superseded Nonevent files and R22 files that were erroneously imported before).
DB FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =========== ======== =========== ========== ======== ======= bbkr14 0T SP signal full 1 0.8 bbkr18 0T Nonevent pointer 20 0.3 bbkr18 0T SP generic full 30 22.7 bbkr18 0T SP signal full 10 2.7 bbkr18 0T SPskims generic full 80 36.7 bbkr18 0T SPskims signal full 13 11.9 bbkr18 1P SP signal full 3 0.7 bbkr18 2 Nonevent pointer 14 1.2 bbkr18 2 SP signal full 5 0.3 ====== =========== ======== =========== ========== ======== ======= Totals 176 77.4
20/03/07 part 1
Marked new signal MC file_status=0S (to remove from disk later).
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
=========== ======== =========== ========== ======== =======
0S SP signal full 1766 367.6
=========== ======== =========== ========== ======== =======
Totals 1766 367.6
Mark new (initially file_status=0) and undeleted (0U, files that weren't on tape before) unused skims file_status=0S.
SKIM_CYCLE SKIM FILE_STATUS #FILE_ID +GBYTES ========== =================== =========== ======== ======= R18b BCCKsPi0Pi003a3body 0S 174 20.0 R18b BCCKsPi0Pi03body 0S 172 27.0 R18b BFourHHPE 0S 183 48.9 R18b BToA1Rho 0S 211 121.2 R18b BtoPhiKPhiTo3pi 0S 180 72.8 R18b ExclEtaP 0S 243 190.5 R18b InclK0s 0S 240 187.1 R18b InclOmega 0S 377 387.0 R18c InclEtapRhoGam 0S 515 612.5 ========== =================== =========== ======== ======= Totals 2295 1667.0
09/02/07 part 2
Marked some unused skims for deletion with status '0U' (to simplify changing some back if necessary).
SKIM_CYCLE SKIM FILE_STATUS #FILE_ID +GBYTES ========== =================== =========== ======== ======= R18b BCCKsPi0Pi003a3body 0U 1540 186.0 R18b BCCKsPi0Pi03body 0U 1542 257.5 R18b BFourHHPE 0U 1541 541.3 R18b BToA1Rho 0U 1843 1196.9 R18b BtoPhiKPhiTo3pi 0U 1542 644.9 R18b ExclEtaP 0U 2125 1637.5 R18b InclK0s 0U 2115 1697.1 R18b InclOmega 0U 3202 3753.5 R18c InclEtapRhoGam 0U 4721 6264.5 ========== =================== =========== ======== ======= Totals 20171 16179.2
Manny will remove them from disk on 15 Feb.
09/02/07
I can't remember why these files (listed in ~bbdatsrv/lists/070209a-0R.lis) were marked '0R':
DFILE_MODIFIED FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =================== =========== ======== =========== ========== ======== ======= 2006/07/31 16:02:09 0R SP generic full 31 28.9 =================== =========== ======== =========== ========== ======== =======
but I changed them to '0S'.
06/02/07 part 2
Marked file_status='0S' (for next time we delete) a few newly created pointer skim collections.
SKIM_CYCLE SKIM FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ========== ============== =========== ======== =========== ========== ======== ======= R18b BToCXX 0 SPskims generic pointer 36 6.4 R18b Bppp0K 0 SPskims generic pointer 36 6.1 R18b BtoKKKL 0 SPskims generic pointer 36 7.2 R18b InclEtapRhoGam 0 SPskims generic pointer 36 6.3 R18b InclPhi 0 SPskims generic pointer 36 2.0 R18b InclProtTight 0 SPskims generic pointer 36 2.3 R18b Inclppbar 0 SPskims generic pointer 36 1.8 ========== ============== =========== ======== =========== ========== ======== ======= Totals 252 32.1
06/02/07
Marked some bad collections (dse_status=2) that are not in any dataset tag for removal.
DSE_RUN_CYCLE FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ============= =========== ======== =========== ========== ======== ======= run5 0S PRskims full 91 100.0 ============= =========== ======== =========== ========== ======== ======= Totals 91 100.0
Removed more files marked '0S', including the above and new SP.
DB FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =========== ======== =========== ========== ======== ======= bbkr14 0T SP signal full 71 43.5 bbkr18 0T PR full 11 15.5 bbkr18 0T PRskims full 266 332.6 bbkr18 0T PRskims generic full 10 7.8 bbkr18 0T PRskims signal full 1 0.0 bbkr18 0T SP generic full 266 242.8 bbkr18 0T SP signal full 2439 777.0 bbkr18 0T SPskims generic full 125 158.7 bbkr18 0T SPskims signal full 3 0.1 ====== =========== ======== =========== ========== ======== ======= Totals 3192 1578.1
These files all had one copy on disk, so this frees up 1.5 TB.
25/01/07
Removed new PR and SP from disk (~bbdatsrv/lists/070125a-del.lis).
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0T PR full 3 2.4 0T PR mini 784 487.5 0T SP generic full 1021 938.7 0T SP signal full 7096 1550.6 =========== ======== =========== ========== ======== ======= Totals 8904 2979.2
These files all had one copy on disk, so this frees up 2.9 TB.
16/11/06
Removed new PR and SP from disk (~bbdatsrv/lists/061116a-del.lis).
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0T SP generic full 1034 675.9 0T SP signal full 10426 2365.1 =========== ======== =========== ========== ======== ======= Totals 11460 3041.0
These files all had one copy on disk, so this frees up 3.0 TB.
02/11/06
Removed new PR and SP from disk (~bbdatsrv/lists/061102a-del.pfn).
DB FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =========== ======== =========== ========== ======== ======= bbkr14 0T SP signal full 31 18.1 bbkr18 0T PR full 9 3.3 bbkr18 0T SP generic full 1898 1388.7 bbkr18 0T SP signal full 19275 3929.2 ====== =========== ======== =========== ========== ======== ======= Totals 21213 5339.3
These files all had one copy on disk, so this frees up 5.2 TB.
csfnfs38 (/stage/bdata-data40 and 41) was having problems, so stopped removal (before it got to those disks), and restarted without the files on those disks.
10/10/06
Removed all 2563 Classic Kanga backup tapes (~bbdatsrv/lists/purge-cm1a.log). These were (and remain) in the RAL skim_archive database.
Removed R14 SP generics and pointer skims from tape. (061010a-deltape-info.lis)
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= D SP generic full 20427 20607.1 D SPskims generic pointer 9913 6381.5 =========== ======== =========== ========== ======== ======= Totals 30340 26988.6
Also removed from the ADS path catalogue.
Of these, 81 files (061010a-deltape-ondisk.lis) were still on disk (not previously removed due to problems with the tape copy). These were also removed.
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= D SP generic full 54 62.1 D SPskims generic pointer 27 10.8 =========== ======== =========== ========== ======== ======= Totals 81 72.9
In total this frees up (1 TByte=2^40 bytes as usual)
Type Tapes TBytes ==================== ===== ====== Classic Kanga 2563 4.1 R14 SP generic 20427 20.1 R14 SP pointer skims 9913 6.2 ==================== ===== ====== Total 32903 30.3
04/10/06
Removed new PR and SP production from disk, much of it SP signal with the F2K bug fix. Includes some older files that couldn't be removed earlier as they weren't yet on tape.
DB FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =========== ======== =========== ========== ======== ======= bbkr14 0T SP signal full 1 0.1 bbkr18 0T PR full 1485 1509.4 bbkr18 0T SP generic full 1480 1146.5 bbkr18 0T SP signal full 28251 6685.8 ====== =========== ======== =========== ========== ======== ======= Totals 31217 9341.8
All files had a single copy on disk, so this should free up 9341.8 GB.
25/08/06
Updated ~bbdatsrv/CM2-Import to 18.6.4e. The main change is that now BbkImport and BbkFiles update the data_files modification date when they update the file status. This follows what BbkUser --set file_status does.
31/07/06
Removed runs 1-4 R18b pointer skims and AllEvents SP.
SKIM_CYCLE SKIM FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
========== ============== =========== ======== =========== ========== ======== =======
SP 0T SP generic full 15950 15682.5
R18b BToCXX 0T SPskims generic pointer 669 310.3
R18b Bppp0K 0T SPskims generic pointer 710 409.9
R18b BtoKKKL 0T SPskims generic pointer 701 362.4
R18b InclEtapRhoGam 0T SPskims generic pointer 669 302.8
R18b InclPhi 0T SPskims generic pointer 669 85.2
R18b InclProtTight 0T SPskims generic pointer 669 165.6
R18b Inclppbar 0T SPskims generic pointer 670 123.8
========== ============== =========== ======== =========== ========== ======== =======
Totals 20707 17442.4
This should free up 17356.7 GB. Files listed are in ~bbdatsrv/lists/060731a-del.pfn.
28/07/06
Removed a few more newly migrated files in categories already marked for removal.
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0S PR full 90 96.9 0S SP generic full 17 5.5 0S SP signal full 211 37.3 =========== ======== =========== ========== ======== ======= Totals 318 139.7
This should free up the full 139.7 GB. Files listed in ~bbdatsrv/lists/060728b-del.pfn
14/07/06
Removed a few more new files or newly migrated files in categories already marked for removal.
DB FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =========== ======== =========== ========== ======== ======= bbkr14 0T SP signal full 98 11.9 bbkr18 0T PR full 78 75.5 bbkr18 0T SP generic full 75 40.4 bbkr18 0T SP signal full 206 37.7 ====== =========== ======== =========== ========== ======== ======= Totals 457 165.5
Files listed in ~bbdatsrv/lists/060714b-del.pfn
27/06/06
Removed any new files or newly migrated files in categories already marked for removal. This includes files still on disk after being reimported 24/06 (marked 0B), though most of those were on csfnfs64.
DB DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== ========== =========== ======== =========== ========== ======== ======= bbkr14 1 0 SP generic full 4 4.6 bbkr14 1 0 SP signal full 5 1.5 bbkr14 1 0 SPskims generic full 3 2.0 bbkr14 1 0 SPskims generic pointer 2 0.1 bbkr18 1 0B PR full 65 41.0 bbkr18 1 0B PRskims pointer 8 1.4 bbkr18 1 0B SP generic full 25 19.2 bbkr18 1 0B SP signal full 28 12.5 bbkr18 1 0B SPskims generic pointer 53 50.9 bbkr18 1 0S PR full 117 114.0 bbkr18 1 0S PRskims pointer 38 2.7 bbkr18 1 0S SP generic full 471 322.9 bbkr18 1 0S SP signal full 2442 537.7 bbkr18 2 0B PR full 2 1.4 bbkr18 2 0B PRskims pointer 1 1.4 bbkr18 2 0B SP generic full 29 31.6 bbkr18 2 0B SP signal full 70 30.0 bbkr18 2 0B SPskims generic full 6 4.2 bbkr18 2 0B SPskims generic pointer 57 16.4 bbkr18 2 0S PRskims full 2 1.1 bbkr18 2 0S SP generic full 16 18.6 ====== ========== =========== ======== =========== ========== ======== ======= Totals 3444 1215.1
This should free up 1016.0 GB on the remaining disks.
26/06/06
csfnfs65 (/stage/bdata-data79, 80, 81, and 82) retired.
24/06/06
on 19/06/06, Tim Folkes reported that 1112 tape files had been lost from the Datastore (060619b-adslost.lis). These have now been reimported (where necessary) and remigrated. Files that aren't needed on disk (because they were previously only on tape) are marked status=0B and can be removed later.
21/06/06
Removed 2.0 TB of status=0T files that were on disk (mostly staged), created more than a week ago, and unused in the last week (according to xrootd monitoring db). 0.3 TB of this is on csfnfs64, so doesn't really help.
Files listed in ~bbdatsrv/lists/060621a-staged-unused.lis
Also removed some 0T dangling symlinks (mostly on broken csfnfs64 disks), listed in ~bbdatsrv/lists/060621a-staged-miss.lis
19/06/06
Removed some AllEventsSkim files that were accidentally imported.
SKIM FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ========= =========== ======== =========== ========== ======== ======= AllEvents 0P SPskims generic pointer 597 555.3
24/06/06: The files that had already been migrated to tape (060624c-AES-migrated.lis) were marked 0T (apart from those changed to 0B as reported above for 24/06/06)
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0B SPskims generic pointer 48 49.9 0T SPskims generic pointer 455 437.6 =========== ======== =========== ========== ======== ======= Totals 503 487.4
and the rest (060624c-AES-deleted.lis) changed back to file_status 2.
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
=========== ======== =========== ========== ======== =======
2 SPskims generic pointer 94 67.9
16/06/06
Removed run5 R18b pointer skims and AllEvents SP and all of BToCXXDCP from disk, as well as a bit of stuff already in a category for removal from disk.
DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
========== =========== ======== =========== ========== ======== =======
1 0T PR full 112 116.1
1 0T PRskims pointer 10 1.1
1 0T SP generic full 5566 4945.6
1 0T SP signal full 174 84.7
1 0T SPskims generic full 392 531.1
1 0T SPskims generic pointer 798 320.3
2 0T SP generic full 89 88.2
========== =========== ======== =========== ========== ======== =======
Totals 7141 6087.1
Discounting files on csfnfs64 and 1 file already missing, and counting duplicates, this should free up 6103.5 GB.
The full list of files removed is in ~bbdatsrv/lists/060616c-del.pfn .
09/06/06
Removed uds G4Bug generic MC from disk (runs 2 and 3, SP and SPskims), as well as a bit of stuff already in a category for removal from disk.
DB DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== ========== =========== ======== =========== ========== ======== ======= bbkr14 1 0T SP signal full 108 32.3 bbkr18 1 0T PR full 533 456.1 bbkr18 1 0T PRskims pointer 35 5.1 bbkr18 1 0T SP signal full 682 241.0 bbkr18 2 0T Nonevent pointer 2 0.0 bbkr18 2 0T SP generic full 1802 1886.7 bbkr18 2 0T SPskims generic full 3450 2867.6 bbkr18 2 0T SPskims generic pointer 400 230.3 ====== ========== =========== ======== =========== ========== ======== ======= Totals 7012 5719.1
Counting a couple of duplicates, and discounting data on csfnfs64 (nearly all now duplicated elsewhere) and 1 file already missing, that should free up 5717.5 GB.
The full list of files removed is in ~bbdatsrv/lists/060609a-del.pfn .
26/05/06
Removed non-uds G4Bug generic MC from disk (runs 2 and 3, SP and SPskims), as well as a bit of stuff already in a category for removal from disk.
DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
========== =========== ======== =========== ========== ======== =======
1 0T PR full 158 117.0
1 0T PRskims full 9 11.0
1 0T PRskims pointer 23 14.5
1 0T SP signal full 2618 837.8
2 0T PRskims full 296 219.9
2 0T SP generic full 4750 4813.1
2 0T SPskims generic full 10879 4362.1
2 0T SPskims generic pointer 1652 500.2
========== =========== ======== =========== ========== ======== =======
Totals 20385 10875.6
This does not include files on /stage/bdata-data41, which is marked read-only due to disk problems.
The full list of files removed is in ~bbdatsrv/lists/060526b-del-nodata41.lis
23/05/06
Removed more stuff newly imported or migrated to tape that was already in a category for removal from disk.
DB DSE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== ========== ======== =========== ========== ======== ======= bbkr18 1 PR full 4 2.3 bbkr18 1 SP signal full 2283 689.7 bbkr18 2 PRskims full 296 219.9 bbkr14 1 SP signal full 15 10.3 ====== ========== ======== =========== ========== ======== ======= Totals 2598 922.2
The full list of files removed is in ~bbdatsrv/lists/060523a-del.lis
11/05/06
Removed R18b PR pointer skims (and BToCXXDCP) which are replaced by R18c equivalents.
SKIM_CYCLE SKIM FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ========== ============== =========== ======== =========== ========== ======== ======= R18b BToCXX 0T PRskims pointer 306 305.7 R18b BToCXXDCP 0T PRskims full 844 1214.8 R18b Bppp0K 0T PRskims pointer 306 297.6 R18b BtoKKKL 0T PRskims pointer 455 340.2 R18b InclEtapRhoGam 0T PRskims pointer 283 285.0 R18b InclPhi 0T PRskims pointer 272 104.6 R18b InclProtTight 0T PRskims pointer 290 294.8 R18b Inclppbar 0T PRskims pointer 271 160.1 ========== ============== =========== ======== =========== ========== ======== ======= Totals 3027 3002.9
Removed R18 PR AllEvents which is no longer needed now the pointer skims are gone.
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0T PR full 26636 14430.4
Removed some more remnants of the R18b Tau/QED skims.
SKIM_CYCLE SKIM FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ========== =================== =========== ======== =========== ========== ======== ======= R18b Tau11 0T PRskims pointer 20 20.8 R18b Tau11 0T SPskims generic pointer 25 3.0 R18b Tau1N 0T PRskims full 26 31.8 R18b Tau1N 0T SPskims generic full 32 13.0 R18b Tau33 0T PRskims full 23 29.5 R18b Tau33 0T SPskims generic full 36 24.1 R18b TwoPhotonPentaquark 0T PRskims pointer 31 31.2 R18b TwoPhotonPentaquark 0T SPskims generic pointer 26 6.0 R18b TwoPhotonPi0Pi0 0T PRskims pointer 10 4.9 R18b TwoPhotonPi0Pi0 0T SPskims generic pointer 27 0.5 R18b TwoPhotonTwoTrack 0T PRskims pointer 15 13.4 R18b TwoPhotonTwoTrack 0T SPskims generic pointer 25 0.5 ========== =================== =========== ======== =========== ========== ======== ======= Totals 296 178.6
Removed some more signal SP.
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0T SP signal full 2636 541.9
Removed all files in bad collections (dse_status=2) not in G4Bug, v02, or v03 dataset tags.
DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
========== =========== ======== =========== ========== ======== =======
2 0T Nonevent pointer 3 0.0
2 0T PRskims full 687 388.9
2 0T SPskims generic full 141 49.5
2 0T SPskims generic pointer 19 3.7
2 0T SPskims signal full 89 16.1
========== =========== ======== =========== ========== ======== =======
Totals 939 458.2
Removed a few more bbkr14 files that are now OK on tape.
FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES =========== ======== =========== ========== ======== ======= 0T SP signal full 3 0.8
The full list of files removed is in ~bbdatsrv/lists/060511a-del-tapeok.lis.
Note that one (Nonevent, CfgDB) file was not on disk anyway, and 749 files were on the broken disks /stage/bdata-data81 and 82. They were still set file_status=0T, and the latter had their symlinks removed from the /stage/bdata-data79/kanga cache filesystem tree. So in total we freed up 18272.1 GB.
02/05/06
Removed more signal MC (~bbdatsrv/lists/060502a-del-pfn.lis)
DB FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ====== =========== ======== =========== ========== ======== ======= bbkr14 0T SP signal full 90 8.9 bbkr18 0T SP signal full 8491 1925.3
Also removed
- ~bbdatsrv/lists/060502c-staged-ago.lis
- 1051 files (393.2 GB): staged (I presume) to disk before 1 April, but marked "tape-only" in the database (file_status=0T).
- ~bbdatsrv/lists/060502e-dup-del.lis
- 487 files (420.1 GB): more than one copy on disk (excluding 2 to be deleted with the list above). If one copy was pinned, I kept that one. NB. doesn't count extra copy on csfnfs64, for which we want another copy to allow csfnfs64's removal.
I checked they are all OK on tape.
21/04/06
Removed bad R18 files that aren't in any tagged dataset (G4Bug, v01, and v02 tags). Ie. everything with dse_status=2 that isn't in
BbkUser --dbname=bbkr18 --dse_status=2 --file_status=0 \
--dse_type=^Nonevent --tag=G4Bug,v01,v02 \
--ds_is_local=1 --ds_name=^prelim-*,^test-*
Also removed some more R18 signal and R14.
DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
========== =========== ======== =========== ========== ======== =======
1 0 SP signal full 108 37.8
2 0 PR full 123 63.6
2 0 PRskims full 747 646.8
2 0 PRskims pointer 22 13.3
2 0 SP generic full 1 0.0
2 0 SP signal full 8 3.1
2 0 SPskims generic full 1649 1374.7
2 0 SPskims generic pointer 209 59.6
2 0 SPskims signal full 204 76.7
========== =========== ======== =========== ========== ======== =======
Totals 3071 2275.8
DSE_STATUS FILE_STATUS DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES
========== =========== ======== =========== ========== ======== =======
1 0 Bkg background 1 0.7
1 0 SP generic full 1 1.2
1 0 SP signal full 10 0.3
1 0 SPskims generic full 1 0.0
========== =========== ======== =========== ========== ======== =======
Totals 13 2.1
19/04/06
Removed more R18 signal and R14 files, excluding files on /stage/bdata-data49 which is currently down.
DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ======== =========== ========== ======== ======= SP signal full 7717 2331.5
DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ======== =========== ========== ======== ======= SP generic full 10 7.1 SP signal full 44 5.4 SPskims generic full 113 4.7 SPskims generic pointer 4 0.5 ======== =========== ========== ======== ======= Totals 171 17.8
05/04/06
Removed R14 PR (plus a few more R14 signal SP, now archived).
DSE_TYPE DSE_MODENUM COMPONENTS #FILE_ID +GBYTES ======== =========== ========== ======== ======= PR full 19607 8329.5 SP signal full 14 1.4 ======== =========== ========== ======== ======= Totals 19621 8330.9
Apart from files on /stage/bdata-data44 and some other files still to be archived, that's the last of the R14/R16 data on disk. R14 is dead! Long live R18!
04/04/06
R18: removed some more new SP signal, and bad PR/PRskims (at least those marked bad before 08/03/2006).
DSE_TYPE DSE_MODENUM COMPONENTS DSE_STATUS #FILE_ID +GBYTES ======== =========== ========== ========== ======== ======= PR full 2 34 23.3 PRskims full 2 1394 1190.5 PRskims pointer 2 172 157.6 SP signal full 1 1878 451.1 SP signal full 2 12 4.0 ======== =========== ========== ========== ======== ======= Totals 3490 1826.5
R14: removed some more non-PR files that are now on tape.
DSE_TYPE DSE_MODENUM COMPONENTS DSE_STATUS #FILE_ID +GBYTES ======== =========== ========== ========== ======== ======= SP signal full 1 105 15.0 SP signal full 2 52 3.9 ======== =========== ========== ========== ======== ======= Totals 157 18.9
All file sizes checked against ADS, and all now marked file_status=0T.
NB. 49 of these files were on /stage/bdata-data44, which is limping along. I stopped the removal at that point, and restarted it excluding data44 files. The files that remain on data44 were marked file_status=0 again.
27/03/06
Removed from disk the remaining R14/R16 skims
SKIM_CYCLE DSE_TYPE SKIM #DSE_ID #FILE_ID +GBYTES ========== ======== =================== ======= ======== ======= R14 PRskims BCCC03a3body 196 196 153.4 R14 PRskims BCCKs03a3body 196 196 81.9 R14 PRskims BCCKs3body 196 196 117.4 R14 PRskims BCCPi003a3body 196 196 96.4 R14 PRskims BCPi0Ks03a3body 196 196 74.6 R14 PRskims BCPi0Ks3body 196 196 105.7 R14 PRskims BKsKsPi003a3body 196 196 13.8 R14 PRskims BPi0Pi0C03a3body 196 196 50.9 R14 PRskims BPi0Pi0C3body 196 196 71.0 R14 PRskims BPi0Pi0Ks03a3body 196 196 16.4 R14 PRskims BPi0Pi0Ks3body 195 195 22.2 R14 PRskims BPi0Pi0Pi003a3body 196 196 7.0 R14 PRskims BPi0Pi0Pi03body 196 196 9.1 R14 SPskims BCCC03a3body 739 752 315.2 R14 SPskims BCCKs03a3body 739 739 165.7 R14 SPskims BCCKs3body 738 743 229.1 R14 SPskims BCCPi003a3body 740 748 200.9 R14 SPskims BCPi0Ks03a3body 737 737 151.0 R14 SPskims BCPi0Ks3body 737 742 207.9 R14 SPskims BKsKsPi003a3body 738 738 26.8 R14 SPskims BPi0Pi0C03a3body 738 738 104.0 R14 SPskims BPi0Pi0C3body 738 740 145.2 R14 SPskims BPi0Pi0Ks03a3body 737 737 31.5 R14 SPskims BPi0Pi0Ks3body 738 738 43.6 R14 SPskims BPi0Pi0Pi003a3body 739 739 11.6 R14 SPskims BPi0Pi0Pi03body 738 738 15.7 R16a PRskims BKsKsPi003a3body 168 168 15.4 R16a SPskims BKsKsPi003a3body 589 589 20.5 R16b PRskims B0ToRhoPRhoM 156 156 25.0 R16b PRskims BCCKsPi0Pi003a3body 167 167 31.9 R16b PRskims BToKsKsKlDefault 167 167 8.7 R16b PRskims InclPhi 167 167 69.2 R16b SPskims B0ToRhoPRhoM 686 686 66.3 R16b SPskims BCCKsPi0Pi003a3body 685 685 80.2 R16b SPskims BToKsKsKlDefault 680 680 25.7 R16b SPskims InclPhi 679 679 93.8 R18b PRskims A0 1 2 3.0 R18b PRskims AllEvents 1 4 6.4 R18c PRskims AllEvents 1 4 6.4 ========== ======== =================== ======= ======== ======= Totals 16289 16326 2914.1
and the remaining/new SP and Background collections
DSE_TYPE DSE_MODENUM #DSE_ID #FILE_ID +GBYTES ======== =========== ======= ======== ======= Bkg 10 31 40.5 SP generic 8 8 6.2 SP signal 198 198 40.6 ======== =========== ======= ======== ======= Totals 216 237 87.3
This leaves just R14 AllEvents PR on disk (and a few collections that aren't on tape).
Removed most of the remaining R18 signal MC.
DSE_TYPE DSE_MODENUM #DSE_ID #FILE_ID +GBYTES ======== =========== ======= ======== ======= SP signal 9785 9785 2570.6 ======== =========== ======= ======== ======= Totals 9785 9785 2570.6
Note that in all cases, data on /stage/bdata-data44 was not removed (nor marked 0T), since that disk is having problems and shouldn't be touched.
Also, after removing the signal MC on 14/03, we discovered that some of those files had not in fact been written to tape (they were in the nameit catalogue, listed in ADSFiles, but the datastore create had failed since moving to the new front-end machines, which weren't in the %ATLAS datastore rule). Those files were reimported, and the above files have all been checked as being in the datastore with the right size.
14/03/06
Removed newly produced R18 signal MC
DSE_TYPE DSE_MODENUM #FILE_ID +GBYTES ======== =========== ======== ======= SP signal 17974 4604.8 ======== =========== ======== ======= Totals 17974 4604.8
and newly produced and recently-archived R14 SP.
DSE_TYPE DSE_MODENUM #FILE_ID +GBYTES ======== =========== ======== ======= SP generic 17 13.3 SP signal 471 89.5 ======== =========== ======== ======= Totals 488 102.8
(~bbdatsrv/lists/060314a-signal.lis)
23/02/06
Removed the rest of the R14 generic SP and the old test User collections (apart from 343 (79.7 GB) that weren't on tape). Marked them all file_status=0T.
DSE_TYPE DSE_MODENUM #DSE_ID #FILE_ID +GBYTES ======== =========== ======= ======== ======= SP B+B- 5160 5160 5494.2 SP B0B0bar 4951 4951 5154.6 SP ccbar 39 39 35.5 SP signal 242 242 66.2 SP tau 33 33 19.9 SP uds 107 107 101.7 User 15 25 29.3 ======== =========== ======= ======== ======= Totals 10547 10557 10901.4
(~bbdatsrv/lists/060223a-sp.lis)
14/02/06
Removed R14 uds, ccbar, and tau generic SP
DSE_TYPE DSE_MODENUM #DSE_ID #FILE_ID +GBYTES ======== =========== ======= ======== ======= SP 1005 3317 3317 3444.9 SP 3429 1884 1884 1305.8 SP 998 4904 4904 5021.1 ======== =========== ======= ======== ======= Totals 10105 10105 9771.8
and R18 Tau/QED skims
SKIM SKIM_CYCLE DSE_TYPE #DSE_ID #FILE_ID +GBYTES =================== ========== ======== ======= ======== ======= Tau11 R18b PRskims 194 353 345.5 Tau11 R18b SPskims 448 448 56.0 Tau1N R18b PRskims 192 348 414.2 Tau1N R18b SPskims 453 484 318.6 Tau33 R18b PRskims 190 354 460.5 Tau33 R18b SPskims 463 581 464.7 TwoPhotonPentaquark R18b PRskims 186 340 335.7 TwoPhotonPentaquark R18b SPskims 439 439 108.0 TwoPhotonPi0Pi0 R18b PRskims 189 189 94.0 TwoPhotonPi0Pi0 R18b SPskims 440 440 7.5 TwoPhotonTwoTrack R18b PRskims 183 183 153.6 TwoPhotonTwoTrack R18b SPskims 452 452 11.1 =================== ========== ======== ======= ======== ======= Totals 3829 4611 2769.3
This is the list of files on tape that are now marked file_status=0T. 22 of them were already missing from disk, probably from a broken disk.
12/02/06
Removed next batch of signal SP and marked file_status=0T.
Database DSE_TYPE #DSE_ID #FILE_ID +GBYTES ======== ======== ======= ======== ======= bbkr14 SP 678 678 195.8 bbkr18 SP 6144 6144 1782.4 ======== ======== ======= ======== ======= Total 6822 6822 1978.2
08/02/06
Removed the following collections, now marked bad (dse_status=2) from R14
DSE_TYPE DSE_STATUS #DSE_ID #FILE_ID +GBYTES ======== ========== ======= ======== ======= PR 2 23 23 7.4 PRskims 2 15 15 3.4 SP 2 10 10 11.6 SPskims 2 11 11 0.2 ======== ========== ======= ======== ======= Totals 59 59 22.6
and R18
DSE_TYPE DSE_STATUS #DSE_ID #FILE_ID +GBYTES ======== ========== ======= ======== ======= PR 2 1121 1121 728.6 PRskims 2 970 1301 1219.8 SP 2 3363 3363 969.6 SPskims 2 943 1004 303.2 ======== ========== ======= ======== ======= Totals 6397 6789 3221.2
All marked file_status=0T. The files are listed in ~bbdatsrv/lists/060208a-del-ontape.lis. A few others with dse_status=2 had just been imported, so aren't yet on tape. I didn't want to confuse matters by removing them too.
04/02/06
Removed A0, Tau1N, and the last AllEventsSkim and marked file_status=0T
SKIM_CYCLE SKIM DSE_TYPE #FILE_ID +GBYTES ========== ========= ======== ======== ======= R14 A0 PRskims 392 486.4 R14 A0 SPskims 1123 1012.7 R14 Tau1N PRskims 357 418.2 R14 Tau1N SPskims 881 615.7 R16c AllEvents PRskims 652 976.6 R16c AllEvents SPskims 828 866.5 ========== ========= ======== ======== ======= Totals 4233 4376.2
30/01/06
Removed R16b BToPPP skim.
SKIM_CYCLE SKIM DSE_TYPE #FILE_ID +GBYTES ========== ====== ======== ======== ======= R16b BToPPP PRskims 617 791.5 R16b BToPPP SPskims 1873 2321.3 ========== ====== ======== ======== ======= Totals 2490 3112.8
Marked file_status=0T 4/2/06
13/01/06
Removed signal MC (R14 and R18 so-far imported and on tape) from disk and marked file_status=0T.
Database DSE_TYPE #FILE_ID +GBYTES
======== ======== ======== =======
bbkr14 SP 42584 6646.9
bbkr18 SP 11735 3503.0
======== ======== ======== =======
54319 10149.9
12/01/06
Removed from disk and marked file_status=0T.
SKIM SKIM_CYCLE DSE_TYPE #FILE_ID +GBYTES ========= ========== ======== ======== ======= BFourBody R14 PRskims 529 682.8 Tau11 R14 PRskims 736 1023.4 Tau11 R14 SPskims 957 597.9 ========= ========== ======== ======== ======= Totals 2222 2304.0
23/12/05
Removed from disk and marked file_status=0T.
SKIM SKIM_CYCLE #FILE_ID +GBYTES ========== ========== ======== ======= BToA1Rho R16b 1208 877.8 BToCCC R16b 1245 1135.3 BToCPP R16b 1338 1313.1 BToKstarXX R16b 935 693.8 BToPP R16b 932 616.7 ========== ========== ======== ======= Totals 5658 4636.7
17/12/05
Removed from disk (marked file_status=0T on 23/12/05).
Skim Cycle Files Gbytes Last RAL xrootd access =================== ============ ===== ====== ====================== BToCXX R16b 3028 4079.9 never BToPXX R16b 2155 2763.4 never BtoKKKL R14+R14a+R16a 3321 3648.3 2005/04/28 Tau33 R14 1323 1263.6 only test jobs (bbdatsrv,olaiya,brew) =================== ============= ==== ======= Totals 9827 11755.1
Olya said I could remove the Tau33 (R14+R18b) and TwoPhotonPentaquark (R18b) skims, but I'll leave the R18b for the moment (it isn't so big yet). The other very big (3TB) skim, BToPPP, is being used now (last access 2005/12/09). AllEvents and SP are also heavily used.
