Service Challenge Transfer Tests
As part of the Tier 2 SC4 Milestones, GridPP will perform transfer tests using the File Transfer Service, between the RAL Tier1 and each Tier 2 (T1<->T2) and Tier 2 to Tier 2 (T2<->T2).
These tests aim to:
- To demonstrate connectivity at high bandwidth from the T1 to the T2. The target rate is 300-500Mb/s.
- To demonstrate connectivity at high bandwidth from each T2 to another T2. The target rate is >250Mb/s.
- To demonstrate that thess bandwidth can be maintained over prolonged periods of intense file transfer (this is an end to end test of components from FTS->SRMs->Network).
Contents
Summary
An abbeviated summary of the best results in these transfer tests is in Service Challenge Transfer Test Summary
Testing that has contined in the runup to LHC startup is described at:
Transfer Tests
Transfer Test Coordinator
After a brief period away, it's, once again, the delightful... Graeme Stewart Andrew Elwell!
Doing A Transfer Test
- We've a Transfer Test Checklist, which tells you what you need in place and walks through the mechanics of doing a transfer.
Script and Settings
Filetransfer.py
Ensure the latest version of the filetransfer script is installed on the machine that the FTS transfers shall be submitted from.
The options/arguments that should be used (the example is from Edinburgh to Glasgow) are...
--duration => time in minutes (or hours and minutes) that the test should run for. --delete => the files are to be deleted from the destination after each transfer has taken place. --background => After the first submission run all the transfers in the background --logfile => Log transfers to this file (can be tailed to monitor progress) --uniform-source-size => the metadata (size) of just one file is checked and not all of the source SURLs. Checking 100 would take a long time and is unnecessary if all source files are the same size. srm://srm.epcc.ed.ac.uk:8443/pnfs/epcc.ed.ac.uk/data/dteam/canned/tfr000-file00[000:099] is the list of source SURLs. srm://se2-gla.scotgrid.ac.uk:8443/dpm/scotgrid.ac.uk/home/dteam/T2toT2test is the destination endpoint. It is formulated from an srm hostname and directory which exists/shall be created in the destination srm namespace.
A typical example (the example is from Edinburgh to Glasgow) would be...
filetransfer.py --duration=24:00 --background --logfile=ed2gla --delete --uniform-source-size \ srm://srm.epcc.ed.ac.uk:8443/pnfs/epcc.ed.ac.uk/data/dteam/canned/tfr000-file00[000:099] \ srm://se2-gla.scotgrid.ac.uk:8443/dpm/scotgrid.ac.uk/home/dteam/24hrInterT2-dest/`date +%Y%m%d_%H%M%S`
Files/Streams per channel
Recommended values are
Number of files = 8
Number of Streams = 1
These can be set as below (obviously using your own FTS channel names)
$ glite-transfer-channel-set -f 8 STAR-SCOTGRIDGLA $ glite-transfer-channel-set -T 1 STAR-SCOTGRIDGLA
and can be checked thus
$ glite-transfer-channel-list STAR-SCOTGRIDGLA |grep files Number of files: 8, streams: 1
Table of Responsibilities
Site | Times | Person Responsible for Submitting Tests |
|
---|---|---|---|
ScotGrid | Graeme Stewart | g.stewart@physics.gla.ac.uk | |
Durham | Mark Nelson | Mark.Nelson@durham.ac.uk | |
Edinburgh | Greig Cowan | G.Cowan@ed.ac.uk | |
Glasgow | User:Graeme stewart | g.stewart@physics.gla.ac.uk | |
NorthGrid | Alessandra forti | Alessandra.Forti@manchester.ac.uk | |
Lancaster | Brian Davies | b.g.davies@lancaster.ac.uk | |
Liverpool | Paul Trepka | pat@hep.ph.liv.ac.uk | |
Manchester | Alessandra forti | Alessandra.Forti@manchester.ac.uk | |
Sheffield | d.g.wilson@sheffield.ac.uk | ||
SouthGrid | Pete Gronbech | gronbech@physics.ox.ac.uk | |
Birmingham | Yves Coppens | yrc@hep.ph.bham.ac.uk | |
Bristol | Yves Coppens Winnie Lacesso |
lcg-site-admin@phy.bris.ac.uk | |
Cambridge | Santanu Das | santanu@hep.phy.cam.ac.uk | |
Oxford | Pete Gronbech | gronbech@physics.ox.ac.uk | |
RAL_Tier2 | Chris Brew | C.A.J.Brew@RL.AC.UK | |
London Tier2 | Olivier Van-der-aa | o.van-der-aa@IMPERIAL.AC.UK | |
Brunel | Duncan Rand | duncan.rand@brunel.ac.uk
| |
IC-HEP | Mona Aggarwal | m.aggarwal@imperial.ac.uk
| |
IC-LeSC | Mona Aggarwal | m.aggarwal@imperial.ac.uk | |
QMUL | Giuseppe Mazza | g.mazza@qmul.ac.uk | |
RHUL | Duncan Rand | duncan.rand@brunel.ac.uk | |
UCL-CENTRAL | Alice Fage William Hay Austin Chamberlain |
is-lcg-support@ucl.ac.uk | |
UCL-HEP | Gianfranco Sciacca | Gianfranco Sciacca <gs@hep.ucl.ac.uk> |
Transfer Tests Endpoints
Edinburgh Endpoints
Birmingham Endpoints
RAL T2 Endpoints
SC4 File Transfer Tests
Site | July 2006 | August 2006 | September 2006 | October 2006 | November 2006 | December 2006 | |
---|---|---|---|---|---|---|---|
RAL Tier1 | 14: 2hrs -> GLA 14: 4hrs -> GLA |
||||||
ScotGrid | |||||||
Durham | 5: 24hrs -> Ed 6: 24hrs <- Ed |
3: 24hrs <-> Ed | |||||
Edinburgh | 28: 6hrs -> RAL T2 28: 6hrs <- RAL T2 |
||||||
Glasgow | 27: 24hrs <- Ed @235Mb/s* |
11: 24hrs <-> Ed | |||||
NorthGrid | |||||||
Lancaster | 11: 24hrs -> Ed 12: 24hrs <- Ed |
4: 24hrs <-> Ed | |||||
Liverpool | 13: 24hrs -> Ed 14: 24hrs <- Ed |
5: 24hrs <-> Ed | |||||
Manchester | 7: 24hrs -> Ed 8: 24hrs <- Ed |
11: 24hrs <-> Ed @271/244 | |||||
Sheffield | 15: 24hrs -> Ed 18: 24hrs <- Ed |
9: 24hrs <-> Ed | 13: 24hrs <-> Ed | ||||
SouthGrid | |||||||
Birmingham | 29: 6hrs -> Ed 29: 6hrs <- Ed |
||||||
Bristol | 1: 24hrs -> BHAM 4: 24hrs <- BHAM |
2: 24hrs <-> BHAM | 9: 24hrs <-> RAL T2 | ||||
Cambridge | 5: 24hrs -> BHAM 6: 24hrs <- BHAM |
3: 24hrs <-> BHAM | 13: 24hrs <-> RAL T2 | ||||
Oxford | 7: 24hrs -> BHAM 8: 24hrs <- BHAM |
4: 24hrs <-> BHAM | 9: 24hrs -> RAL T2 14: 24hrs <-> RAL T2 |
||||
RAL Tier2 | 30: 6hrs -> BHAM 30: 6hrs <- BHAM |
||||||
London Tier2 | |||||||
Brunel | 1: 24hrs -> RAL T2 4: 24hrs <- RAL T2 |
2: 24hrs <-> RAL T2 | |||||
IC-HEP | 11: 24hrs -> BHAM 12: 24hrs <- BHAM |
5: 24hrs <-> BHAM | |||||
IC-LeSC | 13: 24hrs -> BHAM 14: 24hrs <- BHAM |
6: 24hrs <-> BHAM | |||||
QMUL | 5: 24hrs -> RAL T2 6: 24hrs <- RAL T2 |
3: 24hrs <-> RAL T2 | |||||
RHUL | 7: 24hrs -> RAL T2 8: 24hrs <- RAL T2 |
4: 24hrs <-> RAL T2 | |||||
UCL-CENTRAL | 11: 24hrs -> RAL T2 12: 24hrs <- RAL T2 |
5: 24hrs <-> RAL T2 | |||||
UCL-HEP | 13: 24hrs -> RAL T2 14: 24hrs <- RAL T2 |
6: 24hrs <-> RAL T2 |
Site | December 2005 | January 2006 | February 2006 | March 2006 | April 2006 | May 2006 | June 2006 |
---|---|---|---|---|---|---|---|
RAL Tier1 | 7: 1TB -> RAL T2 @197Mb/s* 13: 0.86TB -> RAL T2 @194Mb/s* 22: 1TB -> MAN @~320Mb/s* |
9: 1TB <- GLA @331Mb/sGlasgow 10: 0.985TB -> RAL T2 @397Mb/s* 12-23: @1200Mb/s <- CERN * 25: 1TB <- RAL T2@388Mb/s* 25: 1TB -> BHAM @289Mb/s 26: 1TB -> GLA@166Mb/s* 26: 1TB -> OX @252Mb/s 27: 1TB <- BHAM @407Mb/s 28: 1TB <- OX @456Mb/s 29: 1TB <- BHAM @461Mb/s 29: 974GB <- MAN@152.755 31: 1TB <- MAN@143.988 30: 1TB <- OX@456Mb/s |
7: 1TB -> ED 8: 1TB -> LIV* 3: 1TB <- LIV* 14: 1TB <- QMUL@118Mb/s 15: 1TB ->QMUL@172Mb/s 16: 96GB <- BRIS @85Mb/s 16: 240GB <- CAM @74Mb/s 17-20: RAL T1 shut down |
28-3 48hr -> BHAM,GLA,MAN,QMUL 7-10 48hr -> BHAM,GLA,MAN,QMUL 15-16 48hr -> BHAM,GLA,MAN,QMUL,LANC@1100Mb/s* 22-23 48hr <- BHAM,GLA,MAN,QMUL,LANC |
|||
NorthGrid | |||||||
Lancaster | 12: 50GB <- T1 @150Mb/s |
9: 1.6TB -> T1 @500Mb/srmcp only(noFTS) |
|||||
Liverpool | 2: 1TB <- RAL T1*
@ 88 Mb/s 8: 1TB -> RAL T1* @ 24 Mb/s |
18 1TB<-GLA 19 1TB->GLA |
|||||
Manchester | 21: 250GB <- RAL @320Mb/s* 22: 1TB <- RAL @~320Mb/s* |
29: 974GB -> RAL@152.755Mb/s* 31: 1TB -> RAL@143.988Mb/s* |
| ||||
Sheffield | 12 0.542TB<-GLA@100.7Mb/s* 13 0.25TB<-GLA@144Mb/s* 13 1TB->GLA@414Mb/s* |
||||||
ScotGrid | |||||||
Durham | 18: 500GB <- GLA@88Mb/s* | 20 194GB<-RAL T1@193Mb/s* | 20 100GB->GLA@176Mb/s*
| ||||
Edinburgh | 2: 100GB -> Gla @200Mb/s 13: 25GB <- T1 @11Mb/s* 14: 500MB <- T1* 14: 10GB <- T1* 14: 1GB <- T1 * 16: various tests to/from T1* 16: 15GB -> T1 @422Mb/s* 17: 15GB <- T1 @156Mb/s* 17: 10GB <- Gla @57Mb/s* 17: 10GB -> Gla @122Mb/s* |
9: 1TB -> RAL @440Mb/s* | 6: 1TB <- RAL | 18: 0.25TB <- SHEF@203Mb/s* | |||
Glasgow | 2: 100GB <- Ed @200Mb/s 22: 5x20GB <- Ed @~210Mb/s* 24: 1TB <- Ed @224Mb/s* |
9: 1TB -> RAL @331Mb/s* 16: 1TB -> BHAM @317Mb/s* 17: 66GB -> QMUL@80Mb/s* 18: 500GB -> DUR@88Mb/s* 20: 200GB -> RHUL@59Mb/s* 26: 1TB <- RAL@166Mb/s* |
|||||
London Tier2 | |||||||
Brunel | 25: 2GB <- GLA @ 32Mb/s | 24: 92GB->RAL T1 @ 59Mb/s | 8: 135GB <- GLA @ 57Mb/s | ||||
IC-HEP | 16: 250MB <- T1@84Mb/s* | 20: 140GB -> RAL@190Mb/s* 22: <- RAL@80Mb/s* |
|||||
IC-LeSC | 10 250GB->GLA@95.27Mbit/sec* 11 250GB<-GLA@156.16Mbit/sec* |
||||||
QMUL | 16: 50GB test. 67Mb/s * |
14: 1TB -> RAL@172Mb/s* 15: 1TB <- RAL@118Mb/s* |
|||||
RHUL | 19: 2GB <- GLA @35Mb/s (Wed day)* 19: 53GB <- GLA @49Mb/s (Wed night)* |
16: 128GB<-Gla @35Mb/s |
|||||
UCL-HEP | 21: 421GB<-RAL T1@71Mb/s* 25: 421GB->RAL T1@63Mb/s* |
||||||
UCL-CENTRAL | 26 (not office hours): 250GB<-GLA@90Mb/s* 27 (not office hours): 250GB->GLA@309Mb/s* |
06/06 (office hours): 8GB<-Edinburgh@266Mb/s* | |||||
SouthGrid | |||||||
Birmingham | 16: 1TB <- GLA @317Mb/s* 25: 1TB <- RAL T1 @ 289Mb/s 28: 1TB -> RAL T1 @407Mb/s 29: 1TB -> RAL T1 @461Mb/s |
||||||
Bristol | 16: 96GB -> RAL @85Mb/s | 17: 500GB->GLA @291Mb/s 17: 500GB <-GLA @117Mb/s |
|||||
Cambridge | 16: 240GB -> RAL @74Mb/s |
20: 787GB->GLA @153Mb/s 21: 1TB<-GLA @293Mb/s |
|||||
Oxford | 26: 1TB <- RAL T1 @252Mb/s 30: 1TB -> RAL T1@456Mb/s |
||||||
RAL Tier2 | 7: 1TB <- T1 @197Mb/s* 8: 2GB <- T1 @309Mb/s* 13: 0.86TB <- T1 @194Mb/s* |
10: 0.985TB <- T1 @397Mb/s* 25: 1TB -> RAL T1@388Mb/s* |
Table Format
To keep this table tidy and legible, please follow these notes on Table Format:
- Use green for completed tests and red for scheduled tests. Get these using, e.g., <span style="color:green">green</span>.
- At your discretion, leave out smaller tests (say <100GB), which can instead be recorded in your site page. This table is meant to give a high level view of the larger tests, not to record every single transfer you submit.
- Prefix each test with the day, then a colon, e.g., for a test starting on the fifth, use "5:".
- Use "->" to indicate "to", and "<-" to indicate "from".
- Use a line break in separate tests done in the same month. (<br>)
- Once a test has been done, indicate a rate in Mb/s (bits, not bytes!).
- Any extra information about a transfer can be put into a link back your local site page, using a * as the link text, e.g., [[Glasgow#Transfer 2005 12 12|*]]. Thanks to Chris for that idea.
- The convention employed when deciding which row to put a transfer planned/result in the tables is that it shall go in the row corresponding to the site being tested. e.g. in August 06 it was the Castor filesystem at RAL T1 being tested whereas in September 06 it was the non-reference sites bandwidth capibilities being tested.