Difference between revisions of "Service Challenge Transfer Tests"

From GridPP Wiki
Jump to: navigation, search
 
(No difference)

Latest revision as of 15:00, 24 January 2008

As part of the Tier 2 SC4 Milestones, GridPP will perform transfer tests using the File Transfer Service, between the RAL Tier1 and each Tier 2 (T1<->T2) and Tier 2 to Tier 2 (T2<->T2).

These tests aim to:

  1. To demonstrate connectivity at high bandwidth from the T1 to the T2. The target rate is 300-500Mb/s.
  2. To demonstrate connectivity at high bandwidth from each T2 to another T2. The target rate is >250Mb/s.
  3. To demonstrate that thess bandwidth can be maintained over prolonged periods of intense file transfer (this is an end to end test of components from FTS->SRMs->Network).

Summary

An abbeviated summary of the best results in these transfer tests is in Service Challenge Transfer Test Summary

Testing that has contined in the runup to LHC startup is described at:

Transfer Tests

Transfer Test Coordinator

After a brief period away, it's, once again, the delightful... Graeme Stewart Andrew Elwell!

Doing A Transfer Test

  • We've a Transfer Test Checklist, which tells you what you need in place and walks through the mechanics of doing a transfer.

Script and Settings

Filetransfer.py

Ensure the latest version of the filetransfer script is installed on the machine that the FTS transfers shall be submitted from.

The options/arguments that should be used (the example is from Edinburgh to Glasgow) are...

 --duration => time in minutes (or hours and minutes) that the test should run for.
 --delete => the files are to be deleted from the destination after each transfer has taken place.
 --background => After the first submission run all the transfers in the background
 --logfile => Log transfers to this file (can be tailed to monitor progress)
 --uniform-source-size => the metadata (size) of just one file is checked and not all of the source SURLs.
 Checking 100 would take a long time and is unnecessary if all source files are the same size.
 srm://srm.epcc.ed.ac.uk:8443/pnfs/epcc.ed.ac.uk/data/dteam/canned/tfr000-file00[000:099] is the list of source SURLs.
 srm://se2-gla.scotgrid.ac.uk:8443/dpm/scotgrid.ac.uk/home/dteam/T2toT2test is the destination endpoint. It is
 formulated from an srm hostname and directory which exists/shall be created in the destination srm namespace.

A typical example (the example is from Edinburgh to Glasgow) would be...

 filetransfer.py --duration=24:00 --background --logfile=ed2gla --delete --uniform-source-size \
 srm://srm.epcc.ed.ac.uk:8443/pnfs/epcc.ed.ac.uk/data/dteam/canned/tfr000-file00[000:099] \
 srm://se2-gla.scotgrid.ac.uk:8443/dpm/scotgrid.ac.uk/home/dteam/24hrInterT2-dest/`date +%Y%m%d_%H%M%S`

Files/Streams per channel

Recommended values are
Number of files = 8
Number of Streams = 1
These can be set as below (obviously using your own FTS channel names)

 $ glite-transfer-channel-set -f 8 STAR-SCOTGRIDGLA
 $ glite-transfer-channel-set -T 1 STAR-SCOTGRIDGLA

and can be checked thus

 $ glite-transfer-channel-list STAR-SCOTGRIDGLA |grep files
 Number of files: 8, streams: 1

Table of Responsibilities

GridPP Tier 2 (ref) - Tier 2 (non-ref) SC4 File Transfer Tests September-December 06
Site Times Person Responsible
for Submitting Tests
email
ScotGrid Graeme Stewart g.stewart@physics.gla.ac.uk
Durham Mark Nelson Mark.Nelson@durham.ac.uk
Edinburgh Greig Cowan G.Cowan@ed.ac.uk
Glasgow User:Graeme stewart g.stewart@physics.gla.ac.uk
NorthGrid Alessandra forti Alessandra.Forti@manchester.ac.uk
Lancaster Brian Davies b.g.davies@lancaster.ac.uk
Liverpool Paul Trepka pat@hep.ph.liv.ac.uk
Manchester Alessandra forti Alessandra.Forti@manchester.ac.uk
Sheffield d.g.wilson@sheffield.ac.uk
SouthGrid Pete Gronbech gronbech@physics.ox.ac.uk
Birmingham Yves Coppens yrc@hep.ph.bham.ac.uk
Bristol Yves Coppens
Winnie Lacesso
lcg-site-admin@phy.bris.ac.uk
Cambridge Santanu Das santanu@hep.phy.cam.ac.uk
Oxford Pete Gronbech gronbech@physics.ox.ac.uk
RAL_Tier2 Chris Brew C.A.J.Brew@RL.AC.UK
London Tier2 Olivier Van-der-aa o.van-der-aa@IMPERIAL.AC.UK
Brunel Duncan Rand duncan.rand@brunel.ac.uk


IC-HEP Mona Aggarwal m.aggarwal@imperial.ac.uk


IC-LeSC Mona Aggarwal m.aggarwal@imperial.ac.uk
QMUL Giuseppe Mazza g.mazza@qmul.ac.uk
RHUL Duncan Rand duncan.rand@brunel.ac.uk
UCL-CENTRAL Alice Fage
William Hay
Austin Chamberlain
is-lcg-support@ucl.ac.uk
UCL-HEP Gianfranco Sciacca Gianfranco Sciacca <gs@hep.ucl.ac.uk>

Transfer Tests Endpoints

Edinburgh Endpoints
Birmingham Endpoints
RAL T2 Endpoints

SC4 File Transfer Tests

GridPP Tier 2 SC4 File Transfer Tests Jul 06 - Dec 06
Site July 2006 August 2006 September 2006 October 2006 November 2006 December 2006
RAL Tier1 14: 2hrs -> GLA

14: 4hrs -> GLA
14: 4hrs -> Ed
15: 5hrs -> GLA
15: 5hrs -> Ed
15: 5hrs -> IC-HEP
16: 5hrs -> GLA
16: 5hrs -> Ed
16: 5hrs -> IC-HEP
16: 5hrs -> BRIS

ScotGrid
Durham 5: 24hrs -> Ed

6: 24hrs <- Ed

3: 24hrs <-> Ed
Edinburgh 28: 6hrs -> RAL T2

28: 6hrs <- RAL T2

Glasgow 27: 24hrs <- Ed @235Mb/s*

28: 24hrs -> Ed @66Mb/s*

11: 24hrs <-> Ed
NorthGrid
Lancaster 11: 24hrs -> Ed

12: 24hrs <- Ed

4: 24hrs <-> Ed
Liverpool 13: 24hrs -> Ed

14: 24hrs <- Ed

5: 24hrs <-> Ed
Manchester 7: 24hrs -> Ed

8: 24hrs <- Ed

11: 24hrs <-> Ed @271/244
Sheffield 15: 24hrs -> Ed

18: 24hrs <- Ed

9: 24hrs <-> Ed 13: 24hrs <-> Ed
SouthGrid
Birmingham 29: 6hrs -> Ed

29: 6hrs <- Ed

Bristol 1: 24hrs -> BHAM

4: 24hrs <- BHAM

2: 24hrs <-> BHAM 9: 24hrs <-> RAL T2
Cambridge 5: 24hrs -> BHAM

6: 24hrs <- BHAM

3: 24hrs <-> BHAM 13: 24hrs <-> RAL T2
Oxford 7: 24hrs -> BHAM

8: 24hrs <- BHAM

4: 24hrs <-> BHAM 9: 24hrs -> RAL T2
14: 24hrs <-> RAL T2
RAL Tier2 30: 6hrs -> BHAM

30: 6hrs <- BHAM

London Tier2
Brunel 1: 24hrs -> RAL T2

4: 24hrs <- RAL T2

2: 24hrs <-> RAL T2
IC-HEP 11: 24hrs -> BHAM

12: 24hrs <- BHAM

5: 24hrs <-> BHAM
IC-LeSC 13: 24hrs -> BHAM

14: 24hrs <- BHAM

6: 24hrs <-> BHAM
QMUL 5: 24hrs -> RAL T2

6: 24hrs <- RAL T2

3: 24hrs <-> RAL T2
RHUL 7: 24hrs -> RAL T2

8: 24hrs <- RAL T2

4: 24hrs <-> RAL T2
UCL-CENTRAL 11: 24hrs -> RAL T2

12: 24hrs <- RAL T2

5: 24hrs <-> RAL T2
UCL-HEP 13: 24hrs -> RAL T2

14: 24hrs <- RAL T2

6: 24hrs <-> RAL T2
GridPP Tier 2 SC4 File Transfer Tests Dec 05 - Jun 06
Site December 2005 January 2006 February 2006 March 2006 April 2006 May 2006 June 2006
RAL Tier1 7: 1TB -> RAL T2 @197Mb/s*
13: 0.86TB -> RAL T2 @194Mb/s*
22: 1TB -> MAN @~320Mb/s*
9: 1TB <- GLA @331Mb/sGlasgow
10: 0.985TB -> RAL T2 @397Mb/s*
12-23: @1200Mb/s <- CERN *
25: 1TB <- RAL T2@388Mb/s*
25: 1TB -> BHAM @289Mb/s
26: 1TB -> GLA@166Mb/s*
26: 1TB -> OX @252Mb/s
27: 1TB <- BHAM @407Mb/s
28: 1TB <- OX @456Mb/s
29: 1TB <- BHAM @461Mb/s
29: 974GB <- MAN@152.755
31: 1TB <- MAN@143.988
30: 1TB <- OX@456Mb/s
7: 1TB -> ED
8: 1TB -> LIV*
3: 1TB <- LIV*
14: 1TB <- QMUL@118Mb/s
15: 1TB ->QMUL@172Mb/s
16: 96GB <- BRIS @85Mb/s
16: 240GB <- CAM @74Mb/s
17-20: RAL T1 shut down
28-3 48hr -> BHAM,GLA,MAN,QMUL
7-10 48hr -> BHAM,GLA,MAN,QMUL
15-16 48hr -> BHAM,GLA,MAN,QMUL,LANC@1100Mb/s*
22-23 48hr <- BHAM,GLA,MAN,QMUL,LANC
NorthGrid
Lancaster 12: 50GB <- T1 @150Mb/s

20: 1TB <- T1 @~800Mb/s
19: 1TB -> T1 @0Mb/s

9: 1.6TB -> T1 @500Mb/srmcp only(noFTS)
Liverpool 2: 1TB <- RAL T1*

@ 88 Mb/s
8: 1TB -> RAL T1* @ 24 Mb/s

18 1TB<-GLA
19 1TB->GLA
Manchester 21: 250GB <- RAL @320Mb/s*
22: 1TB <- RAL @~320Mb/s*
29: 974GB -> RAL@152.755Mb/s*
31: 1TB -> RAL@143.988Mb/s*


Sheffield 12 0.542TB<-GLA@100.7Mb/s*
13 0.25TB<-GLA@144Mb/s*
13 1TB->GLA@414Mb/s*
ScotGrid
Durham 18: 500GB <- GLA@88Mb/s* 20 194GB<-RAL T1@193Mb/s* 20 100GB->GLA@176Mb/s*


Edinburgh 2: 100GB -> Gla @200Mb/s
13: 25GB <- T1 @11Mb/s*
14: 500MB <- T1*
14: 10GB <- T1*
14: 1GB <- T1 *
16: various tests to/from T1*
16: 15GB -> T1 @422Mb/s*
17: 15GB <- T1 @156Mb/s*
17: 10GB <- Gla @57Mb/s*
17: 10GB -> Gla @122Mb/s*
9: 1TB -> RAL @440Mb/s* 6: 1TB <- RAL 18: 0.25TB <- SHEF@203Mb/s*
Glasgow 2: 100GB <- Ed @200Mb/s
22: 5x20GB <- Ed @~210Mb/s*
24: 1TB <- Ed @224Mb/s*
9: 1TB -> RAL @331Mb/s*
16: 1TB -> BHAM @317Mb/s*
17: 66GB -> QMUL@80Mb/s*
18: 500GB -> DUR@88Mb/s*
20: 200GB -> RHUL@59Mb/s*
26: 1TB <- RAL@166Mb/s*
London Tier2
Brunel 25: 2GB <- GLA @ 32Mb/s 24: 92GB->RAL T1 @ 59Mb/s 8: 135GB <- GLA @ 57Mb/s
IC-HEP 16: 250MB <- T1@84Mb/s* 20: 140GB -> RAL@190Mb/s*
22: <- RAL@80Mb/s*
IC-LeSC 10 250GB->GLA@95.27Mbit/sec*
11 250GB<-GLA@156.16Mbit/sec*
QMUL 16: 50GB test. 67Mb/s *

17: 10GB test. 64Mb/s *
17: 66GB test. ~80Mb/s *

14: 1TB -> RAL@172Mb/s*
15: 1TB <- RAL@118Mb/s*
RHUL 19: 2GB <- GLA @35Mb/s (Wed day)*

19: 53GB <- GLA @49Mb/s (Wed night)*
20: 200 GB <- GLA @59Mb/s (Fri night)*

16: 128GB<-Gla @35Mb/s

18: 250GB->Gla @58Mb/s
19: 250GB<-Gla

UCL-HEP 21: 421GB<-RAL T1@71Mb/s*
25: 421GB->RAL T1@63Mb/s*
UCL-CENTRAL 26 (not office hours): 250GB<-GLA@90Mb/s*
27 (not office hours): 250GB->GLA@309Mb/s*
06/06 (office hours): 8GB<-Edinburgh@266Mb/s*
SouthGrid
Birmingham 16: 1TB <- GLA @317Mb/s*
25: 1TB <- RAL T1 @ 289Mb/s
28: 1TB -> RAL T1 @407Mb/s
29: 1TB -> RAL T1 @461Mb/s
Bristol 16: 96GB -> RAL @85Mb/s 17: 500GB->GLA @291Mb/s
17: 500GB <-GLA @117Mb/s
Cambridge 16: 240GB -> RAL @74Mb/s
20: 787GB->GLA @153Mb/s
21: 1TB<-GLA @293Mb/s
Oxford 26: 1TB <- RAL T1 @252Mb/s
30: 1TB -> RAL T1@456Mb/s
RAL Tier2 7: 1TB <- T1 @197Mb/s*
8: 2GB <- T1 @309Mb/s*
13: 0.86TB <- T1 @194Mb/s*
10: 0.985TB <- T1 @397Mb/s*
25: 1TB -> RAL T1@388Mb/s*

Table Format

To keep this table tidy and legible, please follow these notes on Table Format:

  1. Use green for completed tests and red for scheduled tests. Get these using, e.g., <span style="color:green">green</span>.
  2. At your discretion, leave out smaller tests (say <100GB), which can instead be recorded in your site page. This table is meant to give a high level view of the larger tests, not to record every single transfer you submit.
  3. Prefix each test with the day, then a colon, e.g., for a test starting on the fifth, use "5:".
  4. Use "->" to indicate "to", and "<-" to indicate "from".
  5. Use a line break in separate tests done in the same month. (<br>)
  6. Once a test has been done, indicate a rate in Mb/s (bits, not bytes!).
  7. Any extra information about a transfer can be put into a link back your local site page, using a * as the link text, e.g., [[Glasgow#Transfer 2005 12 12|*]]. Thanks to Chris for that idea.
  8. The convention employed when deciding which row to put a transfer planned/result in the tables is that it shall go in the row corresponding to the site being tested. e.g. in August 06 it was the Castor filesystem at RAL T1 being tested whereas in September 06 it was the non-reference sites bandwidth capibilities being tested.