Tests indicated a ~100Mb/s write speed to Durham's DPM from Glasgow. Decided to scale back test to 500GB to avoid transferring during the day.
Initiated transfer (5 files, 5 streams) at ~9.30pm. Glasgow lost networking (power cut) at 10pm, currting the transfer. When Glasgow came back up next morning the transfer kicked in again and completed.
Rate was 88Mb/s (measured from Ganglia). Mark assures me the SE is connected to a Gb link, so we need to investigate why there's a bottleneck. Should use iperf to do some Network Testing.
Resolved a long standing problem with Durham's DPM enabling the transfer test to take place. Mark had a broken UI, so Graeme triggered the test.
Started a 1000x1GB transfer from RAL to Durham at ~2200. Transfer went well until about 2400, then died horribly, with FTS reporting:
Failed on SRM put: Failed SRM put call. Error is Permission denied
Subsequent investigations show the Durham DPM is giving permission denied errors on all attempts to transfer into the /dpm space.
On the bright side, the transfer rate was ok while it lasted:
Transfer Bandwidth Report: 194/1000 transferred in 8009.78204894 seconds 194000000000.0 bytes transferred. Bandwidth: 193.763075015Mb/s
After some investgations the DPM problem was revealed to be an issue with a directory on the underlying filesystem having the wrong ownership (root:root).
Tests the previous month revealed a severe outbound rate limitation (<10Mb/s). This was tracked to an anachronistic rate limitation imposed by NORMAN on Durham. This has been limted to Durham's nominal fraction of the MAN (200Mb/s).
Avoiding a major daytime test, 100GB was transfered to Glasgow:
Transfer Bandwidth Report: 100/100 transferred in 4526.39037299 seconds 100000000000.0 bytes transferred. Bandwidth: 176.741273747Mb/s
An approach will now be made to NORMAN to lift the 200Mb/s limit to allow the grid to use spare bandwidth on the MAN (it would be fine for this to be at a lower priority than other users' traffic).