PhEDEx high-throughput data transfer management system
Authors: TUURA, Lassi (Northeastern University), BARRASS, Tim (Bristol University), BONACORSI, Daniele - on behalf of the CMS Collaboration (INFN-CNAF, Bologna, Italy)
Track classification: Distributed Event production and processing
Distributed data management at LHC scales is a staggering task, accompanied by equally challenging practical management issues with storage systems and wide-area networks. CMS data transfer management system, PhEDEx, is designed to handle this task with minimum operator effort, automating the workflows from large scale distribution of HEP experiment datasets down to reliable and scalable transfers of individual files over frequently unreliable infrastructure. Over the last year PhEDEx has matured to the point of handling virtually all CMS production data transfers. CMS pushes equally its own components to perform and the heavy investment into peer projects at all levels, from technical details to grid standards to world-wide projects, to ensure the endto-end service is of sufficient quality. We present the throughput and service quality we have reached in the current daily 24/7 production work, the steps taken in LCG service challenges for the next generation transfer service, and the resulting changes in performance. We also report results from our scalability stress tests on PhEDEx alone. We offer an analysis of transfer-related problems we have encountered and how they have been affecting CMS data management.
Last modified Wed 7 December 2005 . View page history
Switch to HTTPS . Website Help . Print View . Built with GridSite 1.4.3