Difference between revisions of "Perfsonar refresh"

From GridPP Wiki
Jump to: navigation, search
 
(68 intermediate revisions by 15 users not shown)
Line 1: Line 1:
 +
==Deployment==
  
 +
We are installing WLCG nodes and deployment instructions may be found on the [https://opensciencegrid.org/networking/#network-monitoring-in-wlcg-and-osg-perfsonar OSG Networking pages]. In the past large sites had two hosts, one for bandwidth tests and one for latency tests. In this refresh both tests run from one host using two NICs, each with its own FQDN. Details of how to configure this new dual-NIC configuration may be found on the OSG site [https://opensciencegrid.org/networking/perfsonar/deployment-models/#multiple-nic-network-interface-card-guidance here] and in the [https://docs.perfsonar.net/manage_dual_xface.html official perfSONAR documentation]. Please make sure to update the [https://goc.egi.eu/ GOCDB] if you change the names of the hosts. You can check basic operation of your hosts on the WLCG Check_MK service:
 +
 +
* [https://psetf.opensciencegrid.org/etf/check_mk/index.py?start_url=%2Fetf%2Fcheck_mk%2Fview.py%3Fhostgroup%3DUK%26opthost_group%3DUK%26view_name%3Dhostgroup UK host status (cert required)]
 +
 +
and also the UK dashboard here
 +
 +
* [https://psmad.opensciencegrid.org/maddash-webui/index.cgi?dashboard=UK%20Mesh%20Config WLCG maddash]
 +
 +
With respect to the existing data, the WLCG installation guide states: "Local measurement archive backup is not needed as OSG/WLCG stores all measurements centrally. In case you'd like to perform the backup anyway please follow the migration [https://docs.perfsonar.net/install_migrate_centos7.html guide]."
  
 
{|border="1" cellpadding="1"
 
{|border="1" cellpadding="1"
 
|+
 
|+
 
 
|-style="background:#7C8AAF;color:white"
 
|-style="background:#7C8AAF;color:white"
 
|Site
 
|Site
 
|Status
 
|Status
|Host names
 
 
|Notes
 
|Notes
 
|Date last reviewed or updated
 
|Date last reviewed or updated
 
 
 
|-
 
|-
 
|RAL  Tier-1
 
|RAL  Tier-1
|<span style="color:red">In progress</span>
+
|<span style="color:orange">In progress</span>
|
+
|Hardware has been installed and cabled up on 100Gb/s capable switch. Implementing host into the new Tier1 network and configuring with dula network https://docs.perfsonar.net/manage_dual_xface.html (~December).
|Hardware has been delivered. Still in boxes as it is not a clear . Realistic time frame of October for deployment.
+
|2021-11-17
|2020-05-29
+
 
+
 
+
 
|-
 
|-
 
|UKI-LT2-Brunel
 
|UKI-LT2-Brunel
|<span style="color:red">In progress</span>
+
|<span style="color:orange">In progress</span>
|
+
|The perfSONAR server has been delivered but access to the data centre has been restricted.  Access to the data centre should be possible by mid October and we hope to have it running soon after.
|We have received our perfsonar, but we have two problems: 1) restrictions on access to racks, due to COVID limitations, 2) restrictions due to network upgrade that should have been done before the lockdown.
+
|2020-10-02
|2020-05-29
+
 
+
+
 
|-
 
|-
 
|UKI-LT2-IC-HEP
 
|UKI-LT2-IC-HEP
 
|<span style="color:green">Done</span>
 
|<span style="color:green">Done</span>
|lt2ps00-lat.grid.hep.ph.ic.ac.uk lt2ps00-bw.grid.hep.ph.ic.ac.uk
+
|
|100G
+
|2020-09-30
|2020-05-29
+
 
+
 
+
 
|-
 
|-
 
|UKI-LT2-QMUL
 
|UKI-LT2-QMUL
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
|
+
|Connected at 100G for Bandwidth and separate link for latency
|New 100Gb-capable perfSONAR hardware has been deployed into rack, connected to 100Gb switch and has IPv4/IPv6 connectivity w/ jumbo frames.  We would like to migrate the PMA (perfSONAR Measurement Archive) from our existing perfSONAR hosts but if this is not necessary, we don’t have a problem in simply changing our DNS records to point to the new host and consider the PMA on the existing hosts as ‘lost’.
+
|2021-12-15
 
+
As the existing hosts require physical access in order to gain shell access, migrating the data is somewhat difficult as our Mile End campus is still under lockdown with limited access available to us.
+
|2020-05-29
+
 
+
 
|-
 
|-
 
|UKI-LT2-RHUL
 
|UKI-LT2-RHUL
|<span style="color:red">In progress</span>
+
|<span style="color:orange">In progress</span>
|
+
|New node was received and set up in our lab, has now been moved to the data centre and is ready for physical installation, along with some accompanying network equipment. No estimate yet as to when we could do this. Currently (Jan 2021) only doing essential maintenance (Simon).
|New node is installed but sitting in a temporary rack in our dept, awaiting the resumption of the portering service to move it to the data centre. (Simon)
+
|2021-01-12
|2020-05-29
+
 
+
|-
+
|UKI-LT2-UCL-HEP
+
|<span style="color:red">In progress</span>
+
|
+
|
+
|YYYY-MM-DD
+
 
+
 
|-
 
|-
 
|UKI-NORTHGRID-LANCS-HEP
 
|UKI-NORTHGRID-LANCS-HEP
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
|
+
|The routing scripts, which set packets to "only go out the interface they came in", break our perfsonar if they are run so results might be affected by this.
|The new Lancaster perfsonar box is in the rack and I hope to have it installed and configured soon, but it's having to take a back seat for a big cluster overhaul happening towards the end of June. I will be online by the end of June, but probably not until right near the end of June.
+
|2020-12-04
|2020-05-29
+
 
+
 
|-
 
|-
 
|UKI-NORTHGRID-LIV-HEP
 
|UKI-NORTHGRID-LIV-HEP
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
|
+
|New node installed with perfsonar, running bandwidth and latency tests. Old nodes shut down. Awaiting test results.
|
+
|2020-10-15
|YYYY-MM-DD
+
 
+
 
|-
 
|-
 
|UKI-NORTHGRID-MAN-HEP
 
|UKI-NORTHGRID-MAN-HEP
|<span style="color:red">In progress</span>
+
|<span style="color:blue">On hold</span>
|
+
|Hardware arrived at office building, needs to be moved to machine room. Installation also depending on network upgrade. No information by when this will be completed (also dependent on the University getting its 100G connections to JISC).
|
+
|2020-12-15
|YYYY-MM-DD
+
 
+
 
+
 
|-
 
|-
 
|UKI-NORTHGRID-SHEF-HEP
 
|UKI-NORTHGRID-SHEF-HEP
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
|
+
|Sheffield Has installed new box and networking team have opened firewall ports and deal with reverse lookup DNS. The box is configured with bonded interface so may need separating out in the future. For now it is "working" attention needs to be elsewhere.
|
+
|2020-10-05
|YYYY-MM-DD
+
 
|-
 
|-
 
|UKI-SCOTGRID-DURHAM
 
|UKI-SCOTGRID-DURHAM
 
|<span style="color:green">Done</span>
 
|<span style="color:green">Done</span>
|
 
 
|They are seeing asymmetric routing, which is being investigated.
 
|They are seeing asymmetric routing, which is being investigated.
 
|2020-05-29
 
|2020-05-29
 
|-
 
|-
 
 
|UKI-SCOTGRID-ECDF
 
|UKI-SCOTGRID-ECDF
 
|<span style="color:green">Done</span>
 
|<span style="color:green">Done</span>
|
 
 
|Completed before Covid-19 caused problems.
 
|Completed before Covid-19 caused problems.
 
|2020-05-29
 
|2020-05-29
 
 
|-
 
|-
 
|UKI-SCOTGRID-GLASGOW
 
|UKI-SCOTGRID-GLASGOW
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
|
+
|Hardware installed, highlighted problems with number of paths due to in process DC migration as well as IPv6 bandwidth issues.
|Hardware has been delivered.  Sam didn’t feel qualified to comment on the exact status.
+
|2020-09-30
|2020-05-29
+
 
+
 
|-
 
|-
 
|UKI-SOUTHGRID-BHAM-HEP
 
|UKI-SOUTHGRID-BHAM-HEP
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
|
+
|Reinstalled perfSONAR toolkit and it appears to be working. Previous server still available if needed.
|
+
|2021-01-28
|YYYY-MM-DD
+
 
+
 
|-
 
|-
 
|UKI-SOUTHGRID-BRIS
 
|UKI-SOUTHGRID-BRIS
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
|
+
|Completed 1 NIC configuration (full replacement of the previous box). Next: request and configure new networks for 2nd NIC with new aliases.
|Building still closed, new Perfsonar hardware still with delivery/vendor
+
|2020-10-30
|2020-05-29
+
 
+
|-
+
|UKI-SOUTHGRID-CAM-HEP
+
|<span style="color:red">In progress</span>
+
|
+
|
+
|YYYY-MM-DD
+
 
+
 
|-
 
|-
 
|UKI-SOUTHGRID-OX-HEP
 
|UKI-SOUTHGRID-OX-HEP
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
|
+
|Completed.
|Managed to rack up the boxes last week.  Will try and install them in the coming couple of weeks.
+
|2020-09-30
|2020-05-29
+
|-
 +
|UKI-SOUTHGRID-SUSX
 +
|<span style="color:orange">In Progress</span>
 +
|Hardware arrived and installed in DC. Still sorting networking out and need to install and configure PerfSonar server
 +
|2020-11-17
 
|-
 
|-
 
|UKI-SOUTHGRID-RALPP
 
|UKI-SOUTHGRID-RALPP
|<span style="color:red">In progress</span>
+
|<span style="color:green">Done</span>
 
|
 
|
|
+
|2020-11-12
|YYYY-MM-DD
+
 
+
 
|-
 
|-
|UKI-SOUTHGRID-SUSX
 
|<span style="color:red">In progress</span>
 
|
 
|
 
|YYYY-MM-DD
 
  
 
|}
 
|}
  
 +
==Installation Caveats==
 +
 +
Please add any comments below.
  
 
[[Category:Sites Status]]
 
[[Category:Sites Status]]
 
[[Category:Networking]]
 
[[Category:Networking]]

Latest revision as of 15:32, 15 December 2021

Deployment

We are installing WLCG nodes and deployment instructions may be found on the OSG Networking pages. In the past large sites had two hosts, one for bandwidth tests and one for latency tests. In this refresh both tests run from one host using two NICs, each with its own FQDN. Details of how to configure this new dual-NIC configuration may be found on the OSG site here and in the official perfSONAR documentation. Please make sure to update the GOCDB if you change the names of the hosts. You can check basic operation of your hosts on the WLCG Check_MK service:

and also the UK dashboard here

With respect to the existing data, the WLCG installation guide states: "Local measurement archive backup is not needed as OSG/WLCG stores all measurements centrally. In case you'd like to perform the backup anyway please follow the migration guide."

Site Status Notes Date last reviewed or updated
RAL Tier-1 In progress Hardware has been installed and cabled up on 100Gb/s capable switch. Implementing host into the new Tier1 network and configuring with dula network https://docs.perfsonar.net/manage_dual_xface.html (~December). 2021-11-17
UKI-LT2-Brunel In progress The perfSONAR server has been delivered but access to the data centre has been restricted. Access to the data centre should be possible by mid October and we hope to have it running soon after. 2020-10-02
UKI-LT2-IC-HEP Done 2020-09-30
UKI-LT2-QMUL Done Connected at 100G for Bandwidth and separate link for latency 2021-12-15
UKI-LT2-RHUL In progress New node was received and set up in our lab, has now been moved to the data centre and is ready for physical installation, along with some accompanying network equipment. No estimate yet as to when we could do this. Currently (Jan 2021) only doing essential maintenance (Simon). 2021-01-12
UKI-NORTHGRID-LANCS-HEP Done The routing scripts, which set packets to "only go out the interface they came in", break our perfsonar if they are run so results might be affected by this. 2020-12-04
UKI-NORTHGRID-LIV-HEP Done New node installed with perfsonar, running bandwidth and latency tests. Old nodes shut down. Awaiting test results. 2020-10-15
UKI-NORTHGRID-MAN-HEP On hold Hardware arrived at office building, needs to be moved to machine room. Installation also depending on network upgrade. No information by when this will be completed (also dependent on the University getting its 100G connections to JISC). 2020-12-15
UKI-NORTHGRID-SHEF-HEP Done Sheffield Has installed new box and networking team have opened firewall ports and deal with reverse lookup DNS. The box is configured with bonded interface so may need separating out in the future. For now it is "working" attention needs to be elsewhere. 2020-10-05
UKI-SCOTGRID-DURHAM Done They are seeing asymmetric routing, which is being investigated. 2020-05-29
UKI-SCOTGRID-ECDF Done Completed before Covid-19 caused problems. 2020-05-29
UKI-SCOTGRID-GLASGOW Done Hardware installed, highlighted problems with number of paths due to in process DC migration as well as IPv6 bandwidth issues. 2020-09-30
UKI-SOUTHGRID-BHAM-HEP Done Reinstalled perfSONAR toolkit and it appears to be working. Previous server still available if needed. 2021-01-28
UKI-SOUTHGRID-BRIS Done Completed 1 NIC configuration (full replacement of the previous box). Next: request and configure new networks for 2nd NIC with new aliases. 2020-10-30
UKI-SOUTHGRID-OX-HEP Done Completed. 2020-09-30
UKI-SOUTHGRID-SUSX In Progress Hardware arrived and installed in DC. Still sorting networking out and need to install and configure PerfSonar server 2020-11-17
UKI-SOUTHGRID-RALPP Done 2020-11-12

Installation Caveats

Please add any comments below.