ATLAS Site Availability and Performance (ASAP)
Since December 2013, a new way for ATLAS computing availability has been introduced. It's called ASAP / ATLAS Site Availability and Performance / ATLAS_AnalysisAvailability (depending where you look). From what I have gleaned about it, ASAP is a metric to replace ADCD site status; it’s not related the SAM tests. The status of the "PandaResource" of an analysis queue is used by ASAP. A site is considered to be unavailable when its analysis queue is in test mode. This document will briefly describe some of the implications of this.
Contents
HammerCloud
Various tests are sent out to sites using a framework called "HammerCloud". The results of various tests are combined using an algorithm. The HammerCloudATLASOperations wiki page describes it all. The upshot is that once a set of tests have failed, the site is set to test, i.e. not available. Ultimately that is used by ASAP to determine the site's unavailability periods.
How to get the important alerts
Unfortunately, at the moment, when a queue is set to test mode, the notification email is sent to cloud support and doesn’t go to to our site admins. Some site admins may be able to subscribe to the list (atlas-support-cloud-uk@cern.ch) at e-groups.cern.ch. Admins without the necessary security credentials can request to be subscribed; ask Elena Korolkova, Alessandra Forti or another GridPP representative of ATLAS.
Once you are getting the alerts, it's usually easy to set up filters that can find the messages for your site by searching the subject field for the name of the site's queues. A list of all Panda queues can be found here: dashb-atlas-ssb.cern.ch.
Alternatively, the HammerCloud document contains a section that tells how get additional individual emails per queue, although this functionality has not been for a while, so you may run into trouble with it.
Where to check ASAP site status
You can check your site's ASAP status here: wlcg-mon.cern.ch. You can use buttons to select various sites and time-scales etc.
Once you have had an alert or otherwise found out that you have a problem, it's time to look into the cause. The HammerCloud Site Overview is a good place to begin. Select your site and some times to example, and choose functional tests (FTs used for blacklisting).You'll get some grid representing the PandaResources (queues) at your site. I can't understand the naming convention, but the import one at present follows this pattern: ANALY_<site_mnemonic>_SL6 , e.g. for Liverpool, the relevant one is ANALY_LIV_SL6. Look for tests that are f (red) or m (orange). Those are the ones that have caused trouble. Clicking on the test takes you to the summary. You use the information in this page, and the ones linked from it, to debug the problem.
How to use HC detailed status records to debug common scenarios
In this section, I describe particular debugging use cases that I've tried.
xxx
fsdfsd
