DCache Security Service Challenge

From GridPP Wiki
Revision as of 07:22, 16 May 2007 by Greig cowan (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Challenge

The material here details the RAL response to a security service challenge (SSC) that was executed in October 2006. A similar procedure will have to be followed by Tier-2 dCache sites if they have to pass the SSC which will be run in May 2007.

Distinguished Name (DN) of Grid credentials used by the submitter:

/C=CH/O=CERN/OU=GRID/CN=Pal Anderssen 4660

Date:

2006-10-10

Approximate time interval:

   between: 07:40 -and- 08:00 (UTC)

Affected storage element:

   dcache.gridpp.rl.ac.uk


Please investigate and respond by providing the following information:

1). Which sequence of storage operations were executed by the challenger

   in the specified time interval (UTC)?

2). What was the IP-address of the User Interface (UI) which was used

   for the job submission?


Response

1. Which sequence of storage operations were executed by the challenger in the specified time interval (UTC)?

At 07:47 (UTC) an srm put request was received by dcache.gridpp.rl.ac.uk from the host lcg0648.gridpp.rl.ac.uk by a certificate who's DN matched that given in the details. A TURL for the host gftp0440.gridpp.rl.ac.uk was returned to the client.

Again at 07:47 (UTC), a gridftp connection to gftp0440.gridpp.rl.ac.uk using the same DN and client host was then received. A data connection to csfnfs63.rl.ac.uk was opened and a 1092 byte file called //pnfs/gridpp.rl.ac.uk/data/dteam/ssc/SSC_66b6d462-9dff-44bc-8f01-e29d9be8ce6e.file was written.

AT 07:48 (UTC) an srm get request was received by dcache.gridpp.rl.ac.uk from the host lcg0648.gridpp.rl.ac.uk by the same DN. Again a TURL for the host gftp0440.gridpp.rl.ac.uk was returned to the client

Again at 07:48 (UTC), a gridftp connection to gftp0440.gridpp.rl.ac.uk using the same DN was received and the 1092 byte file was read from csfnfs63.rl.ac.uk

The file was then deleted, dCache does not appear to log the method use to delete, but the message to pools to remove the file was logged at 07:48 (UTC)

2. What was the IP-address of the User Interface (UI) which was used for the job submission

The user interface used to submit the job was lxplus008.cern.ch (137.138.5.77)


Working

Having been told the DN of the user, we need to find dCache's internal id for that DN:

Connect to the database for dcache.gridpp.rl.ac.uk on pg350.gridpp.rl.ac.uk

#psql -U srmdcache dcache
> >select id from srmrequestcredentials where credentialname like '/C=CH/O=CERN/OU=GRID/CN=Pal Anderssen 4660';
     id
-------------
 -2147482565
(1 row)


Now we have the internal id for the DN we can find what requests have been received


dcache=> select * from putrequests_b where credentialid=-2147482565 order by creationtime;
     id      | nextjobid | creationtime  | lifetime | state | errormessage | creatorid | schedulerid | schedulertimestamp | numofretr | maxnumofretr | credentialid |  retrydeltatime | shouldupdateretrydeltatime
-------------+-----------+---------------+----------+-------+--------------+-----------+-------------+--------------------+-----------+--------------+--------------+----------------+----------------------------
 -2147482564 |           | 1150116890934 |  7200000 |     9 |              | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0
 -2147482244 |           | 1150117668122 |  7200000 |     9 |              | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0
 -2147404770 |           | 1150295022169 |  7200000 |     9 |              | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0
 -2147404768 |           | 1150295022396 |  7200000 |     9 |              | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0
 -2144446720 |           | 1160466467955 |  7200000 |     9 |              | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0
(5 rows)
dcache=> select * from getrequests_b where credentialid=-2147482565 order by creationtime;
     id      | nextjobid | creationtime  | lifetime | state |   errormessage   | creatorid | schedulerid | schedulertimestamp | numofretr | maxnumofretr | credentialid | retrydeltatime | shouldupdateretrydeltatime
-------------+-----------+---------------+----------+-------+------------------+-----------+-------------+--------------------+-----------+--------------+--------------+----------------+----------------------------
 -2147482520 |         0 | 1150116925952 |  7200000 |    11 | lifetime expired | dteam001  |             |                  0 |         0 |           10 |  -2147482565  |              4 |                           1
 -2147482218 |         0 | 1150117792522 |  7200000 |    11 | lifetime expired | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              4 |                          1
 -2147482117 |         0 | 1150118198504 |  7200000 |    11 | lifetime expired | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              4 |                          1
 -2147481936 |         0 | 1150119093133 |  7200000 |    11 | lifetime expired | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |             13 |                          1
 -2147404658 |           | 1150295226999 |  7200000 |     9 |                  | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0
 -2147404654 |           | 1150295230633 |  7200000 |     9 |                  | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0
 -2144446718 |           | 1160466481742 |  7200000 |     9 |                  | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0
(7 rows)
dcache=> select * from copyrequests_b where credentialid=-2147482565 order by creationtime;
 id | nextjobid | creationtime | lifetime | state | errormessage | creatorid | schedulerid | schedulertimestamp | numofretr | maxnumofretr | credentialid | retrydeltatime | shouldupdateretrydeltatime
----+-----------+--------------+----------+-------+--------------+-----------+-------------+--------------------+-----------+--------------+--------------+----------------+----------------------------
(0 rows)


Only care about this put request :

-2144446720 |           | 1160466467955 |  7200000 |     9 |              | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0

And this get request

-2144446718 |           | 1160466481742 |  7200000 |     9 |                  | dteam001  |             |                  0 |         0 |           10 |  -2147482565 |              1 |                          0

All others are too old - creationtime field is milliseconds after epoch, 1150295230633 is Wednesday, June 14th 2006, 14:27:10 (UTC); 1160466467955 is Tuesday, October 10th 2006, 07:47:47 (UTC)

Now we look for specific file requests associated with these requests ids

Put request specifics:

dcache=> select * from putfilerequests_b where requestid=-2144446720;
    id      | nextjobid | creationtime  | lifetime | state | errormessage | creatorid | schedulerid | schedulertimestamp | numofretr | maxnumofretr |  requestid  | credentialid |                                                      surl                                                      |                                                           turl                                                            | fileid |       parentfileid       | spacereservationid |       clienthost        | size

+-----------+---------------+----------+-------+--------------+-----------+-------------+--------------------+-----------+--------------+-------------+--------------+----------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+--------+--------------------------+--------------------+-------------------------+------
 -2144446719 |           | 1160466468175 |  7200000 |     9 |              | dteam001  |             |      1156410969815 |         0 |           10 | -2144446720 |   -2147482565 | srm://dcache.gridpp.rl.ac.uk/pnfs/gridpp.rl.ac.uk/data/dteam/ssc/SSC_66b6d462-9dff-44bc-8f01-e29d9be8ce6e.file | gsiftp://gftp0440.gridpp.rl.ac.uk:2811//pnfs/gridpp.rl.ac.uk/data/dteam/ssc/SSC_66b6d462-9dff-44bc-8f01-e29d9be8ce6e.file |        | 0002000000000000001B04F0 |                    | lcg0648.gridpp.rl.ac.uk | 1092
(1 row)


Get request specifics:

dcache=> select * from getfilerequests_b where requestid=-2144446718;
     id      | nextjobid | creationtime  | lifetime | state | errormessage | creatorid | schedulerid | schedulertimestamp | numofretr | maxnumofretr |  requestid  | credentialid |                                                      surl                                                      |                                                           turl                                                            |          fileid          |        pinid

+-----------+---------------+----------+-------+--------------+-----------+-------------+--------------------+-----------+--------------+-------------+--------------+----------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------+--------------------------+----------------------
 -2144446717 |           | 1160466481960 |  7200000 |     9 |              | dteam001  |             |      1156410969790 |         0 |           10 | -2144446718 |  -2147482565 | srm://dcache.gridpp.rl.ac.uk/pnfs/gridpp.rl.ac.uk/data/dteam/ssc/SSC_66b6d462-9dff-44bc-8f01-e29d9be8ce6e.file | gsiftp://gftp0440.gridpp.rl.ac.uk:2811//pnfs/gridpp.rl.ac.uk/data/dteam/ssc/SSC_66b6d462-9dff-44bc-8f01-e29d9be8ce6e.file | 000200000000000001171328 |  -9223372036854181423


These entries give us the SURL for the file : srm://dcache.gridpp.rl.ac.uk/pnfs/gridpp.rl.ac.uk/data/dteam/ssc/SSC_66b6d462-9dff-44bc-8f01-e29d9be8ce6e.file and a size - 1092 bytes And dCache's internal id for the file  : 000200000000000001171328 We also have a host used : lcg0648.gridpp.rl.ac.uk

We can now check the dcache billing logs on dcache-head.gridpp.rl.ac.uk in /opt/d-cache/billing/2006/10/ for entries about this pnfsid

> > grep 000200000000000001171328 billing-2006.10.10
10.10 08:47:52 [pool:csfnfs63_4@csfnfs63Domain:transfer] [000200000000000001171328,1092] dteam:dteam@osm 1092 198 true {GFtp-1.0 gftp0440.gridpp.rl.ac.uk 52856} {0:""}
10.10 08:47:51 [door:GFTP-gftp0440-Unknown-2452@gridftp-gftp0440Domain:request] ["/C=CH/O=CERN/OU=GRID/CN=Pal Anderssen 4660":36300:24311:lcg0648.gridpp.rl.ac.uk]  [000200000000000001171328,0] <unknown> 1160466471993 0 {0:""}
10.10 08:48:08 [pool:csfnfs63_4:transfer] [000200000000000001171328,0] <unknown> true {null} {0:""}
10.10 08:48:08 [pool:csfnfs63_4@csfnfs63Domain:transfer] [000200000000000001171328,1092] dteam:dteam@osm 1092 98 false {GFtp-1.0 128.142.175.73 20026} {0:""}
10.10 08:48:07 [door:GFTP-gftp0440-Unknown-2455@gridftp-gftp0440Domain:request] ["/C=CH/O=CERN/OU=GRID/CN=Pal Anderssen 4660":36300:24311:lcg0648.gridpp.rl.ac.uk]  [000200000000000001171328,0] <unknown> 1160466487760 0 {0:""}
10.10 08:48:23 Pool=broadcast;RemoveFiles=,000200000000000001171328

Times are UK local so match the given range. This appears consistent with the srm database logs - a put then a get. We now know that the file was stored in pool csfnfs63_4 which is on host csfnfs63.rl.ac.uk. We have also logged a RemoveFiles message for that pnfsid, so it appears to have been removed. Checking the virtual pnfs fs confirms this, as does checking the pool filesystem on csfnfs63.rl.ac.uk.

This is all we can determine from dCache.

lcg0648 is a RAL-T1 worker node therefore it is likely a job was submitted to the Tier 1 farm to store this file, however double check that it wasn't done by logging into the sysem


Using last on the system shows no logins to that host at the relevant time

> > last
root     pts/0        escpc07.esc.rl.a Wed Oct 11 13:28   still logged in
root     pts/0        dumbledore.cis.r Tue Oct 10 11:12 - 07:55  (20:42)

wtmp begins Tue Oct 10 11:12:21 2006

> > last -f /var/log/wtmp.1
root     pts/0        dumbledore.cis.r Wed Sep 27 13:54 - 08:06  (18:12)

Checking central syslog server confirms last log.

Oct 10 11:12:21 lcg0648 sshd[15760]: Accepted publickey for root from 130.246.77

.4 port 52034 ssh2

Oct 10 11:12:21 lcg0648 sshd(pam_unix)[15760]: session opened for user root by (

uid=0)

Oct 10 17:47:44 lcg0648 sshd[28580]: Accepted publickey for root from 130.246.18

1.138 port 34344 ssh2

Oct 10 17:47:44 lcg0648 sshd(pam_unix)[28580]: session opened for user root by (

uid=0)

Oct 10 17:47:44 lcg0648 sshd(pam_unix)[28580]: session closed for user root

It appears the requests were done by a job therefore check the PBS server logs on csflnx353.rl.ac.uk In /var/spool/pbs/server_priv/accounting,

grep ';E;' 20061010 | grep lcg0648 | less

This job stands out: its a dteam job, matching the vo writing the file and ends shortly after the last action we've seen performed on the SE, however we cannot be sure yet.

10/10/2006 08:48:30;E;3912656.csflnx353.rl.ac.uk;user=dteam036 group=dteam jobname=STDIN queue=dteamL ctime=1160466435 qtime=1160466435 etime=1160466435 start=1160466453 exec_host=lcg0648.gridpp.rl.

ac.uk/2 Resource_List.cput=200:00:00 Resource_List.neednodes=1 Resource_List.nodect=1 Resource_List.nodes=1 Resource_List.pcput=200:00:00 Resource_List.pmem=500mb Resource_List.walltime=72:00:00 ses sion=7655 end=1160466510 Exit_status=0 resources_used.cput=00:00:05 resources_used.mem=22292kb resources_used.vmem=60596kb resources_used.walltime=00:01:13

pbs job id is 3912656.csflnx353.rl.ac.uk, user is dteam036

Checking on CE for mapped user

lcgce01 root]# ls -i /etc/grid-security/gridmapdir/ | grep `ls -i /etc/grid-security/gridmapdir/dteam036 | cut -f1 -d' '`
3091260 %2fc%3dch%2fo%3dcern%2fou%3dgrid%2fcn%3dpal%20anderssen%204660
3091260 dteam036

This matches with the DN supplied to us, very likely that we have the job that performed the actions.

Find GRAM_SCRIPT_JOBID from pbs jobid in CE syslog messages

grep 3912656.csflnx353.rl.ac.uk /var/log/messages*

/var/log/messages:Oct 10 08:47:15 lcgce01 gridinfo: [20926-23922] Submitted job 1160466364:lcgpbs:internal_2339032324:20608.1160466358 to batch system lcgpbs with ID 3912656.csflnx353.rl.ac.uk /var/log/messages:Oct 10 08:50:12 lcgce01 gridinfo: [20926-20926] Job 1160466364:lcgpbs:internal_2339032324:20608.1160466358 (ID 3912656.csflnx353.rl.ac.uk) has finished

GRAM_SCRIPT_JOB_ID is 1160466364:lcgpbs:internal_2339032324:20608.1160466358 

> > grep 1160466364:lcgpbs:internal_2339032324:20608.1160466358 /var/log/globus-gatekeeper.log JMA 2006/10/10 08:46:04 GATEKEEPER_JM_ID 2006-10-10.08:45:53.0000002637.0000056007 has GRAM_SCRIPT_JOB_ID 1160466364:lcgpbs:internal_2339032324:20608.1160466358 manager type lcgpbs

GATEKEEPER_JM_ID is 2006-10-10.08:45:53.0000002637.0000056007 
> > grep 2006-10-10.08:45:53.0000002637.0000056007 /var/log/globus-gatekeeper.log
Notice: 0: GATEKEEPER_JM_ID 2006-10-10.08:45:53.0000002637.0000056007 for /C=CH/O=CERN/OU=GRID/CN=Pal Anderssen 4660 on 130.246.183.184
JMA 2006/10/10 08:45:53 GATEKEEPER_JM_ID 2006-10-10.08:45:53.0000002637.0000056007 has EDG_WL_JOBID 'https://lcgrb01.gridpp.rl.ac.uk:9000/8Zh3mjHbJGsZ5eyoD5oJNg'
JMA 2006/10/10 08:46:04 GATEKEEPER_JM_ID 2006-10-10.08:45:53.0000002637.0000056007 for /C=CH/O=CERN/OU=GRID/CN=Pal Anderssen 4660 on 130.246.183.184
JMA 2006/10/10 08:46:04 GATEKEEPER_JM_ID 2006-10-10.08:45:53.0000002637.0000056007 mapped to dteam036 (36335, 24311)
JMA 2006/10/10 08:46:04 GATEKEEPER_JM_ID 2006-10-10.08:45:53.0000002637.0000056007 has GRAM_SCRIPT_JOB_ID 1160466364:lcgpbs:internal_2339032324:20608.1160466358  manager type lcgpbs
JMA 2006/10/10 08:46:04 GATEKEEPER_JM_ID 2006-10-10.08:45:53.0000002637.0000056007 JM exiting

Job was submitted through RAL-LCG2 RB lcgrb01.gridpp.rl.ac.uk Convert GRAM_SCRIPT_JOB_ID into JM-Contact-String

1160466364:lcgpbs:internal_2339032324:20608.1160466358 => https://lcgce01.gridpp.rl.ac.uk/???????/20608/1160466358

Look on rb at CondorG logs in /var/edgwl/logmonitor/CondorG.log

[root@lcgrb01 CondorG.log]# grep -C2 "lcgce01.gridpp.rl.ac.uk.*/20608/1160466358" *
CondorG.1160464329.log-017 (345407.000.000) 10/10 08:46:03 Job submitted to Globus
CondorG.1160464329.log-    RM-Contact: lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs
CondorG.1160464329.log:    JM-Contact: https://lcgce01.gridpp.rl.ac.uk:50041/20608/1160466358/
CondorG.1160464329.log-    Can-Restart-JM: 1
CondorG.1160464329.log-...

CondorG id is (345407.000.000)

Try to find some more info using the CondorG id:

# grep -A2 "(345407.000.000)" *
CondorG.1160464329.log:000 (345407.000.000) 10/10 08:45:48 Job submitted from host: <130.246.183.184:49160>
CondorG.1160464329.log-    () (UI=000003:NS=0000000003:WM=000004:BH=0000000000:JSS=000003:LM=000000:LRMS=000000:APP=000000) (0)
CondorG.1160464329.log-...
--
CondorG.1160464329.log:017 (345407.000.000) 10/10 08:46:03 Job submitted to Globus
CondorG.1160464329.log-    RM-Contact: lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs
CondorG.1160464329.log-    JM-Contact: https://lcgce01.gridpp.rl.ac.uk:50041/20608/1160466358/
--
CondorG.1160464329.log:001 (345407.000.000) 10/10 08:49:33 Job executing on host: lcgce01.gridpp.rl.ac.uk
CondorG.1160464329.log-...
--
CondorG.1160464329.log:005 (345407.000.000) 10/10 08:50:56 Job terminated.
CondorG.1160464329.log- (1) Normal termination (return value 0)
CondorG.1160464329.log-         Usr 0 00:00:00, Sys 0 00:00:00  -  Run Remote Usage

EDG Job id is https://lcgrb01.gridpp.rl.ac.uk:9000/8Zh3mjHbJGsZ5eyoD5oJNg

Check the events log for the EDG job id

# grep 8Zh3mjHbJGsZ5eyoD5oJNg events.log*
events.log:10 Oct, 08:45:31 [8] -I- "CFSI::crStagingDirectories": Creating staging dirs - job: https://lcgrb01.gridpp.rl.ac.uk:9000/8Zh3mjHbJGsZ5eyoD5oJNg
events.log:10 Oct, 08:45:38 [19] -F- "NS2WM::convertProtocol": Converted String: [ arguments = [ ad = [ requirements = ( other.GlueCEUniqueID ==  "lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs-dteamL" ) && ( other.GlueCEStateStatus == "Production" ); RetryCount = 3; edg_jobid =   "https://lcgrb01.gridpp.rl.ac.uk:9000/8Zh3mjHbJGsZ5eyoD5oJNg"; Arguments = "cat"; OutputSandboxPath = "/var/edgwl/SandboxDir/8Z/https_3a_2f_2flcgrb01.gridpp.rl.ac.uk_3a9000_2f8Zh3mjHbJGsZ5eyoD5oJNg/output"; JobType = "normal"; Executable = "cat"; CertificateSubject = "/C=CH/O=CERN/OU=GRID/CN=Pal Anderssen 4660"; StdOutput = "std.out"; X509UserProxy = "/var/edgwl/SandboxDir/8Z/https_3a_2f_2flcgrb01.gridpp.rl.ac.uk_3a9000_2f8Zh3mjHbJGsZ5eyoD5oJNg/user.proxy"; OutputSandbox = { "std.out","std.err" }; LB_sequence_code = "UI=000003:NS=0000000003:WM=000000:BH=0000000000:JSS=000000:LM=000000:LRMS=000000:APP=000000"; InputSandboxPath = "/var/edgwl/SandboxDir/8Z/https_3a_2f_2flcgrb01.gridpp.rl.ac.uk_3a9000_2f8Zh3mjHbJGsZ5eyoD5oJNg/input"; VirtualOrganisation = "dteam"; rank =  -other.GlueCEStateEstimatedResponseTime; Type = "job"; StdError = "std.err"; DefaultRank =  -other.GlueCEStateEstimatedResponseTime; InputSandbox = { "cat",".storacc","profiler.cfg","file","SE.cfg" } ] ]; command = "jobsubmit"; version = "1.0.0" ]

events.log:10 Oct, 08:45:39 [19] -I- "NS2WM::stSeqCode": Sequence Code file: /var/edgwl/SandboxDir/8Z/https_3a_2f_2flcgrb01.gridpp.rl.ac.uk_3a9000_2f8Zh3mjHbJGsZ5eyoD5oJNg/.edg_wll_seq events.log:10 Oct, 08:51:26 [7] -W- "CFSI::doPurge": JobId object for purging created: https://lcgrb01.gridpp.rl.ac.uk:9000/8Zh3mjHbJGsZ5eyoD5oJNg

Executable run was called 'cat'

Connect to mysql server on RB (db lbserver20)

> > select event,code,prog,host,time_stamp,arrived from events where jobid='8Zh3mjHbJGsZ5eyoD5oJNg';
+-------+------+-----------------+-------------------------+---------------------+---------------------+
| event | code | prog            | host                    | time_stamp          | arrived             |
+-------+------+-----------------+-------------------------+---------------------+---------------------+
|     0 |   17 | UserInterface   | lxplus008.cern.ch       | 2006-10-10 07:45:24 | 0000-00-00 00:00:00 |
|     1 |    1 | UserInterface   | lxplus008.cern.ch       | 2006-10-10 07:45:25 | 0000-00-00 00:00:00 |
|     2 |    2 | NetworkServer   | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:31 | 0000-00-00 00:00:00 |
|     3 |    4 | NetworkServer   | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:38 | 0000-00-00 00:00:00 |
|     4 |    5 | WorkloadManager | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:39 | 0000-00-00 00:00:00 |
|     5 |    1 | UserInterface   | lxplus008.cern.ch       | 2006-10-10 07:45:38 | 0000-00-00 00:00:00 |
|     6 |   15 | WorkloadManager | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:40 | 0000-00-00 00:00:00 |
|     7 |    4 | WorkloadManager | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:42 | 0000-00-00 00:00:00 |
|     8 |    4 | WorkloadManager | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:43 | 0000-00-00 00:00:00 |
|     9 |    5 | JobController   | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:44 | 0000-00-00 00:00:00 |
|    10 |    1 | JobController   | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:46 | 0000-00-00 00:00:00 |
|    11 |    1 | JobController   | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:45:48 | 0000-00-00 00:00:00 |
|    12 |    2 | LogMonitor      | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:46:22 | 0000-00-00 00:00:00 |
|    13 |    1 | LogMonitor      | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:46:25 | 0000-00-00 00:00:00 |
|    14 |    8 | LRMS            | lcg0648.gridpp.rl.ac.uk | 2006-10-10 07:47:44 | 0000-00-00 00:00:00 |
|    15 |   10 | LRMS            | lcg0648.gridpp.rl.ac.uk | 2006-10-10 07:48:26 | 0000-00-00 00:00:00 |
|    16 |    8 | LogMonitor      | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:49:45 | 0000-00-00 00:00:00 |
|    17 |   10 | LogMonitor      | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:51:06 | 0000-00-00 00:00:00 |
|    18 |   13 | NetworkServer   | lcgrb01.gridpp.rl.ac.uk | 2006-10-10 07:51:26 | 0000-00-00 00:00:00 |
+-------+------+-----------------+-------------------------+---------------------+---------------------+ 

User interface was lxplus008.cern.ch lxplus008.cern.ch has address 137.138.5.77