DPM Log File Tracing

DPM Logging

The log files for, say, an srmcp transaction are spread over dpns, dpm, srmv1 and gridftp logs. In addition, note that gridftp transactions on pool nodes are logged on that host, not on the admin node. It's then far from easy to work out where one should look. DNs get recorded in multiple places by multiple daemons - usually information appears more than once (confusing but 2 is much better than 0 in this case).

Process IDs and thread numbers

Most of the DPM logs contain lines like:

01/24 18:33:58 25057,0 Cns_srv_stat: NS092 - stat request by

The 25057 is the unix process ID of the daemon that processed the request. The numer after this (0 above) is the number of the DPM daemon thread that actually did the processing. You can use this information to track multiple requests that are being processed by the same daemon simultaneously. For an example, by default, the DPNS daemon has 20 "fast" threads and "20" slow threads for dealing with different types of requests.

Tracing Transactions (Put/Get/Delete)

The answer to the basic security question who put file X on your system is much easier if SRM was used (normally it will be).

1. For SRM v1 look at the /var/log/srmv1/* logs (on the headnode). If the file was transferred using srm then the DN and client host will be recorded. This is probably all you need.
   1. For incoming files look for a put request.
   2. For outgoing files look for a get request.
   3. For deletes look for an advisoryDelete request.
2. If the SRM v2.2 interface was used the logfile is, naturally, /var/log/srmv2.2/*. (N.B. The original srmv2 daemon should be switched off.)
   1. For incoming files look for a PrepareToPut request.
   2. For outgoing files look for a PrepareToGet request.
   3. For deletes look for an Rm request.
   4. There are other SRM v2.2 operations with obvious meanings: Ls, MkDir, RmDir.
3. If you need more information, or the file was griftped, then grep for file name X in the dpns logfile. This should show the name server transaction, and will also show the grid certificate used. Importantly it will also show the TURL (look for the Cns_srv_{add,del,list}replica: line) and the pool onto which the file was placed.
4. If required, go to the disk server and look in /var/log/dpm-gsiftp/gridftp.log - you should find the gridftp transfer which placed the file onto the system. Again, the certificate DN is recorded. The dpm-gridftp.log file does not record the DN information - it just logs some parameters of the transfer, e.g size, streams. However, very_importantly, it records the host which transferred the file (under DEST=[IP.ADDRESS]). (There is host information also stored in the dpns logs, however this is normally the DPM admin host for file copies, which is doing the lookups for the user, not the user's client machine.)

Caveats and Notes

Direct GridFTP Operations

1. If the client did a globus-url-copy or an edg-gridftp-rm directly into DPM name space, then the receiving gridftp server is on the admin node, not the pool node (the server then uses RFIO to conduct the operation on the disk server hosting the file). Look at the admin node's /var/log/dpm-gsiftp/gridftp.log logs. Naturally, there's nothing in the srm logs in this case.
2. The gridftp server on a pool is also normal gridftp server, i.e., it can transfer files into the normal file space of your system (try globus-url-copy file:/some/file gsiftp://pool.node/tmp/SomeFile). If this happens then there's nothing logged in the nameserver. The only logs will be /var/log/dpm-gsiftp/ on the pool node.

90 Days Later...

As ever with Grid Security one requires logfiles to be kept for 90 days. The DPM RPMs now include the appropriate logrotate entries to do this.

Note there seems to be a minor bug in logging with gridftp - although the logfile is rotated every day, the daemon keeps loging to an old file for a week. Don't worry, everything is being logged, just not quite in the place you might expect.

Conclusions

The above proceedures should satisfy security logging requirements. We can find: time stamps, certificate used, source host, destination filename and operation performed.