Template

Service: GridPP nameserver

Location: Manchester

Incident Date: 11th June 2008

Impacted: Anyone using a .gridpp.ac.uk domain name

Timeline

First seen: Seen by: Several GridPP website users - email query

Time to first response (communications): Time to first intervention:

Incident duration: 4 hours

Related URL: None

Incident summary:

What seems to have happened is that the default named.conf was reinstalled by the upgrade of BIND to deal with the recently publicised DNS cache poisoning vulnerabilities (patches have been released for all DNS implementations in the last couple of weeks.)

This had the effect of resetting the authoritative GridPP nameserver to be a cache-only nameserver. This meant that queries continued to work initially from the secondary name servers and from local caching nameservers at other sites, and that our own automatic checking of the name service continued to work (it _was_ functioning as a caching, recursive name server correctly - just not in the authoritative way from files as required.)

Several people spotted problems during the morning of the 11th and we were able to reset the configuration correctly by late morning.

Future mitigation:

To avoid this kind of problem in the future, we have added the line OPTIONS='-c /etc/named.gpp.conf' to the file /etc/sysconfig/named which is not modified by any RPM upgrades, and which forces the use of our own, uniquely named, config file for the authoritative server. Unfortunately, this is one of those configuration problems that redundancy doesn't help us with.

Related issues:

DNS cache poisoning vulnerabilities now being widely discussed and prompted the upgrade that caused the problem:

http://www.kb.cert.org/vuls/id/800113. Services that are not using certificates would be most vulnerable, and that, for instance, large numbers of near identical worker nodes all doing YUM updates at about the same time each day and not checking GPG signatures, might make an attractive target. We've prompted some steps to guard against this class of attacks in the OSCT, but The Correct Solution to some of these depend on RPM package signing which isn't being done in ETICS etc. As an interim, it's also possible for RPMs to be signed further downstream, with a GPG key specific to the YUM repository (eg with a script that just loops over the binary RPMs and runs rpm --addsign on each existing binary RPM in turn.) That's an option that people running local RPM repositories could consider.

Report date: 22nd June 2008