Ned over at the Barking Seal uses the recent Macondo example to illustrate what Richard Bejtlich calls Building Visibility In:
Steven Newman, the CEO of Transocean, said during a recent senate hearing, “There is some delay in the replication of our data, so our operational data, our sequence of events ends at 3 o’clock in the afternoon on the 20th. And so the VMS system, along with the logs of the VMS system, would have gone down with the vessel.” The blowout and massive explosion happened at 10, taking eleven lives and seven hours of VMS data to the bottom of the ocean. Representative Bruce Braley from Iowa followed up with “So you have no mirrored backup data device so that that information is recorded at some other location than on the rig itself?”. Newman replied, “We do not have real-time off-rig monitoring of what’s going on on the vessel”.
In the
class I teach for developers to build Audit Logging into their applications, we build on the good work that's been driven out of PCI DSS - namely the spec created demand for best of breed audit logging tools - a competitive marketplace. So now there are legions of tools in the space, at a relatively decent cost. People in security love to slag off PCI, but you know what - if you went back 8 years pre PCI, you would not find a market for audit logging tools, it would have been two guys in a basement in West Texas. Now there's a nice niche market, and real tools.
But as always, the tools only go so far and so its necessary to build the audit loggers into the code so that you can make the audit log manager useful. It would be nice if you had consistent event models, types and reporting as well. What we spend a lot of time on in training is - placement of the audit logger.
Location, Location, Location
Think about about a three tier architecture. Now think about an attack - let's SQL injection. What does SQL Injection look like at the presentation tier? Probably not much HTTP request, possibly with some funky characters, but its likely a "nothing to report" situation. (Yes I know I am glossing over the possibility of the input validator catching it up stream, but bear with me) A little lower in the middle tier, mapping to business objects and applying business rules, it might trip a wire, but even there maybe not. Then at the data tier, formatting the query the logger may finally catch an exception or see something malformed and be able to 1) identify it as such and 2) report it.
Then we do other exercises trying to audit log CSRF, XSS and other areas. In each case you will likely find that where in the code you choose to locate your audit logger is just as important as the events you are looking to gather.
Richard Bejtlich mentioned the topic of monitoring in a post, where he was posed a simple question (always the hardest kind) by a CISO Can you tell me when something bad happens to any of my 100 servers?"
Its worth reading Richard's whole post, but the part I want to include here is this part of his answer:
- Can we collect host and application logs?
- Do we have instrumentation in place to collect data for the servers in question?
- Are the logs standard? Nonstandard? Obscure? Binary?
- Are the logs complete? Useful?
- What volume of data do we need to analyze?
- What retention period do we have for this data?
- What laws, regulations, or other restrictions affect collecting and analyzing this data?
Its a good list, and I would add a few more -
- Where in the stack is the instrumentation?
- What is the event model? What event types are visible?
- How are the events, requesters and providers correlated?
- What event payload types are used?
- How are transactions and sessions handled?
- How are the identity information and authorities handled?
- What service interfaces, databases are monitored?
- Can you correlate identity providers, relying parties that vouch for transactions?
One of the smartest clients I have asked me to put together the Audit Logging training class (rule 1 in consulting - listen to your clients, especially the smart ones). He had been down the PCI road, had tooling and a basic event model, but needed concrete ideas, examples, patterns and practices on how hands on developers could integrate the audit logging beast into their apps. I put the class together and was leery that even though it seemed important topic, that anyone else would care, but it turned out to be a popular class.
One thing I have learned from reading Richard Bejtlich and studying real world security (ever lose a credit card?) responses, is that access control can only get you so far. When the stuff hits the fan its all about monitoring and response. Its hard to get the mindset to build security into systems up front and harder still to get people to think about building visibility in, but that's the best shot at mapping auditable events to the customers, users, identities, apps and data that you care about. To those about to audit log, we salute you.