The VERT Daily Post has a post on the Paradox of Host-based Security Measures. The post describes an important architectural consideration:
A PROBLEM CANNOT BE PROPERLY RESOLVED AT THE SAME LOGICAL LEVEL IT WAS CREATED.
Kurt Godel, Albert Einstein, John Boyd, Gregory Bateson, and many other great thinkers have said in one way or another that certain problems cannot be resolved at the same level they originate. If you apply this to our client-side problem, you will also see that there is a similar paradox. Once the client has been compromised and the adversary now has complete control over that computing base, ALL security related detection, auditing, and any type of safeguard must be also not trusted since the underlaying computing base is not to be trusted. I call it the Paradox Host-Based Security Measures: How can the computer detect it has been compromised when as soon as it has been compromised the ability to detect it has been compromised can no longer be trusted?
I can't leave you without an answer. Although a brief one, the answer is vitualization. The security controls for the OS must live at a higher logical level. Security measures must exist outside of the world in which it is providing services. We have to return to the roots of "The Theory of Logical Types" which says that no class can be a member of itself; that a class of classes cannot be one of the classes which are its members. I can go on and on about this but you get the idea.
I think virtualization is one solution that can be used, but first the architect should use separation of concerns and think terms of a layered system. Use logical and physical separation to enforce policy. Mitigate risk by spreading protection, detection, and response mechanisms across layers and domains. Log files are a great example. If you rely on log files to detect attacks, and then store them on the same system that you are trying to protect, how can you verify that they are correct in the event of an attack? Piping logs out to other servers, using integrity services based on keys and functions in other logical and physical zones my provide more resiliency.
Update: There are many ways to use indirection to increase security, I addressed smart cards in this context here.
I also disagree that the security controls must "live" at a higher logical level. There are always ways to cross security levels, even down to the point where hardware can be damaged.
The layered protections approach isn't perfect, but it's the best we have. Properly designed, each of the layers should protect the weak spots in the others.
Posted by: joat | November 16, 2005 at 08:26 PM