Limitations of Certification

There's been a lot of threads recently about infosec certification, education and training. I believe in training for infosec, I have trained several thousand people myself. Greater knowledge, professionalism and skills definitely help, but are not enough by themselves.

We saw in the case of the Great Recession and in Enron where the skilled, certified accounting and rating professions totally sold out and blessed bogus accounting practices and non-existent earning.

To take another example, lets take money management. There are legions of professional, certified MBAs that are happy to take your money and invest it on your behalf into mutual funds. However, around 80% of mutual funds *underperform* the S&P index. Not only that they charge fees to manage your money. So you pay extra to a professional so you can underperform a passive index! Mutual funds are where most people (and even, I suspect, most of the people reading this) put their money for their 401ks and IRAs, and since they are hiring a pro, they assume they are getting value, but they are definitely not.

"The general systems of money management today require people to pretend to do something they can't do and like something they don't. It's a funny business because on a net basis, the whole investment management business together gives no value added to all buyers combined. That's the way it has to work. Mutual funds charge two percent per year and then brokers switch people between funds, costing another three to four percentage points. The poor guy in the general public is getting a terrible product from the professionals. I think it's disgusting. It's much better to be part of a system that delivers value to the people who buy the product."

-Charlie Munger

So education and professionalism alone does not really solve much. The reasons it has not worked in the financial world (from a consumer point of view) are perverse incentives (the croupier gets paid, everyone else loses) plus behavioral issues (few people have the right temperament to buy when blood is running the street[1]).

Since in infosec we have both problems in spades - perverse incentives (of auditors, middle managers, companies, loss sufferers, stack vendors to name a few) and behavior (CYA plus grasping for silver bullets [2] plus not good at dealing with fear and complexity), I think professionalism only offers a partial solution until some of the structural issues are changed, and worse case is a professional, certified industry that churns out crummy products that subtract value from their customers [3]. 

I am not disagreeing with the fundamental point around need for certification and education, but just like anything else it has its limits. I would say of the two its easier to solve the incentive issue than the human behavioral issues.

One of the things that Petroski explores is that form follows failure:

"But whereas the shortcomings of an existing thing may be expressed in terms of a need for improvement, it is really want rather than need that drives the process of technological evolution. Thus we may need air and water, but generally we do not require air conditioning and ice water in any fundamental way. We may find food indispensable, but it is not necessary to eat it with a fork. Luxury, rather than necessity, is the mother of invention. Every artifact is somewhat wanting in its function, and this is what drives its evolution."

The above paragraph should be mandatory reading for every infosec pro that wants to deliver in the real world. And to a large extent education about security cannot increase our desire for a specific property, whether or not its "needed." 

At the end of the day, it looks to me that education can provide a partial solution, and the kind of education that is needed is engineering and integration:

"There may or may not be new breakthroughs in new security functions, but I don't think we really need many new functions, new primitives. If they come, we will use them. That's nice. If they don't, we can make do with what we have. What we really need, but are not likely to get is greater levels of assurance. That is sad because despite of the real need of assurance technology, we fail to use that which we already have in hand. We need to better use those assurance techniques that we do have" 

-Brian Snow

To me, the big problems in security are not about new mechanisms, its about identifying the ones you want to use (through threat models), locating them (through attack surface) and building them. In short, the biggest problem in infosec is integration. Education around security engineering for integration would be most welcome. For the other concerns around incentives and behavior, those solutions, if they come, are likely to come from outside of infosec.

1. When I went to holiday parties in winter 2008, when I brought up stocks people would get a deer in head light look and walk away diagonally across the room. When I went back to the same holiday parties in 2009 the same people (mainly doctors and lawyers as it turns out) were all excited to talk about investing! When the market goes up 70% people feel better about buying!

2. cf. RSA trade show floor

3. Ibid

Comments

[Iang]

Funnily enough, MBAs are taught from the get-go that funds are likely to underperform the market; it's a central tenet of the CAPM theory, which is foundational for finance theory.

But, the urge to think that we are better than average is high, amongst MBAs as well as doctors, dentists and others with spare cash.