Do not discount what matters

From the FT today (emphasis added):

When you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.” This observation scores high on the list of silly remarks made by clever men, to which its author, Lord Kelvin, made several sterling contributions. The knowledge possessed by Plato and Shakespeare, Austen and Darwin was neither meagre nor unsatisfactory.

Bogus quantification attempts to compress complex problems and analyses into single observations. The annual report of the United Nations Human Development Programme is an entirely admirable endeavour. The agency focuses on three broad dimensions of development – health, education and material standard of living – and reports on progress towards these goals among UN members.

Then the curse of Kelvin strikes. Health is measured easily enough. Take life expectancy at birth, subtract 25 years, then divide by 60. This score has a one-third weight in the final total. Educational development is measured as a combination of attainment – adult literacy – and opportunity – enrolment. Material standard of living is measured by gross domestic product per head at purchasing power parity. But differences in GDP are much larger than differences in life expectancy, so you prevent this measure from swamping the whole calculation by using the logarithm of GDP rather than the level. Finally, you give equal weight to each of the three components and come up with your ratings. Iceland, ironically, comes top, and Sierra Leone bottom.

If you must undertake an exercise of this kind, then what the UNDP does is sensible enough. But suppose I thought that more weight should be given to health and less to per capita income? Or that a measure of human development should include freedom of speech, or democracy, or religious tolerance, or religious observance? The problem is not just that reasonable people would have different views on these things, but that there are no objective criteria by which such disagreement could ever be conducted, far less resolved.

And there we have the basic problem with security metrics as well. How to weight software, data, host, and network when they are all intertwingled? 

The first problem is lots of stuff that matters doesn't get counted, the second problem is that some things that are counted get overweighted because you can count them. 

Comments

[Swatne]

Jeez Gunnar - Now I'll have to take the Lord Kelvin quote out of my security strategy! :) Swatne.

[Iang]

Like, having key sizes greater than 1024 bits is more important than a protocol that uses a key in the right way? Because it has a number? Well, I never :)