Models and Reality

Richard Bejtlich has a post talking about the risks of when models and reality diverge - 

Did we not just suffer a global recession exacerbated by clowns who thought they could model risk "with a high degree of accuracy"?

This of course is the financial crisis in a nutshell, we've had plenty of real estate crisis in the past, but they never lead to 10% unemployment and cratering the whole financial system. This was the first time that the geniuses had built mortgage backed security models on top of real estate (key underlying assumption - house prices never go down in the US), add in some false precision and perverse incentives, stir - and voila.

Its not to say models are bad, just when they're confused with reality. We don't need to look any further than the world of college sports. College football's championship is based solely on a model where you calculate strength of schedules, margins of victory, strength of opponents, and so on, but what you get is an unholy mess that leads to someone getting hosed every year, and endless legitimate arguments. Its all model. (Plus a zillion lame bowl games - the Swiffer Sweeper bowl?)

College basketball otoh starts with a model - picking top 64 teams (pretty large sample set, probably fair to say if you are not in the top 64 then you don't deserve to be national champ), but then the model is stress tested and determines the champ by who actually wins the game (note - Kansas looks good this year). Many times (most?) the winner is not the #1 ranked team going in, is this bad? Not at all, the model gives a logical starting point but its not an end in and of itself.

Comments

[Alex]

Did you see where Madden's simulations have been right on for the past 7 years? Even this year where the Saints weren't the "obvious" favorite?

Also, there are many retrospective baseball posts about PECOTA, CHONE, ZiPS and the like. Which is also very interesting.

Most notably, I very much enjoy the comparisons of how much better these models do than baseball analysts who don't use rigor.

[Dtauzell]

Many people knew the "model" was bad. Many brokers knew they were making bad loans that would likely never be paid back. But ... everybody else was doing it. Make bad loans, or be crushed by the competition. At least, that was the mentality driving the low level brokers who then sold everything back up the chain.

If you have time, this episode of "This American Life" talks to several of those brokers: http://www.thisamericanlife.org/Radio_Episode.aspx?episode=390.