The DBIR is a net plus for the industry. When Wade Baker and team first released it, the infosec industry was dominated by BS hallway conversations "if you knew what I knew", this still happens obviously, but there is a lot more out in the open.
There have been very few events that I have seen that have moved the whole industry forward, the original release of the DBIR was one of those events. It was a great move forward the industry as a whole, Wade and team deserve full credit for that and none of my critiques should color that. We are now in a period of something resembling transparency with companies disclosing far more than they did in the past, DBIR was the pointy tip of the spear to getting us to this point. What follows is hopefully constructive criticism of a useful asset in our industry and really my attempt at a brainstorm of ways it could evolve to be more useful
Sharing the dataset was and is a huge net plus for the industry. The data in the report now has been sliced and diced many ways. I have no doubt that the people doing the slicing and dicing have good intentions, but much of the work falls into the category of trying too hard and has the knock on effect of delivering higher confidence with not necessarily higher accuracy. Any author or consumer needs to keep in mind the distinction between precision and accuracy, otherwise you risk leading people in potentially the wrong direction. After all, the only value of reports is as a decision making tool.
The report uses data with an authoritative tone, but there are many undocumented assumptions baked into the presentation throughout. So what is presented as objective is the result of a very subjective and not communicated modeling process. So readers should be careful here, as always beware of geeks bearing models.
While the overall report is presented as authoritative and comprehensive, it needs further refinement and scrutiny to be useful for drawing conclusions. The report is biased based on the sales team coverage (and that of their partners) so if they have a great Real Estate sales guy and a poor one in Utilities, you get heavy skew. Further some important, regulated industries simply do not follow the same reporting constructs and so DBIR is blind to whole segments.
Deployment context matters! To take one random example, in frequency Mining appears to have a huge problem with Cyberespionage (whatever that is), but then you realize there are no POS in Mining (unless you count the Gold ATM in London, I guess?) or web apps, so by eliminating two categories you get a heavier percent weighted in the others. They cannot correct for their sales team, but this one should be weighted to correct for absence of deployments.
I am sure the authors are trying to be helpful, but the categorizations may be too fine and the trebling of efforts on frequency may miss what is really important and actionable - risk and countermeasures.
Mixing subjective and objective data is a must avoid. As Charlie Munger says, if you mix raisins with turds, you've got turds.The report aligns a set of countermeasures with the breach findings. The report's intentions are top notch, what really matters for most is countermeasures going forward. However, its not clear that the countermeasures would always be the best way for companies to address some of the problems. I applaud the authors for including prescriptive guidance, but there should be a disclaimer of sorts that there is not data that rules these in as blocking factors.
My Thoughts on How to Improve
* Solving problems - aligning countermeasures to issues would be the single biggest improvement, but this is likely very hard to do. Still its a worthy long term, North Star goal.
* Establish a hard separation throughout of what is data driven versus not. My favorite example here is the famous Value Line reports. These reports distill a tremendous amount data into one page to use for analysis. Click the link to see their free report on any of the Dow 30. Value Line draws a bright line between historical data, analyst opinion and future forecasts. The rule of thumb here is to ignore the latter two categories.
* Warning label - data is no good if you cannot describe the boundaries of it. What it's not telling you. Charles Wheelan in Naked Economics makes a great point that any report should accompanied with a warning label about what isn't represented.
* Risk - one good warning to include is that whatever your view of the likelihood shown in the report, risk is mainly absent. The impact of an account is not remotely equal across companies. A breach with 35 stolen account with limited credentials maybe amounts to a couple hundred dollars. A breach that hits 35 accounts in a clinical trial could be worth billions of dollars. LIkewise the report has tried for years to draw distinctions in the insider/outsider attack landscape. Using frequency here is a pretty weak measure, there should be lower frequency for insider attacks (or you have a massive HR problem), but the downside, tail risk for rogue insiders is not remotely comparable either.
This probably reads more negative than my feelings about the work. None of the critiques should be read as negative in the main, DBIR is a worthy endeavor run by high quality people. What I am hoping to get to in this piece is that I would like to see DBIR become a touchstone that can be used to inform better strategy and tactics. Its already delivering a lot of value, I hope to see it continue to evolve in a way that it can be used as part of a decision making process to deliver better code and protect our users.