I've had a few conversations in the past week that essentially followed the same path. Since there seems to be more than one person out there interested in return on investment (ROI) for software security, I thought I'd write about it here.
The conversation always starts with a question along these lines: "How do you build an argument for investing in software security?" When I hear it, it's usually coming from a security person who needs to explain to non-security people why they need money for software security tools. My answer comes in three parts:
Part 1: Don't try to justify the need for security using ROI
Trying to justify investment in security with an ROI argument is a losing proposition. I don't think anyone would actually go down this path if it weren't for the siren song of the risk equation, which, in its simplest form looks like this:
risk = probability * cost
It's an appealing formula because
- It's simple.
- If you compute cost in dollars, then you get to measure risk in dollars, and that's something you can talk to non-security people about.
The thinking goes like this: all you need to do is enumerate your risks, estimate some probabilities and costs, and now you know how much you should expect to lose. You could compute ROI for any countermeasure by showing how the cost of the countermeasure is small compared to the reduced risk that brings.
It's a nice idea, but in the security world the probabilities are small, and the costs are large. Taken together, that means that small errors in estimates of probability or cost mean big errors in estimated risk. How much would you like to estimate for "brand and reputation"? Not an easy problem.
What's worse is that, in order to sum over a set of risks, you have to assume that the probabilities involved are independent, and that's rarely true. The bottom line is that you can use the risk equation to explain why the security budget should be either zero or equal to the GDP of California, as you see fit.
It's just not time to talk about ROI until everyone agrees that software security is important. Need help getting people to that point? Start by handing out copies of Bruce Schneier's Secrets and Lies.
Once everyone agrees that software security is a worthy endeavor, you can begin to talk about how to make it cheap.
Part 2: It pays to find defects early.
This point is widely accepted as fact in the software engineering community, but it is worth tracking down a few references none the less. None of the three papers referenced below are focused specifically on security, but they all make a strong case for the increased cost associated with having to make changes to a deployed system. Does this apply to security defects? In spades.
- Software Defect Reduction Top 10 List (PDF) The cost of finding a defect after delivery is 100 times more expensive than finding it during the design/requirements phase. (Ratio is 5:1 rather than 100:1 for small non-critical projects.)
- Case Study: Finding Defects Earlier Yields Enormous Savings
Case study showing costs increasing the longer a defect exists. - What Does It Cost You To Fix A Defect? And Why Should You Care?
Compares cost to fix a defect at different stages for a variety of different companies.
The one additional point I should mention here is that many security defects cannot be found at design/requirements time because they are not present in the design; they are introduced when the code is written.
Part 3: People find more defects faster when they use tools.
I don't know of any published studies on the topic. If you do, please send them my way. Part of the problem with publishing studies about this stuff is that organizations with real security concerns have some legitimate reasons to be secretive about the effectiveness of their practices.
So instead of looking to the literature, I looked for data we have collected at Fortify based on doing large deployments of our static analysis tools. My sample size is small, and if I shared my sources I'd be strung up, but here is what I've learned:
- For a security team charged with doing source code reviews, using a source code analysis tool almost immediately doubles the amount of code the team can review. Some teams use this advantage to review more projects. Some use it to do reviews more often. Some use it to overcome the fact that they can't find enough qualified security engineers to fully staff up.
- Security teams use source code analysis to do a more complete job of finding bugs. Whereas a manual review process might entail doing some spot checks on pieces of the code, source code analysis tools apply the same level of rigor to every line of code. In theory you might hope that you could teach a development team "this is what path manipulation is and why it's a bad deal" and then have them go off and make the code secure, but in practice, you'll reduce the number of times that code has to be re-submitted for review if you point out all of the occurrences of each type of bug.
- When developers use source code analysis tools before the security team gets the code, the security team finds many fewer critical defects. In the past, many development teams weren't doing anything specific to prevent security problems, so it's no big surprise that when they get to find their own bugs, there's less work left for the security team to do. The result is that everyone is happier. The developers don't have the security team stepping in and screwing up their release schedule, and the security team feels less like they're being forced to teach a remedial version of Security 101 every day.
The bottom line is that you can make a good case for software security spending, but it has to be driven by the need for efficiency, not the need for security.
