Black box testing is software testing. It's time for black box testing to start applying some of the same discipline used in the software testing world.
Most organizations admit they don't know how thorough their black box testing is, but most people assume their tests are comprehensive and effective. More than half of our survey participants believe their testing covered at least 60% of their applications, but the data suggest they're not doing as well as they think.
We looked at two of the leading black box tools as they probed five small web applications. Our data suggests that even for small applications, people using black box tools are not even getting 30% coverage. (Looking at larger apps generally makes the situation even worse.) Applying manual effort to customize the tools can significantly improve their effectiveness, but the tests still failed to achieve coverage numbers greater than 50%. Conclusion: black box tools aren't a good a substitute for good testing techniques.
In the past, people have used black box testing as a way to attract attention to the security problem. For that purpose, coverage doesn't matter: all you have to do is find some security holes. Software security is starting to mature a little bit, and now we need to start looking at whether or not our security testing is effective. We can borrow a lot from software testing. It's time for security testing to start growing up!
