Black Box vs White Box. You are doing it wrong.

Wednesday, October 28, 2009

A longstanding debate in Web application security, heck all of application security, is which software testing methodology is the best -- that is -- the best at finding the most vulnerabilities. Is it black box (aka: vulnerability assessment, dynamic testing, run-time analysis) or white box (aka: source code review, static analysis)? Some advocate that a combination of the two will yield the most comprehensive results. Indeed, they could be right. Closely tied into the discussion is the resource (time, money, skill) investment required, because getting the most security bang for the buck is obviously very important.

In my opinion, choosing between application security testing methodologies based upon a vulnerabilities-per-dollar metric is a mistake. They are not substitutes for each other, especially in website security. The reasons for choosing one particular testing methodology over the other are very different. Black and white box testing measure very different things. Identifying vulnerabilities should be considered a byproduct of the exercise, not the goal. When testing is properly conducted, the lack or reduction of discovered vulnerabilities demonstrates improvement of the organization, not the diminished value of the prescribed testing process.

If you reached zero vulnerabilities (unlikely), would it be a good idea to stop testing? Of course not.

Black box vulnerability assessments measure the hackability of a website given an attacker with a certain amount of resources, skill, and scope. We know that bad guys will attack essentially all publicly facing websites at some point in time, so it makes sense for us to learn about the defects before they do. As such, black box vulnerability assessments are best defined as an outcome based metric for measuring the security of a system with all security safeguards in place.

White box source code reviews, on the other hand, measure and/or help reduce the number of security defects in an application resulting from the current software development life-cycle. In the immortal words of Michael Howard regarding Microsoft’s SDL mantra, “Reduce the number of vulnerabilities and reduce the severity of the bugs you miss.” Software has bugs, and that will continue to be the case. Therefore it is best to minimize them to the extent we can in effort to increase software assurance.

Taking a step back, you might reasonably select a particular product/service using vulns-per-dollar as one of the criteria, but again, not the testing methodology itself. Just as you wouldn’t compare the value of network pen-testing against patch management, firewalls against IPS, and so on. Understanding first what you want to measure should be the guide to testing methodology selection.

WhiteHat Security is a leading provider of website security services.

Originally Posted by Jeremiah Grossman here:
http://jeremiahgrossman.blogspot.com/2009/10/black-box-vs-white-box-you-are-doing-it.html

Possibly Related Articles:
10365
Vulnerabilities Webappsec->General
Consulting Information Security
Pen Testing Methodologies
Post Rating I Like this!