White box, black box, and gray box vulnerability testing: What’s the difference and why does it matter?

by | Jan 19, 2018 | Software Vulnerabilities

Share This Story, Choose Your Platform

2017 saw a record number of security vulnerabilities, with more than 16,000 vulnerabilities reported by the end of Q3. That is more than in all of 2016 combined. While some organizations are addressing these application security risks quickly, others are failing to do so. A recent SANS survey found that 10% of enterprises are not testing their applications for security at all, with another 24% testing only once per year or less. A mere 12% of organizations are performing application security testing on a regular basis.  

Why are businesses neglecting application security testing? Rapid development cycles are a major contributing factor, with application security testing viewed as an obstacle that hinders development and deployment. A 2017 report on mobile and IoT application security testing found that up to 75% of application security professionals felt pressure to rush a release, at the expense of security.

The dichotomy between these statistics is alarming. Enterprises need to focus more attention on application security testing, even if it means slowing down. Fixing vulnerabilities after application deployment costs businesses more time and resources, not to mention negative publicity, lost revenue, lost customer data and trust, a tarnished reputation, and even possible legal action should those vulnerabilities get exploited.

Knowing the different approaches that can be taken to application security testing can help you develop a strategy that will deliver the biggest ROI, balancing improvement in security with the extra time and resources required.

There are three flavors of application security testing, generally referred to as black box security testing, white box security testing, and gray box security testing. Understanding the differences between each type will help you form a strong plan for application security testing that will decrease your chances of being exposed to potential threats and maximize resources, funds, and time.

Black box security testing

Black box security testing mimics the actions of a hacker, attacking the application from the outside. In doing so, it tests the functionality of the application. It is performed based on the assumption that the attacker has no knowledge of the inner workings of the application: it treats the application like a “black box” whose contents you can’t see, just as a hacker can’t see into your application.

Black box security testing can mimic hackers of various skill levels, such as:

  • Script kiddies (skids) – Untrained individuals (“kiddies”) who use programs or scripts developed by other people to attack an application, and scan for known vulnerabilities.
  • Mid-level hackers – Individuals who possess more knowledge about development, systems, and networking, and can therefore wreak more havoc.
  • Elite hackers – Individuals who are more likely to be sponsored by a nation-state. They have more time and resources to devote to hacking.

This testing looks for known vulnerabilities, weak access controls, and weak defense mechanisms in the application. To be thorough, it is important to test for all variables and scenarios that may come into play.

The advantages of black box security testing are:

  • This testing can be done faster than white box vulnerability testing because it is more limited in scope, only attacking from outside the application.
  • It can be less costly, although this does depend on the specific application and the scope of testing.

Disadvantages include:

  • It is not comprehensive, as it is only testing from the outside.
  • Test cases can be difficult to design without clear guidance on what areas and/or vulnerabilities to specifically test.
  • If you want to be thorough, it can take a lot of time to try different scenarios and use cases. Hackers often have more time on their hands than you do, so you can’t always anticipate everything they might try.

White box security testing

White box security testing is performed based on having all knowledge of the application, testing the application’s internal workings. It is frequently performed with access to the full source code, so source code scans and reviews are often included as part of the testing process. Full access internally (locally) to the application is provided, including log-in credentials and full authentication. White box security testing should be done before an application is released into production, so vulnerabilities can be identified and corrected prior to deployment.

White box vulnerability testing has several advantages:

  • Finds potential vulnerabilities everywhere, from the source code to the application architecture and design
  • Provides a thorough testing of the application’s security
  • Can be started early in the development process

The disadvantages of this testing are:

  • Any potential vulnerabilities that are found need to be further tested via black box testing to see if they are actually exploitable, which requires additional time. Taking this step helps you prioritize your remediation work.
  • This type of testing is lengthier, slowing down the development process and conflicting with rapid development lifecycles.
  • It is more expensive due to the time and highly-skilled resources required. White box security testing requires knowledge of what makes applications insecure, ability to read code, and to see security exposures in software architecture.

Gray box security testing

As its name would imply, gray box security testing is a combination of white box and black box testing. It is the most ideal approach to take when you need to balance time, cost, and impact.

Gray box security testing can be employed in various ways, depending on the application and the testing goals. It may, for example, be performed by testing the exploitability of potential vulnerabilities identified in white box testing.

This testing is performed on the basis that a hacker has some limited knowledge of the internal workings of the application, including but not limited to the following scenarios:

  • The hacker has a high-level understanding of the application’s dataflow and architecture.
  • The hacker may have access to parts of the source code or compiled binaries, and may know the languages and specific technologies the application is using.
  • The hacker may have user or admin accounts with which he/she can login.

These are not unreasonable assumptions: seasoned attackers have ways of discovering things like this.

Scenarios are then created to test specific areas of higher priority or concern, mimicking a real hacker who may have partial knowledge of the application’s inner workings. In terms of advantages, gray box security testing:

  • Attacks certain parts of the application in a more targeted manner
  • Delivers benefits of both black box and white box testing
  • Takes less time than white box security testing, making it a better balance of time, effort, and cost for application security testing
  • Delivers the best ROI for time and resources
  • Provides a better representation of a determined hacker who is likely to have at least some knowledge of the internal workings of the application, and more time to focus on an attack, possibly researching the company, developers, and the application.

There are, of course, some disadvantages to gray box security testing:

  • It will not cover all aspects of the application, as that would be too time-consuming.
  • It lacks the full coverage of the source code that white box testing provides.
  • Test cases can be difficult to design.

Why does all of this matter?

It is important to understand the basic differences between the three types of application security testing, and what each approach does to improve application security. Having multiple options helps in formulating an approach that best fits your project’s budget and timeline.

In an ideal world, businesses will employ a blended approach, employing some of each type of testing based on the specific application and the resources available.

To assist in crafting an application security testing strategy, keep in mind the best uses for each type of testing:

  • Black box security testing – Emulates short-term hack attempts of script “kiddies” (attackers without a lot of time, resources, or skills).
  • White box security testing – Ensures your developers are not introducing vulnerabilities into the application
  • Gray box security testing
    • Simulates attacks from hackers with more knowledge and skill
    • Simulates attacks from insider threats
    • Tests potential vulnerabilities found in white box testing

Understanding all three flavors and leveraging the strengths of each helps you create a more resilient application security profile. Taking the time up front to develop an application security testing plan, and incorporating it into the development process, can keep your application from contributing to the rise of security vulnerabilities in today’s market.

Share This Story, Choose Your Platform