Each SAST tool only discovers about 14% of the vulnerabilities in your code

by | Apr 11, 2019 | AppSec Classroom, Tool Studies

Share This Story, Choose Your Platform

An essential element of the application development process is scanning the software to find potential vulnerabilities. Static Application Security Testing tools are notorious for returning lots of results (often thousands, even for relatively small applications), which can overwhelm a developer. But no matter how they feel about the results, software developers must understand that by running only one application security testing tool—even the best on the market—they are missing most of the weaknesses in their code.

One tool only covers the tip of the iceberg. There could be thousands of flaws that a single analysis tool does not find, which may result in serious weaknesses being missed, potentially putting valuable data at risk of being exploited.

One SAST tool only provides 14 percent coverage

According to a study done by the National Security Agency’s (NSA) Center for Assured Software (CAS), the average static code analysis tool covers 8 of the 13 weakness classes (e.g. buffer handling, file handling, initialization and shutdown, and number handling), which is 61.5 percent. This study also found that the average tool covers only 22 percent of the flaws in each of the 13 weakness classes. If the percentage of the flaws is multiplied by the percentage of weakness classes covered, the total coverage of the average tool is only 14 percent.  

This should be an eye-opening statistic for many software developers who have assumed that their vulnerability scanners cover a much larger area. Missing more than 80 percent of the weaknesses in the application code should not be acceptable for any organization.

Additional research supports these findings. One study evaluated three commercial static code analysis tools and determined they performed close to or worse than average at detecting application vulnerabilities. Another evaluation of nine tools found an average recall of 0.527 and precision of 0.7, while the results of yet another evaluation determined the highest recall of any one tool as a standalone to be 18.4 percent.

In addition to discovering that each of the analysis tools failed to report a significant portion of the flaws studied, the NSA CAS and subsequent studies found that the tools perform differently on various languages and weakness classes.  

SAST weaknesses

Given the results of these studies, we think it is important to have a firm understanding of the general weaknesses of Static Application Security Testing tools. This knowledge can be used to make sure you choose the right combination of SAST tools for application security testing.

SAST weaknesses include an inability to:

  • Interpret the results—you still need to determine if a finding is a true positive and what the impact on your application is.
  • Recognize flaws in the application design or architecture.
  • Identify new types of bugs. This may be possible with some customization, but this requires some extra work on the part of your team.
  • Certify that the application code is free of defects. In other words, just because a given SAST tool does not find a problem, it does not mean one does not exist.

This is why it is so important to take a combined approach and use more than one SAST tool. In fact, the studies found that complementary tools can be combined to achieve better results.

Not only will two or more tools cover a larger area of your code, the fact that each tool specializes in different weakness classes and different languages eliminates much of the overlap among the tools. When there is an overlap, that is a strong indicator that the identified flaws are not false positives, and developers can focus on ensuring those weaknesses are fixed.

It is also critical to not rely solely on SAST tools. Use other types of tools, such as Dynamic Application Security Testing (DAST) tools and Interactive Application Security Testing (IAST) tools, along with manual code review for comprehensive application security.

How to manage multiple application security testing tools

Using multiple tools does come with its challenges, namely in the additional time required to set up and run the tools, compare the results, and the cost required to add more tools. Comparing the results can be painstaking, as each tool produces a set of weaknesses with its own naming conventions and severity ratings.

This is one of the reasons we developed Code Dx: to identify the overlap in the various tools used. Whether using commercial scanners, open-source vulnerability tools, or a combination of both, Code Dx shows the results of each and identifies the vulnerabilities that were found by each tool (and were found by more than one).

The Code Dx application vulnerability manager correlates and normalizes the results from commercial and open-source tools to deliver a consolidated set of results that provides greater coverage of potential vulnerabilities in the source code and a better assessment of an organization’s overall enterprise risk.

Deduplication, remediation management, reporting, and compliance checks are handled in one streamlined tool. Workflow integration options allow your developers to stay in their preferred environment—Eclipse, Jira, Jenkins, and others—while addressing application vulnerability issues.

One of the most valuable benefits of this tool is Application Vulnerability Correlation (AVC), also called Hybrid Analysis. This refers to the combination of SAST results (which identify potential vulnerabilities) with DAST results (which identify which threats are actually exploitable). This allows you to determine which threats exist in your code and can be exploited by an outside attacker, so you can address them first.

Don’t settle for 14 percent coverage; this leaves your application and your reputation exposed to serious threats. A well-rounded approach that involves multiple SAST, DAST, and other application security testing tools yields the most secure application, the benefits of which will trickle down to your bottom line.

Fortunately, using multiple tools is easier than it seems. Code Dx streamlines the application security testing process. Your team can use all of the tools necessary to obtain comprehensive coverage of your application’s code without the headache of managing multiple tools and the disjointed results they deliver.

Share This Story, Choose Your Platform