As business guru Peter Drucker said, “If you can’t measure it, you can’t improve it.” This quote rings especially true when it comes to application security.
Many businesses are finally starting to give AppSec the attention it deserves, but the only way to know if your approach is effective is with application security metrics. There are a number of vulnerability management metrics you can record and analyze; choosing the right ones for your business is important.
It’s the only way for CISOs and other executives to manage and monitor the effectiveness of the program and make it more efficient.
Why application security metrics matter
Perhaps the most important part of any application security metrics plan is selecting the right metrics to capture. It’s important to make sure the metrics you choose align with your goals and objectives.
For example, many CISOs want to see a reduction in the number of new vulnerabilities introduced to an application (making this an important metric to record). A decrease in this number over time indicates an improvement in secure coding practices within the team and contributes to a stronger defense against an attack.
It is also important for your development and security teams to have access to metrics that help them perform their jobs better. This may include data relating to the types of threats being found, how they are discovered, and how long it takes to resolve them.
While each organization has unique goals for application security and vulnerability management, there are a few best practices every CISO should keep in mind when it comes to application security metrics.
- Don’t measure everything. It can be tempting to measure everything you can in relation to application security, but this is not going to help you improve the program. Too much data can be just as bad as too little.
- Identify the goal of the program. Who will be looking at security metrics? What type of data will they want to see? Use the answers to these questions to identify the right metrics to measure.
- Make it easy to digest. Whatever metrics you decide to capture, viewing and sharing results should be simple. A concise report that makes it easy to see strengths, weaknesses, and overall improvement in application security is the best way to use vulnerability metrics to your advantage. Visual displays are often easier to interpret.
- Include appropriate benchmark data. As you gather application security metrics, you should also include data that shows how your organization is performing over time. You may want to show internal improvement as compared to an industry average, to better demonstrate how your team is doing compared to others in the field. However you choose to present metrics, it is important to show how they relate to the program’s objectives and the progress made over time.
- Make sure all of your applications are covered. Once you design and implement AppSec metrics, make sure you apply it across all applications in the enterprise, including legacy applications. We recommend making a list of all applications. The list should be revisited on a regular basis to make sure it is current.
Keep in mind that there are both direct and indirect metrics that you can measure. Direct metrics measure the security of the software application itself and include such items as the total number of vulnerabilities identified. Indirect metrics look beyond the application and instead focus on tools, people, and processes. An example would be the average time it takes to correct known issues. A combination of direct and indirect metrics yields the most comprehensive picture of how your AppSec program is performing.
5 application vulnerability metrics that will strengthen your team’s effectiveness
Number of application vulnerabilities and the severity
This is one of the most important application security metrics for your business. It is always critical to know how many vulnerabilities exist within an application, and—even more importantly—just how severe each weakness is. Severity is based on the impact the threat can have on the application (and the business) and how likely it is to occur.
One of the best ways to identify the biggest threats is to cross-reference results from Static Application Security Testing (SAST) tools and Dynamic Application Security Testing (DAST) tools. SAST tools identify potential vulnerabilities, while DAST tools tell you which of these possible threats are actually exploitable. You can read more about these tools and what they offer here.
When you combine the results from these two types of tools via Hybrid Analysis Mapping (HAM), you get a list of the application vulnerabilities that pose the biggest risk. This allows your team to prioritize which issues should be addressed first.
Average days to resolution
The longer it takes to remediate an issue, the more time attackers have to exploit it. AppSec teams and executives need data on how long it typically takes to address issues after they are discovered.
It is even better if this data is broken down by severity, so you can see how long it takes to remediate critical issues versus those that fall into the categories of high, medium, and low severity. This information allows managers to pinpoint inefficiencies in the remediation process. Finding and fixing bottlenecks improves the overall security of your program.
Number of new vulnerabilities
New releases and updates are a constant with Agile development. It’s important to know how many new threats are introduced when a new release is deployed.
This metric helps CISOs monitor risk and also helps managers assess how well developers are performing when it comes to writing secure code. You may find your team could benefit from additional training or guidance on secure coding practices or the importance of application security in the development process.
How vulnerabilities are identified
It can be helpful to know which tools and processes are finding vulnerabilities. Data pertaining to the number of threats found by SAST tools, DAST tools, threat modeling, and other detection methods will give you a sense of how comprehensive your application security testing is and if there are holes that need to be addressed.
The types of vulnerabilities identified
It is important to know the most common types of application vulnerabilities discovered. For example, knowing whether SQL Injection is your most prevalent threat helps you properly allocate resources to improve remediation times.
Tools that assist with vulnerability assessment metrics
Capturing, analyzing, and displaying application security metrics require the right tools. It should be easy to view results and share them with stakeholders so everyone can see the progress being made.
The dashboard presents all of your metrics in one place, so you can easily see the status of your project’s security. Interactive images let security team members and executives dig deeper into areas of interest. These visual displays encourage communication and transparency across the organization. It is easier to coordinate remediation efforts and uncover issues and trends before they become larger security threats.
In addition to the security metrics mentioned above, the dashboard offers other metrics to help your team improve application security.
- A risk score is assigned to each project so you can quickly assess the overall quality. The score is based on the number of vulnerabilities found in both custom code and third-party components. You can also see whether the score has improved or worsened over the past week. This helps keep your team on track for secure coding practices within the Agile development process.
- A visual display of open findings uses colored dots in a waffle chart so you can quickly see the number of open vulnerabilities within each severity level and the relative age of each finding. This chart is an interactive snapshot of how well the triage process is going on a given project.
- Analysis frequency tells you how many analyses were run on a project over the past week, month, and three months, and how long each took to complete. You can also see how many tools were used in an analysis. Projects using hybrid analysis can also see the percentage of custom code covered by dynamic analysis.
- A graph and table that displays the number of newly created issues versus resolved ones gives you a clear picture as to whether a project is moving in the right direction with security.
These additional metrics provide a complete picture of security for the entire team—from the CISO all the way down to individual coders.
When the right metrics are captured, analyzed, and presented in a clear manner, all stakeholders can benefit from application security metrics. CISOs and managers can look for inefficiencies and prove the value of the program, while developers and security team members can see how they are progressing over time toward the goal of secure coding.