Vulnerability Metrics – Which Matter and Which Don’t
The best way to share information about the risks associated with vulnerabilities is via quantifying these risks – i.e. metrics. The question is, which metrics? In order to communicate a cohesive vulnerability narrative, in this post we’ve grouped some of the more common metrics – with the aim of helping you leverage the most useful ones, and steer clear of those that are not.
Vulnerability Metrics That You Can Ignore
We’re diving straight to examples of metrics that should not capture your attention and take your eye off the 8 ball. As we go through these examples, you’ll quickly see that the common denominator here is using exclusively quantitative measurements; vulnerability management and remediation must include metrics that at once measure quantity and quality.
- How many vulnerabilities? – Many companies are feeling the squeeze of addressing such a massive number of vulnerabilities, and 74% of security teams are reported to feel overwhelmed by the vulnerability maintenance work. Yet the fact is that a simple count of vulnerabilities doesn’t tell you anything about either their severity, or more importantly, how likely they are to be exploited.
- Average CVSS scores in the network – Offering a standardized method for rating vulnerabilities, CVSS is a metric designed to assist organizations in prioritizing and coordinating their response by communicating the various properties of each vulnerability. However, the average CVSS score in the network does not reflect a given vulnerability’s objective technical severity, nor does it say anything about whether there are exploits in the wild. It also doesn’t reflect whether the vulnerability may impact anything that’s business critical in your unique IT ecosystem.
- Number of scans run – Organizations often use penetration testers (a.k.a. white-hat hackers) to run vulnerability scans, with the aim of finding vulnerabilities. These scans are no simple undertakings, and security teams often take pride in the number of scans they manage to run in a given timeframe. Yet the number of scans run is not a metric to be relied on. How much you scan doesn’t say anything about how vulnerable you may be to an exploit.
- Critical vs non-critical vulnerabilities in the system – These scores, usually released by software vendors about their own products, provide a general picture of vulnerabilities for a given software package. However, only your company can determine what is actually mission-critical. These scores don’t reflect the context of the vulnerability.
The Metrics that DO Matter
These are the metrics that can help improve your vulnerability and remediation program:
- Coverage – For business-critical systems or applications, coverage is an extremely relevant metric, which includes understanding the type of scanning. Is it, for example agent based, authenticated with a username and password, or is it unauthenticated altogether? This gives a qualitative view in culling the asset’s data – while also clarifying the scope of unknown risks that security teams may be unaware of.
- Vulnerability dwell time – The main focus here is the time that a known vulnerability lives in the customer’s environment. Often, the longer the vulnerability’s dwell time in the environment, the costlier the attack will be. The more critical the application, the more important this metric is. “If your board isn’t already asking for dwell time reporting…the questions are coming,” noted Rafal Los, VP of Solution Strategy at Armor, in a recent article. “If you don’t have those answers you should probably start evaluating the tools, the processes, and the staff you have to see how you’re going to come up with that data.”
- Average number of vulnerabilities per asset over time – The importance piece with this metric is ‘over time’. If you do not measure vulnerabilities over a continuous period of time, you will be incorrectly relying on scan results which may have not seen all the assets during a scan and reflect drops that in actuality are simply deviations. In other words, you need to be measuring the number of vulnerabilities continuously on the entire infrastructure in order to avoid mistakenly relying on scanning gaps.
- SLAs – How quickly an organization successfully (or unsuccessfully) remediates vulnerabilities and the speed at which they do has a tremendous impact on business objectives. By evaluating remediation results vs an SLA, you can evaluate how effective you’ve been in providing your business the time and resources it needs to reach its goals
The Key Question to Answer: Will You Be Affected?
Security professionals understand the importance of effectively and accurately communicating threat severity and overall risk to senior management. Yet the tendency to focus on traditional ‘critical’ vulnerability ratings can easily mislead as well as the tendency to play the numbers game instead of the quality game.
Here is the point we want to drive home: the metrics that we use must reflect the quality of our efforts; we cannot succumb solely to the quantity metric. By focusing on metrics that tell the story best for your organization’s unique IT environment, you can more effectively empower organizational decision makers to make your network – and your company overall – a safer space.