The new InfoSec efficiency metric
Measuring InfoSec efficiency has always been problematic, both in terms of technology and terminology. For example, reducing false positives seems like a great starting point. But after more thought, isn’t the real issue detecting false negatives and increasing true positives?
Introducing “Pattern Detection Ratio,” (PDR) the most efficient means of measuring your company’s efficacy in detecting attacks. PDR can be used with analyst-driven and machine learning anomaly detection solutions, yielding a single stat that can be tracked over time or used as a comparator.
PDR compares the recall (or true positive rate) of an InfoSec system against the investigation cost at various cost settings. In simple terms, it measures “how many attacks did I detect out of the total number of attacks, vs. how many alerts did my system generate to detect those attacks?” This analysis is directly related to the cost/benefit analysis done in decision making and provides a way to evaluate solutions independently from the investigation budget or the class (benign or attack) distribution.
In simple terms, PDR measures “how many attacks did I detect out of the total number
of attacks vs. how many alerts did my system generate to detect those attacks?”
To compare the overall efficiency of two or more solutions independently from the specific investigation budget, we compute the PDR, which can be graphically interpreted as the area under the recall/cost curve. The PDR can be approximated with the trapezoidal rule and captures the efficiency of the system for different costs. In order to normalize results (between 0 and 1), we divide the computed area by the maximum achievable area, that is, by the area that an optimum/ideal solution would achieve. The biggest the area, or equivalently the closest to 1, the more efficient the system is. In fact, an increased area translates into an improved recall/cost trade-off.
In the figure above, for example, the PatternEx AI2 approach achieves a PDR of 0.833 while the Anomaly Detection strategy reaches 0.292.
As a final point, when we plot the recall/cost curves for two or more approaches, we obtain a series of comparison points. We can fix a desired recall and compare the cost at which it is achieved. In the same way, we can fix a maximum cost, and see which of the compared solutions yields the highest recall.