MIT put a spotlight on a paper co-written by researchers from PatternEx and MIT CSAIL. The paper compared PatternEx's Active Learning approach with generic Anomaly Detection approaches. (Click Here to Read MIT's article.)
The research validates the artificial intelligence technology that we have developed during the last two-plus years.
We often say that the PatternEx platform is ten times better at detecting attacks and does so with five times fewer alerts. Today we’d like to provide some more detail on how we came up with those numbers and we’d like to highlight two important points.
The heart of the PatternEx solution is an active learning technology called AI2. One of the key pieces of the AI2 technology are the supervisory models that are trained in real time by analyst feedback.
In order to test this technology, we used real-world data sets containing real-world attacks. We compared the recall rate of the PatternEx solution with an Anomaly Detection solution. The recall rate is the number of detected attacks divided by total attacks.
In theory, a model trained by humans would show improvements over time, and that is exactly what happened.
We controlled the number of alerts to a constant of 100 alerts per day and observed the results over a 12 week period.
Along the X axis you can see we have increments designating weeks. And along the Y Axis we have increments associated with recall rates. Over the full twelve weeks of the experiment, the predictions made by the PatternEx platform jump to 60% recall rate.
Generic Outlier Detection approaches, by contrast, do not learn. See below:
The green dots show the recall rates at consecutive weeks for an Outlier Detection solution. As you can see, these approaches hover around 10% recall rate over the twelve-week period.
The next graph shows what happens if we increase or decrease the number of alerts and observe the recall rate.
This graph has the recall rate along the Y axis and along the X axis has various levels of alerts shown to analysts. The blue line is the curve described by the PatternEx technology and the green line is the curve described by Anomaly Detection technology.
As the number of alerts are increased, both solutions have an increasing recall rate. At 200 alerts shown, PatternEx has a recall rate of 85% and Anomaly Detection has a recall rate of 8%. That’s over a 10x improvement in attack detection.
At all points along the X axis, Anomaly Detection is inferior to PatternEx AI2.
At 1000 daily alerts, the Anomaly Detection solution reaches a 77% recall rate. Interestingly, AI2 exceeds that recall rate at only 200 alerts. This is why we say PatternEx has 5x fewer alerts than Anomaly Detection approaches.
This is a good opportunity to acknowledge the hard work of the entire PatternEx team: our world-class engineers, security experts, and data science/AI team. Our vision and work does not stop here. We will keep working in close collaboration with MIT to develop innovative solutions to face the challenges within Cyber Security.
This research was based on using real-world data sets and comparing the efficacy and efficiency of a generic anomaly detection solution and PatternEx’s solution. One of the researchers works at MIT’s CSAIL and the other researcher is the Chief Data Scientist of PatternEx. To remove any vendor bias, we presented it to IEEE for publishing and peer review. We presented April 9, 2016 at the IEEE International Conference on Big Data Security in New York City.
Use the form at the top of this page to let us know you'd like to speak with us about PatternEx in your enterprise!