Artificial Reality | The PatternEx Blog

Collections of the thoughts and the people behind the PatternEx Virtual Analyst Platform powered by AI2.

Help for SOC Analysts - Autocorrelation

Security Incident and Event Management (SIEM) solutions have been in use for almost two decades, but the promise of SIEMs and other log search solutions remains unfulfilled. e-Security, arguably the first SIEM company, was founded in 1999 in Vienna, Virginia.

SIEMs deliver analytics tools with search capability, but these tools remain limited to to providing responses to manually created questions / queries / correlations by human analysts and have not evolved beyond rule-based correlations. SIEMs have made claims about increased complexity and sophistication of such correlations through the use of wildcards, Boolean logic, RegEx, and other techniques. However, the SOC analyst remains constrained to receiving responses about his or her specific query and the correlation must be very specific in order for the signal-to-noise ratio to be acceptable. 

As a result, SIEMs lead to alert overload, generating thousands or millions of false positives for analysts to manually filter, investigate, and take action. In addition to being a huge drain on resources, this workflow often misses true risks (false negatives) in the deluge of alerts.

SIEM Workflow vs. PatternEx AutoCorrelate

Let’s then compare the current state-of-the-art workflow with the PatternEx Virtual Analyst platform. 

Suppose a particular domain has been flagged as suspicious. Then an analyst investigating this domain may seek answers to the following queries during an investigation:

  • What source IP addresses visited this domain in the last few days?
  • Which destination IP addresses are associated with the domain?
  • Are other domains associated with these destination IP addresses?

In a current SIEM workflow, each of those questions would need to be answered via queries. After each investigative query, an analyst must decide on the next step of the investigation based on the outcome of that query. The problem, however, is that there may be hundreds or even thousands of next entities to investigate. Therefore, deciding the next step of the investigation is often rule-based (i.e., investigate entities / relationships that follow a known malicious patterns).

However, in the PatternEx platform, the AutoCorrelate method instead has the capability to automatically explore entity relationships in parallel, using information from not only first-order correlations, but also second and even third order correlations. This allows simultaneous processing of all relationships involved in malicious and suspicious predictions. And, this allows looking beyond direct connections to find additional entities potentially involved in malicious behavior—going beyond first-degree relationships to surface entities not found by simple queries.

AutoCorrelate Goals

So what goals does AutoCorrelate achieve?

1. The first goal is to provide richer context to existing alerts.

After malicious and suspicious predictions are surfaced from the models, AutoCorrelate automatically provides the analyst with context and relationships between these malicious entities (e.g., Host 1 and Host 2 both visit malicious domains Domain 1 and Domain 2. Host 1 is accessed by User 1, which is flagged as suspicious). When viewing these relationships in a graph, the context and investigation into root-cause analysis becomes much more straightforward:


2. The second goal of AutoCorrelate is to surface new suspicious entities.

To do so, we need to provide further contextual information and relationships between those entities (e.g., domain z, domain a, user Bob are flagged as malicious) which can be mined to identify additional suspicious IP addresses and domains. After an analyst has performed an investigation and determined an entity to be malicious, she can then utilize that label to surface additional new suspicious entities. An example of such could look like:


Which then reveals:



Thus, autocorrelation not only provides significantly better detection capabilities, but it also provides far more context than previously, greatly increasing the speed with which investigations can be conducted.  For example, attacks can be tracked across the "kill chain," greatly increasing the efficacy of an investigation.


Real-World Results

So, how well does this new capability work? Our real-world / live data from customers shows that this new capability speeds up investigation times by 20x compared to existing threat-analytics platforms at ⅕ the cost. Really? How so? Well, let’s look at some of that real-world math.

In one use case (delivery - either malware or phishing), we compared predictions of malicious domains made by PatternEx’s Virtual Analyst Platform against VirusTotal. In those few cases where VirusTotal already had confirmed the predictions made by our Virtual Analyst Platform, an average of 20 additional investigations would have been required to find the additional malicious domains that PatternEx predicted, which were later confirmed by VirusTotal as indeed being malicious. Hence, speeding up investigation times by 20x compared to existing threat-analytics platforms.

Given the number of headcount required to conduct such additional investigations, and the average length of an investigation, PatternEx’s Virtual Analyst Platform costs ⅕ the fully loaded cost of that headcount.

With regard to a 10x improvement in detection by PatternEx’s Virtual Analyst Platform compared to existing threat-analytics platforms, that improvement has been previously validated in a peer-reviewed paper published by IEEE (“AI2: Training a big data machine to defend”). Together, these improvements in detection and investigation efficacy are significant.

Interested in finding out more?  For further information about these capabilities, please contact us.  

Request a PatternEx Demo


Topics: Threat Detection infosec AI SOC

Subscribe Now