What Threat Detection is NOT about — before they sell it to you!

Detection Engineering is really making its way into the lexicon of what Cybersecurity customers demand today. After advocating for that practice for many years, I thought it would be great to share a few bullets on that.

This might not only serve practitioners in the field but also product managers and other threat detection personas involved in research, product development sales engineering cycles.

Every threat detection solution requires engineering efforts.

Yes, even the 'AI driven' ones. If you outsource detection to a provider, that's where engineering is hopefully happening then.

So before you buy or even build another snake oil, here's my two cents.

Detection Engineering is sexy

Besides being one of the most exciting jobs on the blue side (super biased), I guess there are a few other aspects contributing towards the momentum:

  • Regulations requiring logging, security monitoring, and data breach reporting (transparency/accountability) capabilities;
  • SIEM/XDR vendors and MSS/MDR providers realizing the competitive advantage of a mature threat detection practice to their business;
  • The less disruptive nature of detection controls implementation.

Peace sells but who is buying?

Delivering high-quality detection controls requires engineering skills that can’t be easily found and retained. And what is one of the outcomes from that? Underdeveloped detection programs, poor detection output, etc.

Despite having the noble goal of helping customers implement security controls, a business (vendor) needs to be profitable. But how to sell the solution to this gap? Selling people or outsourcing won't scale.

I don’t think you need to work in Cyber or even in IT to understand how the game is played. Once a problem or rather, a (bad) symptom, is identified, that becomes an opportunity to be explored in the market.

Alert Output symptoms as a distraction

This is part of human behavior, instead of practicing healthy habits or good hygiene, it's easier to tackle the symptoms and carry on.

So watch out in case you are buying a product that simply alleviate a symptom without helping solve the actual challenge (detecting threats).

Detection is about reducing the alert volume — NOT!

I believe this is the #1 scenario.

This is the infamous Alert Fatigue problem: too many alerts to be handled leading to overwhelmed teams handling the detection output.

It's hard to engineer good detections. It's easy to tune down bad ones.

Believe me, I have already seen "Reduce the number of alerts by X%" set as a project milestone. What's even worse is the potential to introduce false-negatives (assuming at least one good detection idea is in place).

We all know the current Professional Services (PS) approach: deploy and run. Once the green lights are blinking and alerts are flowing in, time to move on to the next product (or the next customer engagement for PS).

What's the follow up from there? What should be the focus?

Use a product (ex.: SIEM) as another tool in the arsenal to address threat detection gaps/model or tune down a dozen detections deployed by PS constantly triggering alerts? That's what some teams are left with.

Detection is about alerting in real-time — NOT!

And it doesn’t change if you say near real-time here.

If LOG Contains ‘BAD’; Then Alert versus If LOG Contains ‘ERROR’; Then Alert.

This is likely another inheritance from NOCs and other SLA driven monitoring workflows. Yes, the SOC should engage as soon as possible, whenever an incident or suspicious activity is spotted.

However, if you consider what is seen in many industry threat reports, it's easy to realize attackers have a big leverage as it takes many days to detect a breach. It's indeed hard to detect, let alone in near-real-time.

The challenge is actually spotting threats in a consistent, reliable way.

Once that's solid, then it's time to work on such metric (Mean Time to Detect). Detecting requires data analysis, threat intelligence/modeling and a combination of curiosity, opportunity, just like attackers employ.

If the detection output is poor, you are not making it any better by detecting it faster! The challenge is not detecting faster. Most teams are far from that stage.

Also, besides multiple data points needed (modeling, correlation), consider log availability and transport (latency) are still an issue before performing the actual data analytics.

Detection is about (triage) automation — NOT!

Just like the previous fallacy or distraction, if the detection output is poor, automating what happens post-alert does not help either!

Don't get me wrong, I am sure there are use cases for SOAR in the triage workflow, but buying it to justify investment in detection makes little sense.

I see Automation here more as a DevOps approach, pushing new detection code via Git, integrating Attack Simulation or Autonomous Penetration Test platforms to the worklfow for both testing and crafting new detections.

Automatically performing real-world attacks will effectively boost detection quality and reliability while helping prioritize the detection backlog.

How many automated (SOAR) or even manual workflows for triaging an alert do terminate when an alert hash is checked against VirusTotal or another CTI IOC repository? No hits, end of the line, close the ticket.

Why not considering that outcome (# of file hash hits in a CTI repo) as yet another indicator, before dispatching an alert? Consider the Shift Left Strategy when designing a detection.

Crawl before you run

The infamous "trash in, trash out" is still applicable today. It's easier to define progress around data on-boarding than it is to leverage that same data for threat detection — where most value is realized.

Aiming at some of the metrics discussed (ex.: mean time to detect) is part of a secondary stage in a detection engineering practice. Most customers of those products still don't know 'What' to detect in the first place. So clear message here is Do the basics first!

Since we are here. What detection engineers actually need?

That easily makes into another blog post but here is my bet:

Better languages or interfaces to design and implement detection logic. Detection engineers need more specialized languages such as SPL or KQL as well as new commands incorporating more ML/Stats related functions.

General purpose languages code tend to be harder to learn and maintain. (Hey, I like Python too! But not in this context…)

And don't get me started on SQL! A language created around 50 years ago…

Do you recommend an interface for time series or logs in general? Feel free to send me a message, happy to exchange ideas!



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alex Teixeira

Alex Teixeira

Blueteamer. Threat Detection Engineering & Security Analytics. https://linkedin.com/in/inode