The idea of this post came after a Slack chat with Ryan Long, a Sr. Security Analyst who had asked this very question highlighted in the blog title.
Ryan as many others is realizing there’s sometimes a very thin line between analysis (operations) and engineering when it comes to threat detection.
Why is Threat Detection so trendy now?
Because the demand is higher? Because Cyber is becoming specialized? I wrote a blog post touching on this topic a few years ago. Let's start with some context first.
Machine Data & Modern Tooling
Logs, logs everywhere! Today we have too much data at our disposal, so much that we need to design data pipelines with routing, filtering and other pre-processing cycles before consuming it in the other end.
I remember working as an architect for a bank in the early 2000s when an item of my product assessments was "Does it provide logs?" Today, we ask "Does it ship JSON logs?", "Does it provide an API to extract logs?"
Any enterprise-grade system today must generate logs.
Besides AVs, what was actually detecting threats — as they happen?
Some years ago, data for detection was basically coming from the wire (NIDS, PCAP analysis) and Syslog — that was it! Going beyond that meant you were on your own, building a sort of log aggregator (shivers!) or custom agents.
The main reason (use case) for log collection used to be Forensics or Compliance. Who was proactively checking Windows Eventlogs 15 years ago? You can probably name a few of your heroes here.
Besides the lack of abundant data, the ability to query on it was also limited.
Even some big names from SIEM industry have (miserably) failed to provide easy and fast ways to question the data with very rudimentary interfaces or by leveraging languages not far from 46 years old (!) SQL.
I believe that's another big shift. IMO, platforms such as Elastic, Splunk and Microsoft's Data Explorer (or Sentinel) leveraging script-like languages to ask questions to data makes a whole difference.
You've probably noticed I'm focusing on SIEM and Big Data.
Yara or Zeek (Bro) rule writing skills do matter — a lot, but the ability to aggregate and extract signals from these and other data sources combined is what brings threat detection to the next level (Security Analytics).
When these are in the hands of skillful teams generating the expected outcome, it's easier to assert "Detection starts where Prevention ends."
Or as Florian Roth beautifully put "Assume Compromise"! His tweet summarizes the shift in paradigm quite well!
Besides the obvious threat evolution, those are among the main drivers for the recent developments in the threat detection landscape.
Is a Threat Detection Engineer a Security Analyst/Engineer with focus on Threat Detection?
I used to write Snort rules, but that has accounted for a tiny fraction of my full-time job as a senior analyst, security hardening accounted for most. Still, it does qualify as a practice of Threat Detection Engineering.
Today, it can be very different, one can easily land a full-time job writing detection content. It also depends on how the team is structured or how the organization strategy is defined.
Some orgs designate a team responsible for deploying and maintaining (fine-tuning) a multitude of rather specialized security products or appliances, from WAFs, to EDRs, to SPAM (anti-phishing) filters.
What would disqualify those from being considered detection engineers?
Anyways, here's my attempt to define it:
A Threat Detection Engineer is someone who applies domain knowledge on designing, building or maintaining detection content in the form of detections generating alerts; or interfaces in the form of dashboards or reports supporting the security monitoring practice within an organization.
As a former security analyst, I keep saying that the security analyst's work is a continuation of the detection engineer's one, since many detections would never be feasible without exception handling granted in the end.
What's your take on that? Feel free to reach out or leave comments!