Threat detection metrics: exploring the true-positive spectrum

I’ve had the chance to work with many great security teams during my career and in 2012, I had the opportunity to join Verizon's SOC in Germany. That was a very challenging experience considering its massive scale SecOps.

It was also by that time when I realized Splunk could be used as a sort of BI/Reporting platform given its ability to quickly generate eye-catching reports or dashboards from case or incident management systems data.

Today, when designing and building detection mechanisms, it's easier to notice the link between threat detection engineering practice and overall SOC services quality, regardless of target customer (internal/external).

The security alert dichotomy: TP x FP

Without going too deep on that, I guess the true-positive (TP) versus false-positive (FP) classification became widespread in Infosec after the introduction of pattern based Network Intrusion Detection Systems (NIDS).

In a nutshell, here's how the assessment is done:

"Does the input (network flow) match a pattern?" If yes, it must be either a TP, when indeed the attack or threat is verified; or a FP, when despite the positive test, the attack or threat expected is not present.

On the other hand, if the answer is no, then someone may later find it to be a false-negative case, when the threat or attack is present but the detection did not trigger an alert. A true-negative case is when "everyone is fine".

It turns out modern detection systems and strategies require more fine-grained TP/FP labels (specification) for tracking important metrics.

In a threat detection context, higher granularity outcomes will make it easier to determine, for instance, how good or bad a rule is performing. Ultimately, better supporting decisions to modify or even shutdown a detection mechanism. More on this later on "metrics" section.

Driving QA from alert states

My work involves developing detection content, so it makes perfect sense to define a methodology to track security alerts quality over time.