SIEM Detection Engineering with Splunk/ELK

A SIEM with no detection engineering behind it is a very expensive log archive that bills you by the gigabyte. Pipe terabytes into Splunk or Elastic, leave the default content running, and you get the worst of both worlds: a thousand alerts a day, most of them a backup agent that looks like exfiltration and a service account that’s been failing every ninety seconds since March. Analysts burn out clicking through them, and the one alert that mattered scrolls off the screen unread.

This is the discipline of turning logs into answers — detections that fire on real behaviour, stay quiet the rest of the time, and hand an analyst something specific enough to act on. The unglamorous truth up front: a SIEM is never finished. Detections decay as the environment changes, and “we deployed the rules” is the start of the work, not the end.

Splunk vs. Elastic—Different Tools, Same Goal

The query language changes; the engineering doesn’t.

• Splunk: SPL, backed by accelerated data models for speed and capable of genuinely complex multi-stage correlation. The catch is that SPL’s power makes it easy to write a search that’s correct and ruinously slow — an unbounded correlation across a wide time window will happily bring a search head to its knees, so performance is part of the rule, not an afterthought.
• Elastic/ELK: KQL and Lucene for search, EQL for sequence-based detections, plus a native detection engine and ML jobs that carry real weight. The flexibility is the open-source upside; the cost is that more of the tuning, and the cluster health it depends on, is your team’s job rather than a vendor’s.

We work in both and tune the detections to whichever you actually run — the platform decision is usually already made by the time detection engineering starts.

How We Build A Detection

A detection is built, validated and maintained — never written once and forgotten.

1. Start With a Hypothesis: Which specific attack are we trying to catch, grounded in your actual threat landscape and what the last pen test walked straight through? A detection with no threat behind it is a rule looking for a reason to fire.
2. Find Your Data: Do the logs even exist? Windows Event Logs, Sysmon, EDR — and specifically, is command-line auditing on, because default 4688 gives you the process name and nothing about what it was told to do. This is where most detection ideas die, and better to find out here than after writing the rule.
3. Write the Query: SPL or KQL, fast enough not to hurt the platform under real load.
4. Tune the Thresholds: Too tight and you drown analysts in false positives; too loose and the real thing slips past. Every false positive is paid for by a human on shift, and a rule that’s wrong nine times in ten trains the team to click through the tenth. Test against historical data before it ever fires live.
5. Add Context: Enrich the alert so it answers its own first questions — which asset, which user, what the threat intel says. Context is the difference between a ten-second triage and a ten-minute one.
6. Test It: Validate against the real technique in a purple-team exercise. “We have a rule for that” and “we watched it fire” are different claims, and only the second one counts.
7. Deploy & Monitor: Ship it, watch how it behaves against live traffic, and adjust. Then keep watching, because the environment that made it accurate will change under it.

Threat Hunting—Finding What The Rules Miss

Automated rules catch the attacks you already anticipated. Hunting is how you find the ones you didn’t — the low-and-slow operator who never trips a threshold because they were careful to stay under every one you set.

Example hunt: Looking for unusual outbound connections

• The Question: Is anything beaconing out on ports or to destinations it shouldn’t?
• Data You Need: VPC Flow Logs, firewall logs, DNS queries — and note the hard limit, that you can only hunt as far back as your retention lets you, so a 30-day window makes “when did this start” unanswerable if it started in month two.
• The Filter: Strip the known-good (office ranges, 80/443 to reputable destinations) and look hard at the remainder.
• What To Examine: Destination IPs, transfer volumes, user-agents, and the regularity of the timing — human traffic is bursty, beacons are metronomic.
• What To Do: Confirm it, block the domain, feed the indicators to threat intel, and — the step people skip — turn the successful hunt into an automated detection so you never hunt for that exact thing by hand again.

Give analysts good, documented hunt templates and they’ll find what the rules miss. The honest trade-off: hunting hours compete directly with triage hours, and they’re the first thing cancelled on a busy week, quietly.

Dashboards That Actually Matter

Three audiences, three different screens — build one dashboard for everyone and it serves none of them:

• For Your Analysts: A triage queue that leads with what needs action and carries the context inline — is this user risky, is this asset critical — so the decision lives on one screen instead of five tabs.
• For Your SOC Manager: Detection and triage speed over time, and whether the team is drowning. One caution worth building in: MTTD and MTTR are trivially gameable, since closing alerts faster improves MTTR whether or not anything was investigated, so pair them with a quality signal.
• For Your Executives: ATT&CK coverage, cost, and trend — with the coverage number read honestly. A heat map showing 70% usually means someone counted rules that exist, not rules proven to fire, and the gap between those is exactly where a real attack lands.

Sample Detection Matrix

Rule Name	MITRE Tactic	Log Source	Alert Logic (High-Level)	False Positive Risk
Suspicious PowerShell Download	Execution (TA0002)	Sysmon (EID 1) / EDR	Detects powershell.exe execution combined with known download flags (Net.WebClient, Invoke-WebRequest).	Medium (Requires tuning against admin scripts)
Mass File Deletion / Modification	Impact (TA0040)	Windows Event (4663)	High volume of file modification events by a single user within a 5-minute window.	Low
Geographically Improbable Login	Initial Access (TA0001)	Azure AD / Okta	Successful authentication from two distinct countries within an impossible travel timeframe.	Low
AWS CloudTrail Configuration Modification	Defense Evasion (TA0005)	AWS CloudTrail	Detects API calls targeting logging infrastructure (StopLogging, DeleteTrail).	Low (High severity, low volume)

Get It Done in 90 Days

• Weeks 1–4 (Audit): Confirm the logs actually parse — a field that never mapped, or mapped as the wrong type, returns zero results rather than an error, and zero results looks identical to “nothing malicious”. Map existing rules to ATT&CK, find the gaps, and silence the ten noisiest alerts first, because that alone buys back analyst attention.
• Weeks 5–8 (Build): Write ten to fifteen high-fidelity detections, enrich them with context, and build the one analyst dashboard the team will genuinely use rather than the impressive one they won’t.
• Weeks 9–12 (Test & Hunt): Validate the detections against real techniques in a purple-team exercise, stand up repeatable hunts, and give leadership a coverage view — labelled as tested coverage, not rule count.

Good detection engineering sits on the seam between offence and defence: you write rules by thinking like the attacker and measure them by whether the defender can act. Done right, the SOC works the alerts that matter instead of drowning in the ones that don’t — and keeps doing so, because the maintenance never actually stops.