ELK Threat Hunting Basics: How to Turn Logs into Security Investigations
A practical, beginner-friendly ELK threat hunting guide covering architecture, data onboarding, hunting workflow, worksheet design, documentation standards, and a 4-week plan to turn hunts into detections.
Most teams that say “we tried threat hunting and got nothing” have the same three problems, and none of them is Elasticsearch. The hunts had no falsifiable question. The fields were inconsistent enough that correlation quietly failed. And nothing was written down, so the second hunt started from the same blank page as the first.
Alerts cover what your rules already understand. Hunting is the part that goes after what they don’t — which means it produces its value on a longer timescale, and it will feel unproductive for the first month. That’s normal. What isn’t normal is a year of hunting that has produced no new detections, no closed telemetry gaps, and no record of what was ruled out.
ELK Threat Hunting Basics
The workflow below is built to produce results you can measure, hand over, and turn into rules.
1) What Threat Hunting Actually Is (And Isn’t)
Hunting is hypothesis testing, not browsing. You state something specific enough to be wrong — “service accounts are authenticating interactively” — and then go looking for evidence that settles it either way. The discipline is in the “either way”. A hunt that can only ever confirm your suspicion isn’t a hunt; it’s a search for something you already believe, and it will find it.
The unglamorous truth is that most hunts come back negative, and a negative hunt with a documented time window and data source is a genuinely useful artefact. It tells the next analyst not to spend a day on it.
How Threat Hunting Differs from Alert Triage
| Activity | What Drives It | What You’re Trying to Do | What You End Up With |
|---|---|---|---|
| Alert Triage | Alerts that already fired | Figure out if they’re real threats or false alarms | An incident ticket to work or a “false positive” note to close |
| Threat Hunting | Your own hypothesis + the logs | Find attackers and patterns your rules miss | New detection rules, data gaps you never knew about, or escalated cases |
Both matter, and they compete for the same analysts. That’s the real trade-off nobody states: hunting time is triage time you didn’t spend, and if the queue is already underwater, protected hunting hours will be the first thing sacrificed. Ring-fence them explicitly or accept that hunting won’t happen.
2) Your ELK Stack: What You Actually Need
You don’t need an enterprise-grade deployment to start. You need data arriving consistently, with field names that mean the same thing on Tuesday as they did last month. Consistency beats coverage — three well-normalised sources will out-hunt fifteen sources where user, user.name and username all appear depending on which pipeline processed the event.
The Five Layers of Your Stack
- Ingestion: Beats, agents, or forwarders pull events from your systems
- Parsing: Logstash or ingest processors normalise the data and map fields consistently — ECS is worth adopting here even if it costs you a fortnight, because it’s what every Elastic-native rule assumes
- Storage: Elasticsearch indexes everything so you can query it fast
- Analysis: Kibana gives you dashboards, saved searches, and investigation timelines
- Detection: Elastic Security rules, cases, and automated response workflows
Before You Hunt: Quality Checks for Each Layer
| Layer | What You Need | Red Flag |
|---|---|---|
| Ingestion | Data consistently flowing from all your sources | A source suddenly goes quiet with no scheduled maintenance |
| Parsing | Field names and types stay the same across logs | The same piece of information appears in different fields with different names |
| Indexing | Data arrives on time and is searchable | You see gaps in data where you expect to see events |
| Analysis | Dashboards and queries you can reuse between hunts | Starting every hunt from scratch because nothing is documented |
| Detection | Your hunt findings easily turn into new detection rules | Good hunts just sit around and never become operational alerts |
Inconsistent field types are the failure that hurts most, because it fails silently. Query a keyword field as though it were text, or filter on a field that’s mapped in one index and absent in another, and Elasticsearch returns zero hits rather than an error. Zero hits looks exactly like “no malicious activity found”. Analysts have closed hunts clean on the strength of a mapping bug. Before you trust a negative result, confirm the query returns something when you deliberately remove the suspicious condition.
3) Which Data Sources to Feed Into ELK First
Useful results come from the right sources, not from all of them. Every additional source costs storage, licence volume and — more importantly — parsing effort you’ll owe forever.
Start with These Sources (In This Order)
- Authentication logs — Identity providers, Active Directory, SSO systems
- Endpoint logs — Process execution, parent-child relationships, network connections
- Firewall and network flow logs — Who’s talking to whom and when
- DNS and proxy logs — What domains are being queried, what traffic goes through
- Web and API access logs — Who hit which endpoints and what they asked for
- Cloud audit logs — Who did what in AWS/GCP/Azure control planes
Why This Order Actually Works
Auth plus endpoint gets you the join that matters most: an identity, a machine, and what ran there. Add DNS and proxy and you can characterise egress — the beaconing and staging behaviour that shows up nowhere else. Cloud audit logs come last only because they’re least useful without the first four to pivot against; a suspicious AssumeRole means little until you can tie it to a session on a host.
Two practical warnings. Endpoint process telemetry is by far the most expensive source per gigabyte, and teams routinely onboard it, watch retention costs, and cut the window to seven days — which is shorter than most dwell times and therefore shorter than the hunts you wanted it for. And DNS logging from a forwarder shows you the forwarder’s IP, not the client’s, unless you collect at the client end. That’s a data gap people discover mid-hunt, three hours in.
4) Your Hunting Workflow (Keep It Repeatable)
Use the same loop every time. Consistency is what makes a hunt defensible six weeks later when someone asks how you concluded the activity was benign, and it’s what makes the conversion to a detection rule mechanical rather than a rewrite.
The Seven-Step Hunt Loop
- Ask a Question (Your Hypothesis)
- Example: “Are our service accounts logging in from interactive terminals at weird times?”
- Pick Your Data Sources
- Choose the logs you need to answer that question.
- Set Your Baseline
- Compare against historical normal — and match the period. Same weekday, same hours. A Tuesday-morning baseline makes Saturday night look like an intrusion.
- Pivot Around the Data
- Start with one user/host/IP/process, then expand to related events.
- Review the Evidence
- Is it suspicious, or is there a benign explanation? Ask the system owner before you decide. Half of “anomalous” turns out to be a backup job, a vulnerability scanner, or a contractor in another timezone.
- Reach a Conclusion
- Is this a real finding, a common false positive, or a data gap you discovered?
- Improve Detection
- If you found something real, turn that hunt logic into a new detection rule or dashboard.
Hunt workflow table
| Step | Key Question | Deliverable |
|---|---|---|
| Hypothesis | What suspicious behaviour are we testing? | Written hypothesis statement |
| Data Selection | Which logs can prove or disprove it? | Source list + required fields |
| Baseline | What does normal look like for this behaviour? | Baseline snapshot with time window |
| Pivot | What related entities/events should be explored? | Pivot map (user, host, process, IP) |
| Evidence | Is the signal suspicious after context checks? | Evidence packet with timestamps |
| Conclusion | Is this incident, benign, or inconclusive? | Hunt outcome classification |
| Improvement | What control should be improved next? | Detection/task backlog item |
5) Real Hunt Examples You Can Actually Run
Six hunts that are worth the afternoon, with the benign explanation you’ll most likely land on.
Hunt A: Unusual Login Behavior
Privileged accounts authenticating outside their established pattern — hour, source geography, device. Then look at what happened on that host in the following ten minutes, because a login on its own tells you almost nothing. Most common benign cause: someone travelling, or a VPN concentrator that changed egress IP.
Hunt B: Rare Process Execution
Rank process names by frequency across the estate and read the tail. The interesting signal isn’t the rare binary; it’s the rare parent-child pair — Office spawning a shell, a web server spawning anything. Expect noise from installers and patch cycles, which is why you check the change calendar before escalating.
Hunt C: Strange Outbound Connection Patterns
First-seen destinations per host, then intersect with reputation data. Beware the base rate here: a busy estate produces hundreds of never-before-seen destinations daily, most of them CDN edges and ad networks. Filter by which internal host is talking before you filter by reputation, or you’ll drown.
Hunt D: DNS Anomalies
Query volume outliers per client, plus unusually long or high-entropy labels. Long subdomains are the classic tunnelling tell — and also exactly what several endpoint-security and antivirus products do for cloud lookups. Establish which vendors in your environment behave that way once, and write it down, so nobody re-investigates it quarterly.
Hunt E: Unexpected Admin Path Access
Rare hits on admin routes and sensitive endpoints. Who, from where, and what surrounds it — failed auth bursts, odd user agents, requests with no referrer. Internet-facing admin paths get scanned constantly, so the question isn’t “was it accessed” but “did anything succeed”.
Hunt F: Privilege Changes
Spikes in group or role assignment. Correlate against change tickets. The finding you actually want is the grant with no ticket behind it — and the follow-on question is what the account did in the hour afterwards, which is where a legitimate-looking grant stops looking legitimate.
All six stay on the defensive side: you’re characterising behaviour in your own logs and improving what you can see.
6) Threat hunting worksheet table (required artifact)
Fill this in for every hunt, including the ones that find nothing. The worksheet is what makes handover, peer review and detection conversion possible; without it a hunt exists only in one analyst’s head until they leave.
| Hunt ID | Hypothesis | Data Sources | Time Window | Baseline Method | Key Fields | Query Notes | Pivots Run | Evidence Collected | Outcome | Detection Candidate | Owner | Next Review Date |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HNT-YYYY-001 | Example: Rare privileged login pattern | Auth + endpoint + proxy | 14 days + current 24h | Same weekday/hour comparison | user, src_ip, host, auth_result | Saved search + filters used | User → host → process | Timestamped log bundle | Suspicious/Benign/Inconclusive | Rule idea summary | Analyst name | YYYY-MM-DD |
Minimum worksheet quality bar
- Hypothesis is explicit and testable
- Data sources are sufficient to disprove hypothesis
- Time window includes baseline and current period
- Outcome includes rationale, not just a label
- Detection candidate is written even for negative hunts (if data gap found)
7) Document Your Hunts Like They’re Code
An undocumented hunt has a shelf life of about a fortnight — after that even the analyst who ran it can’t reconstruct which time window they used. Treat the notes as engineering documentation, because that’s what the detection engineer will be reading.
Every Hunt Needs This Written Down
- Hypothesis: What you thought was happening and why you cared
- Query logic: In plain English, what you searched for
- How you set your baseline: What “normal” meant for this hunt
- Why you ruled out false positives: What benign explanations you considered
- Where your evidence came from: Links to dashboards, query IDs, case numbers
- What you concluded: What your confidence level is
- What’s next: Detection rule idea or telemetry gap you found
Confidence Levels (Keep It Simple)
| Level | What It Means | What You Do About It |
|---|---|---|
| Low | Something weird happened but you’re not sure | Ask for more data or watch longer before deciding |
| Medium | Multiple things point to the same suspicious behaviour | Escalate to someone who can triage it properly |
| High | The evidence clearly shows malicious or policy-breaking activity | Open an incident and start responding |
8) Turn Your Hunts Into Permanent Detection Rules
The point of hunting is to find patterns worth automating — but not every finding survives the transition. A hunt tolerates a hundred results you read by hand. A production rule firing a hundred times a night will be muted within the week, and a muted rule is worse than no rule because everyone believes it’s covering you.
How to Convert a Hunt Into a Detection
- Find the core pattern: What single behaviour separated normal from suspicious? If you can’t name it in a sentence, it isn’t ready to be a rule.
- Define your requirements: What fields do you need? How good do they need to be?
- Set your threshold: Based on your baseline, when should this alert?
- Add context: Tag by criticality, owner, environment—anything that helps triage
- Write the triage steps: What should an analyst check when this alert fires?
- Test it quietly first: Run it for a fortnight without alerting and count what it would have produced. This step gets skipped more than any other, and it’s the one that prevents a rule being muted on day three
- Release it: Promote to production with a named owner and a review date. An unowned rule decays as the environment changes around it, and nobody notices until it stops firing entirely
Detection conversion table
| Hunt Outcome Type | Detection Action | KPI to Track |
|---|---|---|
| Confirmed suspicious behaviour | Build new alert rule | Detection precision and incident conversion rate |
| Repeated benign pattern | Add suppression or context filter | False-positive reduction |
| Inconclusive due to missing fields | Create telemetry improvement task | Data quality completion rate |
| Rare but valid admin operation | Add approval/change-ticket correlation | Analyst triage time reduction |
9) Mistakes That Derail Hunts (And How to Avoid Them)
- Hunting without a falsifiable question: browsing dashboards until something looks odd. Something always looks odd, and you’ll spend the day on it.
- Inconsistent fields: correlation fails silently and returns nothing, which reads as “all clear”.
- Not asking the system owner: you can’t call behaviour abnormal for a team whose normal you’ve never seen.
- Mismatched time windows: comparing a Tuesday to a Saturday, or a working day to a bank holiday. The baseline is then decoration.
- Concluding from one event: a single odd log line is a starting point, not a finding. Corroborate across a second source before you escalate.
- Never converting findings: you find it, you don’t rule it, it recurs in March and someone hunts it again from scratch.
- Measuring nothing: without hunt counts and conversion rates, “is this worth the analyst hours?” has no answer, and the first budget review settles it for you.
Before You Hunt, Check This
- Write down your hypothesis before you touch the query tool
- Document what normal looks like before you call something abnormal
- For anything suspicious, dig at least one level deeper with a pivot
- When you wrap up, say what your confidence is and who owns the next step
- Create a ticket for any detection rule or data fix that came out of the hunt
10) Metrics That Actually Show Your Hunting Maturity
You get better at what you measure, and worse at everything you measure badly. “Hunts per month” is the obvious trap: make it a target and you’ll get thirty shallow hunts instead of six that mattered. Track it, but read it alongside conversion and outcome quality, and never put it in someone’s objectives on its own.
| Metric | Why It Tells You Something | What You Want to See |
|---|---|---|
| Hunts per month | Are analysts actually hunting, or just reacting to alerts? | Steady increase as you mature |
| New detections from hunts | Are hunts turning into better rules? | Increasing over time |
| False positives you eliminated | Is your tuning making rules better? | Reduction rate going up |
| Real incidents from hunting | Is hunting actually catching bad stuff? | Depends on your environment |
| Data gaps you fixed | Are you closing the holes hunting reveals? | Closure rate going up |
| Hours from hunt start to done | Are analysts getting faster? | Decreasing without sacrificing quality |
11) Get Started: A 4-Week Hunting Plan
Week 1: Make Sure Your Data Is Clean
- Confirm auth, endpoint and DNS/proxy logs are arriving — and arriving now, not eleven hours late
- Check field names and, more importantly, field types; also check every source is on UTC. Mixed timezones make timeline reconstruction quietly wrong rather than obviously broken
- Build two dashboards: one auth, one endpoint activity
You end this week with: A list of data quality defects and two baseline snapshots. Resist hunting this week. Hunting on data you haven’t validated produces conclusions you’ll have to withdraw.
Week 2: Run Your First Hunts
- Do a hunt on login anomalies
- Do a hunt on weird process execution
- Use the worksheet format for both
- Show your team what you found
You end this week with: Two completed hunt worksheets with confidence levels
Week 3: Turn Findings Into Rules
- Pick one suspicious pattern you found and draft a detection rule for it
- Find one false positive pattern you saw and write a suppression for it
- Document the triage questions someone should ask when the rule fires
You end this week with: One detection draft and one tuning improvement you can implement
Week 4: Make It Operational
- Deploy your tuned detection and actually monitor it
- Run one more hunt, this time looking for data gaps
- Review what you ran, what you found, and what it cost in hours
You end this week with: A monthly summary and a prioritised list of detections to build next
None of this depends on ELK specifically. What it depends on is doing the same thing every time: a question you could be wrong about, evidence from fields you’ve verified, notes another analyst could follow, and an outcome that changes something in production. Do that six times and you’ll have a detection backlog built from your own environment rather than someone else’s blog post.
Hunt operations worksheet for team consistency
| Workstream | Owner | First Action | Validation Signal |
|---|---|---|---|
| Hypothesis quality | Hunt lead | Require testable hunt question before querying | Fewer aimless hunts, clearer outcomes |
| Data reliability | SIEM/platform owner | Validate key fields and ingestion continuity | Reduced inconclusive hunts due to missing data |
| Evidence standards | Analysts | Enforce worksheet completion for all hunts | Better handoff and peer review quality |
| Detection conversion | Detection engineer | Track hunt-to-rule backlog with owners | More hunts converted into production detections |
Daily hunt discipline checklist
- Start with one explicit hypothesis and time window
- Record baseline method before interpreting anomalies
- Capture at least one pivot path per suspicious signal
- End each hunt with decision + next action owner
Hunt-to-detection handoff pack
| Artifact | Minimum Content | Consumer |
|---|---|---|
| Hunt summary | Hypothesis, data sources, confidence, outcome | SOC lead |
| Query logic notes | High-level logic and required fields | Detection engineering |
| Evidence bundle | Timestamped events and pivot trail | Incident responders |
| Improvement task list | New rules, suppressions, data-gap fixes | Platform + detection teams |
Quality checks
- Would another analyst reach the same conclusion from your worksheet?
- Are recommended detection changes specific and implementable?
- Are data gaps documented with actionable owners?
Your 90-Day Hunting Roadmap
Month 1: Get the Process Right
- Make sure everyone uses the same hunting worksheet
- Do one hunt a week on your biggest questions
- Track how many hunts you do, what outcome you get, how many become rules
Month 2: Fix the Data, Build the Rules
- Fill in the gaps in your field names and log sources
- Turn your best hunt findings into draft detection rules
- Start having the team review each hunt before you close it
Month 3: Operationalize and Report
- Deploy your new detections and watch how they perform
- See which new rules are generating signal, which are noisy
- Write up a summary of what you accomplished and where the gaps still are
| Thing to Track | Why It Matters |
|---|---|
| Hunts with full documentation | Shows the team is doing this right |
| New detections from hunts | Shows hunting is creating value |
| Hunts stuck because of missing data | Shows you what telemetry to invest in |
| Average time to finish a hunt | Shows if analysts are getting faster |
By the end of ninety days the honest question is whether your detection coverage looks different than it did on day one. If it does, hunting is working. If the only artefact is a folder of worksheets, the loop is broken at the conversion step, and that’s a scheduling problem rather than an analysis one.
Hunt-to-detection pipeline (turn investigations into lasting coverage)
Every hunt should end in one of three states: a new detection, a new control requirement, or a documented “not an issue” with the reasoning attached. That third one is a real outcome, not a failure — an undocumented negative gets re-hunted by the next analyst who has the same idea.
Hunt card template (repeatable)
| Field | What to capture |
|---|---|
| Hypothesis | What you believe is happening and why |
| Data prerequisites | Required logs, fields, and time range |
| Query approach | High-level logic (not just one query) |
| Validation | How you confirm true vs false positives |
| Outcome | Detection/control/documentation |
| Follow-ups | Owners and due dates |
Converting a hunt into a detection
- Extract the core signal separating malicious from normal behaviour.
- Identify the best data source, and be honest about what you’d have to onboard if it isn’t there.
- Write triage steps deterministic enough that a tired analyst at 3 a.m. reaches your conclusion.
- Define suppressions carefully — service accounts, known scanners, expected automation. Every suppression is also a blind spot an attacker can occupy, so scope them narrowly and review them.
- Add a regression test, because rules die quietly. A detection that stopped firing in February because a log source changed format looks identical to a detection that’s working.
Quality gates
| Gate | Pass condition |
|---|---|
| Data quality | Required fields present in > 95% of events |
| Analyst usability | Triage can be completed in < 15 minutes |
| Noise control | Alert volume is operationally sustainable |
| Documentation | Hunt card + reasoning are stored and searchable |
The gates are the difference between hunting as a discipline and hunting as a series of interesting afternoons. Each hunt leaves something behind that outlives the analyst who ran it.
