Modern EDR Stack: Trellix, Sysmon & OpenSearch

The fastest way to spot an attacker who’s done this before: they never drop a file. They run rundll32 or certutil, they execute in memory, and they use the signed Windows binaries already on the box — and Trellix, CrowdStrike or Defender let them through because everything they touched is legitimate. Modern EDR is good at the malware that looks like malware. Living-off-the-land is the category built specifically to look like normal administration.

The answer isn’t a better single product; it’s layers that fail differently, so what one misses another catches:

1. Trellix (Automated Prevention): Blocks known malware, quarantines suspicious executables, stops the signature-based attacks so your team never has to look at them.
2. Sysmon (Deep Visibility): Records every process creation, network connection and key system change in detail — capturing the activity Trellix allowed because it looked legitimate but wasn’t.
3. OpenSearch (Pattern Detection): Uses ML to flag behaviour that deviates from a machine’s own baseline, catching what matches no signature at all.

The trade-off is honest and worth stating: this is two agents plus a log pipeline plus ML tuning. It costs a little endpoint footprint and real engineering time, and in exchange you get depth a single vendor console can’t give you. If your team won’t own the tuning, a managed EDR is the cheaper choice.

---

1. Trellix: Your First Line of Defense

Trellix Endpoint Security is the prevention layer, and its job is to make the easy decisions so nothing obvious ever reaches a human queue.

How to configure it effectively:

• Block unsigned code: Stop unsigned executables running from the directories attackers stage in — AppData\Local\Temp, C:\Users\Public. Legitimate software rarely runs from there; malware routinely does.
• Enable AMSI scanning: Turn on AMSI integration so malicious PowerShell and VBScript are inspected after de-obfuscation, at the point they’d execute.
• Send logs to OpenSearch: Forward Trellix threat events to OpenSearch via syslog or JSON, so the prevention layer’s decisions sit alongside Sysmon’s telemetry for correlation.

What Trellix won’t catch: A renamed system binary, or a living-off-the-land chain through rundll32.exe or certutil.exe, sails past because every component is signed and legitimate. Trellix isn’t failing here — it’s doing exactly what a signature-and-reputation engine does. Catching the misuse of a trusted tool is a different problem, and it’s the reason the next layer exists.

---

2. Sysmon: Detailed System Telemetry

Sysmon (System Monitor) is a free Sysinternals utility that writes detailed system activity to the Windows Event Log. It doesn’t block anything — it’s pure visibility, and that’s the point.

Why add Sysmon to Trellix? Trellix lets powershell.exe run because PowerShell is a legitimate tool. Sysmon records the context Trellix didn’t judge: which process spawned it, the full command line it received, and every IP it reached out to. That parent-process-plus-command-line detail is what turns “PowerShell ran” — meaningless — into “Word spawned PowerShell, which downloaded from an IP nobody’s seen before”, which is an incident.

Configuring Sysmon Without Drowning in Logs

Never deploy Sysmon with a bare configuration. Unconfigured, it logs everything, floods your indices, and spikes CPU on the endpoints you’re trying to protect — the fastest way to get the whole project switched off by an annoyed desktop team. Start from SwiftOnSecurity’s community baseline and tune from there; it encodes years of “log this, ignore that” you’d otherwise learn the hard way.

An illustrative fragment of what to include:

<!-- Catch suspicious process creations (Event ID 1) -->
<RuleGroup name="" groupRelation="or">
  <ProcessCreate onmatch="include">
    <!-- Monitor living-off-the-land binaries -->
    <OriginalFileName condition="is">rundll32.exe</OriginalFileName>
    <OriginalFileName condition="is">regsvr32.exe</OriginalFileName>
    <OriginalFileName condition="is">certutil.exe</OriginalFileName>
    <!-- Monitor document applications spawning shells -->
    <ParentImage condition="image">winword.exe</ParentImage>
    <ParentImage condition="image">excel.exe</ParentImage>
  </ProcessCreate>
</RuleGroup>

<!-- Catch attempts to dump LSASS memory (Event ID 10) -->
<RuleGroup name="" groupRelation="include">
  <ProcessAccess onmatch="include">
    <TargetImage condition="is">C:\Windows\system32\lsass.exe</TargetImage>
    <GrantedAccess condition="is">0x1410</GrantedAccess>
  </ProcessAccess>
</RuleGroup>

Installation: Deploy via Group Policy or SCCM:

sysmon.exe -accepteula -i sysmonconfig.xml

Log Collection: Use Winlogbeat to forward Sysmon events from the Microsoft-Windows-Sysmon/Operational channel to OpenSearch.

---

3. OpenSearch: Finding Anomalies Automatically

With Trellix and Sysmon both feeding OpenSearch, you need a way to surface the suspicious patterns without hand-writing a rule for every technique — static rules are always one campaign behind. OpenSearch’s ML anomaly detection learns each machine’s normal and flags the deviation. Worth setting expectations: out of the box these models are noisy, and the first few weeks are baselining and tuning, not clean signal. ML anomaly detection tells you something changed, not something is malicious — the analyst still supplies that judgement.

Two Essential Anomaly Detectors

Detector 1: Process Explosion

• What it watches: Process-creation rate per endpoint (Sysmon Event ID 1)
• Why it matters: Ransomware and wipers spawn hundreds of processes in minutes as they enumerate and encrypt. The model learns each machine’s normal rate and flags the spike — which is often the earliest automated signal you’ll get, though by the time a process explosion registers, encryption is usually already underway, so pair it with the earlier-stage detections above.

Detector 2: Suspicious Network Connections

• What it watches: Outbound connections from LOLBins like PowerShell to IPs the host has never contacted (Sysmon Event ID 3)
• Why it matters: PowerShell reaching a brand-new external IP is the network half of the living-off-the-land pattern Sysmon’s process data catches on the host — seeing both together is far stronger than either alone.

When an Alert Fires

1. Investigate the anomaly: OpenSearch shows the exact process or connection that triggered it — Sysmon’s context is what makes this fast rather than a guessing game.
2. Cross-reference with Trellix: Look for Trellix events on the same host around the same time. Two independent layers agreeing is a strong signal; the anomaly firing while Trellix stayed silent is the living-off-the-land case the whole stack exists for.
3. Isolate if needed: On confirmation, Trellix can isolate the endpoint. Keep automated isolation gated by confidence — an ML anomaly is a reason to look, not on its own a reason to auto-quarantine a finance workstation mid-quarter-close.

---

Building a Practical Defense Stack

The three layers earn their place by covering each other’s blind spots: Trellix blocks the known-bad so humans never see it, Sysmon exposes how trusted tools get abused, and OpenSearch’s ML adapts to behaviour no signature describes yet. Each is weak alone — Trellix misses LOLBins, Sysmon only records and never blocks, ML only flags change — and the combination is what closes the gaps.

For most SOCs this is the pragmatic middle: real depth against fileless and living-off-the-land attacks, without a full commercial SIEM’s licensing bill. The cost you’re accepting in return is ownership — the Sysmon config, the ML tuning and the log pipeline are yours to maintain, and left untended they drift back into noise within months. Budget for the upkeep, not just the build.