Modern EDR Stack: Trellix, Sysmon & OpenSearch

Commercial EDR products are powerful, but no single agent catches everything. Advanced adversaries routinely bypass primary endpoint defenses (like Trellix, CrowdStrike, or Defender) by utilizing “living off the land” binaries (LOLBins) or operating strictly in memory.

This guide outlines a highly practical, defense-in-depth endpoint architecture:

1. Trellix (Primary AV/EPP): Handles the automated blocking, quarantine actions, and signature-based prevention.
2. Microsoft Sysmon (Deep Telemetry): Silently logs the granular process creations, network connections, and memory allocations that Trellix might allow but are contextually suspicious.
3. OpenSearch (Analytics & ML): Aggregates the logs and applies Anomaly Detection to identify malicious deviations in baseline behavior.

---

1. Trellix: The Prevention Layer

Trellix Endpoint Security provides the foundational block-and-tackle layer. Your goal here is to let Trellix handle the commodities so your analysts don’t have to.

Practical Implementation Steps:

• Aggressive Execution Control: Ensure Trellix is configured to block unsigned executables executing from AppData\Local\Temp and C:\Users\Public.
• AMSI Integration: Ensure Trellix AMSI (Antimalware Scan Interface) integration is enabled to catch script-based payloads (PowerShell, VBScript) before execution.
• Log Forwarding: Configure Trellix ePO to forward threat events directly to your centralized OpenSearch cluster via syslog/JSON.

Note: Trellix catches the known bads. However, when an attacker drops a renamed calc.exe or uses rundll32.exe to execute a malicious payload, traditional EPP often fails. This is where Sysmon takes over.

---

2. Sysmon: The Forensic Telemetry Layer

System Monitor (Sysmon) is a free Windows Sysinternals tool. Once installed, it remains resident across reboots to log system activity to the Windows Event Log.

Why Sysmon alongside Trellix? Trellix might see powershell.exe running and allow it. Sysmon will log exactly which parent process spawned it, the exact command-line arguments, and what IP address it connected to.

Practical Sysmon Configuration (SwiftOnSecurity Baseline)

Never run Sysmon without a tuned configuration file, or you will exhaust endpoint resources and flood your SIEM with noise. Start with the industry-standard SwiftOnSecurity configuration and tune it to your environment.

<!-- Example: Catching Suspicious Process Creations (Event ID 1) -->
<RuleGroup name="" groupRelation="or">
  <ProcessCreate onmatch="include">
    <!-- Catch LOLBins -->
    <OriginalFileName condition="is">rundll32.exe</OriginalFileName>
    <OriginalFileName condition="is">regsvr32.exe</OriginalFileName>
    <OriginalFileName condition="is">certutil.exe</OriginalFileName>
    <!-- Catch suspicious shell spawns -->
    <ParentImage condition="image">winword.exe</ParentImage>
    <ParentImage condition="image">excel.exe</ParentImage>
  </ProcessCreate>
</RuleGroup>

<!-- Example: Catching LSASS Memory Dumps (Event ID 10) -->
<RuleGroup name="" groupRelation="include">
  <ProcessAccess onmatch="include">
    <TargetImage condition="is">C:\Windows\system32\lsass.exe</TargetImage>
    <GrantedAccess condition="is">0x1410</GrantedAccess>
  </ProcessAccess>
</RuleGroup>

Deployment: Deploy Sysmon silently via GPO or SCCM: sysmon.exe -accepteula -i sysmonconfig.xml

Log Shipping: Use Winlogbeat to read the Microsoft-Windows-Sysmon/Operational event log and ship it natively to OpenSearch.

---

3. OpenSearch: Anomaly Detection & Alerting

With Trellix handling known threats and Sysmon logging deep telemetry, OpenSearch acts as the brain. Instead of writing hundreds of static rules, we use OpenSearch’s built-in Random Cut Forest (RCF) Machine Learning algorithms.

Setting up Practical Anomaly Detectors

Static rules fail when attack patterns change. Anomaly detection establishes a baseline of normal behavior and flags significant deviations.

Detector 1: Unusual Process Execution Volume

• Index: winlogbeat-*
• Feature Field: Count of winlog.event_data.Image
• Filter: winlog.event_id: 1 (Process Creation)
• Why it works: Ransomware or wiper malware generates a massive spike in process creations/terminations in seconds. The ML model instantly flags the deviation from the endpoint’s baseline.

Detector 2: Rare Network Connections from LOLBins

• Index: winlogbeat-*
• Feature Field: winlog.event_data.DestinationIp cardinality
• Filter: winlog.event_id: 3 (Network Connection) AND winlog.event_data.Image: *powershell.exe
• Why it works: PowerShell making network connections isn’t inherently malicious, but if it suddenly connects to a never-before-seen foreign IP address, the ML model will trigger a high-severity alert.

Rapid Triage Workflow

When an anomaly triggers:

1. Investigate in OpenSearch Dashboards: The analyst reviews the anomaly. OpenSearch highlights exactly which feature (e.g., the specific command-line argument) contributed to the anomaly score.
2. Review Trellix Context: The analyst queries OpenSearch for any concurrent Trellix events for that hostname.
3. Take Action: If confirmed malicious, the analyst isolates the host via Trellix ePO.

---

Conclusion

This hybrid stack represents the “sweet spot” for modern SOCs. By leveraging Trellix for robust prevention, augmenting it with Sysmon’s surgical visibility, and applying OpenSearch’s machine learning, teams can detect highly sophisticated, evasive threats without drowning in the configuration debt of complex SIEM rules engine tuning.