Phishing Detection with Explainable Machine Learning

A tier-1 analyst opens their queue and the top item is a vendor appliance verdict: MALICIOUS, confidence high, no reasoning attached. So they open the email and investigate it from scratch anyway — because a label with no evidence behind it is a label you can’t act on. The appliance didn’t save the analyst any work. It just moved the item to the top of the pile and told them to trust it. That’s the failure this solution is built around.

The Phishing Detection with Explainable Machine Learning approach is grounded in one principle: the model has to show its working. It uses a Random Forest rather than a black-box neural network specifically because a forest can tell you which features drove the decision. Instead of a bare score, the analyst sees the reasoning — reply-to doesn’t match the sender, the language leans on manufactured urgency, the sending domain was registered yesterday. Now there’s something to act on, or to overrule.

One honest caveat before anything else: a Random Forest’s predicted probability is a vote fraction across trees, not a calibrated likelihood. An “89%” is 89% of the trees agreeing, not an 89% chance the mail is phishing, and those two numbers can diverge sharply. It’s fine for ranking a queue; treat it as a true probability when you set blocking thresholds and you’ll be wrong in ways nobody notices until a legitimate invoice gets quarantined.

Why Behavior Matters More Than Signatures

Signature-based detection loses this race by design. Attackers rotate domains and sending IPs faster than blocklists update — a domain registered this morning sends its campaign this afternoon and is abandoned by tomorrow. Chase the infrastructure and you’re always a day behind. The behaviour, though, is stubborn: phishing works by pairing psychological pressure (urgency, fear, authority) with structural giveaways (broken headers, a mismatched sender, a freshly minted domain). The attacker can change the domain in seconds. Changing the shape of the manipulation is much harder, because the manipulation is the product.

A Random Forest fits this problem because it stays interpretable at useful accuracy. A deep neural network might edge it on raw score, but it can’t tell you why it flagged anything — and a detection your analysts can’t reason about is one they’ll eventually stop trusting and start clicking through. The trade-off is worth stating plainly: you give up a little headline accuracy for the ability to explain every decision. In a SOC, a slightly-less-accurate model people act on beats a better one they ignore.

Feature Engineering: What Actually Gets Checked

The model reads each message from several angles at once. One thing to keep in mind across all of them: these feature families are correlated — a newly registered domain, a reply-to mismatch and an unfamiliar sending IP tend to travel together in the same campaign. That correlation quietly distorts impurity-based feature importance, so a genuinely strong signal can look unimportant simply because a correlated twin absorbed the credit. It matters when you’re deciding which features to trust, and it’s the reason SHAP (below) earns its place.

Email Headers Does the Reply-To match the sender? Do DKIM and SPF pass, and does DMARC actually align — not just pass one check while the visible From says something else? Is the routing path plausible? Header inconsistencies are among the most reliable structural tells, precisely because they’re awkward for the attacker to fake when they don’t own the spoofed domain.

Sender Behaviour Has this address emailed you before? Is it arriving at an odd hour — 3 a.m. from someone nominally in finance? Is this a genuine first contact? Legitimate correspondence has rhythm. Phishing breaks the rhythm because the sender has no history to imitate.

Language & Linguistics NLP scans for manufactured urgency (“immediate action required”), financial framing (“invoice attached”), and phrasing that doesn’t sound like your organisation. Worth being careful here: “phishers aren’t native speakers” was true a decade ago and is now a weak signal — LLM-generated lures are fluent, so lean on the urgency and structure, not the grammar.

URL Patterns Long URLs stuffed with dots and hyphens, raw IP links instead of hostnames, typosquats like amaz0n.com. The URL structure carries signal even when the destination domain is brand new.

Impersonation & Display Names Display name reads “IT Helpdesk” while the actual address is a consumer webmail account or a lookalike domain. Real internal mail comes from real internal addresses; the gap between the name someone reads and the address the mail client hides is a favourite lever.

Attachments Double extensions like .pdf.exe, macro-bearing Office documents, password-protected archives (encrypted so the gateway can’t scan them), odd file metadata. The attachment is often the whole point, and everything else in the message exists to get it opened.

Explainability: The Game Changer

When the model flags a message it hands over the reasons, not just a verdict. Instead of:

Alert: Malicious Email

the analyst gets:

Confidence: 89% | Urgency_Language detected | Reply-To mismatch | Domain < 48 hours old

That’s the difference between a ten-minute investigation and a ten-second one. Does this sender normally write with this kind of urgency? Is a reply-to mismatch usual for them? Is the domain age genuinely concerning here? The features frame the questions the analyst already knows how to answer. The score orders the queue; the features let them clear it.

The transparency is also what earns the tool a place in the workflow at all. Nobody stakes their shift on a black box they can’t interrogate — the first time it’s wrong and won’t say why, they stop trusting it and go back to reading every mail by hand. A system that shows its reasoning survives being wrong, because the analyst can see exactly where it went wrong and correct for it.

How It Works: Step by Step

1. Data Preparation Collect years of historical corporate mail labelled legitimate or phishing, and combine it with open-source phishing corpora. Sanitise it so you’re not training on real passwords or account numbers. The label quality is the whole game here — and the uncomfortable truth is that your “phishing” labels come almost entirely from attacks your existing controls already caught, which means the training set systematically excludes the mail that got through. User-reported phishing is the highest-value label source precisely because it captures what the gateway missed.

2. Feature Extraction Python that parses EML files and pulls the features above — headers, language, URLs, attachment metadata. Automate it so the exact same extraction runs in training and in production; a feature computed one way offline and another way live is the quiet source of a model that scored beautifully in the notebook and does nothing in deployment.

3. Training the Model Train the Random Forest on the labelled data and tune it to hold false positives down, because a legitimate mail blocked is a business disruption someone will trace back to you. Split the data by time, not at random — a random split scatters members of the same campaign across train and test, leaks the answer, and inflates every metric you report. And be honest about the corpus: an F1 above 0.98 on a balanced 50/50 dataset is an artefact of the balance, not a production expectation. Real inbound mail is a fraction of a percent phishing, and the model will behave very differently against that ratio.

4. Deploy as an API Expose the model behind an HTTP endpoint. When the gateway sees a borderline message it posts the mail over and gets back a score plus the contributing features.

5. Analyst Review The output lands in your SIEM/SOAR — Splunk or ELK, TheHive or Shuffle. The analyst sees the score and the features and makes the call. The human stays in the loop by design, not as a courtesy: this is a ranking aid, not an autonomous blocker.

6. Continuous Improvement Every analyst decision feeds back into the training set. A “false positive” verdict is a labelled example. Over time the model adapts to your organisation’s own patterns — but watch for the failure mode this hides: drift degrades recall while precision holds steady, so more phishing slips through while every dashboard stays green. Monitor recall against fresh user-reported catches, not just the metrics the model reports about itself.

The Tech Stack

Data Science Python with Scikit-learn for the Random Forest, Pandas for wrangling, NumPy underneath, Jupyter for the experimentation phase. Nothing exotic — the point is that the whole stack is open-source and inspectable, so there’s no vendor black box and no licence renewal holding your detection logic hostage.

Feature Engineering NLTK or spaCy for the language features — urgency, semantic patterns — plus custom Python to parse headers and pull metadata. The header parsing is where most of the reliable signal lives, so it’s worth more care than the NLP.

SOC Integration The model slots into Splunk or ELK on the SIEM side and TheHive or Shuffle on the SOAR side, running as a Python API between the email gateway and the alert pipeline. Keep it advisory at first. If you ever wire it to automated retroactive purge — pulling mail from inboxes on a model score — that capability needs a blast-radius limit and a one-step reversal by someone on shift, because a miscalibrated threshold turns “delete this phish” into “delete the quarterly invoice from everyone’s inbox”.

Evaluation Metrics

Vendor marketing quotes one big number, usually accuracy, which is meaningless on imbalanced data — a model that flags nothing scores 99%+ accuracy when phishing is under 1% of mail. Here’s what actually tells you whether it works:

Metric	What It Means	Why It Matters
Precision	Of all emails we flagged as phishing, how many really were?	High precision = analysts aren’t wasting time on false alarms.
Recall	Of all actual phishing emails in the inbox, how many did we catch?	High recall = sophisticated attacks don’t slip through.
F1 Score	The balanced score between precision and recall.	This is usually the single number you optimize for.
False Positive Rate (FPR)	How often we block legitimate emails.	This must stay low or the business will disable the system.

90-Day Implementation Roadmap

Days 1–30: Data Collection & Baseline Gather and label 10,000+ corporate emails, legitimate and phishing. Sanitise out real passwords and account data. Build the extraction pipeline. Then train a plain logistic regression first as a baseline — if the Random Forest can’t beat a linear model by a meaningful margin, the extra complexity isn’t earning its keep, and you want to know that on day 30, not day 90.

Days 31–60: Train the Model & Explain It Train the Random Forest. Use SHAP (SHapley Additive exPlanations) for feature attribution rather than the built-in impurity importance — SHAP is far more robust to the correlated-feature problem flagged earlier, which is exactly the problem this dataset has. Tune hyperparameters, but treat any F1 above ~0.95 on a balanced set as a lab figure, not a promise, and re-measure on a realistically imbalanced hold-out before you believe it.

Days 61–90: Shadow Mode & Integration Run the model against live mail in shadow mode — predict, log, block nothing. Route the explainable outputs into TheHive so analysts can see what it would have caught and, just as importantly, what it would have wrongly caught. Gather false-positive feedback, adjust thresholds against real traffic, and only switch to active blocking once the false-positive rate is low enough that the business won’t disable the system the first Monday an invoice goes missing.

The Bottom Line Machine learning here is applied statistics, not magic, and treating it that way is what keeps it useful. Choosing an interpretable model over a black box costs a little accuracy and buys a detection your team will actually act on, argue with, and correct. It also does not, on its own, solve phishing — it ranks a queue faster and gives analysts a reason for each verdict. Pair it with user reporting, DMARC enforcement and awareness training, keep a human on the trigger, and it earns its place. Sell it as autonomous protection and it won’t survive its first bad week.