Tristan’s Cybersecurity Substack
Tristan’s Cybersecurity Substack Podcast
SIEM 201 — Automating Threat Response: A Practical Guide
0:00
-16:58

SIEM 201 — Automating Threat Response: A Practical Guide

Introduction — The Case for Automated Response

If your SIEM only sends alerts, you're leaving value—and security—on the table.

In every incident I’ve investigated, there’s a window between detection and response that determines the scale of impact. In some cases, this delay is a matter of hours, sometimes minutes. But in fast-moving attacks like ransomware deployment, credential stuffing, or lateral movement, seconds might be all it takes to go from contained to compromised.

That’s where automated response comes in.

Rather than waiting for someone to read a Slack message or open their inbox, a well-placed automation can block an attacker IP, disable a vulnerable user, or trigger a workflow instantly. When configured right, automation gives defenders their time back—and gives attackers one chance less.

In this post, I’ll show you how to implement effective, low-risk automated actions in your SIEM. I’ll walk through examples like blocking brute force source IPs, sending adaptive notifications, and integrating webhooks. You don’t need a full SOAR platform to take meaningful action. You just need a reliable alert, a clear goal, and good guardrails.

Let’s explore what’s possible when your alerts don’t just end in a ticket—they end in action.

Tristan’s Cybersecurity Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.


What You Can Automate (and Why It Helps)

Automated response in your SIEM doesn’t have to mean deploying a full SOAR platform. Even with basic webhook support, you can trigger meaningful actions when alerts match the right conditions. Here's a breakdown of the most common types of automated actions—each with practical benefits and varying levels of risk.


1. Send a Notification (Email, Slack, Teams)

🟢 Low-friction, low-risk

This is the simplest form of automation.

Your SIEM detects something unusual—like a user logging in from a suspicious IP—and sends a message to your Slack channel or a dedicated security inbox.

This doesn't stop anything yet, but it dramatically shortens response time by reaching humans faster (and in the right place).

👉 Example:

  • Alert: Office login from TOR node

  • Action: Send message to #sec-team on Slack with a link to verify user


2. Trigger a Webhook or API Call

🟡 Medium complexity, highly flexible

If your SIEM supports webhooks or external HTTP calls, you can trigger actions beyond notifications. For instance, you could call a small API you wrote that places an IP in your firewall’s blocklist, or notify a central webhook handler in your detection pipeline.

👉 Example:

  • Alert: Multiple failed SSH attempts from a single IP

  • Action: Send that IP to an internal API that appends it to iptables ban list via fail2ban

(This is the type I use most, and I’ll walk through it later in the blog.)


3. Execute a Script or Runbook

🟠 Higher risk, more automation

Some systems can run a script directly in response to an alert—especially if you’re using a SIEM combined with an EDR or agent on the machine. This could restart a process, disable a user account, or isolate a host.

While powerful, scripts need rigorous testing. Misfires can create outages.

👉 Example:

  • Alert: PsExec lateral movement detected

  • Action: Run PowerShell to isolate the machine from the domain

💡 Tip: These are best combined with tagging or human-in-the-loop review at first.


4. Push to a Ticketing System or IR Workflow Tool

🟢 Low-to-medium risk, includes humans

Another useful automation is pushing summarized alert data into your incident response or case management platform (like Jira, TheHive, PagerDuty, etc.). This ensures follow-up, adds context, and lets responders add manual judgment without missing the event.

👉 Example:

  • Alert: Phishing link clicked

  • Action: Open case in TheHive with log URLs, domain observed, user’s IP


Real-World Use Cases (That Aren’t Too Risky)

Getting started with SIEM automation doesn’t mean plugging your alert pipeline straight into production-blocking scripts. That’s a great way to lock yourself out of an important service—or worse, cause a false positive outage. I like to start with low-friction, high-confidence alerts with clear outcomes.

Here are some use cases I’ve implemented or advised others to adopt first:


🚫 Block Brute-Force IPs Automatically

Brute force attempts are a perfect starter for automation because they’re noisy, repetitive, and usually easy to fingerprint.

🧠 Trigger:

An alert that detects 10+ failed SSH or Windows logon attempts from the same IP within 5 minutes.

⚙️ Action:

Send a webhook to a lightweight API receiving bad IPs, which in turn updates your firewall, fail2ban, or a zero-trust ACL.

✅ Why It Works:

  • You can whitelist trusted IP ranges up front

  • Blocking via firewall doesn’t affect internal systems

  • These attacks are usually automated and follow patterns


🌍 GeoIP + Impossible Travel Notifications

User logs in from Toronto. Three minutes later, they try accessing a sensitive application from Singapore. Unless they have a teleportation pod, something's wrong.

🧠 Trigger:

Alert when two logins are detected from geographically distant locations within an impossible timeframe.

⚙️ Action:

  • Automatically send a Slack message or email to the affected user’s manager

  • Optionally open a ticket in the incident response system

✅ Why It Works:

  • No impact, just visibility

  • Helps build context if ongoing investigation is needed


🧪 Tagging or Labeling High-Risk Hosts

Not all automations need to stop attackers directly. You can enrich telemetry by tagging things as risky the moment they're detected.

🧠 Trigger:

Host exhibits lateral movement behavior (e.g., PsExec or WMIExec detection via Sysmon)

Or: Endpoint shows signs of crypto-ransomware behavior

⚙️ Action:

Attach a tag to the host in your log system or asset tracker

Or: write an enrichment flag in Sentinel/Humio to surface in dashboards

✅ Why It Works:

  • Non-destructive

  • Makes hunting and triage faster

  • Builds starting points for weekly reviews or ML training


These ideas are deliberately safe to start with. As you implement these and trust their accuracy, you can progressively increase risk tolerance and expand to more disruptive actions (like user disabling or host isolation).


Hands-On Example: Blocking an IP from a Humio Alert

Let’s walk through a real-world example: a brute force detection alert that automatically blocks the source IP by sending it to a small script via webhook.

This approach works with any SIEM that supports HTTP-based alert actions—Humio makes it simple, but you could do the same with Logz.io, Graylog, or even a homegrown setup.


🎯 The Goal

  • Detect failed login attempts coming from a single IP more than 10 times in 5 minutes.

  • When that happens, send the IP to a webhook endpoint.

  • The webhook triggers a script to block the IP on the firewall.

  • The action is logged, monitored, and reversible.

Low risk, high reward.


🗂 Step 1: Create the Detection Query in Humio

In Humio or your SIEM, build the alert like this.

Query Example (Sysmon/Winlogbeat logs):

event.code=4625 
| groupby([beat_agent.hostname, winlog.event_data.IpAddress], function=count()) 
| where([count > 10]) 

This detects 10+ failed login attempts from the same source IP on the same host.

🔍 Tips:

  • Avoid internal IPs or your own jump-boxes by filtering with where(not winlog.event_data.IpAddress in (your allowlist))

  • Tune down to 5 attempts in testing, tune up to 20+ in prod


⚠️ Step 2: Configure the Alert Action

In Humio:

  • Go to Alerts

  • Choose your query, click “Create Alert”

  • Set threshold = 1 (fires whenever there’s a matching group)

  • Throttle: >5 mins per host+IP tuple

Action Setup (Webhook):

  • Type: “HTTP POST”

  • URL: https://your-automation-host/api/block-ip

  • Payload:

{
  "hostname": "{{beat_agent.hostname}}",
  "ip": "{{winlog.event_data.IpAddress}}"
}

✔️ Set content-type as application/json

✔️ Secure it with an API key in the header, or only allow network-restricted source IPs


🛠 Step 3: Write the Webhook Script

Here’s a very simple example in Python (Flask-based API):

from flask import Flask, request, jsonify
import subprocess
import logging

app = Flask(__name__)
logging.basicConfig(filename='ipblocks.log', level=logging.INFO)

@app.route('/api/block-ip', methods=['POST'])
def block_ip():
    data = request.json
    ip = data.get('ip')

    # Validate the IP (simple check)
    # This is not enough to safeguards against RCE.
    # TODO: add more checks!
    # This is only an example!
    if not ip or '..' in ip or '/' in ip or ';' in ip or '&' in ip or '|' in ip:
        return jsonify({"error": "invalid IP"}), 400

    # Block the IP using iptables (Linux)
    # One would imagine a CTF challenge here!
    cmd = f"iptables -A INPUT -s {ip} -j DROP"
    subprocess.run(cmd, shell=True)

    logging.info(f"Blocked IP {ip} via automation")
    return jsonify({"status": "blocked", "ip": ip})

💡 Save and run this on a secured automation server. Rotate IPs out after X days or add removal logic.

Alternatives:

  • Use UFW, fail2ban custom chains, or cloud firewall/blocklist APIs (e.g., AWS, Cloudflare)

  • If you don’t want actual blocking: just log or simulate the response action!


🔎 Step 4: Monitor & Tune

Now that you’ve automated your first response, be sure to:

  • Log every action (separate audit file or SIEM event)

  • Monitor alerts for false positives—adjust thresholds

  • Use alert “Preview” and dry runs to test

  • Sanitize inputs and control webhook access

Optional additions:

  • Store blocked IPs in SQLite or Redis to allow removal later

  • Add Slack/Email alerts from the script for visibility


🔐 Bonus: Add a Safety Switch

Build a toggle in the script (“MAINTENANCE_MODE=True”) that allows you to disable enforcement instantly. Or only allow from specific alert sources (e.g., trusted Humio instance UUIDs).


🎉 Boom — you’ve now built an early warning system that defends itself.

Up next, we’ll talk about how to scale this approach and avoid common mistakes.


Best Practices & Guardrails

Before you get too deep into automating blockade systems and forensic scripts, let’s cover the most important part of any automation effort: treating it like production code.

Automated actions affect your systems—sometimes permanently. Without proper guardrails, a bug in detection or a misfiring rule could put your security team in the spotlight for all the wrong reasons. Here's how to do it right.


✅ 1. Start with Notifications, Then Actions

The temptation to go straight from alert to action is real—but don’t skip the dry run phase.

Always begin by triggering a notification or logging-only action, and monitor the alert fidelity over time. Only after verifying that your false positive rate is low should you consider moving to blocking or remediation automation.

📝 Pro tip:

“Log + Notify + Review” for a week before enabling the final blocking logic.


🛡 2. Whitelist Known Good Behavior

Always protect critical systems, known testers, and internal IPs from being impacted by automated actions unless absolutely required. You’ll want to include a whitelist—either inline or backend-controlled—which the automation must check before enforcement.

Examples:

  • Don’t block your VPN endpoint

  • Don’t isolate your own monitoring agent

  • Don’t trigger changes on development or staging logs


🧪 3. Log Everything the Automation Does

If your automation blocks an IP, disables a user, or restarts a service, you need a clear audit trail of:

  • What triggered the action

  • When it ran

  • What was actually done

  • Who or what performed it

  • Whether it succeeded or failed

📁 Tip: Log all of this to your SIEM itself.

Bonus: Create a dashboard of recent automation actions to catch errors early.


🔁 4. Add Rate Limiting and Throttling

Just because 50 alerts fire at once doesn’t mean you want 50 API calls and actions firing simultaneously.

Add throttling—or use your SIEM’s built-in alert throttle—to avoid alert storms. This ensures you don’t overwhelm your firewall with too many rule changes or bounce through rate limits on an external system.

📏 Example:

Only execute the action once per unique IP per hour, or once per system per 10 minutes.


🔂 5. Make It Reversible (and Pauseable)

You WILL make a mistake. Your automation might react to a misparsed event or a data ingestion bug.

Plan for rollback. Here’s how much:

  • Can you remove a blocked IP with a single call?

  • Is there a safe list to re-allow access?

  • Can you “pause” automation with one toggle if you’re under pressure?

Most mistakes in security are made in fear. Don’t let your automation increase panic—build kill switches and clear recovery steps.


Automating threat response means trusting your SIEM’s alerts with real power. These best practices are like a flight checklist: they ensure it flies where and when you want it to—and lands safely when it’s done.

Up next, let’s look at the typical automation pitfalls I see in the wild—and how you can avoid them.


Common Pitfalls to Avoid

When I talk to teams about SIEM automation, they’re usually excited. And with good reason—automation turns alerts into actions and allows teams to scale. But I’ve also seen well-meaning automation wreak havoc because a few basic rules were overlooked.

Here are the most common mistakes I see when teams launch automated responses too early—or without thinking through the consequences.


🔁 Acting on Noisy Alerts

This is the classic mistake: building automations on top of alerts that still generate false positives. If your brute force alert is misfiring, feeding it into an auto-blocking script will just create chaos.

You might end up:

  • Blocking your own security scanner

  • Disabling perfectly legitimate logins

  • Filling your firewall ruleset unnecessarily

🛑 Fix it first: Make sure your detection logic is solid, tuned, and meaningful before giving it power.

💡 Pro tip: Review fired alerts over 7–14 days before moving to response.


🌪 Forgetting Alert Throttling

Without rate limiting, a flood of log-ins from one device, a SIEM parsing bug, or log ingestion issue could trigger hundreds of simultaneous actions.

I once saw a mis-parsed Linux event trigger 300+ alerts in under 60 seconds—which then triggered an overload in the iptables automation service.

✔ Add guardrails:

  • Alert throttling (native in many SIEMs)

  • IP block counters (rate-limits per source)

  • Timeout queues on your action scripts


👁️ No Visibility on What’s Happening

If you're not logging the actions taken—or you’re only logging success—you’re working blind.

Imagine:

  • You blocked 42 IPs… but which ones?

  • A script failed to run… was it retried?

  • Someone paused your webhook host… and now no action is running

🌟 Always log:

  • What triggered the action (alert details)

  • What was attempted (API call, command)

  • The result (success/failure)

Put this in a shared log store, or even feed “meta events” to the SIEM itself.


🔒 No Authentication or Input Validation

If your webhook is listening for JSON from anywhere—make sure it's authentic.

Imagine an attacker figures out your webhook URL… and submits their own IP with your same automation call. Or worse, they inject shell code as part of a manipulated payload.

Secure your endpoints:

  • API keys or HMAC validation

  • Restrict source IP to the SIEM only

  • Sanitize all fields before running commands

If you're passing variables into a shell script, validate and escape everything.


🧨 Over-Automating Detonation

Some teams go all-in: "If this alert fires, disconnect the user, disable the account, and terminate the EC2 instance."

That’s tempting, but only safe at maturity. Without escalation logic, human review, or an understanding of business impact, automation might react more aggressively than intended.

Instead:

  • Automate gently (alert, log, escalate)

  • Use tagging, enrichment, and visibility automation

  • Add human-in-the-loop steps before destructive actions

You can scale trust with automation—but only if everyone trusts it over time.


TL;DR — Don’t Put a Bazooka on an Alert Until it Knows What It’s Aiming At

Automation is powerful—but only when it obeys the rules of reliability, auditability, and control. Treat your response logic like a developer treats production code: observability, rollback, and safety nets should always be included.


Where to Go From Here

You’ve built one automation, logged its behavior, and avoided tripping your own firewalls—great work. Now what?

One of the strengths of building lean, scriptable SIEM automations is that you can grow them gradually. The goal isn’t to automate everything. It’s to automate the right things consistently and recoverably.

Here’s how to take your setup further:


🧰 1. Expand Your Webhook Handler

Don’t let your blocking script do just one thing. You can support more actions using conditional logic:

  • Block an IP

  • Put a tag in a shared asset tracker

  • Write a note into a shared Google Sheet or case system

  • Notify an MDR or response team externally

  • Populate a blocklist file or feed

Small automations like this scale well and create glue between tools.

🛠 Tip: Build modularly — write clean, multi-source-capable input handlers and you won’t need to rebuild later.


🎩 2. Integrate with Your Ticketing or Workflow Tool

If you’re not ready to block things automatically, pipe your alert context into your response platform.

Examples:

  • Create a Jira ticket for phishing-link clicks

  • Trigger a case in TheHive for suspicious process chains

  • Send markdown-formatted incident summaries to Mattermost

This acts as “automated assurance”—you won’t forget the incident, and responders can add human judgment if needed.


🔄 3. Enrich Alerts — Not Just Respond

Your SIEM should grow more informed over time.

When an alert fires, automation doesn’t have to only take action. It can:

  • Append GeoIP context based on source IP

  • Mark an asset’s risk score in the SIEM

  • Add domain reputation or VirusTotal score

  • Link to previous similar alerts

Automation = context at machine speed.


🧠 4. Create a Lightweight “SOAR Without SOAR”

If you’re not buying a full commercial SOAR (Security Orchestration, Automation and Response) platform, you’re not alone.

With the right stack, you can build your own SOAR-lite with:

  • Your SIEM alerting engine (Humio, Elasticsearch, etc.)

  • A webhook layer / action handler (Flask, Lambda, etc.)

  • Basic storage (SQLite, Google Sheets, JSON-logs)

  • Dashboards or markdown playbooks

This gives you enough control and extensibility for most use cases—especially in a small or mid-size team.

📝My blog used this model before the move to Substack. Works well. Cost? $0.


🔭 5. Experiment, Measure, and Refine

Great SIEM automations are never built in one pass. I recommend:

  • Logging every action, even if it “does nothing”

  • Holding monthly reviews of automation performance (e.g. top 10 triggered rules)

  • Keeping a chart of false positives vs. enforcement success

  • Versioning your detection + automation rules together

This turns security automation into a feedback loop—not just a trigger.


Conclusion

Security automation isn’t reserved for large enterprises or teams with a SOAR budget. You don’t need to over-invest, overbuild, or over-trust untested systems.

With just a few hours and the right SIEM alert, you can build useful automations that:

  • Respond faster to obvious threats (like brute-force attempts)

  • Help defenders focus on high-value work

  • Reduce fatigue from repetitive incidents

Whether you're defending your home lab, managing detection pipelines for clients, or running SecOps in a growing organization, your SIEM can do more than notify—it can act.

Start with one use case. Get it right. Monitor its behavior. And when you’re confident, scale it up to your next alert. You’ll be amazed at how quickly “just an alert” becomes something that actually helps you sleep at night.

If you’d like to explore more, I’ve also written about detecting PsExec lateral movement, configuring alert thresholds, or tuning SIEM correlation rules to reduce false positives.

☑ Build one automation. Review. Repeat.


🎯 Want more hands-on detection walkthroughs and security strategies?

to get the next blog posts as I publish them!

💬 Did you try an automation and learn something from it? Share it in the comments! I’d love to hear it.

Leave a comment

📬 Need help building detection + action pipelines for your team? I run detection engineering for MDR platforms and help teams implement lightweight but effective automation. Feel free to reach out.

🧡 Donation

If you like my blog and my posts, please consider donating!

Imagine this: each minute of reading a post takes about 1–2 hours to create!

Donate 🧡

Discussion about this episode