Real-Time Anomaly Detection With Kafka and Faust: From Stream to Slack Alert in Under 2 Seconds

Last Updated on July 23, 2026 by Editorial Team Author(s): EMMANUEL NWANGUMA Originally published on Towards AI. There’s a category of problem where being right tomorrow is the same as being wrong. A fraudulent transaction clears. A server starts throwing errors at 2pm and nobody notices until the morning report. A sensor drifts out of spec and the machine it’s attached to grinds itself apart over six hours. In every one of those cases the detection logic might be perfect — but if it runs as a nightly batch job, the answer arrives after the damage. So I built the opposite: a streaming pipeline where events flow in continuously, get scored the moment they arrive, and turn into a Slack alert in under two seconds. It handles three genuinely different data types — card transactions, server metrics, and IoT sensor readings — on one pipeline. Along the way I found two bugs that had my LSTM detector performing at 4% recall, and the fix for the second one had nothing to do with the model at all. More on that below. Why batch is the wrong shape for this problem The instinct is to treat anomaly detection as a data science problem: get data, train model, evaluate, ship. But in production it’s mostly a systems problem. Three things matter more than the model: Latency — how long between the event happening and a human knowing. Noise — whether the alerts are still worth reading after a week. Adaptability — whether you can change the detector without taking the system down. A batch job fails all three. It’s slow by construction, it dumps a pile of findings with no grouping, and updating it means a redeploy. The pipeline Data sources: transactions, server metrics, IoT sensors │ ▼ Redpanda topics (Kafka API) anomaly.fraud / anomaly.metrics / anomaly.iot │ ▼ Faust stream processor ├── rolling windows (1m / 5m / 1h, per entity) ├── route event type → detector(s) └── real-time inference │ ┌─────────────────┴─────────────────┐ ▼ ▼ Detection models TimescaleDB ├── Isolation Forest (fraud) (events + flags, ├── LSTM Autoencoder (IoT) hypertables) └── Z-score / EWMA (metrics) │ │ ▼ ▼ Grafana dashboard Alert engine ├── severity scoring ├── deduplication └── Slack + email Redpanda gives me the Kafka API without the JVM. Faust does the stream processing in Python. TimescaleDB stores everything as hypertables so time-bucketed queries stay fast. Grafana reads both TimescaleDB and Prometheus. Rolling windows, and why they’re per-entity A single event usually isn’t enough to judge anything. A £2,000 transaction is unremarkable — unless that card has already made eleven transactions in the last hour. So the stream keeps rolling windows (1 minute, 5 minutes, 1 hour) and derives counts, means, standard deviations, and deltas on top of the raw fields. The subtle part is the key. Windows are kept per source:entity, not per entity: window_key = f"{source}:{event.entity_id}"self._features.add(window_key, event) I found this the hard way. My first version keyed windows by entity_id alone, and a test that reused the same ID across two source types blew up with a KeyError. A server's window had been filled with fraud features. Scoping by source makes the collision structurally impossible rather than merely unlikely. Three detectors, three different jobs Routing is per source type: ROUTING = { "fraud": ["isolation_forest", "zscore"], "metrics": ["zscore", "ewma", "isolation_forest"], "iot": ["lstm_autoencoder", "zscore"],} Z-score / EWMA for server metrics. They track a running mean and standard deviation per feature and flag deviations. No training run, no model file, cheap enough to run inline on every event. For high-volume metrics where “normal” is a stable band, this is genuinely hard to beat. Isolation Forest for fraud. Fraud rarely looks wrong on any single dimension — it’s the combination that’s off. A large amount is fine. A foreign transaction is fine. A 3am transaction is fine. All three together on a card that’s already been used eleven times this hour is not. Isolation Forest handles that interaction; a per-feature threshold never will. LSTM Autoencoder for IoT. Sensors produce sequences, and the anomaly is often a pattern rather than a value — a temperature that’s climbing at the wrong rate is a problem long before it crosses any single threshold. The autoencoder learns to reconstruct a window of normal readings; when reconstruction error spikes, the pattern is off. That last one is where things got interesting. Bug #1: the autoencoder that couldn’t detect anything My first backtest of the LSTM came back with 4% recall. It was catching essentially nothing. The cause was in one line of my training setup: I was training the autoencoder on the full labeled dataset — which included the anomalies. An autoencoder detects anomalies by learning to reconstruct normal data well, then flagging inputs it reconstructs badly. The detection threshold is set at, say, the 99th percentile of reconstruction error observed during training. But if 5% of your training data is anomalous, those anomalies produce the largest reconstruction errors, and they drag the 99th-percentile threshold up to their own level. You end up with a threshold that only the most extreme anomalies could ever exceed. The fix is one line, and it’s a methodological rule rather than a tuning trick: # Autoencoders must train on NORMAL data only — training on the# contaminated set pushes the reconstruction-error threshold up to the# anomalies themselves and collapses recall.normal_events = [e for e, lab in zip(events, y_true) if lab == 0]det = train_lstm_autoencoder(normal_events, fn, seq_len=seq_len, epochs=10) Recall went from 0.04 to 1.00. Bug #2: the model was fine, my evaluation was wrong With recall fixed, precision came back at 0.14. The detector was now flagging roughly seven times more windows than there were anomalies. I nearly started tuning the threshold. Then I looked at how I was scoring it. The autoencoder consumes a window of 10 events and produces one verdict about that window. I was comparing that verdict against the label of the last event in the window only. With a 5% anomaly rate and a 10-event window, […]