Explainable Network Anomaly Detection

Summarize network anomalies with evidence in under one minute

A data-product case that moves from PCAP/CSV ingestion to LightGBM classification, calibrated probabilities, Unknown routing, SHAP/FAISS evidence, and PDF/web reporting.

LightGBMSHAPFAISSFastAPIReportLab

KPI/ROI scenarios are conservative estimates based on the public sources listed below.

How It Works

  1. PCAP/CSV IntakeCollect flow-level inputs from files or pipelines.
  2. Feature AlignmentNormalize schemas and guarantee model-ready inputs.
  3. LGBM + CalibrationGenerate calibrated probabilities through temperature scaling.
  4. Unknown RoutingRoute low-confidence events into a safe triage queue.
  5. SHAP / FAISS / KBExplain “why” using top factors, similar flows, and KB snippets.
  6. SummaryProduce operator-readable summaries from evidence.
  7. ReportGenerate PDF/web reports with logs and evidence.

KPI & ROI

Detection Quality

  • Calibrated probability and threshold-based Unknown routing.
  • SHAP top-K reasons and FAISS similar-case retrieval.
  • Model output is paired with evidence before reporting.

Operational Time

  • Flow analysis, explanation, summary, and report in one path.
  • Analyst review starts from evidence instead of raw alerts.
  • Quarterly alert-triage improvement can be tracked as a KPI.

ROI Basis

  • KOSA hourly wage baseline used for labor-cost scenarios.
  • Scenario values should be replaced by real alert/report logs.
  • Designed for security-team estimation, not fixed financial claims.

Core Features

Probability Calibration

Temperature scaling provides confidence probabilities and Unknown routing.

Explainability

SHAP top-K, FAISS similar cases, and KB snippets make the reason visible.

Product Surface

FastAPI /analyze, Gradio UI, and ReportLab PDF reporting.

Role: Designed the KPI/ROI frame, evidence links, report draft templates, evidence panel, and downstream report flow.

API Sketch

POST /analyze (JSON)

{ "features": { "duration": 12.1, "bytes": 1024 }, "top_k": 5 }

POST /analyze (file upload)

curl -X POST http://localhost:8000/analyze \
  -F "file=@sample.pcap"

Module Summary

serve/api.py

  • POST /analyze
  • request validation and report handles

serve/model.py

  • LightGBM loader and temperature scaling
  • calibrated probabilities

serve/router.py

  • Unknown routing
  • thresholds, fallbacks, and cache

serve/explain.py

  • SHAP top-K and FAISS similar cases
  • KB snippets

serve/report.py

  • ReportLab PDF generation
  • web report

Utilities

  • Gradio UI
  • LLM-based summary generation

Sources

ROI/KPI scenarios are conservative estimates based on public materials such as KOSA wage data, SANS/NIST incident response guidance, ESG reporting statistics, Asana work-management research, and APQC process references. Actual savings vary with alert volume and reporting frequency.