Security Analytics

Overview

The Security Analytics module exposes the internals of the ThreatOps classical machine learning pipeline to operators and analysts. Rather than treating AI triage as a black box, this module provides a live window into model health, training progress, version history, feature importance rankings, and the ensemble scoring strategy that drives every alert disposition decision on the platform.

The frontend page at /analytics polls the ML backend every 30 seconds and renders the state of three sklearn models — an Alert Classifier, an Anomaly Detector, and a Threat Scorer — alongside a training buffer progress bar, the 70/30 ensemble blend visualization, a ranked bar chart of the top 17 RandomForest feature importances, and a summary of the four possible disposition outcomes and their confidence thresholds. The backend is served entirely through the /api/v1/ml router, which also handles real-time predictions, analyst feedback recording, forced retraining, and model version history retrieval for rollback workflows.

What Was Proposed

• A classical ML pipeline with three specialized models covering alert classification, anomaly detection, and threat risk scoring
• Ensemble scoring that blends ML confidence with rule-based scores at a 70/30 ratio to balance data-driven decisions with deterministic expert rules
• Automatic model retraining triggered by accumulated analyst feedback, without requiring manual intervention or scheduled jobs
• Analyst feedback loop: every resolve, escalate, or false-positive action feeds labeled training data back to the models
• Feature importance transparency so analysts can audit which alert attributes drive high-risk scores
• Model versioning with persistent storage (Azure Blob) so retrained models can be rolled back if performance regresses
• Real-time prediction endpoint for ad-hoc analysis and integration testing
• A frontend analytics page showing live model stats, training buffer fill, ensemble weights, and feature importance rankings

What's Built

Architecture

ML Router

File: platform/api/app/routers/ml.py — Prefix: /api/v1/ml

A thin FastAPI router that delegates all logic to the ml_pipeline singleton imported from app/ml/training_pipeline.py. The router is stateless — it does not hold any model state itself. The pipeline singleton manages model loading from Azure Blob storage at startup, maintains the training feedback buffer in memory, and schedules retraining when the buffer crosses the configured threshold (default: 100 samples).

ML Training Pipeline (app/ml/training_pipeline.py)

The singleton ml_pipeline encapsulates three sklearn model wrappers:

• AlertClassifier — wraps a RandomForestClassifier; provides predict(alert) returning a classification label and confidence score, and get_feature_importances() returning a ranked dict of feature names to importance weights
• AnomalyDetector — wraps an IsolationForest with contamination=0.10 (10% expected anomaly rate); unsupervised, so it does not use labeled training data
• ThreatScorer — wraps a GradientBoostingClassifier with 100 estimators; outputs a continuous risk score in the 0–100 range

The pipeline exposes predict_ensemble(alert: dict), which runs all three models and blends results using the 70/30 formula: confidence = 0.7 * ml_score + 0.3 * rule_score. It also manages record_feedback(), which appends analyst-labeled samples to the training buffer and triggers trigger_retrain() when the buffer reaches threshold.

Model Store (app/ml/model_store.py)

Handles persistence of trained model artifacts to and from Azure Blob storage (container: ml-models on storage account stroconmlmodels). Each model version is saved with a timestamp and version number. The store provides get_all_model_versions() and get_version_history(model_name) for the rollback UI.

ML Models

API Endpoints

ML Router — /api/v1/ml

Frontend Route

The page is a Next.js App Router client component ("use client"). On mount it fires two requests in parallel via Promise.allSettled: GET /api/v1/ml/stats and GET /api/v1/ml/feature-importances. Either can fail independently without breaking the page — the component renders whatever data arrives. Auto-refreshes every 30 seconds via setInterval. API calls use the shared api client from @/lib/api-client.

Ensemble Strategy

The ensemble combines outputs from all three models into a single blended confidence score that drives the final alert disposition. The formula is applied in ml_pipeline.predict_ensemble():

blended_confidence = (0.7 × ml_score) + (0.3 × rule_score)

The 70% ML weight reflects confidence in the trained models when sufficient labeled data is available. The 30% rule-based weight preserves expert-authored detection logic as a safety net, particularly for novel attack patterns that the models have not yet seen in training data. The split is visualized in the frontend as a two-segment progress bar (cyan for ML, emerald for rule-based).

Disposition Logic

The final blended confidence score maps to one of four disposition labels. These labels drive what the Autonomous SOC Engine does with the alert — auto-close it, notify an analyst, open a SOAR playbook, or page the incident response team.

The thresholds are: >85% blended confidence → auto-resolve; 50–85% → investigate; <50% → escalate; high threat score + high severity → critical immediate. These thresholds are implemented in TriageService in app/services/triage_service.py.

Data Models

The analytics module does not define its own SQLAlchemy database models. All data is maintained in-memory by the ml_pipeline singleton, with model artifacts persisted to Azure Blob via model_store. The structures below document the API response shapes consumed by the frontend.

MLStats Response (GET /api/v1/ml/stats)

{
  "classifier": {
    "version": "1.3.0",
    "accuracy": 0.923,
    "samples_trained": 1450
  },
  "anomaly_detector": {
    "version": "1.1.0",
    "accuracy": null,              // unsupervised — no accuracy metric
    "samples_trained": null
  },
  "threat_scorer": {
    "version": "1.2.0",
    "accuracy": 0.891,
    "samples_trained": 1450
  },
  "training_buffer": 34,           // samples accumulated since last retrain
  "retrain_threshold": 100,        // samples needed to trigger auto-retrain
  "total_retrains": 14,
  "last_retrain": "2026-02-28T18:42:00",
  "feature_importances": {
    "severity_encoded": 0.187,
    "rule_confidence": 0.142,
    "correlation_count": 0.118,
    // ... up to 17 features
  }
}

FeedbackRequest (POST /api/v1/ml/feedback)

{
  "alert": {
    "title": "Lateral movement detected on DC01",
    "severity": "high",
    "source_siem": "sentinel",
    "mitre_tactic": "lateral_movement",
    "rule_confidence": 0.72,
    "correlation_count": 5
  },
  "disposition": "true_positive",   // true_positive | false_positive | suspicious | benign
  "risk_score": 87.5                // optional analyst-provided override
}

PredictResponse (POST /api/v1/ml/predict)

{
  "classification": "suspicious",
  "anomaly_score": -0.31,           // IsolationForest raw score (lower = more anomalous)
  "risk_score": 74.2,               // ThreatScorer output (0–100)
  "ml_confidence": 0.68,
  "rule_score": 0.72,
  "blended_confidence": 0.692,      // 0.7 * ml_confidence + 0.3 * rule_score
  "disposition": "requires_investigation",
  "reasoning": "High rule confidence, moderate ML confidence. Correlation count of 5 suggests related activity."
}

FeatureImportances Response (GET /api/v1/ml/feature-importances)

{
  "feature_importances": {
    "severity_encoded": 0.187,
    "rule_confidence": 0.142,
    "correlation_count": 0.118,
    "hour_of_day": 0.094,
    "source_siem_encoded": 0.082,
    "mitre_tactic_encoded": 0.079,
    "ioc_count": 0.071,
    "asset_count": 0.063,
    // ... additional features
  }
}

ModelVersionHistory (GET /api/v1/ml/versions/{model_name})

{
  "model_name": "alert_classifier",
  "latest": {
    "version": "1.3.0",
    "saved_at": "2026-02-28T18:42:00",
    "accuracy": 0.923,
    "samples": 1450,
    "blob_path": "ml-models/alert_classifier/v1.3.0.pkl"
  },
  "history": [ /* array of version records */ ],
  "total_versions": 14
}

Prerequisites

• ML Training Pipeline — app/ml/training_pipeline.py singleton ml_pipeline; must be initialized at startup; models loaded from Azure Blob or initialized fresh if no stored versions exist
• AlertClassifier — sklearn RandomForestClassifier wrapper with predict(), get_feature_importances(), and versioned state
• AnomalyDetector — sklearn IsolationForest wrapper (contamination=0.10)
• ThreatScorer — sklearn GradientBoostingClassifier wrapper (100 estimators) with 0–100 score output
• Model Store — app/ml/model_store.py; requires Azure Blob Storage credentials (storage account stroconmlmodels, container ml-models); falls back to local /tmp/ storage if unavailable (ephemeral)
• scikit-learn — Python package; required for all three model types; must be present in requirements.txt
• numpy / pandas — Required for feature engineering in the training pipeline
• Triage Service — app/services/triage_service.py; calls ml_pipeline.predict_ensemble() during the 6-stage alert triage pipeline; provides the rule_score that feeds the 30% blend weight
• Alerts Router — app/routers/alerts.py; calls ml_pipeline.record_feedback() during bulk actions so analyst dispositions reach the training buffer
• Admin Ops Router — app/routers/admin_ops.py; uses ml_pipeline.get_model_stats(), trigger_retrain(), and get_performance_log() for the admin dashboard and activity log

UI Layout