Data Sources

Real-time monitoring dashboard for all connected log sources with health tracking, auto-discovery, self-healing, and connector management.

Overview

A SOCaaS platform is only as good as the data it ingests. If a log source silently stops sending events, detection coverage degrades and threats go unnoticed. The Data Sources module provides continuous visibility into every connected data source -- its status, event throughput, health percentage, and last event timestamp. It also surfaces auto-discovered sources and maintains a self-healing log that documents automated remediation attempts.

What Was Proposed

Centralized data source management dashboard
Real-time health monitoring with per-source metrics (events/hour, health %, last event)
Auto-discovery of new log sources in the environment
Self-healing log source manager that auto-reconnects degraded sources
Connector template library for quick provisioning
24-hour health timeline visualization
Multi-SIEM adapter support (Sentinel, Splunk, Elastic, Chronicle, QRadar)

What's Built Complete

Full data sources dashboard with summary cards (total sources, connected, degraded/disconnected, events/hour)
2-column source grid showing status badges (Connected/Degraded/Disconnected), event throughput, last event time, and health bar
7 connector templates: Microsoft Sentinel, Syslog/CEF, Windows Events, Microsoft Defender, Cloud APIs, Webhook, File-based
"Add Source" panel with template selection and connection flow
Auto-Discovery panel showing pending discovered sources with Approve/Dismiss actions
Self-Healing Log showing recent remediation actions with success/fail status
24-hour health timeline bar chart with color-coded health bands
Auto-refresh polling every 30 seconds
8 mock data sources: Windows Security Events, Azure AD Sign-in, Palo Alto Firewall, AWS CloudTrail, Office 365, Cisco ASA, DNS Query Logs, Defender for Endpoint
Backend data source endpoints via the onboarding engine (shared API)

Architecture

Multi-SIEM Adapter Layer

The Data Sources module operates as the management plane for the multi-SIEM adapter layer. Each SIEM adapter (Sentinel, Splunk, Elastic, Chronicle, QRadar) connects through a standardized connector interface. The frontend polls the onboarding API for source status and health metrics, while the backend's self-healing monitor automatically attempts reconnection for degraded sources.

Frontend Dashboard          API Layer                  SIEM Adapters
+------------------+   +---------------------+   +------------------+
| DataSourcesPage  |-->| /api/v1/onboarding/ |   | Sentinel Adapter |
|  Summary Cards   |   |  datasources        |   | Splunk Adapter   |
|  Source Grid     |   |  datasources/health  |   | Elastic Adapter  |
|  Health Timeline |   |  datasources/discover|   | Chronicle Adapter|
|  Auto-Discovery  |   |  self-healing/log    |   | QRadar Adapter   |
|  Self-Heal Log   |   +---------------------+   +------------------+
+------------------+

Routing

Frontend Route

/data-sources

API Prefix

/api/v1/onboarding/datasources

Prerequisites

Authenticated user with admin, analyst, or super_admin role
At least one tenant with onboarded data sources
Network connectivity to the SIEM platform endpoints

Data Model

Field	Type	Description
id	string	Unique source identifier
name	string	Human-readable source name
source_type	string	endpoint \| cloud \| network \| identity \| email \| firewall \| siem \| dns
connector_type	string	sentinel_connector \| syslog_cef \| api_poll \| webhook \| file_based \| windows_events \| cloud_api
status	enum	connected \| degraded \| disconnected
events_per_hour	number	Current event throughput
last_event	ISO 8601	Timestamp of most recent event
health_pct	number	Health percentage (0-100)

Discovered Source Model

Field	Type	Description
id	string	Discovered source identifier
name	string	Device/service name with IP
source_type	string	Detected source category
discovered_at	ISO 8601	When the source was found

Self-Healing Action Model

Field	Type	Description
id	string	Action identifier
source_name	string	Affected source name
action	string	Remediation action description
timestamp	ISO 8601	When action was attempted
result	enum	success \| fail

UI Description

The page header shows a database icon with "Data Sources" title and "Monitor and manage log source connections" subtitle. Two action buttons appear: "Refresh" and "Add Source" (orange CTA).

Summary Cards

Four cards across the top: Total Sources, Connected (green), Degraded/Disconnected (yellow/red split), and Events/Hour (orange, formatted as "50.6k").

Add Source Panel

Clicking "Add Source" reveals a bordered panel with a 4-column grid of connector templates. Each template shows its icon and name. Clicking initiates a connection flow with loading spinner.

Source Grid

A 2-column grid of source cards. Each card displays: source icon, name, connector type, status badge (Connected/Degraded/Disconnected with appropriate colors), and a 3-column metrics row (events/hr, last event, health %). A thin health bar at the bottom is color-coded: green >=80%, yellow >=50%, red below 50%.

Health Timeline

A bar chart showing 24 hourly health data points. Bars are color-coded (green/yellow/red) with hover tooltips. Time labels appear every 6 hours.

Auto-Discovery Panel

Shows pending discovered sources with device name, discovery time, and Approve/Dismiss buttons. Only visible when discovered sources exist.

Self-Healing Log

A list of recent remediation actions with source name, action description, time ago, and success/fail badge.

Source Files

Component	Path
Frontend Page	platform/frontend/src/app/data-sources/page.tsx
API Router	platform/api/app/routers/onboarding.py (shared)
Service Engine	platform/api/app/services/onboarding_engine.py