Nexus-Guard-Suite/.omo/plans/archived/realtime-traffic-monitoring.md

Real-Time Traffic Monitoring (Optimized)

TL;DR

Real-time device/node status via SSE + HTTP streaming, traffic monitoring with PostgreSQL, historical charts with daily aggregation, toggle controls. Optimized for low resource usage — SSE only active when tab is focused, charts lazy-loaded.

Deliverables:

• HTTP streaming for device-agent → server (Rx/Tx data)
• SSE endpoint for dashboard real-time updates
• PostgreSQL schema for traffic logging
• Traffic recorder (Redis → DB batch)
• Dashboard traffic chart with historical data (lazy-loaded)
• Toggle to disable real-time display (per device/global)
• Tab visibility API — SSE disconnects when tab inactive

Estimated Effort: Medium Parallel Execution: YES - 3 waves Critical Path: T1 → T2 → T3 → T4 → T5

---

Context

Original Request

User wants real-time device/node online status without page refresh, Rx/Tx traffic with charts, daily/historical logging, and toggle controls. System scales to 1000+ devices.

Architecture Decision (Updated)

• Device-Agent → Server: HTTP POST streaming (no protoc needed, uses existing HTTP)
• Dashboard ← Server: SSE (browser native, auto-reconnect, tab-aware)
• Real-time state: Redis (fast in-memory, pub/sub)
• Traffic recording: PostgreSQL (plain, TimescaleDB can be added later)
• Historical query: PostgreSQL with time_bucket aggregation

Optimization Strategy

1. Tab Visibility API — SSE disconnects when browser tab is inactive
2. Lazy-load charts — TrafficChart only mounts when user clicks "Show Chart"
3. Polling interval — SSE pushes every 5s, not every 1s
4. Redis TTL — Traffic data expires after 24h (batch sync to DB)
5. Minimal DOM updates — Chart only re-renders on data change

---

Work Objectives

Core Objective

Real-time device status + traffic monitoring for 1000+ devices with historical charts, optimized for low resource usage.

Must Have

• HTTP streaming for agent traffic data
• SSE for dashboard real-time updates
• Tab-aware SSE (disconnect when tab inactive)
• PostgreSQL for traffic logging
• Traffic chart per device/node (lazy-loaded)
• Toggle to disable chart display
• Historical data query (daily/hourly)

Must NOT Have

• Do NOT use gRPC (no protoc dependency)
• Do NOT add heavy chart libraries (use lightweight SVG)
• Do NOT keep SSE connections open when tab is inactive
• Do NOT render charts when not visible

---

Verification Strategy

Test Decision

• Infrastructure exists: YES (Go, Vue 3, PostgreSQL)
• Automated tests: Tests-after
• Framework: Go test + npm test

---

Execution Strategy

Parallel Execution Waves

Wave 1 (Foundation):
├── T1: PostgreSQL schema + migration
├── T2: HTTP traffic endpoint
└── T3: Traffic recorder (Redis → DB)

Wave 2 (Backend + Frontend):
├── T4: SSE endpoint (tab-aware)
├── T5: Dashboard traffic chart (lazy-loaded)
├── T6: Toggle controls
└── T7: Historical data view

---

TODOs

• 1. PostgreSQL schema + migration

  What to do:

  • Create migration file apps/server-core/migrations/003_device_traffic.sql
  • Create device_traffic table:

    CREATE TABLE IF NOT EXISTS device_traffic (
        id BIGSERIAL PRIMARY KEY,
        time TIMESTAMPTZ NOT NULL DEFAULT NOW(),
        device_id UUID NOT NULL,
        node_id UUID,
        rx_bytes BIGINT DEFAULT 0,
        tx_bytes BIGINT DEFAULT 0,
        rx_rate BIGINT DEFAULT 0,
        tx_rate BIGINT DEFAULT 0
    );
    

  • Create daily aggregate view
  • Create hourly aggregate view
  • Add indexes on device_id + time

  Must NOT do:

  • Do NOT use TimescaleDB extension (not installed)
  • Do NOT remove existing tables

  Recommended Agent Profile:

  • Category: quick
  • Skills: []

  Parallelization:

  • Can Run In Parallel: YES (with T2, T3)
  • Parallel Group: Wave 1
  • Blocks: T4
  • Blocked By: None

  References:

  • apps/server-core/migrations/ - existing migration pattern

  Acceptance Criteria:

  • go build -tags dev ./... passes
  • Migration file created with correct SQL

  Commit: YES

  • Message: feat(db): add device_traffic table and views
  • Files: apps/server-core/migrations/003_device_traffic.sql

• 2. HTTP traffic endpoint

  What to do:

  • Create apps/server-core/api/traffic_stream.go:
    • POST /api/v1/traffic/report — receive traffic data from agent
    • GET /api/v1/traffic/stream — SSE for dashboard
  • Traffic report endpoint accepts JSON: {device_id, rx_bytes, tx_bytes}
  • Stores to Redis via TrafficRecorder
  • No protoc needed — pure HTTP

  Must NOT do:

  • Do NOT require authentication for traffic reports (agent → server)
  • Do NOT block on Redis write

  Recommended Agent Profile:

  • Category: quick
  • Skills: []

  Parallelization:

  • Can Run In Parallel: YES (with T1, T3)
  • Parallel Group: Wave 1
  • Blocks: T4
  • Blocked By: None

  References:

  • apps/server-core/api/heartbeat.go - existing HTTP pattern
  • apps/server-core/internal/traffic/recorder.go - TrafficRecorder

  Acceptance Criteria:

  • go build -tags dev ./... passes
  • POST /api/v1/traffic/report accepts traffic data
  • Data stored to Redis

  Commit: YES

  • Message: feat(api): add HTTP traffic report endpoint
  • Files: apps/server-core/api/traffic_stream.go

• 3. Traffic recorder (Redis → DB batch)

  What to do:

  • Create apps/server-core/internal/traffic/recorder.go:
    • TrafficRecorder struct with Redis client + DB connection
    • Record(deviceID, rxBytes, txBytes) — fast Redis write
    • StartBatchSync(ctx, interval) — batch insert to DB every 60s
    • GetDeviceTraffic(deviceID, from, to) — query historical data
    • GetNodeTraffic(nodeID, from, to) — aggregate per node
  • Redis key: traffic:{device_id}:{timestamp}
  • Batch insert: collect from Redis, insert to DB, delete from Redis

  Must NOT do:

  • Do NOT block on Redis write
  • Do NOT query DB on every traffic report

  Recommended Agent Profile:

  • Category: quick
  • Skills: []

  Parallelization:

  • Can Run In Parallel: YES (with T1, T2)
  • Parallel Group: Wave 1
  • Blocks: T4
  • Blocked By: None

  References:

  • apps/server-core/internal/heartbeat/redis.go - Redis pattern

  Acceptance Criteria:

  • go build -tags dev ./... passes
  • Traffic recorded to Redis on Report()
  • Batch sync inserts to DB

  Commit: YES

  • Message: feat(traffic): add Redis → PostgreSQL recorder
  • Files: apps/server-core/internal/traffic/recorder.go

• 4. SSE endpoint (tab-aware)

  What to do:

  • Create apps/server-core/api/sse.go:
    • SSEHandler struct with Redis + recorder
    • StreamStatus(c *gin.Context) — SSE endpoint
    • Pushes device status updates every 5s
    • Heartbeat ping every 30s (keep-alive)
  • Register route: GET /api/v1/devices/stream
  • Frontend optimization: Use Page Visibility API
    • document.addEventListener('visibilitychange', ...)
    • When tab hidden → disconnect SSE
    • When tab visible → reconnect SSE

  Must NOT do:

  • Do NOT keep SSE open when tab is inactive
  • Do NOT store SSE clients in memory

  Recommended Agent Profile:

  • Category: quick
  • Skills: []

  Parallelization:

  • Can Run In Parallel: NO (depends on T1, T2, T3)
  • Parallel Group: Wave 2
  • Blocks: T5
  • Blocked By: T1, T2, T3

  References:

  • apps/server-core/api/heartbeat.go - existing pattern
  • SSE spec: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events

  Acceptance Criteria:

  • go build -tags dev ./... passes
  • curl -N http://localhost:8080/api/v1/devices/stream returns SSE stream
  • SSE disconnects when tab inactive (frontend)

  Commit: YES

  • Message: feat(sse): add device status streaming endpoint
  • Files: apps/server-core/api/sse.go, apps/server-core/main.go

• 5. Dashboard traffic chart (lazy-loaded)

  What to do:

  • Create apps/dashboard-ui/src/components/TrafficChart.vue:
    • SVG line chart (no heavy libraries)
    • Props: deviceId, height, showToggle
    • Lazy-load: Only render when showChart prop is true
    • Time range selector (1h, 6h, 24h, 7d, 30d)
    • Toggle to enable/disable real-time updates
  • Add chart to DeviceDetail.vue (per-device, behind toggle)
  • Add chart to Dashboard.vue (per-node aggregate)

  Must NOT do:

  • Do NOT add heavy chart libraries (use SVG)
  • Do NOT render chart when showChart is false
  • Do NOT block UI on chart render

  Recommended Agent Profile:

  • Category: visual-engineering
  • Skills: []

  Parallelization:

  • Can Run In Parallel: YES (with T6, T7)
  • Parallel Group: Wave 2
  • Blocks: None
  • Blocked By: T4

  References:

  • apps/dashboard-ui/src/views/DeviceDetail.vue - existing page
  • SVG chart pattern

  Acceptance Criteria:

  • npm run build passes
  • Chart only renders when toggle is ON
  • Time range selector works

  Commit: YES

  • Message: feat(ui): add lazy-loaded traffic chart component
  • Files: apps/dashboard-ui/src/components/TrafficChart.vue

• 6. Toggle controls

  What to do:

  • Add toggle to DeviceDetail.vue:
    • "Show Traffic Chart" toggle (per device)
    • When OFF: chart hidden, no data fetched
    • When ON: chart visible, data fetched
  • Add global toggle to Dashboard.vue:
    • "Show All Charts" toggle
    • Saves preference to localStorage
  • Tab visibility: Implement Page Visibility API
    • document.addEventListener('visibilitychange', handler)
    • When tab hidden → disconnect SSE, stop polling
    • When tab visible → reconnect SSE, resume polling

  Must NOT do:

  • Do NOT render charts when toggle is OFF
  • Do NOT fetch data when chart is hidden
  • Do NOT keep SSE open when tab is inactive

  Recommended Agent Profile:

  • Category: visual-engineering
  • Skills: []

  Parallelization:

  • Can Run In Parallel: YES (with T5, T7)
  • Parallel Group: Wave 2
  • Blocks: None
  • Blocked By: T5

  References:

  • Page Visibility API: https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API
  • localStorage pattern

  Acceptance Criteria:

  • npm run build passes
  • Per-device toggle works
  • Global toggle works
  • SSE disconnects when tab hidden
  • Charts hidden when toggle OFF

  Commit: YES

  • Message: feat(ui): add toggle controls + tab-aware SSE
  • Files: apps/dashboard-ui/src/views/DeviceDetail.vue, apps/dashboard-ui/src/views/Dashboard.vue

• 7. Historical data view

  What to do:

  • Create apps/dashboard-ui/src/views/TrafficHistory.vue:
    • Full-page traffic history view
    • Date range picker
    • Device/node selector
    • Export to CSV
    • Daily/hourly aggregation
  • Add route: /traffic-history
  • Query backend traffic API
  • Lazy-load: Only fetch data when view is active

  Must NOT do:

  • Do NOT fetch data on page load (wait for user action)
  • Do NOT expose raw data

  Recommended Agent Profile:

  • Category: visual-engineering
  • Skills: []

  Parallelization:

  • Can Run In Parallel: YES (with T5, T6)
  • Parallel Group: Wave 2
  • Blocks: None
  • Blocked By: T4

  References:

  • apps/dashboard-ui/src/router/index.ts - routing

  Acceptance Criteria:

  • npm run build passes
  • History page accessible at /traffic-history
  • Date range filter works
  • Data only fetched on user action

  Commit: YES

  • Message: feat(ui): add traffic history view
  • Files: apps/dashboard-ui/src/views/TrafficHistory.vue, apps/dashboard-ui/src/router/index.ts

---

Final Verification Wave

• F1. Plan Compliance Audit — oracle
• F2. Code Quality Review — unspecified-high
• F3. Real Manual QA — unspecified-high
• F4. Scope Fidelity Check — deep

---

Commit Strategy

• Commit #1: Backend — PostgreSQL schema + HTTP endpoint + recorder
• Commit #2: Frontend — SSE + charts + toggles + history

---

Success Criteria

Verification Commands

go build -tags dev ./...  # Expected: no errors
cd apps/dashboard-ui && npm run build  # Expected: no errors

Final Checklist

• HTTP traffic endpoint works (no protoc needed)
• SSE pushes real-time status to dashboard
• SSE disconnects when tab inactive
• Charts lazy-loaded (only when toggle ON)
• PostgreSQL stores traffic data
• Toggle controls work (per device + global)
• Performance: minimal resource usage