nexusguard/Nexus-Guard-Suite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity
Files
d737d400aec7d97990bfd570c76f4546af4c4ff2
Nexus-Guard-Suite/.sisyphus/plans/realtime-traffic-monitoring.md
T

13 KiB
Raw Blame History

Real-Time Traffic Monitoring (Optimized)

TL;DR

Real-time device/node status via SSE + HTTP streaming, traffic monitoring with PostgreSQL, historical charts with daily aggregation, toggle controls. Optimized for low resource usage — SSE only active when tab is focused, charts lazy-loaded.

Deliverables:

  • HTTP streaming for device-agent → server (Rx/Tx data)
  • SSE endpoint for dashboard real-time updates
  • PostgreSQL schema for traffic logging
  • Traffic recorder (Redis → DB batch)
  • Dashboard traffic chart with historical data (lazy-loaded)
  • Toggle to disable real-time display (per device/global)
  • Tab visibility API — SSE disconnects when tab inactive

Estimated Effort: Medium Parallel Execution: YES - 3 waves Critical Path: T1 → T2 → T3 → T4 → T5

Context

Original Request

User wants real-time device/node online status without page refresh, Rx/Tx traffic with charts, daily/historical logging, and toggle controls. System scales to 1000+ devices.

Architecture Decision (Updated)

  • Device-Agent → Server: HTTP POST streaming (no protoc needed, uses existing HTTP)
  • Dashboard ← Server: SSE (browser native, auto-reconnect, tab-aware)
  • Real-time state: Redis (fast in-memory, pub/sub)
  • Traffic recording: PostgreSQL (plain, TimescaleDB can be added later)
  • Historical query: PostgreSQL with time_bucket aggregation

Optimization Strategy

  1. Tab Visibility API — SSE disconnects when browser tab is inactive
  2. Lazy-load charts — TrafficChart only mounts when user clicks "Show Chart"
  3. Polling interval — SSE pushes every 5s, not every 1s
  4. Redis TTL — Traffic data expires after 24h (batch sync to DB)
  5. Minimal DOM updates — Chart only re-renders on data change

Work Objectives

Core Objective

Real-time device status + traffic monitoring for 1000+ devices with historical charts, optimized for low resource usage.

Must Have

  • HTTP streaming for agent traffic data
  • SSE for dashboard real-time updates
  • Tab-aware SSE (disconnect when tab inactive)
  • PostgreSQL for traffic logging
  • Traffic chart per device/node (lazy-loaded)
  • Toggle to disable chart display
  • Historical data query (daily/hourly)

Must NOT Have

  • Do NOT use gRPC (no protoc dependency)
  • Do NOT add heavy chart libraries (use lightweight SVG)
  • Do NOT keep SSE connections open when tab is inactive
  • Do NOT render charts when not visible

Verification Strategy

Test Decision

  • Infrastructure exists: YES (Go, Vue 3, PostgreSQL)
  • Automated tests: Tests-after
  • Framework: Go test + npm test

Execution Strategy

Parallel Execution Waves

Wave 1 (Foundation):
├── T1: PostgreSQL schema + migration
├── T2: HTTP traffic endpoint
└── T3: Traffic recorder (Redis → DB)

Wave 2 (Backend + Frontend):
├── T4: SSE endpoint (tab-aware)
├── T5: Dashboard traffic chart (lazy-loaded)
├── T6: Toggle controls
└── T7: Historical data view

TODOs

  • 1. PostgreSQL schema + migration

    What to do:

    • Create migration file apps/server-core/migrations/003_device_traffic.sql
    • Create device_traffic table: 
      CREATE TABLE IF NOT EXISTS device_traffic (
    id BIGSERIAL PRIMARY KEY,
    time TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    device_id UUID NOT NULL,
    node_id UUID,
    rx_bytes BIGINT DEFAULT 0,
    tx_bytes BIGINT DEFAULT 0,
    rx_rate BIGINT DEFAULT 0,
    tx_rate BIGINT DEFAULT 0
);
    • Create daily aggregate view
    • Create hourly aggregate view
    • Add indexes on device_id + time

    Must NOT do:

    • Do NOT use TimescaleDB extension (not installed)
    • Do NOT remove existing tables

    Recommended Agent Profile:

    • Category: quick
    • Skills: []

    Parallelization:

    • Can Run In Parallel: YES (with T2, T3)
    • Parallel Group: Wave 1
    • Blocks: T4
    • Blocked By: None

    References:

    • apps/server-core/migrations/ - existing migration pattern

    Acceptance Criteria:

    • go build -tags dev ./... passes
    • Migration file created with correct SQL

    Commit: YES

    • Message: feat(db): add device_traffic table and views
    • Files: apps/server-core/migrations/003_device_traffic.sql

  • 2. HTTP traffic endpoint

    What to do:

    • Create apps/server-core/api/traffic_stream.go:
      • POST /api/v1/traffic/report — receive traffic data from agent
      • GET /api/v1/traffic/stream — SSE for dashboard
    • Traffic report endpoint accepts JSON: {device_id, rx_bytes, tx_bytes}
    • Stores to Redis via TrafficRecorder
    • No protoc needed — pure HTTP

    Must NOT do:

    • Do NOT require authentication for traffic reports (agent → server)
    • Do NOT block on Redis write

    Recommended Agent Profile:

    • Category: quick
    • Skills: []

    Parallelization:

    • Can Run In Parallel: YES (with T1, T3)
    • Parallel Group: Wave 1
    • Blocks: T4
    • Blocked By: None

    References:

    • apps/server-core/api/heartbeat.go - existing HTTP pattern
    • apps/server-core/internal/traffic/recorder.go - TrafficRecorder

    Acceptance Criteria:

    • go build -tags dev ./... passes
    • POST /api/v1/traffic/report accepts traffic data
    • Data stored to Redis

    Commit: YES

    • Message: feat(api): add HTTP traffic report endpoint
    • Files: apps/server-core/api/traffic_stream.go

  • 3. Traffic recorder (Redis → DB batch)

    What to do:

    • Create apps/server-core/internal/traffic/recorder.go:
      • TrafficRecorder struct with Redis client + DB connection
      • Record(deviceID, rxBytes, txBytes) — fast Redis write
      • StartBatchSync(ctx, interval) — batch insert to DB every 60s
      • GetDeviceTraffic(deviceID, from, to) — query historical data
      • GetNodeTraffic(nodeID, from, to) — aggregate per node
    • Redis key: traffic:{device_id}:{timestamp}
    • Batch insert: collect from Redis, insert to DB, delete from Redis

    Must NOT do:

    • Do NOT block on Redis write
    • Do NOT query DB on every traffic report

    Recommended Agent Profile:

    • Category: quick
    • Skills: []

    Parallelization:

    • Can Run In Parallel: YES (with T1, T2)
    • Parallel Group: Wave 1
    • Blocks: T4
    • Blocked By: None

    References:

    • apps/server-core/internal/heartbeat/redis.go - Redis pattern

    Acceptance Criteria:

    • go build -tags dev ./... passes
    • Traffic recorded to Redis on Report()
    • Batch sync inserts to DB

    Commit: YES

    • Message: feat(traffic): add Redis → PostgreSQL recorder
    • Files: apps/server-core/internal/traffic/recorder.go

  • 4. SSE endpoint (tab-aware)

    What to do:

    • Create apps/server-core/api/sse.go:
      • SSEHandler struct with Redis + recorder
      • StreamStatus(c *gin.Context) — SSE endpoint
      • Pushes device status updates every 5s
      • Heartbeat ping every 30s (keep-alive)
    • Register route: GET /api/v1/devices/stream
    • Frontend optimization: Use Page Visibility API
      • document.addEventListener('visibilitychange', ...)
      • When tab hidden → disconnect SSE
      • When tab visible → reconnect SSE

    Must NOT do:

    • Do NOT keep SSE open when tab is inactive
    • Do NOT store SSE clients in memory

    Recommended Agent Profile:

    • Category: quick
    • Skills: []

    Parallelization:

    • Can Run In Parallel: NO (depends on T1, T2, T3)
    • Parallel Group: Wave 2
    • Blocks: T5
    • Blocked By: T1, T2, T3

    References:

    Acceptance Criteria:

    • go build -tags dev ./... passes
    • curl -N http://localhost:8080/api/v1/devices/stream returns SSE stream
    • SSE disconnects when tab inactive (frontend)

    Commit: YES

    • Message: feat(sse): add device status streaming endpoint
    • Files: apps/server-core/api/sse.go, apps/server-core/main.go

  • 5. Dashboard traffic chart (lazy-loaded)

    What to do:

    • Create apps/dashboard-ui/src/components/TrafficChart.vue:
      • SVG line chart (no heavy libraries)
      • Props: deviceId, height, showToggle
      • Lazy-load: Only render when showChart prop is true
      • Time range selector (1h, 6h, 24h, 7d, 30d)
      • Toggle to enable/disable real-time updates
    • Add chart to DeviceDetail.vue (per-device, behind toggle)
    • Add chart to Dashboard.vue (per-node aggregate)

    Must NOT do:

    • Do NOT add heavy chart libraries (use SVG)
    • Do NOT render chart when showChart is false
    • Do NOT block UI on chart render

    Recommended Agent Profile:

    • Category: visual-engineering
    • Skills: []

    Parallelization:

    • Can Run In Parallel: YES (with T6, T7)
    • Parallel Group: Wave 2
    • Blocks: None
    • Blocked By: T4

    References:

    • apps/dashboard-ui/src/views/DeviceDetail.vue - existing page
    • SVG chart pattern

    Acceptance Criteria:

    • npm run build passes
    • Chart only renders when toggle is ON
    • Time range selector works

    Commit: YES

    • Message: feat(ui): add lazy-loaded traffic chart component
    • Files: apps/dashboard-ui/src/components/TrafficChart.vue

  • 6. Toggle controls

    What to do:

    • Add toggle to DeviceDetail.vue:
      • "Show Traffic Chart" toggle (per device)
      • When OFF: chart hidden, no data fetched
      • When ON: chart visible, data fetched
    • Add global toggle to Dashboard.vue:
      • "Show All Charts" toggle
      • Saves preference to localStorage
    • Tab visibility: Implement Page Visibility API
      • document.addEventListener('visibilitychange', handler)
      • When tab hidden → disconnect SSE, stop polling
      • When tab visible → reconnect SSE, resume polling

    Must NOT do:

    • Do NOT render charts when toggle is OFF
    • Do NOT fetch data when chart is hidden
    • Do NOT keep SSE open when tab is inactive

    Recommended Agent Profile:

    • Category: visual-engineering
    • Skills: []

    Parallelization:

    • Can Run In Parallel: YES (with T5, T7)
    • Parallel Group: Wave 2
    • Blocks: None
    • Blocked By: T5

    References:

    Acceptance Criteria:

    • npm run build passes
    • Per-device toggle works
    • Global toggle works
    • SSE disconnects when tab hidden
    • Charts hidden when toggle OFF

    Commit: YES

    • Message: feat(ui): add toggle controls + tab-aware SSE
    • Files: apps/dashboard-ui/src/views/DeviceDetail.vue, apps/dashboard-ui/src/views/Dashboard.vue

  • 7. Historical data view

    What to do:

    • Create apps/dashboard-ui/src/views/TrafficHistory.vue:
      • Full-page traffic history view
      • Date range picker
      • Device/node selector
      • Export to CSV
      • Daily/hourly aggregation
    • Add route: /traffic-history
    • Query backend traffic API
    • Lazy-load: Only fetch data when view is active

    Must NOT do:

    • Do NOT fetch data on page load (wait for user action)
    • Do NOT expose raw data

    Recommended Agent Profile:

    • Category: visual-engineering
    • Skills: []

    Parallelization:

    • Can Run In Parallel: YES (with T5, T6)
    • Parallel Group: Wave 2
    • Blocks: None
    • Blocked By: T4

    References:

    • apps/dashboard-ui/src/router/index.ts - routing

    Acceptance Criteria:

    • npm run build passes
    • History page accessible at /traffic-history
    • Date range filter works
    • Data only fetched on user action

    Commit: YES

    • Message: feat(ui): add traffic history view
    • Files: apps/dashboard-ui/src/views/TrafficHistory.vue, apps/dashboard-ui/src/router/index.ts

Final Verification Wave

  • F1. Plan Compliance Audit — oracle
  • F2. Code Quality Review — unspecified-high
  • F3. Real Manual QA — unspecified-high
  • F4. Scope Fidelity Check — deep

Commit Strategy

  • Commit #1: Backend — PostgreSQL schema + HTTP endpoint + recorder
  • Commit #2: Frontend — SSE + charts + toggles + history

Success Criteria

Verification Commands

go build -tags dev ./...  # Expected: no errors
cd apps/dashboard-ui && npm run build  # Expected: no errors

Final Checklist

  • HTTP traffic endpoint works (no protoc needed)
  • SSE pushes real-time status to dashboard
  • SSE disconnects when tab inactive
  • Charts lazy-loaded (only when toggle ON)
  • PostgreSQL stores traffic data
  • Toggle controls work (per device + global)
  • Performance: minimal resource usage
Reference in New Issue View Git Blame Copy Permalink