Nexus-Guard-Suite/.sisyphus/plans/archived/realtime-traffic-monitoring.md

# Real-Time Traffic Monitoring (Optimized)

## TL;DR
> Real-time device/node status via SSE + HTTP streaming, traffic monitoring with PostgreSQL, historical charts with daily aggregation, toggle controls. **Optimized for low resource usage** — SSE only active when tab is focused, charts lazy-loaded.

**Deliverables**:
- HTTP streaming for device-agent → server (Rx/Tx data)
- SSE endpoint for dashboard real-time updates
- PostgreSQL schema for traffic logging
- Traffic recorder (Redis → DB batch)
- Dashboard traffic chart with historical data (lazy-loaded)
- Toggle to disable real-time display (per device/global)
- **Tab visibility API** — SSE disconnects when tab inactive

**Estimated Effort**: Medium
**Parallel Execution**: YES - 3 waves
**Critical Path**: T1 → T2 → T3 → T4 → T5

---

## Context

### Original Request
User wants real-time device/node online status without page refresh, Rx/Tx traffic with charts, daily/historical logging, and toggle controls. System scales to 1000+ devices.

### Architecture Decision (Updated)
- **Device-Agent → Server**: HTTP POST streaming (no protoc needed, uses existing HTTP)
- **Dashboard ← Server**: SSE (browser native, auto-reconnect, **tab-aware**)
- **Real-time state**: Redis (fast in-memory, pub/sub)
- **Traffic recording**: PostgreSQL (plain, TimescaleDB can be added later)
- **Historical query**: PostgreSQL with time_bucket aggregation

### Optimization Strategy
1. **Tab Visibility API** — SSE disconnects when browser tab is inactive
2. **Lazy-load charts** — TrafficChart only mounts when user clicks "Show Chart"
3. **Polling interval** — SSE pushes every 5s, not every 1s
4. **Redis TTL** — Traffic data expires after 24h (batch sync to DB)
5. **Minimal DOM updates** — Chart only re-renders on data change

---

## Work Objectives

### Core Objective
Real-time device status + traffic monitoring for 1000+ devices with historical charts, optimized for low resource usage.

### Must Have
- HTTP streaming for agent traffic data
- SSE for dashboard real-time updates
- **Tab-aware SSE** (disconnect when tab inactive)
- PostgreSQL for traffic logging
- Traffic chart per device/node (lazy-loaded)
- Toggle to disable chart display
- Historical data query (daily/hourly)

### Must NOT Have
- Do NOT use gRPC (no protoc dependency)
- Do NOT add heavy chart libraries (use lightweight SVG)
- Do NOT keep SSE connections open when tab is inactive
- Do NOT render charts when not visible

---

## Verification Strategy

### Test Decision
- **Infrastructure exists**: YES (Go, Vue 3, PostgreSQL)
- **Automated tests**: Tests-after
- **Framework**: Go test + npm test

---

## Execution Strategy

### Parallel Execution Waves

```
Wave 1 (Foundation):
├── T1: PostgreSQL schema + migration
├── T2: HTTP traffic endpoint
└── T3: Traffic recorder (Redis → DB)

Wave 2 (Backend + Frontend):
├── T4: SSE endpoint (tab-aware)
├── T5: Dashboard traffic chart (lazy-loaded)
├── T6: Toggle controls
└── T7: Historical data view
```

---

## TODOs

- [x] 1. **PostgreSQL schema + migration**

  **What to do**:
  - Create migration file `apps/server-core/migrations/003_device_traffic.sql`
  - Create `device_traffic` table:
    ```sql
    CREATE TABLE IF NOT EXISTS device_traffic (
        id BIGSERIAL PRIMARY KEY,
        time TIMESTAMPTZ NOT NULL DEFAULT NOW(),
        device_id UUID NOT NULL,
        node_id UUID,
        rx_bytes BIGINT DEFAULT 0,
        tx_bytes BIGINT DEFAULT 0,
        rx_rate BIGINT DEFAULT 0,
        tx_rate BIGINT DEFAULT 0
    );
    ```
  - Create daily aggregate view
  - Create hourly aggregate view
  - Add indexes on device_id + time

  **Must NOT do**:
  - Do NOT use TimescaleDB extension (not installed)
  - Do NOT remove existing tables

  **Recommended Agent Profile**:
  - **Category**: `quick`
  - **Skills**: `[]`

  **Parallelization**:
  - **Can Run In Parallel**: YES (with T2, T3)
  - **Parallel Group**: Wave 1
  - **Blocks**: T4
  - **Blocked By**: None

  **References**:
  - `apps/server-core/migrations/` - existing migration pattern

  **Acceptance Criteria**:
  - [x] `go build -tags dev ./...` passes
  - [x] Migration file created with correct SQL

  **Commit**: YES
  - Message: `feat(db): add device_traffic table and views`
  - Files: `apps/server-core/migrations/003_device_traffic.sql`

- [x] 2. **HTTP traffic endpoint**

  **What to do**:
  - Create `apps/server-core/api/traffic_stream.go`:
    - `POST /api/v1/traffic/report` — receive traffic data from agent
    - `GET /api/v1/traffic/stream` — SSE for dashboard
  - Traffic report endpoint accepts JSON: `{device_id, rx_bytes, tx_bytes}`
  - Stores to Redis via TrafficRecorder
  - No protoc needed — pure HTTP

  **Must NOT do**:
  - Do NOT require authentication for traffic reports (agent → server)
  - Do NOT block on Redis write

  **Recommended Agent Profile**:
  - **Category**: `quick`
  - **Skills**: `[]`

  **Parallelization**:
  - **Can Run In Parallel**: YES (with T1, T3)
  - **Parallel Group**: Wave 1
  - **Blocks**: T4
  - **Blocked By**: None

  **References**:
  - `apps/server-core/api/heartbeat.go` - existing HTTP pattern
  - `apps/server-core/internal/traffic/recorder.go` - TrafficRecorder

  **Acceptance Criteria**:
  - [x] `go build -tags dev ./...` passes
  - [x] POST /api/v1/traffic/report accepts traffic data
  - [x] Data stored to Redis

  **Commit**: YES
  - Message: `feat(api): add HTTP traffic report endpoint`
  - Files: `apps/server-core/api/traffic_stream.go`

- [x] 3. **Traffic recorder (Redis → DB batch)**

  **What to do**:
  - Create `apps/server-core/internal/traffic/recorder.go`:
    - `TrafficRecorder` struct with Redis client + DB connection
    - `Record(deviceID, rxBytes, txBytes)` — fast Redis write
    - `StartBatchSync(ctx, interval)` — batch insert to DB every 60s
    - `GetDeviceTraffic(deviceID, from, to)` — query historical data
    - `GetNodeTraffic(nodeID, from, to)` — aggregate per node
  - Redis key: `traffic:{device_id}:{timestamp}`
  - Batch insert: collect from Redis, insert to DB, delete from Redis

  **Must NOT do**:
  - Do NOT block on Redis write
  - Do NOT query DB on every traffic report

  **Recommended Agent Profile**:
  - **Category**: `quick`
  - **Skills**: `[]`

  **Parallelization**:
  - **Can Run In Parallel**: YES (with T1, T2)
  - **Parallel Group**: Wave 1
  - **Blocks**: T4
  - **Blocked By**: None

  **References**:
  - `apps/server-core/internal/heartbeat/redis.go` - Redis pattern

  **Acceptance Criteria**:
  - [x] `go build -tags dev ./...` passes
  - [x] Traffic recorded to Redis on Report()
  - [x] Batch sync inserts to DB

  **Commit**: YES
  - Message: `feat(traffic): add Redis → PostgreSQL recorder`
  - Files: `apps/server-core/internal/traffic/recorder.go`

- [x] 4. **SSE endpoint (tab-aware)**

  **What to do**:
  - Create `apps/server-core/api/sse.go`:
    - `SSEHandler` struct with Redis + recorder
    - `StreamStatus(c *gin.Context)` — SSE endpoint
    - Pushes device status updates every 5s
    - Heartbeat ping every 30s (keep-alive)
  - Register route: `GET /api/v1/devices/stream`
  - **Frontend optimization**: Use Page Visibility API
    - `document.addEventListener('visibilitychange', ...)`
    - When tab hidden → disconnect SSE
    - When tab visible → reconnect SSE

  **Must NOT do**:
  - Do NOT keep SSE open when tab is inactive
  - Do NOT store SSE clients in memory

  **Recommended Agent Profile**:
  - **Category**: `quick`
  - **Skills**: `[]`

  **Parallelization**:
  - **Can Run In Parallel**: NO (depends on T1, T2, T3)
  - **Parallel Group**: Wave 2
  - **Blocks**: T5
  - **Blocked By**: T1, T2, T3

  **References**:
  - `apps/server-core/api/heartbeat.go` - existing pattern
  - SSE spec: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events

  **Acceptance Criteria**:
  - [x] `go build -tags dev ./...` passes
  - [x] `curl -N http://localhost:8080/api/v1/devices/stream` returns SSE stream
  - [x] SSE disconnects when tab inactive (frontend)

  **Commit**: YES
  - Message: `feat(sse): add device status streaming endpoint`
  - Files: `apps/server-core/api/sse.go`, `apps/server-core/main.go`

- [x] 5. **Dashboard traffic chart (lazy-loaded)**

  **What to do**:
  - Create `apps/dashboard-ui/src/components/TrafficChart.vue`:
    - SVG line chart (no heavy libraries)
    - Props: `deviceId`, `height`, `showToggle`
    - **Lazy-load**: Only render when `showChart` prop is true
    - Time range selector (1h, 6h, 24h, 7d, 30d)
    - Toggle to enable/disable real-time updates
  - Add chart to `DeviceDetail.vue` (per-device, behind toggle)
  - Add chart to `Dashboard.vue` (per-node aggregate)

  **Must NOT do**:
  - Do NOT add heavy chart libraries (use SVG)
  - Do NOT render chart when `showChart` is false
  - Do NOT block UI on chart render

  **Recommended Agent Profile**:
  - **Category**: `visual-engineering`
  - **Skills**: `[]`

  **Parallelization**:
  - **Can Run In Parallel**: YES (with T6, T7)
  - **Parallel Group**: Wave 2
  - **Blocks**: None
  - **Blocked By**: T4

  **References**:
  - `apps/dashboard-ui/src/views/DeviceDetail.vue` - existing page
  - SVG chart pattern

  **Acceptance Criteria**:
  - [x] `npm run build` passes
  - [x] Chart only renders when toggle is ON
  - [x] Time range selector works

  **Commit**: YES
  - Message: `feat(ui): add lazy-loaded traffic chart component`
  - Files: `apps/dashboard-ui/src/components/TrafficChart.vue`

- [x] 6. **Toggle controls**

  **What to do**:
  - Add toggle to `DeviceDetail.vue`:
    - "Show Traffic Chart" toggle (per device)
    - When OFF: chart hidden, no data fetched
    - When ON: chart visible, data fetched
  - Add global toggle to `Dashboard.vue`:
    - "Show All Charts" toggle
    - Saves preference to localStorage
  - **Tab visibility**: Implement Page Visibility API
    - `document.addEventListener('visibilitychange', handler)`
    - When tab hidden → disconnect SSE, stop polling
    - When tab visible → reconnect SSE, resume polling

  **Must NOT do**:
  - Do NOT render charts when toggle is OFF
  - Do NOT fetch data when chart is hidden
  - Do NOT keep SSE open when tab is inactive

  **Recommended Agent Profile**:
  - **Category**: `visual-engineering`
  - **Skills**: `[]`

  **Parallelization**:
  - **Can Run In Parallel**: YES (with T5, T7)
  - **Parallel Group**: Wave 2
  - **Blocks**: None
  - **Blocked By**: T5

  **References**:
  - Page Visibility API: https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API
  - localStorage pattern

  **Acceptance Criteria**:
  - [x] `npm run build` passes
  - [x] Per-device toggle works
  - [x] Global toggle works
  - [x] SSE disconnects when tab hidden
  - [x] Charts hidden when toggle OFF

  **Commit**: YES
  - Message: `feat(ui): add toggle controls + tab-aware SSE`
  - Files: `apps/dashboard-ui/src/views/DeviceDetail.vue`, `apps/dashboard-ui/src/views/Dashboard.vue`

- [x] 7. **Historical data view**

  **What to do**:
  - Create `apps/dashboard-ui/src/views/TrafficHistory.vue`:
    - Full-page traffic history view
    - Date range picker
    - Device/node selector
    - Export to CSV
    - Daily/hourly aggregation
  - Add route: `/traffic-history`
  - Query backend traffic API
  - **Lazy-load**: Only fetch data when view is active

  **Must NOT do**:
  - Do NOT fetch data on page load (wait for user action)
  - Do NOT expose raw data

  **Recommended Agent Profile**:
  - **Category**: `visual-engineering`
  - **Skills**: `[]`

  **Parallelization**:
  - **Can Run In Parallel**: YES (with T5, T6)
  - **Parallel Group**: Wave 2
  - **Blocks**: None
  - **Blocked By**: T4

  **References**:
  - `apps/dashboard-ui/src/router/index.ts` - routing

  **Acceptance Criteria**:
  - [x] `npm run build` passes
  - [x] History page accessible at /traffic-history
  - [x] Date range filter works
  - [x] Data only fetched on user action

  **Commit**: YES
  - Message: `feat(ui): add traffic history view`
  - Files: `apps/dashboard-ui/src/views/TrafficHistory.vue`, `apps/dashboard-ui/src/router/index.ts`

---

## Final Verification Wave

- [x] F1. **Plan Compliance Audit** — `oracle`
- [x] F2. **Code Quality Review** — `unspecified-high`
- [x] F3. **Real Manual QA** — `unspecified-high`
- [x] F4. **Scope Fidelity Check** — `deep`

---

## Commit Strategy

- Commit #1: Backend — PostgreSQL schema + HTTP endpoint + recorder
- Commit #2: Frontend — SSE + charts + toggles + history

---

## Success Criteria

### Verification Commands
```bash
go build -tags dev ./...  # Expected: no errors
cd apps/dashboard-ui && npm run build  # Expected: no errors
```

### Final Checklist
- [x] HTTP traffic endpoint works (no protoc needed)
- [x] SSE pushes real-time status to dashboard
- [x] **SSE disconnects when tab inactive**
- [x] **Charts lazy-loaded (only when toggle ON)**
- [x] PostgreSQL stores traffic data
- [x] Toggle controls work (per device + global)
- [x] Performance: minimal resource usage