Nexus-Guard-Suite/.sisyphus/plans/archive/traffic-performance.md

# Traffic Performance Optimization

## TL;DR

> **Quick Summary**: Fix TrafficHistory performance — silent auto-refresh (no loading flash), limit data fetched, optimize chart rendering, and add server-side pagination.
> 
> **Deliverables**:
> - Silent auto-refresh (no loading state during background updates)
> - API limit parameter to cap data fetched
> - Client-side chart downsampling (max 200 points)
> - Smart CSV export (current page or all with progress)
> - Auto-refresh interval increased to 30s
> 
> **Estimated Effort**: Medium
> **Parallel Execution**: YES - 2 waves
> **Critical Path**: Backend limit → Frontend fetch → Chart/table optimizations

---

## Context

### Original Request
User reports: "terlalu banyak data record. realtime tidak smooth (masih ada warna loading). record table terdownload semua."

### Architecture Finding
- **SSE endpoint exists** (`/devices/stream`) but only streams device STATUS, not traffic data
- **Frontend has NO EventSource consumer** — SSE endpoint is orphaned
- **Traffic uses pure REST polling** — every 10s, fetch ALL records → loading flash
- **No WebSocket anywhere** in the codebase
- **TrafficRecorder** stores data in Redis (24h TTL) → syncs to PostgreSQL every 5min

### Why NOT WebSocket/SSE for Traffic (Yet)
1. SSE doesn't support custom `Authorization` headers — token must be query param or cookie (security tradeoff)
2. Existing SSE only handles device status — would need new SSE channel for traffic
3. Traffic data is already in Redis with 24h TTL — REST with limit is sufficient
4. **Recommended**: Fix REST performance first → evaluate SSE for traffic in future iteration

---

## Work Objectives

### Core Objective
Make TrafficHistory page smooth, fast, and non-blocking — no loading flash, limited data, optimized rendering.

### Concrete Deliverables
- `apps/server-core/api/traffic.go` — Add `limit` query parameter
- `apps/dashboard-ui/src/views/TrafficHistory.vue` — Silent refresh, smart pagination, optimized export
- `apps/dashboard-ui/src/components/TrafficChart.vue` — Downsample data for SVG rendering

### Must Have
- Auto-refresh does NOT show loading state (silent background update)
- Auto-refresh does NOT reset pagination page
- API supports `limit` parameter (default 500, max 5000)
- Chart downsamples to max 200 data points
- CSV export shows progress or limits to current page
- Auto-refresh interval 30s (was 10s)

### Must NOT Have (Guardrails)
- Do NOT add WebSocket infrastructure (future iteration)
- Do NOT change TrafficRecorder or Redis storage
- Do NOT change the SSE device status endpoint
- Do NOT change the POST /traffic/report endpoint
- Do NOT change TrafficChart's visual appearance
- Do NOT remove the auto-refresh feature

---

## Verification Strategy

### QA Policy
- Frontend: `npm run build` passes
- Backend: `go build ./...` passes
- Grep: no `setInterval` with < 20000ms interval
- Manual check: no loading flash during auto-refresh

---

## Execution Strategy

### Parallel Execution Waves

```
Wave 1 (Backend + Frontend foundation):
├── Task 1: Add limit param to traffic API [quick]
├── Task 2: Silent auto-refresh + pagination fix [quick]
├── Task 3: Chart downsampling [quick]

Wave 2 (Integration + Polish):
├── Task 4: CSV export optimization [quick]
├── Task 5: Build verify [quick]
```

---

## TODOs

- [x] 1. Add limit parameter to traffic API

  **What to do**:
  - In `apps/server-core/api/traffic.go`, modify `parseTimeRange` to also parse `limit` query parameter
  - Add `limit` parameter to `GetSummary`: `limit := c.DefaultQuery("limit", "500")`
  - Parse limit as int, cap at 5000 max
  - Apply `.Limit(limit)` to the GORM query in `GetSummary`
  - Also add limit to `GetDeviceTraffic` and `GetNodeTraffic`
  - Return `total_count` in response alongside `total_records` (total available before limit)

  **Must NOT do**:
  - Do NOT change TrafficRecorder
  - Do NOT change Redis storage
  - Do NOT change POST /traffic/report

  **References**:
  - `apps/server-core/api/traffic.go` — Full file (90 lines). `parseTimeRange` at line 75, `GetSummary` at line 55
  - `apps/server-core/internal/traffic/recorder.go` — `TrafficRecord` struct at line 15

  **QA Scenarios:**
  ```
  Scenario: API respects limit parameter
    Tool: Bash (curl)
    Steps:
      1. Start dev server
      2. curl -H "Authorization: Bearer <token>" "http://localhost:8080/api/v1/traffic/summary?from=...&to=...&limit=10"
    Expected Result: Response contains max 10 records, total_count shows actual total
    Evidence: .sisyphus/evidence/task-1-api-limit.txt

  Scenario: Build passes
    Tool: Bash
    Steps:
      1. cd apps/server-core && go build ./...
    Expected Result: Exit code 0
    Evidence: .sisyphus/evidence/task-1-build.txt
  ```

  **Commit**: YES (groups with 2-5)

---

- [x] 2. Silent auto-refresh + pagination fix

  **What to do**:
  - Modify `fetchTrafficData` to accept optional `silent` parameter (default false)
  - When `silent=true`: skip `loading.value = true`, skip `currentPage.value = 1`
  - Auto-refresh interval calls `fetchTrafficData(true)` — silent mode
  - Manual "Apply Filters" calls `fetchTrafficData()` — shows loading, resets page
  - Change interval from 10000ms to 30000ms
  - Add `?limit=500` to API URLs
  - Store `totalCount` from API response for pagination display
  - Update pagination display to show "of X total" using totalCount

  **Must NOT do**:
  - Do NOT remove auto-refresh
  - Do NOT change the date filtering logic
  - Do NOT change the chart component

  **References**:
  - `apps/dashboard-ui/src/views/TrafficHistory.vue` — Lines 177-205 (fetchTrafficData), 283-293 (interval)
  - `apps/dashboard-ui/src/services/api.ts` — Base axios instance

  **QA Scenarios:**
  ```
  Scenario: No loading flash during auto-refresh
    Tool: Playwright
    Steps:
      1. Open Traffic History page
      2. Wait for initial load
      3. Observe for 35 seconds — no loading bar should appear after initial load
    Expected Result: Loading indicator does NOT flash during background refresh
    Evidence: .sisyphus/evidence/task-2-no-flash.txt

  Scenario: Build passes
    Tool: Bash
    Steps:
      1. cd apps/dashboard-ui && npm run build
    Expected Result: Exit code 0
    Evidence: .sisyphus/evidence/task-2-build.txt
  ```

  **Commit**: YES (groups with 1, 3-5)

---

- [x] 3. Chart downsampling

  **What to do**:
  - In `TrafficHistory.vue`, add a `chartDataLimited` computed that limits chart data to max 200 points
  - If data > 200 points, downsample by averaging every N points (N = Math.ceil(data.length / 200))
  - Pass `chartDataLimited` to TrafficChart instead of `chartData`
  - Keep full `trafficData` for table pagination and CSV export

  **Must NOT do**:
  - Do NOT change TrafficChart.vue component
  - Do NOT change the SVG rendering logic

  **References**:
  - `apps/dashboard-ui/src/views/TrafficHistory.vue` — Lines 150-157 (chartData computed)

  **QA Scenarios:**
  ```
  Scenario: Chart receives max 200 data points
    Tool: Bash
    Steps:
      1. Grep for chartDataLimited in TrafficHistory.vue
    Expected Result: Computed property exists with 200-point cap
    Evidence: .sisyphus/evidence/task-3-downsample.txt

  Scenario: Build passes
    Tool: Bash
    Steps:
      1. cd apps/dashboard-ui && npm run build
    Expected Result: Exit code 0
    Evidence: .sisyphus/evidence/task-3-build.txt
  ```

  **Commit**: YES (groups with 1-2, 4-5)

---

- [x] 4. CSV export optimization

  **What to do**:
  - Change exportToCSV to export only `paginatedData` (current page) by default
  - Add a confirmation: "Export all X records or just current page?"
  - Or simpler: always export current filtered data (not limited by pagination)
  - Keep export fast by limiting to filtered dataset

  **Must NOT do**:
  - Do NOT add async CSV generation (overkill)
  - Do NOT change the download mechanism

  **References**:
  - `apps/dashboard-ui/src/views/TrafficHistory.vue` — Lines 239-263 (exportToCSV)

  **QA Scenarios:**
  ```
  Scenario: CSV export works
    Tool: Bash
    Steps:
      1. cd apps/dashboard-ui && npm run build
    Expected Result: Exit code 0
    Evidence: .sisyphus/evidence/task-4-build.txt
  ```

  **Commit**: YES (groups with 1-3, 5)

---

- [x] 5. Build verify all changes

  **What to do**:
  - Run `cd apps/server-core && go build ./...`
  - Run `cd apps/dashboard-ui && npm run build`
  - Grep for `setInterval` with interval < 20000ms in TrafficHistory.vue
  - Verify no `loading.value = true` in auto-refresh path

  **References**:
  - All modified files

  **QA Scenarios:**
  ```
  Scenario: Full build passes
    Tool: Bash
    Steps:
      1. cd apps/server-core && go build ./...
      2. cd apps/dashboard-ui && npm run build
    Expected Result: Both exit code 0
    Evidence: .sisyphus/evidence/task-5-full-build.txt

  Scenario: No aggressive polling
    Tool: Bash
    Steps:
      1. grep -n "setInterval" apps/dashboard-ui/src/views/TrafficHistory.vue
    Expected Result: Interval >= 20000ms
    Evidence: .sisyphus/evidence/task-5-polling-check.txt
  ```

  **Commit**: YES (final commit)

---

## Commit Strategy

- **Commit E**: All traffic performance changes
  - Files: `api/traffic.go`, `TrafficHistory.vue`
  - Pre-commit: `go build ./... && cd ../dashboard-ui && npm run build`

---

## Success Criteria

### Verification Commands
```bash
cd apps/server-core && go build ./...          # Expected: exit 0
cd apps/dashboard-ui && npm run build           # Expected: ✓ built in Xs
grep "setInterval" apps/dashboard-ui/src/views/TrafficHistory.vue  # Expected: 30000
grep -c "loading.value = true" apps/dashboard-ui/src/views/TrafficHistory.vue  # Expected: 1 (only manual refresh)
```

### Final Checklist
- [ ] Auto-refresh is silent (no loading flash)
- [ ] Auto-refresh does not reset pagination
- [ ] API supports limit parameter
- [ ] Chart renders max 200 data points
- [ ] Auto-refresh interval 30s
- [ ] Both backend and frontend build successfully