26 KiB
Device Agent Reliability Overhaul — Implementation Plan
TL;DR
Quick Summary: Improve device-agent reliability with state machine architecture, failover, health checks, better documentation, and cross-platform support (Linux, Windows, Android).
Deliverables:
- State machine lifecycle management
- Server/endpoint failover
- Health check system
- CLI help menu and documentation
- Cross-platform builds (Linux, Windows, Android)
- Server-side status API
Estimated Effort: Medium Parallel Execution: YES - 4 waves Critical Path: State Machine → Health Checks → Failover → Server Integration
Context
Original Request
Improve device-agent reliability (better reconnection, failover) with terminal UI and cross-platform support.
Interview Summary
Key Discussions:
- Device-agent is a small daemon (no database)
- Logs sent to NexusGuard server
- Cross-platform: Linux, Windows, Android
- Phase approach: terminal first, GUI later
- Library architecture for future UI development
Research Findings:
- Current handshake monitor stops after first failure (critical bug)
- No graceful tunnel restart
- No endpoint failover
- Linux-only currently
Work Objectives
Core Objective
Transform device-agent from fragile single-server daemon to reliable cross-platform daemon with automatic failover and health monitoring.
Concrete Deliverables
apps/device-agent/internal/statemachine.go— State machine coreapps/device-agent/internal/healthcheck.go— Health check systemapps/device-agent/internal/failover.go— Failover managerapps/device-agent/README.md— Installation and usage documentationapps/device-agent/internal/tunnel/wireguard.go— Updated with restart capabilityapps/device-agent/internal/client/heartbeat.go— Updated with status reportingapps/server-core/api/heartbeat.go— Updated to accept status reportapps/server-core/api/status.go— New status API endpoints
Definition of Done
- Agent gracefully handles tunnel restart
- Handshake monitor restarts after recovery
- Agent fails over to secondary server
- Agent tries multiple endpoints per server
- CLI help menu is clear and comprehensive
- Installation docs cover Linux, Windows, Android
- Cross-compilation works for all platforms
Must Have
- State machine with Idle/Connected/Recovering/Stopped states
- Server failover (multiple servers)
- Endpoint failover (multiple endpoints per server)
- Health checks (handshake, heartbeat, tunnel, network)
- CLI help menu
- Installation documentation
Must NOT Have (Guardrails)
- No database changes (device-agent is stateless)
- No GUI in this phase (terminal/library only)
- No complex terminal UI (just help menu and status)
- No Android-specific code (just cross-compile)
Verification Strategy
ZERO HUMAN INTERVENTION - ALL verification is agent-executed.
Test Decision
- Infrastructure exists: YES (existing test files in
internal/client/) - Automated tests: Tests-after
- Framework: Go testing
QA Policy
Every task MUST include agent-executed QA scenarios.
Execution Strategy
Parallel Execution Waves
Wave 1 (Start Immediately - foundation):
├── Task 1: State Machine Core [deep]
├── Task 2: Tunnel Restart Fix [quick]
├── Task 3: Handshake Monitor Fix [quick]
└── Task 4: CLI Help Menu [quick]
Wave 2 (After Wave 1 - reliability):
├── Task 5: Health Check System [deep]
├── Task 6: Failover Manager [deep]
├── Task 7: Provisioning Timeout [quick]
└── Task 8: Enhanced Logging [quick]
Wave 3 (After Wave 2 - integration):
├── Task 9: Server Heartbeat API Update [quick]
├── Task 10: Server Status API [quick]
├── Task 11: Installation Documentation [writing]
└── Task 12: Configuration Documentation [writing]
Wave 4 (After Wave 3 - cross-platform):
├── Task 13: Windows Support [deep]
├── Task 14: Android Support [deep]
└── Task 15: Cross-Compile Pipeline [quick]
Wave FINAL (After ALL tasks):
├── Task F1: Plan compliance audit [oracle]
├── Task F2: Code quality review [unspecified-high]
├── Task F3: Real manual QA [unspecified-high]
└── Task F4: Scope fidelity check [deep]
-> Present results -> Get explicit user okay
Critical Path: Task 1 → Task 5 → Task 6 → Task 9 → Task 13 → F1-F4
Parallel Speedup: ~60% faster than sequential
Max Concurrent: 4 (Waves 1 & 2)
Dependency Matrix
| Task | Depends On | Blocks |
|---|---|---|
| 1 | None | 5, 6 |
| 2 | None | 5 |
| 3 | None | 5 |
| 4 | None | None |
| 5 | 1, 2, 3 | 6 |
| 6 | 1, 5 | 9 |
| 7 | None | 5 |
| 8 | None | 9 |
| 9 | 6, 8 | None |
| 10 | 9 | None |
| 11 | None | None |
| 12 | None | None |
| 13 | 1 | None |
| 14 | 1 | None |
| 15 | 13, 14 | None |
Agent Dispatch Summary
- Wave 1: 4 tasks — T1
deep, T2quick, T3quick, T4quick - Wave 2: 4 tasks — T5
deep, T6deep, T7quick, T8quick - Wave 3: 4 tasks — T9
quick, T10quick, T11writing, T12writing - Wave 4: 3 tasks — T13
deep, T14deep, T15quick - FINAL: 4 tasks — F1
oracle, F2unspecified-high, F3unspecified-high, F4deep
TODOs
-
1. State Machine Core
What to do:
- Create
apps/device-agent/internal/statemachine.go - Define states: Idle, Connected, Recovering, Stopped
- Implement state transitions with event triggers
- Add context propagation for clean cancellation
- Add state change logging
Must NOT do:
- No database integration
- No complex state history (just current state)
Recommended Agent Profile:
- Category:
deep - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 1 (with Tasks 2, 3, 4)
- Blocks: Tasks 5, 6
- Blocked By: None
References:
apps/device-agent/internal/client/heartbeat.go— Current reconnect logicapps/device-agent/main.go— Current lifecycle flow
Acceptance Criteria:
- State machine file created
- All 4 states defined
- Transitions work correctly
- Context cancellation works
- State changes are logged
QA Scenarios:
Scenario: State transitions work correctly Tool: Bash (go test) Preconditions: State machine implemented Steps: 1. Run unit tests for state machine 2. Verify all transitions are covered 3. Verify context cancellation works Expected Result: All tests pass Evidence: .sisyphus/evidence/task-1-state-machine-tests.txtCommit: YES
- Message:
feat(agent): add state machine core - Files:
apps/device-agent/internal/statemachine.go
- Create
-
2. Tunnel Restart Fix
What to do:
- Add
StopStealthTunnel()method toTunnelManager - Implement graceful restart logic in
StartStealthTunnel - Add interface cleanup wait (2s)
- Verify tunnel is up after restart
Must NOT do:
- No changes to WireGuard config format
- No changes to stealth architecture
Recommended Agent Profile:
- Category:
quick - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 1 (with Tasks 1, 3, 4)
- Blocks: Task 5
- Blocked By: None
References:
apps/device-agent/internal/tunnel/wireguard.go— Current tunnel manager
Acceptance Criteria:
StopStealthTunnel()method exists- Graceful restart works
- Interface cleanup wait implemented
- Tunnel verification after restart
QA Scenarios:
Scenario: Tunnel restart works Tool: Bash (go test) Preconditions: Tunnel manager updated Steps: 1. Run tunnel restart tests 2. Verify old tunnel is stopped 3. Verify new tunnel starts Expected Result: Restart completes without error Evidence: .sisyphus/evidence/task-2-tunnel-restart.txtCommit: YES
- Message:
fix(agent): add graceful tunnel restart - Files:
apps/device-agent/internal/tunnel/wireguard.go
- Add
-
3. Handshake Monitor Fix
What to do:
- Fix
MonitorHandshaketo restart after recovery - Add handshake monitor restart in reconnect flow
- Ensure monitoring continues after tunnel restart
Must NOT do:
- No changes to handshake timeout values
- No changes to IPC parsing
Recommended Agent Profile:
- Category:
quick - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 1 (with Tasks 1, 2, 4)
- Blocks: Task 5
- Blocked By: None
References:
apps/device-agent/internal/client/heartbeat.go:47-82— Current MonitorHandshake
Acceptance Criteria:
- Handshake monitor restarts after recovery
- Monitoring continues after tunnel restart
- No duplicate monitors running
QA Scenarios:
Scenario: Handshake monitor restarts Tool: Bash (go test) Preconditions: Heartbeat code updated Steps: 1. Run handshake monitor tests 2. Simulate failure and recovery 3. Verify monitoring resumes Expected Result: Monitoring restarts correctly Evidence: .sisyphus/evidence/task-3-handshake-monitor.txtCommit: YES
- Message:
fix(agent): restart handshake monitor after recovery - Files:
apps/device-agent/internal/client/heartbeat.go
- Fix
-
4. CLI Help Menu
What to do:
- Improve CLI help output
- Add command descriptions and examples
- Add global flags documentation
- Add version information
Must NOT do:
- No complex terminal UI
- No interactive prompts
Recommended Agent Profile:
- Category:
quick - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 1 (with Tasks 1, 2, 3)
- Blocks: None
- Blocked By: None
References:
apps/device-agent/main.go— Current CLI parsing
Acceptance Criteria:
- Help menu is comprehensive
- Commands have descriptions
- Examples are provided
- Global flags are documented
QA Scenarios:
Scenario: Help menu works Tool: Bash Preconditions: CLI updated Steps: 1. Run `nexusguard-agent --help` 2. Verify all commands listed 3. Verify descriptions present Expected Result: Help output is clear Evidence: .sisyphus/evidence/task-4-help-menu.txtCommit: YES
- Message:
docs(agent): improve CLI help menu - Files:
apps/device-agent/main.go
-
5. Health Check System
What to do:
- Create
apps/device-agent/internal/healthcheck.go - Implement handshake, heartbeat, tunnel, network checks
- Add failure thresholds and actions
- Integrate with state machine
Must NOT do:
- No external health check dependencies
- No complex metrics collection
Recommended Agent Profile:
- Category:
deep - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 2 (with Tasks 6, 7, 8)
- Blocks: Task 6
- Blocked By: Tasks 1, 2, 3
References:
apps/device-agent/internal/client/heartbeat.go— Current health checks
Acceptance Criteria:
- All 4 check types implemented
- Failure thresholds work
- Actions triggered correctly
- Integration with state machine
QA Scenarios:
Scenario: Health checks work Tool: Bash (go test) Preconditions: Health check system implemented Steps: 1. Run health check tests 2. Simulate failures 3. Verify actions triggered Expected Result: All checks work correctly Evidence: .sisyphus/evidence/task-5-health-checks.txtCommit: YES
- Message:
feat(agent): add health check system - Files:
apps/device-agent/internal/healthcheck.go
- Create
-
6. Failover Manager
What to do:
- Create
apps/device-agent/internal/failover.go - Implement server failover (multiple servers)
- Implement endpoint failover (multiple endpoints per server)
- Add priority-based server selection
- Add exponential backoff
Must NOT do:
- No DNS-based failover
- No geographic-based failover
Recommended Agent Profile:
- Category:
deep - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 2 (with Tasks 5, 7, 8)
- Blocks: Task 9
- Blocked By: Tasks 1, 5
References:
apps/device-agent/internal/client/provisioning.go— Current provisioning
Acceptance Criteria:
- Server failover works
- Endpoint failover works
- Priority-based selection works
- Exponential backoff works
QA Scenarios:
Scenario: Failover works Tool: Bash (go test) Preconditions: Failover manager implemented Steps: 1. Run failover tests 2. Simulate server failure 3. Verify failover to next server Expected Result: Failover completes successfully Evidence: .sisyphus/evidence/task-6-failover.txtCommit: YES
- Message:
feat(agent): add failover manager - Files:
apps/device-agent/internal/failover.go
- Create
-
7. Provisioning Timeout
What to do:
- Add context cancellation to provisioning
- Add HTTP request timeouts
- Improve error handling
Must NOT do:
- No changes to provisioning protocol
- No changes to encryption
Recommended Agent Profile:
- Category:
quick - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 2 (with Tasks 5, 6, 8)
- Blocks: Task 5
- Blocked By: None
References:
apps/device-agent/internal/client/provisioning.go— Current provisioning
Acceptance Criteria:
- Context cancellation works
- HTTP timeouts implemented
- Error handling improved
QA Scenarios:
Scenario: Provisioning timeout works Tool: Bash (go test) Preconditions: Provisioning updated Steps: 1. Run provisioning tests 2. Simulate timeout 3. Verify cancellation works Expected Result: Timeout handling works Evidence: .sisyphus/evidence/task-7-provisioning-timeout.txtCommit: YES
- Message:
fix(agent): add provisioning timeout - Files:
apps/device-agent/internal/client/provisioning.go
-
8. Enhanced Logging
What to do:
- Implement structured JSON logging
- Add component-based logging
- Add state change logging
- Add error context logging
Must NOT do:
- No logging of sensitive data (tokens, keys)
- No external logging dependencies
Recommended Agent Profile:
- Category:
quick - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 2 (with Tasks 5, 6, 7)
- Blocks: Task 9
- Blocked By: None
References:
apps/device-agent/main.go— Current logging
Acceptance Criteria:
- Structured JSON logging works
- Component-based logging works
- State changes are logged
- No sensitive data logged
QA Scenarios:
Scenario: Logging works Tool: Bash Preconditions: Logging implemented Steps: 1. Run agent with --json flag 2. Verify JSON output format 3. Verify no sensitive data in logs Expected Result: Logging works correctly Evidence: .sisyphus/evidence/task-8-logging.txtCommit: YES
- Message:
feat(agent): add structured JSON logging - Files:
apps/device-agent/main.go
-
9. Server Heartbeat API Update
What to do:
- Update heartbeat handler to accept status report
- Store status in existing heartbeat tables
- Add validation for status fields
Must NOT do:
- No database schema changes
- No new database tables
Recommended Agent Profile:
- Category:
quick - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 3 (with Tasks 10, 11, 12)
- Blocks: Task 10
- Blocked By: Tasks 6, 8
References:
apps/server-core/api/heartbeat.go— Current heartbeat handler
Acceptance Criteria:
- Heartbeat API accepts status report
- Status stored in existing tables
- Validation works
QA Scenarios:
Scenario: Heartbeat API accepts status Tool: Bash (curl) Preconditions: API updated Steps: 1. Send heartbeat with status 2. Verify 200 response 3. Verify status stored Expected Result: API works correctly Evidence: .sisyphus/evidence/task-9-heartbeat-api.txtCommit: YES
- Message:
feat(server): update heartbeat API for status - Files:
apps/server-core/api/heartbeat.go
-
10. Server Status API
What to do:
- Create status API endpoints
- Add GET device status
- Add GET list all device statuses
Must NOT do:
- No database schema changes
- No complex queries
Recommended Agent Profile:
- Category:
quick - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 3 (with Tasks 9, 11, 12)
- Blocks: None
- Blocked By: Task 9
References:
apps/server-core/api/— Existing API patterns
Acceptance Criteria:
- GET device status works
- GET list all device statuses works
- Response format consistent
QA Scenarios:
Scenario: Status API works Tool: Bash (curl) Preconditions: API implemented Steps: 1. Call GET device status 2. Verify response format 3. Call GET list all Expected Result: API works correctly Evidence: .sisyphus/evidence/task-10-status-api.txtCommit: YES
- Message:
feat(server): add status API endpoints - Files:
apps/server-core/api/status.go
-
11. Installation Documentation
What to do:
- Write Linux installation guide
- Write Windows installation guide
- Write Android installation guide
- Add troubleshooting section
Must NOT do:
- No complex diagrams
- No video tutorials
Recommended Agent Profile:
- Category:
writing - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 3 (with Tasks 9, 10, 12)
- Blocks: None
- Blocked By: None
References:
apps/device-agent/README.md— Current documentation
Acceptance Criteria:
- Linux guide complete
- Windows guide complete
- Android guide complete
- Troubleshooting section added
QA Scenarios:
Scenario: Documentation is complete Tool: Bash (read) Preconditions: Documentation written Steps: 1. Read documentation files 2. Verify all sections present 3. Verify examples are correct Expected Result: Documentation is complete Evidence: .sisyphus/evidence/task-11-documentation.txtCommit: YES
- Message:
docs(agent): add installation guides - Files:
apps/device-agent/README.md
-
12. Configuration Documentation
What to do:
- Document config file format
- Document all configuration options
- Add configuration examples
Must NOT do:
- No complex configuration schemas
- No environment variable documentation (already exists)
Recommended Agent Profile:
- Category:
writing - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 3 (with Tasks 9, 10, 11)
- Blocks: None
- Blocked By: None
References:
apps/device-agent/.env.example— Current config
Acceptance Criteria:
- Config format documented
- All options documented
- Examples provided
QA Scenarios:
Scenario: Configuration docs complete Tool: Bash (read) Preconditions: Documentation written Steps: 1. Read config documentation 2. Verify all options listed 3. Verify examples work Expected Result: Documentation is complete Evidence: .sisyphus/evidence/task-12-config-docs.txtCommit: YES
- Message:
docs(agent): add configuration guide - Files:
apps/device-agent/docs/configuration.md
-
13. Windows Support
What to do:
- Add Windows Service integration
- Add WireGuard NT driver support
- Add WMI UUID detection
- Add Windows firewall integration
Must NOT do:
- No GUI components
- No complex Windows-specific features
Recommended Agent Profile:
- Category:
deep - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 4 (with Tasks 14, 15)
- Blocks: Task 15
- Blocked By: Task 1
References:
apps/device-agent/internal/identity/— Current identity detectionapps/device-agent/internal/tunnel/— Current tunnel implementation
Acceptance Criteria:
- Windows Service works
- WireGuard NT integration works
- WMI UUID detection works
- Windows firewall integration works
QA Scenarios:
Scenario: Windows support works Tool: Bash (cross-compile) Preconditions: Windows support implemented Steps: 1. Cross-compile for Windows 2. Verify binary runs on Windows 3. Verify Service installation works Expected Result: Windows support works Evidence: .sisyphus/evidence/task-13-windows-support.txtCommit: YES
- Message:
feat(agent): add Windows support - Files:
apps/device-agent/internal/platform/windows.go
-
14. Android Support
What to do:
- Add gomobile binding
- Add Foreground Service integration
- Add Android ID detection
Must NOT do:
- No Android UI components
- No complex Android-specific features
Recommended Agent Profile:
- Category:
deep - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 4 (with Tasks 13, 15)
- Blocks: Task 15
- Blocked By: Task 1
References:
apps/device-agent/internal/identity/— Current identity detectionapps/device-agent/internal/tunnel/— Current tunnel implementation
Acceptance Criteria:
- gomobile binding works
- Foreground Service works
- Android ID detection works
QA Scenarios:
Scenario: Android support works Tool: Bash (gomobile) Preconditions: Android support implemented Steps: 1. Build AAR library 2. Verify library compiles 3. Verify Android ID detection Expected Result: Android support works Evidence: .sisyphus/evidence/task-14-android-support.txtCommit: YES
- Message:
feat(agent): add Android support - Files:
apps/device-agent/internal/platform/android.go
-
15. Cross-Compile Pipeline
What to do:
- Add cross-compilation scripts
- Add build matrix for all platforms
- Add release packaging
Must NOT do:
- No CI/CD changes (separate task)
- No code signing
Recommended Agent Profile:
- Category:
quick - Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 4 (with Tasks 13, 14)
- Blocks: None
- Blocked By: Tasks 13, 14
References:
apps/device-agent/.gitea/workflows/build.yml— Current build
Acceptance Criteria:
- Cross-compilation works for all platforms
- Build matrix configured
- Release packaging works
QA Scenarios:
Scenario: Cross-compilation works Tool: Bash Preconditions: Build scripts created Steps: 1. Run cross-compilation 2. Verify all binaries created 3. Verify binaries run Expected Result: Cross-compilation works Evidence: .sisyphus/evidence/task-15-cross-compile.txtCommit: YES
- Message:
ci(agent): add cross-compilation pipeline - Files:
apps/device-agent/Makefile
Final Verification Wave
-
F1. Plan Compliance Audit —
oracleRead the plan end-to-end. For each "Must Have": verify implementation exists. For each "Must NOT Have": search codebase for forbidden patterns. Compare deliverables against plan. Output:Must Have [N/N] | Must NOT Have [N/N] | Tasks [N/N] | VERDICT: APPROVE/REJECT -
F2. Code Quality Review —
unspecified-highRungo vet,go test,go build. Review all changed files for: error handling, logging, documentation. Check for AI slop: excessive comments, over-abstraction. Output:Build [PASS/FAIL] | Tests [N pass/N fail] | Files [N clean/N issues] | VERDICT -
F3. Real Manual QA —
unspecified-highStart from clean state. Execute EVERY QA scenario from EVERY task. Test cross-task integration. Test edge cases. Save to.sisyphus/evidence/final-qa/. Output:Scenarios [N/N pass] | Integration [N/N] | Edge Cases [N tested] | VERDICT -
F4. Scope Fidelity Check —
deepFor each task: read "What to do", read actual diff. Verify 1:1 — everything in spec was built, nothing beyond spec was built. Check "Must NOT do" compliance. Flag unaccounted changes. Output:Tasks [N/N compliant] | Unaccounted [CLEAN/N files] | VERDICT
Commit Strategy
- Task 1:
feat(agent): add state machine core— statemachine.go - Task 2:
fix(agent): add graceful tunnel restart— wireguard.go - Task 3:
fix(agent): restart handshake monitor after recovery— heartbeat.go - Task 4:
docs(agent): improve CLI help menu— main.go - Task 5:
feat(agent): add health check system— healthcheck.go - Task 6:
feat(agent): add failover manager— failover.go - Task 7:
fix(agent): add provisioning timeout— provisioning.go - Task 8:
feat(agent): add structured JSON logging— main.go - Task 9:
feat(server): update heartbeat API for status— heartbeat.go - Task 10:
feat(server): add status API endpoints— status.go - Task 11:
docs(agent): add installation guides— README.md - Task 12:
docs(agent): add configuration guide— configuration.md - Task 13:
feat(agent): add Windows support— windows.go - Task 14:
feat(agent): add Android support— android.go - Task 15:
ci(agent): add cross-compilation pipeline— Makefile
Success Criteria
Verification Commands
# Build
cd apps/device-agent && go build -o nexusguard-agent .
# Test
cd apps/device-agent && go test ./...
# Cross-compile
GOOS=linux GOARCH=amd64 go build -o nexusguard-agent-linux-amd64 .
GOOS=windows GOARCH=amd64 go build -o nexusguard-agent-windows-amd64.exe .
# Run
./nexusguard-agent --help
./nexusguard-agent status
Final Checklist
- All "Must Have" present
- All "Must NOT Have" absent
- All tests pass
- Cross-compilation works
- Documentation complete
- CLI help menu works