feat: add ESP32 embedded agent + namespace VRF reference docs + update android-agent submodule
- apps/device-agent-embedded/: ESP32 WireGuard agent (C, ESP-IDF, mbedTLS) - reference/namespace-vrf/: Linux namespace/VRF network isolation docs - apps/android-agent: VPN permission check, StartResult sealed class
This commit is contained in:
@@ -0,0 +1,492 @@
|
||||
# Namespace + VRF Troubleshooting Guide
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Issue 1: Namespace Creation Fails
|
||||
|
||||
**Symptoms:**
|
||||
```bash
|
||||
$ ip netns add ns_test
|
||||
RTNETLINK answers: Operation not permitted
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- Not running as root
|
||||
- Kernel namespace support disabled
|
||||
- Container limitations
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Solution 1: Run as root
|
||||
sudo ip netns add ns_test
|
||||
|
||||
# Solution 2: Check kernel support
|
||||
modprobe vrf
|
||||
modprobe wireguard
|
||||
|
||||
# Solution 3: Check container capabilities
|
||||
# Docker needs --cap-add=NET_ADMIN --cap-add=SYS_ADMIN
|
||||
docker run --cap-add=NET_ADMIN --cap-add=SYS_ADMIN ...
|
||||
```
|
||||
|
||||
### Issue 2: VRF Creation Fails
|
||||
|
||||
**Symptoms:**
|
||||
```bash
|
||||
$ ip netns exec ns_test ip link add vrf:main type vrf table 100
|
||||
RTNETLINK answers: No such file or directory
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- VRF module not loaded
|
||||
- Table ID already in use
|
||||
- Namespace doesn't exist
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Solution 1: Load VRF module
|
||||
sudo modprobe vrf
|
||||
|
||||
# Solution 2: Check existing VRFs
|
||||
ip netns exec ns_test ip link show type vrf
|
||||
|
||||
# Solution 3: Use different table ID
|
||||
ip netns exec ns_test ip link add vrf:main type vrf table 200
|
||||
|
||||
# Solution 4: Verify namespace exists
|
||||
ip netns list | grep ns_test
|
||||
```
|
||||
|
||||
### Issue 3: WireGuard Interface Won't Bind to VRF
|
||||
|
||||
**Symptoms:**
|
||||
```bash
|
||||
$ ip netns exec ns_test ip link set wg0 vrf vrf:main
|
||||
RTNETLINK answers: Invalid argument
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- VRF not up
|
||||
- Interface doesn't exist
|
||||
- VRF name mismatch
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Solution 1: Ensure VRF is up
|
||||
ip netns exec ns_test ip link set vrf:main up
|
||||
|
||||
# Solution 2: Verify interface exists
|
||||
ip netns exec ns_test ip link show
|
||||
|
||||
# Solution 3: Check VRF name
|
||||
ip netns exec ns_test ip link show type vrf
|
||||
|
||||
# Solution 4: Create interface first
|
||||
ip netns exec ns_test ip link add wg0 type wireguard
|
||||
ip netns exec ns_test ip link set wg0 vrf vrf:main
|
||||
```
|
||||
|
||||
### Issue 4: No Connectivity in Namespace
|
||||
|
||||
**Symptoms:**
|
||||
```bash
|
||||
$ ip netns exec ns_test ping 10.0.0.1
|
||||
connect: Network is unreachable
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- No default route
|
||||
- Interface not configured
|
||||
- Routing table missing
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Solution 1: Add default route
|
||||
ip netns exec ns_test ip route add default via 10.0.0.1
|
||||
|
||||
# Solution 2: Configure interface
|
||||
ip netns exec ns_test ip addr add 10.0.0.2/24 dev eth0
|
||||
ip netns exec ns_test ip link set eth0 up
|
||||
|
||||
# Solution 3: Check routing table
|
||||
ip netns exec ns_test ip route show
|
||||
|
||||
# Solution 4: Check VRF routes
|
||||
ip netns exec ns_test ip route show table 100
|
||||
```
|
||||
|
||||
### Issue 5: nftables Rules Not Working
|
||||
|
||||
**Symptoms:**
|
||||
```bash
|
||||
$ ip netns exec ns_test nft list table ip nexusguard
|
||||
table ip nexusguard {
|
||||
chain forward {
|
||||
type filter hook forward priority filter; policy accept;
|
||||
}
|
||||
}
|
||||
# But traffic not being filtered
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- Rules in wrong table
|
||||
- Chain not hooked
|
||||
- Interface mismatch
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Solution 1: Verify chain hook
|
||||
ip netns exec ns_test nft list chain ip nexusguard forward
|
||||
|
||||
# Solution 2: Add rules correctly
|
||||
ip netns exec ns_test nft add rule ip nexusguard forward iifname wg0 accept
|
||||
|
||||
# Solution 3: Check interface names
|
||||
ip netns exec ns_test ip link show
|
||||
|
||||
# Solution 4: Flush and re-add rules
|
||||
ip netns exec ns_test nft flush chain ip nexusguard forward
|
||||
```
|
||||
|
||||
### Issue 6: Cross-Namespace Communication Fails
|
||||
|
||||
**Symptoms:**
|
||||
```bash
|
||||
# From ns_alice
|
||||
$ ip netns exec ns_alice ping 10.100.0.1
|
||||
connect: Network is unreachable
|
||||
|
||||
# But 10.100.0.1 is in ns_zerocode
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- No routing between namespaces
|
||||
- VRF isolation working correctly
|
||||
- Need veth pair for cross-namespace
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Solution 1: Create veth pair (if needed)
|
||||
ip link add veth-alice type veth peer name veth-zerocode
|
||||
ip link set veth-zerocode netns ns_zerocode
|
||||
ip link set veth-alice netns ns_alice
|
||||
|
||||
# Solution 2: Configure veth
|
||||
ip netns exec ns_alice ip addr add 10.0.0.1/24 dev veth-alice
|
||||
ip netns exec ns_alice ip link set veth-alice up
|
||||
ip netns exec ns_alice ip route add default via 10.0.0.1
|
||||
|
||||
ip netns exec ns_zerocode ip addr add 10.0.0.2/24 dev veth-zerocode
|
||||
ip netns exec ns_zerocode ip link set veth-zerocode up
|
||||
ip netns exec ns_zerocode ip route add default via 10.0.0.2
|
||||
|
||||
# Solution 3: Add static routes
|
||||
ip netns exec ns_alice ip route add 10.100.0.0/24 via 10.0.0.2
|
||||
ip netns exec ns_zerocode ip route add 10.100.0.0/24 via 10.0.0.1
|
||||
```
|
||||
|
||||
### Issue 7: WireGuard Handshake Fails
|
||||
|
||||
**Symptoms:**
|
||||
```bash
|
||||
$ ip netns exec ns_test wg show
|
||||
interface: wg0
|
||||
peer: <public_key>
|
||||
endpoint: 1.2.3.4:51820
|
||||
allowed ips: 10.100.0.2/32
|
||||
latest handshake: (none)
|
||||
transfer: 0 B received, 0 B sent
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- Firewall blocking UDP
|
||||
- NAT not configured
|
||||
- Endpoint unreachable
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Solution 1: Check firewall
|
||||
ip netns exec ns_test nft list ruleset
|
||||
|
||||
# Solution 2: Allow UDP in firewall
|
||||
ip netns exec ns_test nft add rule ip nexusguard input udp dport 51820 accept
|
||||
|
||||
# Solution 3: Configure NAT (if needed)
|
||||
ip netns exec ns_test nft add table ip nat
|
||||
ip netns exec ns_test nft add chain ip nat postrouting '{type nat hook postrouting priority 100;}'
|
||||
ip netns exec ns_test nft add rule ip nat postrouting oifname eth0 masquerade
|
||||
|
||||
# Solution 4: Check connectivity
|
||||
ip netns exec ns_test ping 1.2.3.4
|
||||
```
|
||||
|
||||
### Issue 8: Permission Denied for API Calls
|
||||
|
||||
**Symptoms:**
|
||||
```json
|
||||
{
|
||||
"error": "permission denied"
|
||||
}
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- User not in namespace
|
||||
- Wrong role
|
||||
- Token expired
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Solution 1: Check user in namespace
|
||||
psql -U nexusguard -d nexusguard -c "
|
||||
SELECT * FROM namespace_users
|
||||
WHERE namespace_id = (SELECT id FROM namespaces WHERE name = 'testns');
|
||||
"
|
||||
|
||||
# Solution 2: Add user to namespace
|
||||
curl -X POST http://localhost:8080/api/v1/ns/testns/users \
|
||||
-H "Authorization: Bearer <admin_token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"user_id": "<user_id>", "role": "admin"}'
|
||||
|
||||
# Solution 3: Check role
|
||||
psql -U nexusguard -d nexusguard -c "
|
||||
SELECT nu.role FROM namespace_users nu
|
||||
JOIN namespaces n ON nu.namespace_id = n.id
|
||||
WHERE n.name = 'testns' AND nu.user_id = '<user_id>';
|
||||
"
|
||||
```
|
||||
|
||||
## Debugging Commands
|
||||
|
||||
### Check Namespace Status
|
||||
|
||||
```bash
|
||||
# List all namespaces
|
||||
ip netns list
|
||||
|
||||
# Check namespace exists
|
||||
ip netns list | grep ns_test
|
||||
|
||||
# Enter namespace shell
|
||||
ip netns exec ns_test bash
|
||||
|
||||
# Check namespace processes
|
||||
ip netns exec ns_test ps aux
|
||||
```
|
||||
|
||||
### Check VRF Status
|
||||
|
||||
```bash
|
||||
# List VRFs in namespace
|
||||
ip netns exec ns_test ip link show type vrf
|
||||
|
||||
# Check VRF details
|
||||
ip -d link show type vrf
|
||||
|
||||
# Check VRF routes
|
||||
ip netns exec ns_test ip route show table 100
|
||||
```
|
||||
|
||||
### Check WireGuard Status
|
||||
|
||||
```bash
|
||||
# Show WireGuard interfaces
|
||||
ip netns exec ns_test wg show
|
||||
|
||||
# Show WireGuard details
|
||||
ip netns exec ns_test wg show wg0
|
||||
|
||||
# Check WireGuard config
|
||||
ip netns exec ns_test wg showconf wg0
|
||||
```
|
||||
|
||||
### Check nftables Status
|
||||
|
||||
```bash
|
||||
# List all tables
|
||||
ip netns exec ns_test nft list tables
|
||||
|
||||
# List specific table
|
||||
ip netns exec ns_test nft list table ip nexusguard
|
||||
|
||||
# List rules with handles
|
||||
ip netns exec ns_test nft -a list table ip nexusguard
|
||||
|
||||
# Check rule counters
|
||||
ip netns exec ns_test nft list chain ip nexusguard forward
|
||||
```
|
||||
|
||||
### Check Connectivity
|
||||
|
||||
```bash
|
||||
# Ping from namespace
|
||||
ip netns exec ns_test ping 10.100.0.1
|
||||
|
||||
# Traceroute from namespace
|
||||
ip netns exec ns_test traceroute 10.100.0.1
|
||||
|
||||
# Capture packets
|
||||
ip netns exec ns_test tcpdump -i wg0 -n
|
||||
|
||||
# Check routing table
|
||||
ip netns exec ns_test ip route show
|
||||
|
||||
# Check ARP table
|
||||
ip netns exec ns_test ip neigh show
|
||||
```
|
||||
|
||||
### Check Database State
|
||||
|
||||
```bash
|
||||
# List namespaces
|
||||
psql -U nexusguard -d nexusguard -c "SELECT * FROM namespaces;"
|
||||
|
||||
# List namespace users
|
||||
psql -U nexusguard -d nexusguard -c "SELECT * FROM namespace_users;"
|
||||
|
||||
# List VRFs
|
||||
psql -U nexusguard -d nexusguard -c "SELECT * FROM vrfs;"
|
||||
|
||||
# List servers with VRF
|
||||
psql -U nexusguard -d nexusguard -c "
|
||||
SELECT s.name, s.interface_name, n.name as namespace, v.name as vrf
|
||||
FROM wg_servers s
|
||||
LEFT JOIN namespaces n ON s.namespace_id = n.id
|
||||
LEFT JOIN vrfs v ON s.vrf_id = v.id;
|
||||
"
|
||||
```
|
||||
|
||||
## Log Analysis
|
||||
|
||||
### Server Logs
|
||||
|
||||
```bash
|
||||
# Check server logs
|
||||
journalctl -u nexusguard-server -f
|
||||
|
||||
# Check for VRF errors
|
||||
journalctl -u nexusguard-server | grep -i vrf
|
||||
|
||||
# Check for namespace errors
|
||||
journalctl -u nexusguard-server | grep -i namespace
|
||||
```
|
||||
|
||||
### Kernel Logs
|
||||
|
||||
```bash
|
||||
# Check kernel messages
|
||||
dmesg | grep -i vrf
|
||||
dmesg | grep -i wireguard
|
||||
dmesg | grep -i namespace
|
||||
```
|
||||
|
||||
### nftables Logs
|
||||
|
||||
```bash
|
||||
# Enable nftables logging
|
||||
ip netns exec ns_test nft add rule ip nexusguard forward log prefix "nft: " accept
|
||||
|
||||
# Check logs
|
||||
journalctl -k | grep "nft:"
|
||||
```
|
||||
|
||||
## Performance Issues
|
||||
|
||||
### High CPU Usage
|
||||
|
||||
```bash
|
||||
# Check CPU usage per namespace
|
||||
top -p $(ip netns exec ns_test pgrep -d,)
|
||||
|
||||
# Check WireGuard CPU usage
|
||||
ip netns exec ns_test wg show | grep -A 5 "transfer"
|
||||
```
|
||||
|
||||
### High Memory Usage
|
||||
|
||||
```bash
|
||||
# Check memory per namespace
|
||||
cat /proc/net/ns/*/mem 2>/dev/null || echo "Not available"
|
||||
|
||||
# Check total namespace memory
|
||||
ps aux | grep "ip netns exec" | awk '{sum+=$6} END {print sum/1024 " MB"}'
|
||||
```
|
||||
|
||||
### Network Latency
|
||||
|
||||
```bash
|
||||
# Measure latency
|
||||
ip netns exec ns_test ping -c 100 10.100.0.1 | tail -1
|
||||
|
||||
# Check interface stats
|
||||
ip netns exec ns_test ip -s link show wg0
|
||||
```
|
||||
|
||||
## Cleanup Commands
|
||||
|
||||
### Remove Namespace
|
||||
|
||||
```bash
|
||||
# Delete namespace (removes everything)
|
||||
ip netns del ns_test
|
||||
|
||||
# Or manually cleanup
|
||||
ip netns exec ns_test ip link del wg0
|
||||
ip netns exec ns_test ip link del vrf:main
|
||||
ip netns exec ns_test ip link del veth0
|
||||
ip netns del ns_test
|
||||
```
|
||||
|
||||
### Remove VRF
|
||||
|
||||
```bash
|
||||
# Delete VRF
|
||||
ip netns exec ns_test ip link del vrf:main
|
||||
|
||||
# Verify deletion
|
||||
ip netns exec ns_test ip link show type vrf
|
||||
```
|
||||
|
||||
### Remove WireGuard Interface
|
||||
|
||||
```bash
|
||||
# Delete WireGuard interface
|
||||
ip netns exec ns_test ip link del wg0
|
||||
|
||||
# Verify deletion
|
||||
ip netns exec ns_test wg show
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
### Check Documentation
|
||||
|
||||
```bash
|
||||
# Kernel documentation
|
||||
man ip-netns
|
||||
man ip-link
|
||||
man wg
|
||||
|
||||
# NexusGuard documentation
|
||||
ls reference/namespace-vrf/
|
||||
cat reference/namespace-vrf/architecture.md
|
||||
```
|
||||
|
||||
### Community Resources
|
||||
|
||||
- WireGuard mailing list: https://lists.zx2c4.com/pipermail/wireguard/
|
||||
- Linux kernel mailing list: https://lkml.org/
|
||||
- NexusGuard GitHub: https://github.com/nexusguard/nexusguard
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-07-10
|
||||
**Author:** NexusGuard Team
|
||||
Reference in New Issue
Block a user