Nexus-Guard-Suite/reference/namespace-vrf/troubleshooting.md

# Namespace + VRF Troubleshooting Guide

## Common Issues

### Issue 1: Namespace Creation Fails

**Symptoms:**
```bash
$ ip netns add ns_test
RTNETLINK answers: Operation not permitted
```

**Causes:**
- Not running as root
- Kernel namespace support disabled
- Container limitations

**Solutions:**

```bash
# Solution 1: Run as root
sudo ip netns add ns_test

# Solution 2: Check kernel support
modprobe vrf
modprobe wireguard

# Solution 3: Check container capabilities
# Docker needs --cap-add=NET_ADMIN --cap-add=SYS_ADMIN
docker run --cap-add=NET_ADMIN --cap-add=SYS_ADMIN ...
```

### Issue 2: VRF Creation Fails

**Symptoms:**
```bash
$ ip netns exec ns_test ip link add vrf:main type vrf table 100
RTNETLINK answers: No such file or directory
```

**Causes:**
- VRF module not loaded
- Table ID already in use
- Namespace doesn't exist

**Solutions:**

```bash
# Solution 1: Load VRF module
sudo modprobe vrf

# Solution 2: Check existing VRFs
ip netns exec ns_test ip link show type vrf

# Solution 3: Use different table ID
ip netns exec ns_test ip link add vrf:main type vrf table 200

# Solution 4: Verify namespace exists
ip netns list | grep ns_test
```

### Issue 3: WireGuard Interface Won't Bind to VRF

**Symptoms:**
```bash
$ ip netns exec ns_test ip link set wg0 vrf vrf:main
RTNETLINK answers: Invalid argument
```

**Causes:**
- VRF not up
- Interface doesn't exist
- VRF name mismatch

**Solutions:**

```bash
# Solution 1: Ensure VRF is up
ip netns exec ns_test ip link set vrf:main up

# Solution 2: Verify interface exists
ip netns exec ns_test ip link show

# Solution 3: Check VRF name
ip netns exec ns_test ip link show type vrf

# Solution 4: Create interface first
ip netns exec ns_test ip link add wg0 type wireguard
ip netns exec ns_test ip link set wg0 vrf vrf:main
```

### Issue 4: No Connectivity in Namespace

**Symptoms:**
```bash
$ ip netns exec ns_test ping 10.0.0.1
connect: Network is unreachable
```

**Causes:**
- No default route
- Interface not configured
- Routing table missing

**Solutions:**

```bash
# Solution 1: Add default route
ip netns exec ns_test ip route add default via 10.0.0.1

# Solution 2: Configure interface
ip netns exec ns_test ip addr add 10.0.0.2/24 dev eth0
ip netns exec ns_test ip link set eth0 up

# Solution 3: Check routing table
ip netns exec ns_test ip route show

# Solution 4: Check VRF routes
ip netns exec ns_test ip route show table 100
```

### Issue 5: nftables Rules Not Working

**Symptoms:**
```bash
$ ip netns exec ns_test nft list table ip nexusguard
table ip nexusguard {
    chain forward {
        type filter hook forward priority filter; policy accept;
    }
}
# But traffic not being filtered
```

**Causes:**
- Rules in wrong table
- Chain not hooked
- Interface mismatch

**Solutions:**

```bash
# Solution 1: Verify chain hook
ip netns exec ns_test nft list chain ip nexusguard forward

# Solution 2: Add rules correctly
ip netns exec ns_test nft add rule ip nexusguard forward iifname wg0 accept

# Solution 3: Check interface names
ip netns exec ns_test ip link show

# Solution 4: Flush and re-add rules
ip netns exec ns_test nft flush chain ip nexusguard forward
```

### Issue 6: Cross-Namespace Communication Fails

**Symptoms:**
```bash
# From ns_alice
$ ip netns exec ns_alice ping 10.100.0.1
connect: Network is unreachable

# But 10.100.0.1 is in ns_zerocode
```

**Causes:**
- No routing between namespaces
- VRF isolation working correctly
- Need veth pair for cross-namespace

**Solutions:**

```bash
# Solution 1: Create veth pair (if needed)
ip link add veth-alice type veth peer name veth-zerocode
ip link set veth-zerocode netns ns_zerocode
ip link set veth-alice netns ns_alice

# Solution 2: Configure veth
ip netns exec ns_alice ip addr add 10.0.0.1/24 dev veth-alice
ip netns exec ns_alice ip link set veth-alice up
ip netns exec ns_alice ip route add default via 10.0.0.1

ip netns exec ns_zerocode ip addr add 10.0.0.2/24 dev veth-zerocode
ip netns exec ns_zerocode ip link set veth-zerocode up
ip netns exec ns_zerocode ip route add default via 10.0.0.2

# Solution 3: Add static routes
ip netns exec ns_alice ip route add 10.100.0.0/24 via 10.0.0.2
ip netns exec ns_zerocode ip route add 10.100.0.0/24 via 10.0.0.1
```

### Issue 7: WireGuard Handshake Fails

**Symptoms:**
```bash
$ ip netns exec ns_test wg show
interface: wg0
  peer: <public_key>
    endpoint: 1.2.3.4:51820
    allowed ips: 10.100.0.2/32
    latest handshake: (none)
    transfer: 0 B received, 0 B sent
```

**Causes:**
- Firewall blocking UDP
- NAT not configured
- Endpoint unreachable

**Solutions:**

```bash
# Solution 1: Check firewall
ip netns exec ns_test nft list ruleset

# Solution 2: Allow UDP in firewall
ip netns exec ns_test nft add rule ip nexusguard input udp dport 51820 accept

# Solution 3: Configure NAT (if needed)
ip netns exec ns_test nft add table ip nat
ip netns exec ns_test nft add chain ip nat postrouting '{type nat hook postrouting priority 100;}'
ip netns exec ns_test nft add rule ip nat postrouting oifname eth0 masquerade

# Solution 4: Check connectivity
ip netns exec ns_test ping 1.2.3.4
```

### Issue 8: Permission Denied for API Calls

**Symptoms:**
```json
{
  "error": "permission denied"
}
```

**Causes:**
- User not in namespace
- Wrong role
- Token expired

**Solutions:**

```bash
# Solution 1: Check user in namespace
psql -U nexusguard -d nexusguard -c "
SELECT * FROM namespace_users 
WHERE namespace_id = (SELECT id FROM namespaces WHERE name = 'testns');
"

# Solution 2: Add user to namespace
curl -X POST http://localhost:8080/api/v1/ns/testns/users \
  -H "Authorization: Bearer <admin_token>" \
  -H "Content-Type: application/json" \
  -d '{"user_id": "<user_id>", "role": "admin"}'

# Solution 3: Check role
psql -U nexusguard -d nexusguard -c "
SELECT nu.role FROM namespace_users nu
JOIN namespaces n ON nu.namespace_id = n.id
WHERE n.name = 'testns' AND nu.user_id = '<user_id>';
"
```

## Debugging Commands

### Check Namespace Status

```bash
# List all namespaces
ip netns list

# Check namespace exists
ip netns list | grep ns_test

# Enter namespace shell
ip netns exec ns_test bash

# Check namespace processes
ip netns exec ns_test ps aux
```

### Check VRF Status

```bash
# List VRFs in namespace
ip netns exec ns_test ip link show type vrf

# Check VRF details
ip -d link show type vrf

# Check VRF routes
ip netns exec ns_test ip route show table 100
```

### Check WireGuard Status

```bash
# Show WireGuard interfaces
ip netns exec ns_test wg show

# Show WireGuard details
ip netns exec ns_test wg show wg0

# Check WireGuard config
ip netns exec ns_test wg showconf wg0
```

### Check nftables Status

```bash
# List all tables
ip netns exec ns_test nft list tables

# List specific table
ip netns exec ns_test nft list table ip nexusguard

# List rules with handles
ip netns exec ns_test nft -a list table ip nexusguard

# Check rule counters
ip netns exec ns_test nft list chain ip nexusguard forward
```

### Check Connectivity

```bash
# Ping from namespace
ip netns exec ns_test ping 10.100.0.1

# Traceroute from namespace
ip netns exec ns_test traceroute 10.100.0.1

# Capture packets
ip netns exec ns_test tcpdump -i wg0 -n

# Check routing table
ip netns exec ns_test ip route show

# Check ARP table
ip netns exec ns_test ip neigh show
```

### Check Database State

```bash
# List namespaces
psql -U nexusguard -d nexusguard -c "SELECT * FROM namespaces;"

# List namespace users
psql -U nexusguard -d nexusguard -c "SELECT * FROM namespace_users;"

# List VRFs
psql -U nexusguard -d nexusguard -c "SELECT * FROM vrfs;"

# List servers with VRF
psql -U nexusguard -d nexusguard -c "
SELECT s.name, s.interface_name, n.name as namespace, v.name as vrf
FROM wg_servers s
LEFT JOIN namespaces n ON s.namespace_id = n.id
LEFT JOIN vrfs v ON s.vrf_id = v.id;
"
```

## Log Analysis

### Server Logs

```bash
# Check server logs
journalctl -u nexusguard-server -f

# Check for VRF errors
journalctl -u nexusguard-server | grep -i vrf

# Check for namespace errors
journalctl -u nexusguard-server | grep -i namespace
```

### Kernel Logs

```bash
# Check kernel messages
dmesg | grep -i vrf
dmesg | grep -i wireguard
dmesg | grep -i namespace
```

### nftables Logs

```bash
# Enable nftables logging
ip netns exec ns_test nft add rule ip nexusguard forward log prefix "nft: " accept

# Check logs
journalctl -k | grep "nft:"
```

## Performance Issues

### High CPU Usage

```bash
# Check CPU usage per namespace
top -p $(ip netns exec ns_test pgrep -d,)

# Check WireGuard CPU usage
ip netns exec ns_test wg show | grep -A 5 "transfer"
```

### High Memory Usage

```bash
# Check memory per namespace
cat /proc/net/ns/*/mem 2>/dev/null || echo "Not available"

# Check total namespace memory
ps aux | grep "ip netns exec" | awk '{sum+=$6} END {print sum/1024 " MB"}'
```

### Network Latency

```bash
# Measure latency
ip netns exec ns_test ping -c 100 10.100.0.1 | tail -1

# Check interface stats
ip netns exec ns_test ip -s link show wg0
```

## Cleanup Commands

### Remove Namespace

```bash
# Delete namespace (removes everything)
ip netns del ns_test

# Or manually cleanup
ip netns exec ns_test ip link del wg0
ip netns exec ns_test ip link del vrf:main
ip netns exec ns_test ip link del veth0
ip netns del ns_test
```

### Remove VRF

```bash
# Delete VRF
ip netns exec ns_test ip link del vrf:main

# Verify deletion
ip netns exec ns_test ip link show type vrf
```

### Remove WireGuard Interface

```bash
# Delete WireGuard interface
ip netns exec ns_test ip link del wg0

# Verify deletion
ip netns exec ns_test wg show
```

## Getting Help

### Check Documentation

```bash
# Kernel documentation
man ip-netns
man ip-link
man wg

# NexusGuard documentation
ls reference/namespace-vrf/
cat reference/namespace-vrf/architecture.md
```

### Community Resources

- WireGuard mailing list: https://lists.zx2c4.com/pipermail/wireguard/
- Linux kernel mailing list: https://lkml.org/
- NexusGuard GitHub: https://github.com/nexusguard/nexusguard

---

**Last Updated:** 2026-07-10
**Author:** NexusGuard Team