Nexus-Guard-Suite/reference/namespace-vrf/troubleshooting.md

Namespace + VRF Troubleshooting Guide

Common Issues

Issue 1: Namespace Creation Fails

Symptoms:

$ ip netns add ns_test
RTNETLINK answers: Operation not permitted

Causes:

• Not running as root
• Kernel namespace support disabled
• Container limitations

Solutions:

# Solution 1: Run as root
sudo ip netns add ns_test

# Solution 2: Check kernel support
modprobe vrf
modprobe wireguard

# Solution 3: Check container capabilities
# Docker needs --cap-add=NET_ADMIN --cap-add=SYS_ADMIN
docker run --cap-add=NET_ADMIN --cap-add=SYS_ADMIN ...

Issue 2: VRF Creation Fails

Symptoms:

$ ip netns exec ns_test ip link add vrf:main type vrf table 100
RTNETLINK answers: No such file or directory

Causes:

• VRF module not loaded
• Table ID already in use
• Namespace doesn't exist

Solutions:

# Solution 1: Load VRF module
sudo modprobe vrf

# Solution 2: Check existing VRFs
ip netns exec ns_test ip link show type vrf

# Solution 3: Use different table ID
ip netns exec ns_test ip link add vrf:main type vrf table 200

# Solution 4: Verify namespace exists
ip netns list | grep ns_test

Issue 3: WireGuard Interface Won't Bind to VRF

Symptoms:

$ ip netns exec ns_test ip link set wg0 vrf vrf:main
RTNETLINK answers: Invalid argument

Causes:

• VRF not up
• Interface doesn't exist
• VRF name mismatch

Solutions:

# Solution 1: Ensure VRF is up
ip netns exec ns_test ip link set vrf:main up

# Solution 2: Verify interface exists
ip netns exec ns_test ip link show

# Solution 3: Check VRF name
ip netns exec ns_test ip link show type vrf

# Solution 4: Create interface first
ip netns exec ns_test ip link add wg0 type wireguard
ip netns exec ns_test ip link set wg0 vrf vrf:main

Issue 4: No Connectivity in Namespace

Symptoms:

$ ip netns exec ns_test ping 10.0.0.1
connect: Network is unreachable

Causes:

• No default route
• Interface not configured
• Routing table missing

Solutions:

# Solution 1: Add default route
ip netns exec ns_test ip route add default via 10.0.0.1

# Solution 2: Configure interface
ip netns exec ns_test ip addr add 10.0.0.2/24 dev eth0
ip netns exec ns_test ip link set eth0 up

# Solution 3: Check routing table
ip netns exec ns_test ip route show

# Solution 4: Check VRF routes
ip netns exec ns_test ip route show table 100

Issue 5: nftables Rules Not Working

Symptoms:

$ ip netns exec ns_test nft list table ip nexusguard
table ip nexusguard {
    chain forward {
        type filter hook forward priority filter; policy accept;
    }
}
# But traffic not being filtered

Causes:

• Rules in wrong table
• Chain not hooked
• Interface mismatch

Solutions:

# Solution 1: Verify chain hook
ip netns exec ns_test nft list chain ip nexusguard forward

# Solution 2: Add rules correctly
ip netns exec ns_test nft add rule ip nexusguard forward iifname wg0 accept

# Solution 3: Check interface names
ip netns exec ns_test ip link show

# Solution 4: Flush and re-add rules
ip netns exec ns_test nft flush chain ip nexusguard forward

Issue 6: Cross-Namespace Communication Fails

Symptoms:

# From ns_alice
$ ip netns exec ns_alice ping 10.100.0.1
connect: Network is unreachable

# But 10.100.0.1 is in ns_zerocode

Causes:

• No routing between namespaces
• VRF isolation working correctly
• Need veth pair for cross-namespace

Solutions:

# Solution 1: Create veth pair (if needed)
ip link add veth-alice type veth peer name veth-zerocode
ip link set veth-zerocode netns ns_zerocode
ip link set veth-alice netns ns_alice

# Solution 2: Configure veth
ip netns exec ns_alice ip addr add 10.0.0.1/24 dev veth-alice
ip netns exec ns_alice ip link set veth-alice up
ip netns exec ns_alice ip route add default via 10.0.0.1

ip netns exec ns_zerocode ip addr add 10.0.0.2/24 dev veth-zerocode
ip netns exec ns_zerocode ip link set veth-zerocode up
ip netns exec ns_zerocode ip route add default via 10.0.0.2

# Solution 3: Add static routes
ip netns exec ns_alice ip route add 10.100.0.0/24 via 10.0.0.2
ip netns exec ns_zerocode ip route add 10.100.0.0/24 via 10.0.0.1

Issue 7: WireGuard Handshake Fails

Symptoms:

$ ip netns exec ns_test wg show
interface: wg0
  peer: <public_key>
    endpoint: 1.2.3.4:51820
    allowed ips: 10.100.0.2/32
    latest handshake: (none)
    transfer: 0 B received, 0 B sent

Causes:

• Firewall blocking UDP
• NAT not configured
• Endpoint unreachable

Solutions:

# Solution 1: Check firewall
ip netns exec ns_test nft list ruleset

# Solution 2: Allow UDP in firewall
ip netns exec ns_test nft add rule ip nexusguard input udp dport 51820 accept

# Solution 3: Configure NAT (if needed)
ip netns exec ns_test nft add table ip nat
ip netns exec ns_test nft add chain ip nat postrouting '{type nat hook postrouting priority 100;}'
ip netns exec ns_test nft add rule ip nat postrouting oifname eth0 masquerade

# Solution 4: Check connectivity
ip netns exec ns_test ping 1.2.3.4

Issue 8: Permission Denied for API Calls

Symptoms:

{
  "error": "permission denied"
}

Causes:

• User not in namespace
• Wrong role
• Token expired

Solutions:

# Solution 1: Check user in namespace
psql -U nexusguard -d nexusguard -c "
SELECT * FROM namespace_users 
WHERE namespace_id = (SELECT id FROM namespaces WHERE name = 'testns');
"

# Solution 2: Add user to namespace
curl -X POST http://localhost:8080/api/v1/ns/testns/users \
  -H "Authorization: Bearer <admin_token>" \
  -H "Content-Type: application/json" \
  -d '{"user_id": "<user_id>", "role": "admin"}'

# Solution 3: Check role
psql -U nexusguard -d nexusguard -c "
SELECT nu.role FROM namespace_users nu
JOIN namespaces n ON nu.namespace_id = n.id
WHERE n.name = 'testns' AND nu.user_id = '<user_id>';
"

Debugging Commands

Check Namespace Status

# List all namespaces
ip netns list

# Check namespace exists
ip netns list | grep ns_test

# Enter namespace shell
ip netns exec ns_test bash

# Check namespace processes
ip netns exec ns_test ps aux

Check VRF Status

# List VRFs in namespace
ip netns exec ns_test ip link show type vrf

# Check VRF details
ip -d link show type vrf

# Check VRF routes
ip netns exec ns_test ip route show table 100

Check WireGuard Status

# Show WireGuard interfaces
ip netns exec ns_test wg show

# Show WireGuard details
ip netns exec ns_test wg show wg0

# Check WireGuard config
ip netns exec ns_test wg showconf wg0

Check nftables Status

# List all tables
ip netns exec ns_test nft list tables

# List specific table
ip netns exec ns_test nft list table ip nexusguard

# List rules with handles
ip netns exec ns_test nft -a list table ip nexusguard

# Check rule counters
ip netns exec ns_test nft list chain ip nexusguard forward

Check Connectivity

# Ping from namespace
ip netns exec ns_test ping 10.100.0.1

# Traceroute from namespace
ip netns exec ns_test traceroute 10.100.0.1

# Capture packets
ip netns exec ns_test tcpdump -i wg0 -n

# Check routing table
ip netns exec ns_test ip route show

# Check ARP table
ip netns exec ns_test ip neigh show

Check Database State

# List namespaces
psql -U nexusguard -d nexusguard -c "SELECT * FROM namespaces;"

# List namespace users
psql -U nexusguard -d nexusguard -c "SELECT * FROM namespace_users;"

# List VRFs
psql -U nexusguard -d nexusguard -c "SELECT * FROM vrfs;"

# List servers with VRF
psql -U nexusguard -d nexusguard -c "
SELECT s.name, s.interface_name, n.name as namespace, v.name as vrf
FROM wg_servers s
LEFT JOIN namespaces n ON s.namespace_id = n.id
LEFT JOIN vrfs v ON s.vrf_id = v.id;
"

Log Analysis

Server Logs

# Check server logs
journalctl -u nexusguard-server -f

# Check for VRF errors
journalctl -u nexusguard-server | grep -i vrf

# Check for namespace errors
journalctl -u nexusguard-server | grep -i namespace

Kernel Logs

# Check kernel messages
dmesg | grep -i vrf
dmesg | grep -i wireguard
dmesg | grep -i namespace

nftables Logs

# Enable nftables logging
ip netns exec ns_test nft add rule ip nexusguard forward log prefix "nft: " accept

# Check logs
journalctl -k | grep "nft:"

Performance Issues

High CPU Usage

# Check CPU usage per namespace
top -p $(ip netns exec ns_test pgrep -d,)

# Check WireGuard CPU usage
ip netns exec ns_test wg show | grep -A 5 "transfer"

High Memory Usage

# Check memory per namespace
cat /proc/net/ns/*/mem 2>/dev/null || echo "Not available"

# Check total namespace memory
ps aux | grep "ip netns exec" | awk '{sum+=$6} END {print sum/1024 " MB"}'

Network Latency

# Measure latency
ip netns exec ns_test ping -c 100 10.100.0.1 | tail -1

# Check interface stats
ip netns exec ns_test ip -s link show wg0

Cleanup Commands

Remove Namespace

# Delete namespace (removes everything)
ip netns del ns_test

# Or manually cleanup
ip netns exec ns_test ip link del wg0
ip netns exec ns_test ip link del vrf:main
ip netns exec ns_test ip link del veth0
ip netns del ns_test

Remove VRF

# Delete VRF
ip netns exec ns_test ip link del vrf:main

# Verify deletion
ip netns exec ns_test ip link show type vrf

Remove WireGuard Interface

# Delete WireGuard interface
ip netns exec ns_test ip link del wg0

# Verify deletion
ip netns exec ns_test wg show

Getting Help

Check Documentation

# Kernel documentation
man ip-netns
man ip-link
man wg

# NexusGuard documentation
ls reference/namespace-vrf/
cat reference/namespace-vrf/architecture.md

Community Resources

• WireGuard mailing list: https://lists.zx2c4.com/pipermail/wireguard/
• Linux kernel mailing list: https://lkml.org/
• NexusGuard GitHub: https://github.com/nexusguard/nexusguard

---

Last Updated: 2026-07-10 Author: NexusGuard Team