The symptom
A freshly installed Linux VM had a seemingly simple network problem.
The VM was configured as:
IP: 172.21.0.7/24
Gateway: 172.21.0.1
Another host on the same subnet was reachable:
172.21.0.7 -> 172.21.0.6 OK
But the default gateway was not:
172.21.0.7 -> 172.21.0.1 FAIL
So Layer 2 connectivity clearly existed — at least to other systems on the same network.
What made it weird
Packet captures made the situation even stranger.
Traffic sent toward the gateway was visible, and replies appeared in tcpdump.
That meant the obvious explanations no longer fit neatly:
- the virtual NIC was working
- the Proxmox bridge was carrying traffic
- the VM was attached to the correct subnet
- the gateway was alive
- packets were apparently coming back
Yet from the VM’s point of view, communication with the gateway still failed.
Useful commands
Start by confirming the local configuration:
ip addr
ip route
Check whether the gateway has a neighbor entry:
ip neigh show 172.21.0.1
Test another host on the same subnet and then the gateway:
ping 172.21.0.6
ping 172.21.0.1
Watch ARP and ICMP traffic while reproducing the problem:
tcpdump -ni any 'arp or icmp'
On the gateway, run a capture on the affected interface as well. The important questions are not just whether packets exist, but:
- does the request reach the gateway?
- does the gateway reply?
- which source and destination IP addresses are used?
- which source and destination MAC addresses are used?
- does the VM actually receive the reply?
Diagnostic path
START
|
v
Can the VM reach another host in the same subnet?
|
+-- NO --> Check VM NIC, bridge, VLAN, subnet mask
| and whether Layer 2 works at all
|
`-- YES
|
v
Does the VM learn a MAC address for the gateway?
|
+-- NO --> Investigate ARP / neighbor discovery
| and Layer 2 toward the gateway
|
`-- YES
|
v
Do packets leave the VM toward the gateway?
|
+-- NO --> Check local routing, policy routing
| and local firewall rules
|
`-- YES
|
v
Does the gateway see the request?
|
+-- NO --> Investigate bridge, VLAN, switching
| and the path between VM and gateway
|
`-- YES
|
v
Does the gateway send a reply?
|
+-- NO --> Check gateway firewall, policy,
| state tracking and interface config
|
`-- YES
|
v
Can the VM see the reply in tcpdump?
|
+-- NO --> Check return path,
| filtering and VLAN state
|
`-- YES
|
v
Is the reply addressed to the expected
IP and MAC identity?
|
+-- NO --> Check:
| - duplicate IPs
| - stale ARP/neighbor state
| - DHCP/static mappings
| - reused MAC addresses
| - hostname changes
|
`-- YES
|
v
Does the kernel accept the packet?
|
+-- NO --> Check:
| - nftables/iptables
| - rp_filter
| - policy routing
| - local state
|
`-- YES
|
v
Was this IP or DHCP reservation previously
assigned to another machine?
|
+-- YES --> Refresh:
| - static mapping
| - DHCP lease
| - ARP/neighbor state
| - MAC association
|
`-- NO --> Continue higher
in the stack
The root cause
The issue turned out to be related to an existing static DHCP mapping on OPNsense that had been reused for the new VM.
The mapping previously belonged to another system. Its MAC address and hostname had been changed for the replacement VM, while existing lease and mapping state still reflected the previous assignment.
After clearing the stale DHCP lease state and refreshing the mapping, connectivity to the default gateway started working normally.
The important point is not that every similar symptom is caused by DHCP. It is that infrastructure may retain identity and state outside the system you are currently debugging.
The lesson
When a replacement system inherits an existing DHCP reservation, changing the MAC address in the static mapping may not be the whole story.
If the symptoms contradict what packet captures appear to show, check the state held by the network infrastructure itself:
- static DHCP mappings
- active and stale leases
- ARP/neighbor state
- MAC address changes
- cached associations between the old and new system
The fault may be outside the machine you are debugging.
FUCKUP.fail
When infrastructure makes no sense, follow the evidence.