The symptom
After a power outage, PoE-powered network clocks restarted and flooded the network with DHCP requests.
The Windows DHCP server stopped returning normal offers, and its logs filled with repeated NACK entries.
From memory, roughly 100 clocks were involved. They were showing the wrong time, and no other affected devices were noticed.
What made it weird
The DHCP console told two different stories. At the time, the affected scope was described as nearly full, but the ordinary lease list showed far fewer leases.
One surviving screenshot complicates the picture: it shows 290 of 1,520 addresses in use, only 19%. It may show combined usage across all pools rather than the affected pool, but too much time has passed to confirm that.
What fixed it
Reconciling the affected scope made previously hidden leases appear in the lease list. After those phantom leases were deleted, the scope had free addresses again.
The PoE switch ports were then cycled to restart the clocks. They came back showing the correct time.
The server-side recovery is documented. The final PoE restart and correct clock display are recalled from memory.
A useful diagnostic path
When DHCP logs, statistics, and the lease list disagree:
- Check the DHCP logs for repeated NACKs or other unusual activity.
- Compare the lease list with the affected scope’s statistics.
- Reconcile the scope and inspect any leases that appear afterward.
- Verify both sides of the recovery: free addresses on the server and working service on the clients.
Deleting leases should follow inspection, not replace it. In this incident, reconciliation exposed previously hidden leases, and removing them restored capacity.
What remains unknown
The exact mechanism was never established. There is no surviving packet capture, and the unusual client identifiers seen at the time could not be classified later.
The evidence does not prove a clock firmware defect, malformed DHCP traffic, or a Windows DHCP bug. The power event and simultaneous restart likely triggered the incident, but what created the hidden lease state remains unknown.
The lesson
When DHCP logs, scope statistics, and the lease list disagree, the lease list is not the whole state.
Reconcile the scope, inspect what becomes visible, and verify the server and clients separately. Restoring capacity proves that the recovery worked; it does not explain how the hidden state was created.