Real-world infrastructure failures. Follow the evidence, find the root cause.
FUCKUP #009: The DHCP Scope Was Full of Leases the Console Could Not See
The symptom After a power outage, PoE-powered network clocks restarted and flooded the network with DHCP requests. The Windows DHCP server stopped returning normal offers, and its logs filled with repeated NACK entries. From memory, roughly 100 clocks were involved. They were showing the wrong time, and no other affected devices were noticed. What made it weird The DHCP console told two different stories. At the time, the affected scope was described as nearly full, but the ordinary lease list showed far fewer leases. ...
FUCKUP #008: The CI Task That Had a Proxy Everywhere Except Where It Needed One
The symptom A task on a CI/automation platform needed to access a public container registry at quay.io. Instead, the task log reported connection refused while it was attempting to reach the registry endpoint at https://quay.io/v2/. The image was public, so registry credentials were not the obvious boundary. The error looked like a remote connectivity failure. What made it weird The surrounding environment already had a working HTTPS proxy configuration. That made the refusal easy to read as evidence against the public endpoint or the network beyond the platform. But the process making the request was not the host or another service with the proxy configured. It was the task itself, running in its own execution context. ...
FUCKUP #007: The VM That Was Almost Out of Memory—According to Proxmox
The symptom The Proxmox GUI showed that a Linux VM was using almost all of its assigned memory: 7.71 / 8 GiB used 96.36% That looked serious enough to trigger a support ticket. The VM itself remained completely healthy. What made it weird Inside the guest, free -h reported approximately: 1 GiB used 4.7 GiB cache 6.4 GiB available Proxmox appeared to show a VM approaching memory exhaustion. The guest showed substantial memory available, and the workload displayed no operational symptoms of memory pressure. ...
FUCKUP #006: The App That Installed from the App Store—but Not Through Company Portal
The symptom One BYOD iPhone could not obtain an application through Microsoft Company Portal. Intune showed the following progression: User requested application VPP App licensing in progress App installation failed 0x87D13B91 Support could not make Company Portal complete the VPP licensing path for that phone. What made it weird The failure was not universal. Another iPhone installed the same application successfully. Intune also had a valid Apple VPP token alongside an expired token, so the visible expired token was not enough to explain the failure. ...
FUCKUP #005: The Expired Certificate Was Only the First Failure
A Nextcloud-to-Paperless upload stopped working in a home test environment. The first error was clear: cURL error 60: SSL certificate problem: certificate has expired The Paperless certificate had indeed expired. The first fix The Nextcloud Paperless integration sends uploads from: custom_apps/integration_paperless/lib/Service/ApiService.php or, depending on the installation: apps/integration_paperless/lib/Service/ApiService.php In the POST request to: /api/documents/post_document/ this option was added: 'verify' => false, Example: $this->client->post( $this->config->url . '/api/documents/post_document/', [ 'verify' => false, 'headers' => $this->getAuthorizationHeaders(), 'multipart' => [ After restarting the Nextcloud/PHP runtime, uploads worked again. ...
FUCKUP #004: The Filesystem Resize That Failed — But Was Already Done
The symptom The incident did not start with somebody watching the VM boot. It started with monitoring: Checkmk reported a problem on the VM, which triggered the investigation. The exact Checkmk service and alert message were not recorded. During that investigation, a recent storage change and an alarming boot-time error became relevant. The Linux VM in Proxmox had its virtual disk expanded from 20 GB to 25 GB. During the next boot, cloud-init attempted to grow the root filesystem. The relevant operation was effectively: ...
FUCKUP #003: The VMs That Broke After Migration — But the Problem Was the Switch
The symptom Two Windows Server 2019 virtual machines were no longer properly reachable after they were moved to newer Proxmox hosts. The timing made the migration itself look suspicious. The VMs had moved, and their network connectivity had failed. What made it misleading Several plausible causes fit the initial symptom: differences between the old and new Proxmox hosts virtual NIC configuration bridge configuration Windows networking migration behavior the Ceph and storage context The incident happened on a weekend. The immediate operational priority was fast service recovery, so restoring service took priority over isolating each variable in the failed state. ...
FUCKUP #002: The Service That Worked Without Its Expected Private-Key ACL Entry
The symptom On a Windows Server 2019 host, a service failed after a second cold restore. The service used a manually provisioned certificate with a private key in Local Computer → Personal. The certificate and associated private key had been present for more than six months and had not been intentionally replaced or reprovisioned. The service had been working normally. Investigation of the failed state showed no explicit Read ACL entry for the service account on the private key. Adding that entry made the service work immediately again. ...
FUCKUP #001: The VM That Could Reach Everything Except Its Gateway
The symptom A freshly installed Linux VM had a seemingly simple network problem. The VM was configured as: IP: 172.21.0.7/24 Gateway: 172.21.0.1 Another host on the same subnet was reachable: 172.21.0.7 -> 172.21.0.6 OK But the default gateway was not: 172.21.0.7 -> 172.21.0.1 FAIL So Layer 2 connectivity clearly existed — at least to other systems on the same network. What made it weird Packet captures made the situation even stranger. ...