[{"content":"The symptom The incident did not start with somebody watching the VM boot. It started with monitoring: Checkmk reported a problem on the VM, which triggered the investigation. The exact Checkmk service and alert message were not recorded.\nDuring that investigation, a recent storage change and an alarming boot-time error became relevant. The Linux VM in Proxmox had its virtual disk expanded from 20 GB to 25 GB. During the next boot, cloud-init attempted to grow the root filesystem. The relevant operation was effectively:\nresize2fs /dev/sda1 It failed with messages that included:\nCouldn\u0026#39;t find valid filesystem superblock short read Those errors suggested that the ext4 filesystem on /dev/sda1 had not been resized. The VM itself still completed its boot.\nThe context The environment had recently been upgraded from Proxmox 8 to Proxmox 9. The VM used cloud-init with the NoCloud datasource, and its config drive was presented through a virtual CD-ROM device.\nThose details define the environment, but they do not establish the cause. The available evidence does not connect the Proxmox upgrade, the NoCloud datasource, or the config-drive device to the transient resize failure.\nWhat made it confusing A virtual disk expansion crosses several separate layers. Success at one layer does not prove success at the next:\nLayer Question to answer Hypervisor virtual disk Was the configured disk expanded from 20 GB to 25 GB? Guest block device Does Linux see the larger device? Partition Does /dev/sda1 occupy the newly available space? Filesystem Does ext4 expose the expanded capacity? The boot-time message came from the filesystem-resize step. Taken alone, it looked like the final layer had failed. It did not describe the final state of all four layers.\nThe evidence After the VM booted, the storage state was checked again:\nThe virtual disk was 25 GB. /dev/sda1 had grown to approximately the full available size. The ext4 filesystem showed the expanded capacity. Running resize2fs manually produced no repair action and effectively reported Nothing to do! The observed final state contradicted the alarming boot-time error. By the time the system was inspected, the partition and filesystem were already at the expected size.\nThe actual outcome No manual partition repair was required. No manual filesystem resize was required, and no filesystem recovery was performed.\nAfter another reboot, the VM came up normally with a consistent disk, partition, and filesystem state. That reboot confirmed the clean state on the next boot; the evidence does not show that the reboot performed the resize.\nWhat probably happened The most defensible explanation is a transient timing or ordering problem during boot. Device discovery, partition-table updates, and the filesystem-resize attempt may not have been observed in the expected order when cloud-init ran.\nThat is a hypothesis, not a captured root cause. The available evidence does not establish whether the message came from cloud-init behavior, device discovery timing, partition update timing, or another boot-ordering effect. It only shows that the resize appears to have completed despite the reported error.\nDiagnostic path When an automated filesystem resize reports failure, verify each storage layer before attempting repair:\nCheck the hypervisor disk size. Confirm that the VM configuration shows the requested capacity. Check the guest block device. Verify that Linux sees the larger virtual disk. If it does not, stop at the device-discovery layer rather than changing partitions or filesystems. Check the partition size. Confirm whether the root partition occupies the newly available space. A larger block device does not mean its partition has grown. Check the filesystem size. Confirm whether ext4 exposes the expanded capacity. A larger partition does not prove the filesystem has been resized. Compare the state with the cloud-init error. Review the failed operation and its timing, but treat it as evidence about that attempt rather than proof of the current state. Retry resize2fs only if the filesystem is still smaller than its partition. If the filesystem already fills the partition and the tool reports that there is nothing to do, do not escalate to partition repair or filesystem recovery without additional evidence. Use a controlled reboot as verification when appropriate. A clean subsequent boot can confirm that the final state persists, but it does not prove which earlier step completed the resize. The root cause The precise transient failure mechanism was not captured. What is established is narrower: cloud-init reported that the ext4 resize failed, while the final disk, partition, and filesystem state became correct without manual repair or resizing.\nThe Proxmox 8-to-9 upgrade remains context, not a proven cause.\nThe lesson Do not confuse a failed automation step with a failed final state.\nCheck the virtual disk, guest block device, partition, and filesystem in order. An automation error tells you that an operation reported failure at a particular moment. It does not, by itself, prove that the intended final state was never reached.\nAggressive repair based only on the earlier error can turn a transient issue into a real one. Verify the current state first, and intervene only at the layer that is still wrong.\nSometimes, a reboot really does do wonders.\n","permalink":"https://fuckup.fail/cases/004-proxmox-cloud-init-upgrade-disk/","summary":"\u003ch2 id=\"the-symptom\"\u003eThe symptom\u003c/h2\u003e\n\u003cp\u003eThe incident did not start with somebody watching the VM boot. It started with monitoring: Checkmk reported a problem on the VM, which triggered the investigation. The exact Checkmk service and alert message were not recorded.\u003c/p\u003e\n\u003cp\u003eDuring that investigation, a recent storage change and an alarming boot-time error became relevant. The Linux VM in Proxmox had its virtual disk expanded from 20 GB to 25 GB. During the next boot, cloud-init attempted to grow the root filesystem. The relevant operation was effectively:\u003c/p\u003e","title":"FUCKUP #004: The Filesystem Resize That Failed — But Was Already Done"},{"content":"A Nextcloud-to-Paperless upload stopped working in a home test environment.\nThe first error was clear:\ncURL error 60: SSL certificate problem: certificate has expired The Paperless certificate had indeed expired.\nThe first fix The Nextcloud Paperless integration sends uploads from:\ncustom_apps/integration_paperless/lib/Service/ApiService.php or, depending on the installation:\napps/integration_paperless/lib/Service/ApiService.php In the POST request to:\n/api/documents/post_document/ this option was added:\n\u0026#39;verify\u0026#39; =\u0026gt; false, Example:\n$this-\u0026gt;client-\u0026gt;post( $this-\u0026gt;config-\u0026gt;url . \u0026#39;/api/documents/post_document/\u0026#39;, [ \u0026#39;verify\u0026#39; =\u0026gt; false, \u0026#39;headers\u0026#39; =\u0026gt; $this-\u0026gt;getAuthorizationHeaders(), \u0026#39;multipart\u0026#39; =\u0026gt; [ After restarting the Nextcloud/PHP runtime, uploads worked again.\nThis was a deliberate test-environment workaround, not a proper TLS fix. verify =\u0026gt; false disables certificate verification for that request and should not be used as a normal production solution.\nThis failure was at a different certificate layer from a Windows service failure where private-key access mattered to recovery. Here, the client rejected the remote server certificate before the upload request could proceed.\nThen the next failure appeared An ODT file now reached Paperless successfully, but document conversion failed with:\n503 Service Unavailable from:\nhttp://gotenberg:3000/forms/libreoffice/convert The Gotenberg log showed:\nLibreOffice listener socket not available: context deadline exceeded with a request duration of almost exactly:\n10.003s LibreOffice itself was present:\n/usr/bin/libreoffice LibreOffice 7.0.4.2 So the problem was not a missing LibreOffice installation. Gotenberg simply did not get its LibreOffice listener ready within the available ten-second window.\nThe next step was therefore to increase the startup timeout, for example:\ncommand: - \u0026#34;gotenberg\u0026#34; - \u0026#34;--uno-listener-start-timeout=60s\u0026#34; - \u0026#34;--api-timeout=120s\u0026#34; and recreate the container:\ndocker compose up -d --force-recreate gotenberg The surviving notes do not record whether that final change was verified successfully.\nThe lesson A different error can be progress.\nThe first failure stopped the request at TLS validation.\nAfter that was bypassed, the document progressed through Nextcloud and Paperless until it hit a completely different failure in Gotenberg.\nSo do not ask only:\nDid the original error disappear?\nAsk:\nHow far did the request get this time?\nThat is often the more useful answer.\n","permalink":"https://fuckup.fail/cases/005-nextcloud-paperless-certificate-timeout/","summary":"\u003cp\u003eA Nextcloud-to-Paperless upload stopped working in a home test environment.\u003c/p\u003e\n\u003cp\u003eThe first error was clear:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ecURL error 60: SSL certificate problem: certificate has expired\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe Paperless certificate had indeed expired.\u003c/p\u003e\n\u003ch2 id=\"the-first-fix\"\u003eThe first fix\u003c/h2\u003e\n\u003cp\u003eThe Nextcloud Paperless integration sends uploads from:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ecustom_apps/integration_paperless/lib/Service/ApiService.php\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eor, depending on the installation:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eapps/integration_paperless/lib/Service/ApiService.php\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIn the POST request to:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e/api/documents/post_document/\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003ethis option was added:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-php\" data-lang=\"php\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#e6db74\"\u003e\u0026#39;verify\u0026#39;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eExample:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-php\" data-lang=\"php\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$this\u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan style=\"color:#a6e22e\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan style=\"color:#a6e22e\"\u003epost\u003c/span\u003e(\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    $this\u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan style=\"color:#a6e22e\"\u003econfig\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan style=\"color:#a6e22e\"\u003eurl\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e \u003cspan style=\"color:#e6db74\"\u003e\u0026#39;/api/documents/post_document/\u0026#39;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    [\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#e6db74\"\u003e\u0026#39;verify\u0026#39;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#e6db74\"\u003e\u0026#39;headers\u0026#39;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u0026gt;\u003c/span\u003e $this\u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan style=\"color:#a6e22e\"\u003egetAuthorizationHeaders\u003c/span\u003e(),\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#e6db74\"\u003e\u0026#39;multipart\u0026#39;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u0026gt;\u003c/span\u003e [\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAfter restarting the Nextcloud/PHP runtime, uploads worked again.\u003c/p\u003e","title":"FUCKUP #005: The Expired Certificate Was Only the First Failure"},{"content":"The symptom A freshly installed Linux VM had a seemingly simple network problem.\nThe VM was configured as:\nIP: 172.21.0.7/24 Gateway: 172.21.0.1 Another host on the same subnet was reachable:\n172.21.0.7 -\u0026gt; 172.21.0.6 OK But the default gateway was not:\n172.21.0.7 -\u0026gt; 172.21.0.1 FAIL So Layer 2 connectivity clearly existed — at least to other systems on the same network.\nWhat made it weird Packet captures made the situation even stranger.\nTraffic sent toward the gateway was visible, and replies appeared in tcpdump.\nThat meant the obvious explanations no longer fit neatly:\nthe virtual NIC was working the Proxmox bridge was carrying traffic the VM was attached to the correct subnet the gateway was alive packets were apparently coming back Yet from the VM\u0026rsquo;s point of view, communication with the gateway still failed.\nUseful commands Start by confirming the local configuration:\nip addr ip route Check whether the gateway has a neighbor entry:\nip neigh show 172.21.0.1 Test another host on the same subnet and then the gateway:\nping 172.21.0.6 ping 172.21.0.1 Watch ARP and ICMP traffic while reproducing the problem:\ntcpdump -ni any \u0026#39;arp or icmp\u0026#39; On the gateway, run a capture on the affected interface as well. The important questions are not just whether packets exist, but:\ndoes the request reach the gateway? does the gateway reply? which source and destination IP addresses are used? which source and destination MAC addresses are used? does the VM actually receive the reply? Diagnostic path START | v Can the VM reach another host in the same subnet? | +-- NO --\u0026gt; Check VM NIC, bridge, VLAN, subnet mask | and whether Layer 2 works at all | `-- YES | v Does the VM learn a MAC address for the gateway? | +-- NO --\u0026gt; Investigate ARP / neighbor discovery | and Layer 2 toward the gateway | `-- YES | v Do packets leave the VM toward the gateway? | +-- NO --\u0026gt; Check local routing, policy routing | and local firewall rules | `-- YES | v Does the gateway see the request? | +-- NO --\u0026gt; Investigate bridge, VLAN, switching | and the path between VM and gateway | `-- YES | v Does the gateway send a reply? | +-- NO --\u0026gt; Check gateway firewall, policy, | state tracking and interface config | `-- YES | v Can the VM see the reply in tcpdump? | +-- NO --\u0026gt; Check return path, | filtering and VLAN state | `-- YES | v Is the reply addressed to the expected IP and MAC identity? | +-- NO --\u0026gt; Check: | - duplicate IPs | - stale ARP/neighbor state | - DHCP/static mappings | - reused MAC addresses | - hostname changes | `-- YES | v Does the kernel accept the packet? | +-- NO --\u0026gt; Check: | - nftables/iptables | - rp_filter | - policy routing | - local state | `-- YES | v Was this IP or DHCP reservation previously assigned to another machine? | +-- YES --\u0026gt; Refresh: | - static mapping | - DHCP lease | - ARP/neighbor state | - MAC association | `-- NO --\u0026gt; Continue higher in the stack The root cause The issue turned out to be related to an existing static DHCP mapping on OPNsense that had been reused for the new VM.\nThe mapping previously belonged to another system. Its MAC address and hostname had been changed for the replacement VM, while existing lease and mapping state still reflected the previous assignment.\nAfter clearing the stale DHCP lease state and refreshing the mapping, connectivity to the default gateway started working normally.\nThe important point is not that every similar symptom is caused by DHCP. It is that infrastructure may retain identity and state outside the system you are currently debugging.\nThe lesson When a replacement system inherits an existing DHCP reservation, changing the MAC address in the static mapping may not be the whole story.\nIf the symptoms contradict what packet captures appear to show, check the state held by the network infrastructure itself:\nstatic DHCP mappings active and stale leases ARP/neighbor state MAC address changes cached associations between the old and new system The fault may be outside the machine you are debugging.\nThat same boundary appears in a VM networking failure caused by a missing VLAN on the physical switch path: the guest showed the symptom, but the decisive state lived elsewhere in the infrastructure.\nFUCKUP.fail\nWhen infrastructure makes no sense, follow the evidence.\n","permalink":"https://fuckup.fail/cases/001-vm-can-reach-subnet-but-not-gateway/","summary":"\u003ch2 id=\"the-symptom\"\u003eThe symptom\u003c/h2\u003e\n\u003cp\u003eA freshly installed Linux VM had a seemingly simple network problem.\u003c/p\u003e\n\u003cp\u003eThe VM was configured as:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eIP:       172.21.0.7/24\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eGateway:  172.21.0.1\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAnother host on the same subnet was reachable:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e172.21.0.7 -\u0026gt; 172.21.0.6   OK\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eBut the default gateway was not:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e172.21.0.7 -\u0026gt; 172.21.0.1   FAIL\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSo Layer 2 connectivity clearly existed — at least to other systems on the same network.\u003c/p\u003e\n\u003ch2 id=\"what-made-it-weird\"\u003eWhat made it weird\u003c/h2\u003e\n\u003cp\u003ePacket captures made the situation even stranger.\u003c/p\u003e","title":"FUCKUP #001: The VM That Could Reach Everything Except Its Gateway"},{"content":"The symptom On a Windows Server 2019 host, a service failed after a second cold restore. The service used a manually provisioned certificate with a private key in Local Computer → Personal. The certificate and associated private key had been present for more than six months and had not been intentionally replaced or reprovisioned. The service had been working normally.\nInvestigation of the failed state showed no explicit Read ACL entry for the service account on the private key. Adding that entry made the service work immediately again.\nWhat made it weird A first incident led to a cold restore, after which the service worked again. A later second cold restore was followed by the failure described here.\nLater clones of two backups from known-good periods showed the expected explicit Read ACL entry was absent. One backup came from before the first incident. The other came from after the first cold restore, when the service was working again. This visible ACL state had therefore coexisted with a working service for a significant period.\nAdding the explicit Read ACL entry fixed the failure, but the same visible ACL state without that entry had previously coexisted with a working service. The remediation explains the recovery, not the complete historical sequence.\nEvidence Observed state Service behavior Private-key ACL observation Later clone of a known-good backup from before the first incident Working at the time of the backup Expected explicit Read entry for the service account absent Later clone of a known-good backup from after the first cold restore Working at the time of the backup Expected explicit Read entry still absent After the second cold restore Failed Expected explicit Read entry absent After granting Read to the service account Worked immediately Explicit Read entry present These are observations from different points in time. They do not reveal which identity or process actually opened the key during the earlier working periods.\nDiagnostic path For a service that appears to depend on a Windows certificate, work from the current failure outward:\nIdentify the failing operation and its runtime identity. Check the service\u0026rsquo;s configured logon account and, where evidence is available, the identity and logon context of the process that actually requests the key. A configured service account alone does not establish that every key operation runs under that identity. Find the certificate in the expected store. On this host it was in Certificates (Local Computer) → Personal. Confirm that the service is selecting the intended certificate. Presence in the store establishes only that the certificate is there; it does not establish that the service can use its private key. Microsoft documents the distinction between machine and user stores. Check the private-key association. Verify that the certificate reports an associated private key. If it does not, investigate the certificate/key association before changing permissions. If it does, continue: association alone does not demonstrate that the service\u0026rsquo;s runtime identity can open the key. Inspect access to the correct key. Use Manage Private Keys for the certificate, or inspect the underlying key object\u0026rsquo;s ACL where appropriate. Check effective access, including group permissions and denies, rather than treating the absence of one explicit account entry as a complete access test. Windows access checks can include group ACEs. Keep the key provider in view. CAPI and CNG can use different key storage mechanisms, so identify the provider before mapping a certificate to an underlying key file. This is diagnostic context, not an established explanation for this incident. Microsoft describes CNG key storage. Test the smallest relevant permission change. If the evidence points to key access, grant only the required Read access to the identity that needs it, then retest the failing service operation. Record both the permission change and the observed result. A successful retest establishes a present-day dependency; it does not, by itself, reconstruct how the service worked in the past. If the certificate, key association, and effective access all check out, continue investigating the actual failing operation instead of assuming an ACL fault.\nWhat we can prove The certificate and associated private key had been present for more than six months and had not been intentionally replaced or reprovisioned. The service worked normally before the failures. The expected explicit Read entry was absent in later clones of two backups from known-good periods, and it was absent after the second cold restore when the service failed. Granting Read to the service account was followed immediately by recovery. Private-key access mattered to the second failure and its remediation. The missing explicit entry alone cannot account for every observed service state, because the service had also worked without that entry visible in the earlier backups. What we cannot prove We cannot reconstruct the exact runtime state during the earlier working periods. The later backup clones show the visible ACL, but not the runtime access path. No additional backups, snapshots, traces, or sufficiently detailed historical logs are available to determine which identity, process, or key access path allowed the service to use the private key then.\nThe evidence does not establish who or what changed the ACL, or even that an ACL change caused the transition from working to failed. It also does not establish a CAPI or CNG mechanism as the historical cause. Those are questions, not findings.\nThe remediation An explicit Read ACL entry was added for the service account on the private key. The service worked immediately afterward. This was an effective fix for the failure after the second cold restore, without resolving why the earlier known-good states worked with that same entry absent from the visible ACL.\nThe lesson Treat certificate presence, private-key association, and private-key usability under the service\u0026rsquo;s actual runtime identity as separate checks. Compare the failed state with known-good evidence before naming a root cause.\nWhen a permission change restores service, report what the test proves: access mattered for the observed recovery. If the same visible ACL state also existed during known-good operation, preserve that contradiction. A successful remediation is not a complete historical explanation.\nThis layer is distinct from a client rejecting an expired TLS server certificate before the request could proceed. That failure happened during remote certificate validation; this incident concerned a local service\u0026rsquo;s ability to use an associated private key.\n","permalink":"https://fuckup.fail/cases/002-service-private-key-acl/","summary":"\u003ch2 id=\"the-symptom\"\u003eThe symptom\u003c/h2\u003e\n\u003cp\u003eOn a Windows Server 2019 host, a service failed after a second cold restore. The service used a manually provisioned certificate with a private key in \u003cstrong\u003eLocal Computer → Personal\u003c/strong\u003e. The certificate and associated private key had been present for more than six months and had not been intentionally replaced or reprovisioned. The service had been working normally.\u003c/p\u003e\n\u003cp\u003eInvestigation of the failed state showed no explicit \u003cstrong\u003eRead\u003c/strong\u003e ACL entry for the service account on the private key. Adding that entry made the service work immediately again.\u003c/p\u003e","title":"FUCKUP #002: The Service That Worked Without Its Expected Private-Key ACL Entry"},{"content":"The symptom Two Windows Server 2019 virtual machines were no longer properly reachable after they were moved to newer Proxmox hosts.\nThe timing made the migration itself look suspicious. The VMs had moved, and their network connectivity had failed.\nWhat made it misleading Several plausible causes fit the initial symptom:\ndifferences between the old and new Proxmox hosts virtual NIC configuration bridge configuration Windows networking migration behavior the Ceph and storage context The incident happened on a weekend. The immediate operational priority was fast service recovery, so restoring service took priority over isolating each variable in the failed state.\nRecovery first A known-good backup was restored as a new VM on the older Proxmox node using ZFS storage. The restored VM became reachable again.\nThat recovery changed several conditions at once. It changed the VM instance, host placement, storage context, and physical network path. The result proved that the recovery path worked, but it did not show that restoring the backup had repaired the guest.\nThe actual root cause The VLAN required by the affected servers was missing from an uplink of the switch serving the newer Proxmox hosts. The older Proxmox node was connected through an older switch path where that VLAN was present.\nPlacement Network path Required VLAN available Result Newer Proxmox hosts Newer switch path No VM not properly reachable Older Proxmox node Older switch path Yes VM reachable Ceph was not the root cause. The guest configuration was not the root cause.\nWhy the restore worked Restoring the VM on the older node also returned it to the older physical network path. That path carried the required VLAN, so the restored VM was reachable.\nA successful rollback does not necessarily prove that VM or hypervisor state was repaired. It may restore another infrastructure dependency at the same time. In this case, moving back to the older node also moved the workload back to a switch path where the VLAN existed.\nDiagnostic path When a VM loses connectivity after migration, trace both the virtual configuration and the physical path:\nCheck guest network state. Confirm the guest-visible adapter, address, route, and the scope of the connectivity failure. Check Proxmox network configuration. Compare the virtual NIC, bridge attachment, and VLAN settings with the known-good placement. Compare source and destination placement. Identify which host the VM left, which host it moved to, and which infrastructure paths differ between them. Trace the VLAN end to end. Follow the path through the Proxmox bridge, physical NIC, switch port, trunk or uplink, and any intermediate switching path. Verify that the required VLAN is carried at every step. Treat rollback as evidence, not root-cause proof. List every condition changed by the rollback. Use those differences to narrow the fault instead of assuming that the VM itself was repaired. This path is reusable for migration-related connectivity failures. In this incident, the actual fault was later identified in the switch uplink configuration.\nThe root cause The required server VLAN was not carried over the uplink of the switch serving the newer Proxmox hosts.\nThe lesson When a VM moves, the relevant change is not limited to the hypervisor. The workload may also move across a different:\nphysical NIC switch port switch trunk VLAN path Do not ask only:\nWhat did the migration break inside the VM?\nAlso ask:\nWhat infrastructure path changed when the VM moved?\nThe gateway failure caused by stale DHCP and static-mapping state is a complementary example: the VM exposed the symptom, while the decisive state remained in the surrounding network infrastructure.\n","permalink":"https://fuckup.fail/cases/003-windows-vms-network-after-proxmox-migration/","summary":"\u003ch2 id=\"the-symptom\"\u003eThe symptom\u003c/h2\u003e\n\u003cp\u003eTwo Windows Server 2019 virtual machines were no longer properly reachable after they were moved to newer Proxmox hosts.\u003c/p\u003e\n\u003cp\u003eThe timing made the migration itself look suspicious. The VMs had moved, and their network connectivity had failed.\u003c/p\u003e\n\u003ch2 id=\"what-made-it-misleading\"\u003eWhat made it misleading\u003c/h2\u003e\n\u003cp\u003eSeveral plausible causes fit the initial symptom:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003edifferences between the old and new Proxmox hosts\u003c/li\u003e\n\u003cli\u003evirtual NIC configuration\u003c/li\u003e\n\u003cli\u003ebridge configuration\u003c/li\u003e\n\u003cli\u003eWindows networking\u003c/li\u003e\n\u003cli\u003emigration behavior\u003c/li\u003e\n\u003cli\u003ethe Ceph and storage context\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe incident happened on a weekend. The immediate operational priority was fast service recovery, so restoring service took priority over isolating each variable in the failed state.\u003c/p\u003e","title":"FUCKUP #003: The VMs That Broke After Migration — But the Problem Was the Switch"},{"content":"Why FUCKUP? For most people, the name is self-explanatory.\nFor those who know:\nFirst Universal Cybernetic-Kinetic Ultramicro Program.\nFUCKUP.fail collects real-world infrastructure failures, strange symptoms, diagnostic paths and the reasoning that leads to the root cause.\nThe focus is not on memorizing commands.\nIt is on answering the more useful question:\nWhat should I test next — and what does the result actually tell me?\n","permalink":"https://fuckup.fail/about/","summary":"\u003ch2 id=\"why-fuckup\"\u003eWhy FUCKUP?\u003c/h2\u003e\n\u003cp\u003eFor most people, the name is self-explanatory.\u003c/p\u003e\n\u003cp\u003eFor those who know:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFirst Universal Cybernetic-Kinetic Ultramicro Program.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFUCKUP.fail collects real-world infrastructure failures, strange symptoms, diagnostic paths and the reasoning that leads to the root cause.\u003c/p\u003e\n\u003cp\u003eThe focus is not on memorizing commands.\u003c/p\u003e\n\u003cp\u003eIt is on answering the more useful question:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWhat should I test next — and what does the result actually tell me?\u003c/strong\u003e\u003c/p\u003e","title":"About FUCKUP.fail"}]