The symptom

The incident did not start with somebody watching the VM boot. It started with monitoring: Checkmk reported a problem on the VM, which triggered the investigation. The exact Checkmk service and alert message were not recorded.

During that investigation, a recent storage change and an alarming boot-time error became relevant. The Linux VM in Proxmox had its virtual disk expanded from 20 GB to 25 GB. During the next boot, cloud-init attempted to grow the root filesystem. The relevant operation was effectively:

resize2fs /dev/sda1

It failed with messages that included:

Couldn't find valid filesystem superblock
short read

Those errors suggested that the ext4 filesystem on /dev/sda1 had not been resized. The VM itself still completed its boot.

The context

The environment had recently been upgraded from Proxmox 8 to Proxmox 9. The VM used cloud-init with the NoCloud datasource, and its config drive was presented through a virtual CD-ROM device.

Those details define the environment, but they do not establish the cause. The available evidence does not connect the Proxmox upgrade, the NoCloud datasource, or the config-drive device to the transient resize failure.

What made it confusing

A virtual disk expansion crosses several separate layers. Success at one layer does not prove success at the next:

Layer Question to answer
Hypervisor virtual disk Was the configured disk expanded from 20 GB to 25 GB?
Guest block device Does Linux see the larger device?
Partition Does /dev/sda1 occupy the newly available space?
Filesystem Does ext4 expose the expanded capacity?

The boot-time message came from the filesystem-resize step. Taken alone, it looked like the final layer had failed. It did not describe the final state of all four layers.

The evidence

After the VM booted, the storage state was checked again:

  • The virtual disk was 25 GB.
  • /dev/sda1 had grown to approximately the full available size.
  • The ext4 filesystem showed the expanded capacity.
  • Running resize2fs manually produced no repair action and effectively reported Nothing to do!

The observed final state contradicted the alarming boot-time error. By the time the system was inspected, the partition and filesystem were already at the expected size.

The actual outcome

No manual partition repair was required. No manual filesystem resize was required, and no filesystem recovery was performed.

After another reboot, the VM came up normally with a consistent disk, partition, and filesystem state. That reboot confirmed the clean state on the next boot; the evidence does not show that the reboot performed the resize.

What probably happened

The most defensible explanation is a transient timing or ordering problem during boot. Device discovery, partition-table updates, and the filesystem-resize attempt may not have been observed in the expected order when cloud-init ran.

That is a hypothesis, not a captured root cause. The available evidence does not establish whether the message came from cloud-init behavior, device discovery timing, partition update timing, or another boot-ordering effect. It only shows that the resize appears to have completed despite the reported error.

Diagnostic path

When an automated filesystem resize reports failure, verify each storage layer before attempting repair:

  1. Check the hypervisor disk size. Confirm that the VM configuration shows the requested capacity.
  2. Check the guest block device. Verify that Linux sees the larger virtual disk. If it does not, stop at the device-discovery layer rather than changing partitions or filesystems.
  3. Check the partition size. Confirm whether the root partition occupies the newly available space. A larger block device does not mean its partition has grown.
  4. Check the filesystem size. Confirm whether ext4 exposes the expanded capacity. A larger partition does not prove the filesystem has been resized.
  5. Compare the state with the cloud-init error. Review the failed operation and its timing, but treat it as evidence about that attempt rather than proof of the current state.
  6. Retry resize2fs only if the filesystem is still smaller than its partition. If the filesystem already fills the partition and the tool reports that there is nothing to do, do not escalate to partition repair or filesystem recovery without additional evidence.
  7. Use a controlled reboot as verification when appropriate. A clean subsequent boot can confirm that the final state persists, but it does not prove which earlier step completed the resize.

The root cause

The precise transient failure mechanism was not captured. What is established is narrower: cloud-init reported that the ext4 resize failed, while the final disk, partition, and filesystem state became correct without manual repair or resizing.

The Proxmox 8-to-9 upgrade remains context, not a proven cause.

The lesson

Do not confuse a failed automation step with a failed final state.

Check the virtual disk, guest block device, partition, and filesystem in order. An automation error tells you that an operation reported failure at a particular moment. It does not, by itself, prove that the intended final state was never reached.

Aggressive repair based only on the earlier error can turn a transient issue into a real one. Verify the current state first, and intervene only at the layer that is still wrong.

Sometimes, a reboot really does do wonders.