Problem with volume resize after upgrade from 4.22.0 to 4.22.1

[sbrueseke]

Hello, we found a strange situation yesterday and hope somebody can try to recreate it.
We are using KVM hypervisor with linstor storage. We needed to resize a volume of an instance, which we successfully did in the past on a running vm. This time it did not work as expected. From ACS and linstor everything looks good, but inside the instance the volume still did had the same size. After stopping the vm and starting it again, the new size was shown and could be used. We see this behavior only on instances which were not stopped and started since our upgrade to 4.22.1. New instances are working as expected.

virsh domblkinfo i-29-423-VM vda
Capacity: 53688164352
Allocation: 53688139776
Physical: 107376365568

It looks like the new Capacity is not shown in virsh domblkinfo and we see some changes in virsh dumpxml regarding source dev.

Is anybody able to recreate this behavior? You need to run on 4.22.1 and you need access to instances which are not being stopped since the upgrade.

[herrickP]

This looks like a libvirt/QEMU live-resize issue rather than a Linstor/ACS problem. When the volume is resized via ACS, it issues a virsh blockresize (or updates the disk via Linstor's DRBD/LVM layer), but if the VM's underlying device mapping or DRBD metadata wasn't refreshed properly post-upgrade, libvirt's cached domblkinfo will keep reporting the old capacity until a fresh dumpxml/device re-attach happens, which only occurs on a full stop/start (since reboot alone doesn't re-read block device geometry).
A few things to check before filing a bug:

Compare virsh dumpxml source dev paths for an affected VM vs. a newly created one, you mentioned seeing differences there, which suggests the older domains may still be referencing a stale Linstor resource path/minor number from before the upgrade.
Check if blockresize was actually invoked on the live domain (libvirt logs) or if ACS only resized the Linstor resource and assumed libvirt would pick it up.
This smells like a regression introduced in how 4.22.1 handles live volume resize for KVM+Linstor combos specifically, worth checking the changelog/diff between 4.21 and 4.22.1 around the LibvirtResizeVolumeCommand handling.

For now, the stop/start workaround seems to be the only reliable fix until this is patched. If you haven't already, definitely open a GitHub issue with your dumpxml diffs, that'll help maintainers spot exactly where the stale reference is held.
Side note - I run a decent chunk of CloudStack on Accuweb.cloud KVM+Linstor in production and haven't hit this yet, but we're still on 4.21.x for exactly this kind of reason, newer ACS minor releases sometimes need a cycle to shake out edge cases like this. Might be worth holding off upgrading further nodes until this is confirmed/fixed.

[sbrueseke]

@herrickP thx for your reply! What we see is the following change in path:
VM before 4.22.1:
source dev='/dev/drbd1039' index='2'

After Start/Stop
source dev='/dev/drbd/by-res/cs-491cc89a-0934-432f-8174-ee82b04e1ae3/0' index='2'

@rp- can you take a look at this?

[rp-]

The source dev name change is intended, it is for supporting the new nas backup feature.

But anyway, I debugged the resize notify in cloudstack and as far as i got see, it doesn't notify libvirsh by block dev name, but rather the logical device name in the guest e.g. (vda, vdb, ...) and on my test cluster libvirt was properly notified about the change.

My question still is, does this not work only on VM's that didn't have before the new device name? or is this happening everytime now?

[sbrueseke]

As far as we see, this issue only applies to instances which where created before the upgrade to 4.22.1 and running since. New created instances are working as expected. Also stopping and starting an instance is solving the issue.

[rp-]

Then I'm afraid qemu/libvirt is confused about the path change and it will only work after a start/stop of the VM.