Recovery of Locked VMDK

This guide explains a recovery procedure for a locked VMDK file under the ESX Hypervisor. Although out of the ordinary, this problem can prevent a VM from booting which makes it particularly nasty.

Log excerpts from vmware.log describing the problem...

Sep 24 08:41:03.048: vmx| DISKLIB-DSCPTR: Failed to open extents for descriptor file in normal mode
Sep 24 08:41:03.049: vmx| DISKLIB-LINK  :
 "/vmfs/volumes/454a6c04-cc731970-4ded-000423c460a7/LUVM21/LUVM21.vmdk" :
 failed to open (Device or resource busy).
Sep 24 08:41:03.049: vmx| DISKLIB-CHAIN :
 "/vmfs/volumes/454a6c04-cc731970-4ded-000423c460a7/LUVM21/LUVM21.vmdk" :
 failed to open (Device or resource busy).
Sep 24 08:41:03.049: vmx| DISKLIB-LIB   : Failed to open
 '/vmfs/volumes/45e781ce-df97e708-4ea4-00145e1c0f80/MYVM/MYVM.vmdk' with 
 flags 0xa (Device or resource busy).
Sep 24 08:41:03.049: vmx| DISK: Cannot open disk
 "/vmfs/volumes/45e781ce-df97e708-4ea4-00145e1c0f80/MYVM/MYVM.vmdk":
 Device or resource busy (1048585).
Sep 24 08:41:03.049: vmx| DISK: Failed to open disk
 '/vmfs/volumes/45e781ce-df97e708-4ea4-00145e1c0f80/MYVM/MYVM.vmdk' :
 Device or resource busy (1048585) 3191
Sep 24 08:41:03.049: vmx| Msg_Post: Error
Sep 24 08:41:03.049: vmx| [msg.disk.noBackEnd] Cannot open the disk
 '/vmfs/volumes/45e781ce-df97e708-4ea4-00145e1c0f80/MYVM/MYVM.vmdk' or one of the
 snapshot disks it depends on.
Sep 24 08:41:03.049: vmx| [msg.disk.configureDiskError] Reason: Device or resource busy.
Sep 24 08:41:03.077: vmx| Module DiskEarly power on failed.

Procedure

ESX 3.5+

In ESX 3.5.x, all you need to do is this:

vmkfstools -L release $PATH_TO_LOCKED_VMDK_FILE/myvm.vmdk

If i'm not mistaken, this is the same in ESX 3.0.x. Leaving the contents for ESX 3.0.x below alone anyway --Long 04:44, 27 August 2010 (UTC)

ESX 3.0.x

We will use vcbExport to pull the data out of the locked VMDK into a fresh unlocked VMDK. Change MYVM to match your VM's name.

export VMNAME=MYVM

Find the folder where it lives

ls /vmfs/volumes/*/$VMNAME/$VMNAME.vmx
/vmfs/volumes/45e781ce-df97e708-4ea4-00145e1c0f80/MYVM/MYVM.vmx
/vmfs/volumes/SAN8/MYVM/MYVM.vmx

Move the old VM folder out of the way, create anew

cd /vmfs/volumes/SAN8/
mv $VMNAME $VMNAME.old
mkdir $VMNAME
cd $VMNAME

time /usr/sbin/vcbExport -M 1 -F 1 -d $VMNAME.vmdk -s ../$VMNAME.old/$VMNAME.vmdk

Expect to wait awhile, this took me 10m for a 12G virtual disk. YMMV.

If there are additional virtual disks, now is the time to convert them also, i.e.,

time /usr/sbin/vcbExport -M 1 -F 1 -d ${VMNAME}_1.vmdk -s ../${VMNAME}.old/${VMNAME}_1.vmdk

Now tweek resulting MYVM.vmdk (hope you know vi)...and perhaps also MYVM_1.vmdk. But not the ones with '-flat' in their names.

createType="monolithicFlat"

becomes

createType="vmfs"

and

RW 25165824 FLAT "MYVM-flat.vmdk" 0

becomes

RW 25165824 VMFS "MYVM-flat.vmdk"

Restore .vmx and .vmxf file

cp ../$VNMAME.old/$VMNAME.vmx* .

Register VM with the host

vmware-cmd -s register `pwd`/$VMNAME.vmx
register(/vmfs/volumes/45e781ce-df97e708-4ea4-00145e1c0f80/MYVM/MYVM.vmx) = 1

Start the VM

vmware-cmd `pwd`/$VMNAME.vmx start
start() = 1

Answer the inevitable UUID question (KEEP)

vmware-cmd `pwd`/$VMNAME.vmx answer

Check the state, it should be on

vmware-cmd `pwd`/$VMNAME.vmx getstate
getstate() = on

Of course, many of the commands above can also be performed using the viclient. But that is a bit harder to explain, eh?