VCSA 7.0 File-based restore failing at 80%

July 14, 2022 | 7 minutes

Recently I had a customer testing their vCenter Server 7.0U3e file-based restore process using the VCSA Restore Wizard.

During Stage 2 (data copy) the wizard hung at 80% and did not progress for hours. No UI errors or hints as to what was happening. I had a look at the restore wizard logs and found the following:

2022-07-12T01:16:19.828Z - debug: [7]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:30.809Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:30.819Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:30.819Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:30.819Z - debug: [8]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:41.821Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:41.847Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:41.847Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:41.847Z - debug: [9]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:52.811Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:52.819Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:52.819Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:16:52.819Z - debug: [10]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:03.800Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:03.808Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:03.809Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:03.809Z - debug: [11]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:14.820Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:14.833Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:14.833Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:14.833Z - debug: [12]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:24.820Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:24.829Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:24.829Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:24.829Z - debug: [13]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:34.840Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:34.866Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:34.866Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:34.866Z - debug: [14]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:44.845Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:44.854Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:44.854Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:44.854Z - debug: [15]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:55.815Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:55.827Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:55.827Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:17:55.827Z - debug: [16]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:06.824Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:06.833Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:06.833Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:06.833Z - debug: [17]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:17.813Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:17.821Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:17.821Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:17.821Z - debug: [18]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:28.815Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:28.823Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:28.824Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:28.824Z - debug: [19]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:39.819Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:39.827Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:39.827Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:39.827Z - debug: [20]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:50.816Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:50.825Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:50.825Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:18:50.825Z - debug: [21]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:19:00.824Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:19:00.834Z - debug: Failed to get fileTransferInfo:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:19:00.834Z - debug: Failed to get url of file in guest vm:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created
2022-07-12T01:19:00.834Z - debug: [22]pollRpmInstallProgress:getGuestFileErr:ServerFaultCode: The object 'vim.VirtualMachine:103' has already been deleted or has not been completely created

These messages repeated endlessly. To understand what these messages mean, it helps to understand how the VCSA wizard deploys the appliances.

When you deploy a vCenter appliance, the wizard will deploy an OVA to an ESXi host (either directly or through vCenter). One of the primary control mechanisms to check the appliance’s guest status is using VIX to peak into the guest OS using VMware Tools. When deploying directly to a host, the APIs are called directly against the host and resolved to the VM on that host.

If that VM is vMotioned from the target host to another during your deployment (maybe during a DRS event), the wizard will fail to find the VM and you’ll end up with log entries similar to the snippet above. That’s why it’s so critical to follow the documentations recommendation if you are deploying directly to a host in a Fully Automated DRS cluster: you have to set DRS to partially automated beforehand to ensure the VM want automatically move while you’re deploying the vCenter appliance (likewise for restores, it’s the same process).

Moral of the story is read the documentation :)

Share this on:

About Stellios Williams

Senior Cloud Solutions Architect - Service Providers VMware

This is my personal tech related blog for anything private and public cloud - including homelabs! My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions. Any technical guidance or advice is given without warranty or consideration for your unique issues or circumstances.

Contact

Comments