Troubleshooting Veeam Application-Aware Backups
Overview In order to get transactionally consistent backups Veeam is capable of using several methods of application-aware guest OS processing. In some cases, issues with application-aware processing can cause backups to fail. These issues can be caused by issues with Microsoft Volume Shadow Copy (VSS), SQL log processing and/or truncation, Hyper-V integration services, or VMware tools. Microsoft VSS VSS Writer Failures Sometimes Veeam can fail with errors similar to the message above. These are typically caused by failed VSS writers on the Guest OS. To check for any failed VSS writers run the following command: In most cases, a reboot will resolve failed writers. Although in certain scenarios the writers can fail again the next time the backup runs. An alternative to a reboot is to restart the services associated with the failed VSS writer. A list of these services can be found here: KB2041: Windows VSS Writers and Corresponding Service Names (veeam.com) VSS Storage Related Failures VSS requires free storage on the guest OS for space that can be used for storing shadow copies. The first thing to check is that there is enough free space on the volumes being backed up. Usually this is around 10% free space, but will depend on the rate of change and total size of the volumes. If there is enough free space on the volumes, the next thing to check is that the “shadow storage” is configured. To find this open command prompt as administrator and run the following command: Make sure that each volume included in the backups is listed with storage configured. If any of the volumes are missing add shadow storage for them with the following command: If any of the volumes with configured shadow storage are configured for less than 10-20%, they can be increased with the following commands: To store shadow copies on a different volume, change the /On=C: flag to the other drive. (For example if the C: drive was close to full but the D: drive had significant free space, the flag would be /On=D:) VSS Provider Failures Sometimes VSS providers can be left over from previous solutions that are no longer in place. In some cases, Veeam will run into problems initiating vss shadow copies when these providers are present. In most cases the fix for these issues is to remove the bad VSS provider. Run the following command to check what providers are installed on the system: This will output a list of the providers, take note of the GUID of non-Microsoft providers or any from unused solutions. Next, open regedit to the following key: Match the previous GUIDs with the keys in the registry. Right click the keys and export them as a backup. Then delete the matching keys. Next, restart the Volume Shadow Copy service (sometimes the machine must be rebooted for the changes to go into effect.) VSS Timeout Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB1377: VSS wait timeout (veeam.com) Multiple VSS Applications Conflict Failed to create snapshot. Error code -2147212522. ‘Backup job failed.Error VSS error: VSS_E_SNAPSHOT_SET_IN_PROGRESS. Code:0x80042316 Both of these warnings indicate that there are multiple applications attempting to use the same writers at the same time. It is recommended that you adjusting the time that you run your backup to not interfere with other applications that utilize the same VSS writers. Veeam has a knowledge base article with information on how to troubleshoot these types of warnings:KB1784: 0x80042316 or Error: VSSControl: -2147212522 VMware Tools This warning message occurs when Veeam tries to perform application aware processing or guest indexing, but it cannot communicate with the Guest OS of the VM being backed up. By default, Veeam will try to communicate over the network from the Guest interaction proxy specified in the backup job. But if that fails, Veeam will try to communicate through VMware tools. This message is indicating that Vmware tools is either not installed or not running. Please make sure that VMware tools is installed and running on the affected VM. If it is installed, but not running, you can typically restart the service from Windows Services. If it fails to start, reviewing the Windows Application logs can provide info on what is causing the service to fail to start. Microsoft SQL Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB2027: Job reports warning “Failed to truncate transaction logs for SQL instances: Possible reasons: lack of permissions, or transaction log corruption.” (veeam.com) Hyper-V Integration Error Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception) Error: Hyper-V Integration Service is not accessible through the network on source host Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB1855: Hyper-V Guest processing skipped (check guest OS VSS state and integration components version) (veeam.com) There is also a great blog post from Veeam on the process to install and setup the Hyper-V integration services: Hyper-V installation and configuration step-by-step (veeam.com)