Author name: Managecast Technologies

Veeam

Troubleshooting Veeam Application-Aware Backups

Overview In order to get transactionally consistent backups Veeam is capable of using several methods of application-aware guest OS processing. In some cases, issues with application-aware processing can cause backups to fail. These issues can be caused by issues with Microsoft Volume Shadow Copy (VSS), SQL log processing and/or truncation, Hyper-V integration services, or VMware tools. Microsoft VSS VSS Writer Failures Sometimes Veeam can fail with errors similar to the message above. These are typically caused by failed VSS writers on the Guest OS. To check for any failed VSS writers run the following command: In most cases, a reboot will resolve failed writers. Although in certain scenarios the writers can fail again the next time the backup runs. An alternative to a reboot is to restart the services associated with the failed VSS writer. A list of these services can be found here: KB2041: Windows VSS Writers and Corresponding Service Names (veeam.com) VSS Storage Related Failures VSS requires free storage on the guest OS for space that can be used for storing shadow copies. The first thing to check is that there is enough free space on the volumes being backed up. Usually this is around 10% free space, but will depend on the rate of change and total size of the volumes. If there is enough free space on the volumes, the next thing to check is that the “shadow storage” is configured. To find this open command prompt as administrator and run the following command: Make sure that each volume included in the backups is listed with storage configured. If any of the volumes are missing add shadow storage for them with the following command: If any of the volumes with configured shadow storage are configured for less than 10-20%, they can be increased with the following commands: To store shadow copies on a different volume, change the /On=C: flag to the other drive. (For example if the C: drive was close to full but the D: drive had significant free space, the flag would be /On=D:) VSS Provider Failures Sometimes VSS providers can be left over from previous solutions that are no longer in place. In some cases, Veeam will run into problems initiating vss shadow copies when these providers are present. In most cases the fix for these issues is to remove the bad VSS provider. Run the following command to check what providers are installed on the system: This will output a list of the providers, take note of the GUID of non-Microsoft providers or any from unused solutions. Next, open regedit to the following key: Match the previous GUIDs with the keys in the registry. Right click the keys and export them as a backup. Then delete the matching keys. Next, restart the Volume Shadow Copy service (sometimes the machine must be rebooted for the changes to go into effect.) VSS Timeout Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB1377: VSS wait timeout (veeam.com) Multiple VSS Applications Conflict Failed to create snapshot. Error code -2147212522. ‘Backup job failed.Error VSS error: VSS_E_SNAPSHOT_SET_IN_PROGRESS. Code:0x80042316 Both of these warnings indicate that there are multiple applications attempting to use the same writers at the same time. It is recommended that you adjusting the time that you run your backup to not interfere with other applications that utilize the same VSS writers. Veeam has a knowledge base article with information on how to troubleshoot these types of warnings:KB1784: 0x80042316 or Error: VSSControl: -2147212522 VMware Tools This warning message occurs when Veeam tries to perform application aware processing or guest indexing, but it cannot communicate with the Guest OS of the VM being backed up. By default, Veeam will try to communicate over the network from the Guest interaction proxy specified in the backup job. But if that fails, Veeam will try to communicate through VMware tools. This message is indicating that Vmware tools is either not installed or not running. Please make sure that VMware tools is installed and running on the affected VM. If it is installed, but not running, you can typically restart the service from Windows Services. If it fails to start, reviewing the Windows Application logs can provide info on what is causing the service to fail to start. Microsoft SQL Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB2027: Job reports warning “Failed to truncate transaction logs for SQL instances: Possible reasons: lack of permissions, or transaction log corruption.” (veeam.com) Hyper-V Integration Error Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception) Error: Hyper-V Integration Service is not accessible through the network on source host Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB1855: Hyper-V Guest processing skipped (check guest OS VSS state and integration components version) (veeam.com) There is also a great blog post from Veeam on the process to install and setup the Hyper-V integration services: Hyper-V installation and configuration step-by-step (veeam.com)

Veeam

Veeam Backup for Salesforce – Now Available!

Veeam Backup for Salesforce provides complete access and control of Salesforce data and metadata, as well as providing powerful, rapid-recovery capabilities for IT departments and Salesforce administrators. Veeam recently announced the launch of its newest product offering – Veeam Backup for Salesforce. Salesforce is one of the most utilized CRMs in the world because of its advanced features, but many don’t realize the importance of backing up Salesforce data. Do I need to back up Salesforce? Some users think Salesforce is responsible for backing up the data. Still, it clearly states, “It is important for Salesforce customers to develop a routine data backup strategy as part of their overall data management and security model.” We often talk about the Microsoft shared responsibility model, and Salesforce follows the same rule. Even though Salesforce manages the infrastructure, this doesn’t replace YOUR responsibility to protect Salesforce from data loss and data corruption. Key Capabilities Veeam Backup for Salesforce eliminates the risk of losing your Salesforce data and metadata — so that your business is reliably protected, giving you complete control. Key capabilities include: Salesforce native: Purpose-built to create backups and restore Salesforce data and metadata Fast and flexible recovery: Restore Salesforce records, hierarchies, fields, files and metadata Secure data: Run a backup environment anywhere, on-premises or in the cloud (AWS, Microsoft Azure, etc.) Custom scheduling: Set granular backup schedules and retention settings at the object level Simplified management: Manage several Salesforce instances from one console Incremental changes: With incremental sync and flexibility scheduling, you can back up your Salesforce data and metadata almost continuously Simple and easy-to-use UI: Run backup policies and restore jobs in minutes See and compare: See versions of records and metadata and quickly compare them with production Restore hierarchy: Granularly restore linked objects to any record, including parent/child records Ready to start? Getting started is simple! Veeam provides a licensing structure that aligns with your Salesforce usage. From now until December 31, 2022, Veeam has special pricing on backup for Salesforce: 50 users or less – FREE! Up to 300 users – $2,000 USD for the first year Unlimited users – $10,000 USD for the first year At Managecast, we fully manage and monitor your backups so that you can focus on more strategic initiatives. Interested in learning how Managecast can help your business with its cloud backup and disaster recovery solutions? Fill out this form for more information!

Veeam

Error: Cannot proceed with the job existing backup meta file is not synchronized with the DB [56]

We recently ran into an issue where a Veeam Cloud Connect replication job started to fail with the error: Restore point is corrupted We noticed that the source for the replication job was set as the offsite backup copy job, but the source backup copy job was also failing with the following error: Looking into the error message, this can happen when the backup server is rebooted or loses connection during the offsite backup, and it causes things to go out of sync. It is possible this can be resolved by just rescanning the repository and re-enabling the backup copy job. We tried this, but the rescan on the repository came back with an error “XYZ Cloud repository: Failed to synchronize 1 backups.” After a bit more investigation, we determined that the VBM file in the customers’ repository had a size of 0 MB. This indicated that the metadata file had been corrupted and needed to be recreated. Veeam can typically recreate the VBM file as long as the job and remaining backup files are intact. In this case, the solution for this issue was to: Disable the backup copy job. From the Cloud Connect provider side, access the customers’ backup repository. Rename the backup metadata file to “XYZ.vbm.bak” Then rescan the repository from the customer’s backup server. Re-enable the backup copy job. This time, the rescan completed successfully. The backup copy and replication jobs were re-enabled and the next run for each completed successfully.

Office 365 Backup

Do You Really Need to Backup Microsoft 365 Data?

I often educate customers on the 3-2-1 rule of backup which is an industry-standard minimum data protection level. The 3-2-1 rule of backup states you should always have: 3 copies of your data (including the original) Stored on 2 different media With 1 copy stored offsite Does Microsoft 365 by itself meet the 3-2-1 rule backup for customers? Do they provide backup at all? First, what Microsoft provides to customers natively does not meet the 3-2-1 rule of backup. Microsoft clearly states the responsibility is yours. Simply Google “Microsoft shared responsibility model” and click the first link. You will note that at the top of the list of those responsibilities always retained by the customer is “information and data”.   Companies are increasingly moving from on-premises file and email servers to Microsoft 365 and other cloud services, but backup and data protection in the cloud is too often a secondary thought.   With the rise of ransomware, other threat actors, and natural disasters, the need for data protection has never been greater.  You should protect your Microsoft 365 data!  For help and assistance with protecting Microsoft 365 data feel free to contact us at sales@managecast.com

Veeam

Error: Item with the same key has already been added.

This is an error that occurs when an instant VM recovery has started. Then, the backup copy job runs while the instant VM recovery is running and has not migrated to production. The original VM is deleted and replaced by the one that is in the instant VM recovery mode. The local job can pick up on this change and backup the new VM that is running on the backup storage. However, the copy job begins to have a bug that causes the whole job to fail. Even if you remove the data from the offsite location and remove the VM from the local job, the job continues to fail. The error also will not list out the VM that is causing the issue, but we have only seen this error occur after one of the VMs in the job has gone through an instant VM recovery. The first thing that needs to be done is to migrate the new VM from the backup storage to production. The new VM should already be in the local job, and it can be left in there. The issue is fixing it on the offsite location. Doing a rescan on the service provider or removing the data for that VM does not work. For this to work, the backups also need to be set to Per VM backup chain. If they are not set to Per VM Backups, then this solution will not work. Next, you need to remove the Per VM chain of the VM that is causing the issue. It cannot be removed from the Veeam Console or else it will display this error: The VM needs to be removed from the service provider side in the repository. Once this is deleted the issue will still be there. Even though all the past data is gone, it will still show the same error for the whole job, and none of the other VMs in the job can be backed up. The final step is taking all the data from the job and moving it outside of the tenant directory on the service provider side. It needs to be moved outside of the tenant directory and rescanned from the tenant side in Veeam. After the rescan, the job data should no longer show up under the backups section. You will then want to move that job data back into the tenant directory from the service provider side in the same way that you moved it out. Next, you will want to do another rescan on the tenant side in Veeam. The job should show back up in the backups section. You will want to ensure that the Copy Job that was getting this error is mapped to the job correctly. After that has been completed, the job should be able to transfer data for the other VMs in the job. I have seen where the VM that was causing the issue fails after this but the other VMs have been able to run. My solution to that was to make a separate local job and offsite job just for that VM and get all new data for it.

Managecast News

Managecast Named Veeam VCSP Competency Partner

Managecast Technologies, Inc., an award-winning Veeam Cloud Service Provider (VCSP), was named a Veeam VCSP Competency Partner earlier this month. This program designates a service provider that offers the highest level of qualification for providing Veeam-as-a-service solutions. Becoming a VCSP Competency Approved Partner involves thorough testing and approval processes and ensures partners follow strict business, technical and documentation requirements. Managecast Technologies, Inc. is a proud Veeam Gold Partner and has been part of the partner program since 2016. Managecast was also presented with the prestigious Veeam Innovation Award, which celebrates creating new and unique ways to add value to Veeam technology, in 2020 at VeeamON. “Not all Veeam service providers are the same, and some focus more on the providing instead of on the service, like we do,” says Nathan Golden, Owner of Managecast. “Managecast is proud to be recognized for our commitment, focus and expertise we provide for Veeam solutions.” Managecast has met the competency requirements for the following services: BaaS for Microsoft 365 Microsoft provides a wide array of powerful services within Microsoft 365, but a comprehensive backup solution is not one of them. Customers who don’t back up their Microsoft 365 data are at risk of accidental deletion, security threats and retention policy gaps. By achieving the Backup as a Service (BaaS) for Microsoft 365 competency, Managecast has been identified as an expert in providing an out-of-the-box BaaS solution that’s specifically built to protect your critical Microsoft 365 data. Off-site backup We make it easy to extend your data protection solutions with whatever storage option makes sense for your recovery needs, whether it’s tape, object storage or sending backups to a service provider’s off-site cloud infrastructure. Managecast offers a ready-made off-site backup solution to help you get started. DRaaS The need for a disaster recovery (DR) solution cannot be understated, particularly with the rise in frequency of ransomware attacks. Managecast delivers a best-in-breed DR solution that enables you to focus your resources on the important projects to grow and innovate your business, spending less time on risk mitigation. —- About ManagecastManagecast Technologies is a data protection business founded in February of 2000. Passionate about providing world-class data protection solutions, we offer a broad spectrum of services around disaster recovery and business continuity solutions. We combine professional management and disaster recovery planning services with the ability to deliver complete offsite disaster recovery solutions for virtually any platform and application. Our data centers are located in Cincinnati, Ohio, and Detroit, Michigan and are HIPAA, SSAE, SOX, and PCI compliant/certified. To learn more, visit www.managecast.com About VeeamVeeam is the leader in Backup solutions that deliver Cloud Data Management. Veeam’s Availability Platform is the most complete backup solution for helping customers on the journey to achieving success in the 5 Stages of Cloud Data Management. To learn more, visit www.veeam.com ContactNathan Golden, Presidentngolden@managecast.com(513) 735-6868 x 4002

General Cloud Backup

What are you using for offsite backups? – Spiceworks Discussion

While scrolling through Spiceworks (an online community where IT professionals can collaborate and seek advice from one another), we found a post requesting options for offsite backups. We see questions like this (below) frequently and wanted to provide a thoughtful and helpful response. If you’re looking for offsite backup options, please take our response into consideration. Nathan Golden – CloudBackupGuy https://community.spiceworks.com/people/nathan-managecast Hey there C.J.  I don’t think you are going to find anything that absolutely prevents bad stuff from being included on the backups as 0-day attacks are often not detected until it’s too late. I do know Sophos makes a great end-point protection product and combined with their firewall offers pretty good defense, but it is not perfect. Nothing is. However, it is wise to do everything possible to keep from being infected such as regularly training your users not to click on bad things (knowb4.com), as well as advanced firewalls and end-point protection, etc.  Backup is the last line of defense! As others mentioned, with Veeam it is possible to run the restore through an AV engine. The idea is that once you’ve found you’ve been compromised you will hopefully find a way to detect the bad stuff. Maybe you need to wait for an updated AV signature, but the idea is once the AV has been updated you can run your restore through it and not restore infected files. Hopefully, only a few files had bad stuff and hopefully, you are restoring to a point in time before the malware has executed and done its thing. The other issue that some are eluding to is how do you protect your past backups from being corrupted/deleted by the malware. A lot of folks will talk about “air gap”, meaning you physically separate your media from the network, which is a great way to protect your data from being compromised by ransomware – though realize air gap does nothing to stop you from backing up malware. However, I see a lot of people do the air gap but then don’t get data offsite frequently. So while negating one threat (ransomware), they are actually introducing additional risk.  Personally, I think anything that relies on human intervention (like rotating media, etc) is prone to error and failure. Our solution is to use a Veeam certified immutable object-store. Yes, it’s still connected to the network, but protected by a different set of credentials and is a solution that is designed to provide unchanging read-only data, even by the network administrator. This maintains full automation and what I call a “virtual airgap”. This is on top of the protections offered by Veeam Insider Protection, etc., so you are really minimizing any risks. As you said there are thousands of companies out there offering these types of services. My suggestion is to ask these questions when choosing an offsite solution: 1) Is the company 100% dedicated to backup and DR, or is it 1 of many different offerings?   2) Is the company a utility-type provider doing their best to keep their humans from interacting with you?  They may provide a service, but little to no expertise – so know what you are doing. 3) Does the company offer Veeam (or other) certified immutable backup capabilities? 4) If you needed your data quickly, how fast can you get it? Is it download only or will they quick ship your data?  Do they offer failover/DR services? Hope this was helpful in your search.

Veeam

Azure File Share with Veeam

In the Azure portal, you will select the storage account that your file share is currently being stored in. Select the File share that you want to backup with Veeam. Click Connect up in the left corner of the screen. Next, you will want to open up Veeam Backup and Replication. Go to Inventory, then under Inventory select File Shares. Select Add File Share. You will want to select SMB Share. When it asks you what shared folder to use, you will have to use the path of \\URL\fileshare In our case, it would be \\teststoragefileshare.file.core.windows.net\blogfileshare. You will then want to check the box next to “This share requires access credentials:” Click Add next to the credentials. Go to the text file that you saved with the connection configuration. In that file you will want to look for the /user: after that will be the username you will use. Then look for the /pass: and after that will be the password. Click Next and it will take you to the processing page. Adjust as you need to then click Apply. Click Finish and your SMB Share will be saved to Veeam. You will Then want to back up the SMB share. Right-click the file share and select Add to backup job, then New Job. Specify a name for the Job, then click next. The Fileshare should already be in there, click next again. Pick the backup repository that you would want the Fileshare backed up to. There are also other settings you can specify here as well. Click next on secondary target and then you will be in schedule. Schedule the backup for whatever time works best for you and click Apply. Finally, check the box for “Run the job when I click Finish” then click finish.

Veeam

Issue: Veeam Issues Nuisance Low Repository Disk Space Warnings

Overview Generally, it is good to be informed of low backup storage situations on Veeam repositories. Running out of disk storage during backup operations can result in corrupted backups. By default, Veeam is configured to trigger a warning notification when backup storage space is 10% or less free space. For example, if you have 1TB of storage and the notification is set to warning at 10% left then it would trigger when there is 100GB or less space.  It is good to be aware of low disk space situations so we can proactively address the issue before it leads to a problem. However, in some scenarios, 10% can trigger nuisance warnings that make all affected jobs turn yellow (warning). For example, if there is 50TB of storage then warnings are going to be issued with 5TB left on the disk. In this situation, it may not be justified to resolve the issue as there is still a lot of free space Solution We recommend changing the notification settings when 10% of free space is still more than adequate for current needs. Changing these settings will return the backup jobs to green instead of showing up as warnings. Our goal is to keep all jobs green so we do not have to investigate why they are not. Green is good!  Do not change these settings if you do not feel comfortable that you will not run out of space!  Changing the value from 10% to 1% may not be wise. We recommend decreasing the threshold to only that which will resolve the issue, for example, changing from 10% to 7%. Note: This guide will work for Hyper-V hosts and VSphere hosts. Changing Veeam repository low-disk notifications First, log into your Veeam appliance. In the top left corner click the drop-down and select General Options. Next, click the notification tab. Then you will see an option for backup storage, there will be a checkbox next to it that says “Warn me when free disk space is below:”. You want to leave this box checked. Next to that will be a percentage. You are going to want to change the percentage from 10% to what fits best for your storage. We recommend cutting it in half to around 5%. You will then want to change the option “Skip VM processing when free disk space is below:”. You will want to lower that percentage to ensure that the VMs are not skipped from processing when there is plenty of storage. Summary Once the notification settings are made to verify future backup jobs return green status.  Take care to leave the threshold at a reasonable percentage as to notify you in the future if disk space gets dangerously low.

Managecast News

Managecast Wins Veeam Innovation Award 2020

CINCINNATI – June 17, 2020 – Managecast Technologies, Inc. was presented with the prestigious Veeam Innovation Award, which celebrates creating new and unique ways to add value to Veeam technology, on June 17, 2020, at VeeamON. Starting in 2018, the Veeam Innovation Award is given out to partners who are making waves in the Veeam space. Managecast created the Managecast Portal for Veeam in 2019 and has been enabling MSPs and clients to more easily monitor backups ever since. This all-encompassing portal utilizes Veeam APIs form Veeam Service Provider Console to provide long-term historical trending of backup jobs and repositories,  ticketing integration, tracking of jobs that have not run, tracking the loss of connection with VBR consoles, a high-level dashboard showing backup activities, fully multi-tenant and MSP enabled to allow third parties to track their client’s backups, acknowledgment and tracking of events, and anomaly detection. “We originally built the Veeam portal for our internal use to monitor client backups and realized other MSP’s could benefit from the tool we built,” says Nathan Golden, Owner of Managecast. “It is a great honor to be recognized by Veeam for our efforts.” You are not required to store backups with us and can use our Veeam portal for your own Veeam service. Managecast welcomes any service provider running Veeam and managing customer backups to see why we were selected as the winner of the Veeam innovation award. ResourcesManagecast Portal for Veeam DemoAnnouncement from Veeam About ManagecastManagecast Technologies is a data protection business founded in February of 2000. Passionate about providing world-class data protection solutions, we offer a broad spectrum of services around disaster recovery and business continuity solutions. We combine professional management and disaster recovery planning services with the ability to deliver complete offsite disaster recovery solutions for virtually any platform and application. Our data centers are located in Cincinnati, Ohio, and Detroit, Michigan and are HIPAA, SSAE, SOX, and PCI compliant/certified. To learn more, visit www.managecast.com About VeeamVeeam is the leader in Backup solutions that deliver Cloud Data Management. Veeam’s Availability Platform is the most complete backup solution for helping customers on the journey to achieving success in the 5 Stages of Cloud Data Management. To learn more, visit www.veeam.com ContactNathan Golden, Presidentngolden@managecast.com(513) 735-6868 x 4002

Scroll to Top