VDI error 1010 - Backup Failures

owenh

I am receiving an intermittant failure during backups, both for full database backups and transaction backups. Here is the email information that is being sent during the failure:

8/15/2005 11:00:06 AM: SQL Backup 3.2.0, (c) Red Gate Software Ltd 2004 - 2005
8/15/2005 11:00:06 AM:
8/15/2005 11:00:06 AM: Backing up Amerex (transaction log) to \SQLBACKUPS\Default\DailyBackups\Amerex\TransactionLogs\Amerex_tlog_200508151100.sqt ...
8/15/2005 11:00:06 AM:
8/15/2005 11:00:06 AM: BACKUP LOG [Amerex] TO DISK = 'D:\SQLBACKUPS\Default\DailyBackups\Amerex\TransactionLogs\Amerex_tlog_200508151100.sqt' WITH NAME = 'LOG Amerex, 2005-08-15 11:00:06.487', DESCRIPTION = 'Backup on 08/15/2005 11:00:06 Database: Amerex Instance: (local) Server: HOU-SQL01', VERIFY, ERASEFILES = 30, MAILTO = 'owenh@amerexenergy.com', COMPRESSION = 2
8/15/2005 11:00:06 AM:
8/15/2005 11:00:06 AM:
8/15/2005 11:00:21 AM: VDI error 1010: Failed to get configuration from server. Check that the SQL Server instance is running, and that you have the SQL Server Systems Administrator server role. Error code: -2139684861 (The api was waiting and the timeout interval had elapsed.)


Additionally, this information is extracted from the System Event Log:

Application popup: sqlbackupc.exe - Application Error : The exception unknown software exception (0x0eedfade) occurred in the application at location 0x77e649d3.


Application Log:

Faulting application sqlbackupc.exe, version 3.2.0.1, faulting module kernel32.dll, version 5.2.3790.0, fault address 0x000249d3.

Application Log:
18210 :
BackupVirtualDeviceSet::Initialize: Open failure on backup device 'SQLBACKUP_6660477'. Operating system error 0x80070002(The system cannot find the file specified.).

3041 :
BACKUP failed to complete the command BACKUP LOG [Amerex] TO VIRTUAL_DEVICE = 'SQLBACKUP_6660477' WITH BLOCKSIZE = 65536, MAXTRANSFERSIZE = 1048576, NAME = 'LOG Amerex, 2005-08-15 11:00:06.487', DESCRIPTION = 'Backup on 08/15/2005 11:00:06 Database: Amerex Instance: (local) Server: HOU-SQL

Reporting queued error: faulting application sqlbackupc.exe, version 3.2.0.1, faulting module kernel32.dll, version 5.2.3790.0, fault address 0x000249d3.

Fault bucket 195403086.


SQLBackup is being run through the Job Scheduler using a domain account with proper credentials and permissions. Could this be a memory shortage problem? I am not sure where to look now. The bad thing is that hte Application popup errors appear to be "stopping" the server processes for backups to complete, thus delaying the backups. As I indicated before,m it doesn't appear to be on any specific backup, transactins or logs, and not always on the same databases.


Help!


Thanks.

Brian Donahue

Hello Owen,

This sounds like SQL Backup could not locate osql.exe to run the backup command again. Can you log on as the user who is configured to run the SQL Agen, then right-click on My Computer and pick properties?

Then go to the advanced tab, click environment variables and make sure that the path to SQL Server Binn is in your path variable (It normally ends in 80/Tools/Binn).

owenh

The Command prompt shows that the path for the OSQL program is available.

When I log on as the userid running SQLAgent and drop to a command prompt, running OSQL command works properly, i.e., it is finding the program.

As I stated previously, this is intermittent. It is running backups usign the identical criteria for bot logs and full backups all day long. It just appears to be failing at seemingly random times. Some other posts indicated that the error could be being caused by "rapidly creating VDI devices " thus causing a conflict.

Brian Donahue

Hi Owen,

I can send the VDI device GUID patch, if you think maybe that will help!

owenh

Will this patch stop the application popup errors that are occuring on the server? If so, please send it.

Thanks

Daniel Handley

Hi Owen

If the application is causing error they may appear as popups. Once we have solved the problems you are having the popups should stop.

I have also sent you the patch now.

Regards
Dan

owenh

The patch for SQLBackup (version 3.2.0.2) appears to have corrected the problems.

Thanks!

joshmurrah

I'm getting this same exact intermittent error; I'm running 3.2.0.1 to fix the GMT ERASEFILE bug... does 3.2.0.2 reduce/eliminate this problem? It seems to be a load problem, I usually get it when multiple backups or multiple log backups are happening at once. Here's an example from the logs:

Thanks!

-Josh Murrah

8/17/2005 1:58:38 PM: SQL Backup 3.2.0, (c) Red Gate Software Ltd 2004 - 2005 
8/17/2005 1:58:38 PM: 
8/17/2005 1:58:38 PM: Backing up SiteCount (full database) to X:\BACKUP\CORP-RSERVER_SiteCount_FULL_200508171358.sqb ...
8/17/2005 1:58:38 PM: 
8/17/2005 1:58:38 PM: BACKUP DATABASE [SiteCount]  TO DISK = 'X:\BACKUP\CORP-RSERVER_SiteCount_FULL_200508171358.sqb' WITH NAME = '', DESCRIPTION = '', ERASEFILES = 1, COMPRESSION = 1
8/17/2005 1:58:38 PM: 
8/17/2005 1:58:38 PM: 
8/17/2005 1:58:53 PM: VDI error 1010: Failed to get configuration from server.  Check that the SQL Server instance is running, and that you have the SQL Server Systems Administrator server role.  Error code:  -2139684861 (The api was waiting and the timeout interval had elapsed.)

devjvc

I'm having the same intermittent problem with Backups "timing out". It seems the command waits for 15 seconds before quitting. Is there some kind of timeout value that can be updated? Is SQL Backup timing out or OSQL (which is being used under the covers)? This is only happening on some of our more taxed SQL Servers. I applied the latest patch to no avail. Any ideas? Thanks!

11/7/2005 11:15:39 AM: SQL Backup 3.2.0.5, (c) Red Gate Software Ltd 2004 - 2005
11/7/2005 11:15:39 AM:
11/7/2005 11:15:39 AM: Backing up PEGADMIN (transaction log) to O:\BACKUP\PEGADMIN\PEGADMIN_LOG_DISK_BACKUP_1100.sqb ...
11/7/2005 11:15:39 AM:
11/7/2005 11:15:39 AM: BACKUP LOG [PEGADMIN] TO DISK = 'O:\BACKUP\PEGADMIN\PEGADMIN_LOG_DISK_BACKUP_1100.sqb' WITH NAME = '', DESCRIPTION = '', INIT, PASSWORD = 'XXXXXXXXXX', ERASEFILES = 30, COMPRESSION = 3
11/7/2005 11:15:39 AM:
11/7/2005 11:15:39 AM:
11/7/2005 11:15:54 AM: VDI error 1010: Failed to get configuration from server. Check that the SQL Server instance is running, and that you have the SQL Server Systems Administrator server role. Error code: -2139684861 (The api was waiting and the timeout interval had elapsed.)

Msg 3013, Level 16, State 1, Server SQLTST04, Line 1
BACKUP LOG is terminating abnormally.


Here is the VDI.log at the time of failure:

---

2005/11/07 11:15:54 pid(1000) tid(5632)
Error on Global\SQLBACKUP_{3906123D-4370-4B12-8F02-1C1821A7DC8F}
Error at CVDS::Close: Abnormal termination state

---

2005/11/07 11:15:54 pid(1000) tid(5632)
Error on Global\SQLBACKUP_{3906123D-4370-4B12-8F02-1C1821A7DC8F}
Error at TriggerAbort: invoked

---

2005/11/07 11:15:56 pid(228) tid(4264)
Error on Global\SQLBACKUP_{3906123D-4370-4B12-8F02-1C1821A7DC8F}
Error at SVDS::Open: Open(control) Status Code: 2, x2
Explanation: The system cannot find the file specified.

Brian Donahue

If the load on the server is the culprit, I can suggest two possible solutions to alleviate the problem. One is to do split file backups to cause SQL Backup to run in multiple threads and thereby spread the load across multiple processord. You can do this in script by specifying more than one file for a backup:

BACKUP DATABASE [Northwind] TO DISK='c:\backup\<AUTO>_1.sqb', DISK='c:\backup\<AUTO>_2.sqb'

You could also try COMPRESSION=2. Compression levels 1 and 2 are much quicker than 3.

devjvc

More times than not, the log backup fails. These log files are usually < 100MB. I'd rather not split them up if at all possible. Is there some kind of "default" timeout that's being hit? Anyway to increase the time that the api waits?

Brian Donahue

As far as I know, that bit is hard-coded.

joshmurrah

I wanted to input that this is the primary failure/problem that I have with Red Gate SQL Backup, are the VDI 1010 errors. It doesn't take much load to cause this failure.

Is the next major revision going to have this timeout adjustable?

@CPU_Busy, but I'd rather have Red Gate wait longer for the API.

Overall, thanks for the great product!

Brian Donahue

Hello Josh,

Have you got any software patches from us? In particular, I believe we have a patch that includes an update that does a retry when it fails to create a virtual device instance.

joshmurrah

I have gotten a few patches, but I don't think I've gotten a patch that includes that... Can you forward that on to me? jmurrah AT gtl.net

Kiowa

Im also getting this intermittant error on large files. Could you please send me the latest patch for 3.2.0

kiowa.apache@gmail.com

Brian Donahue

Hi,

There is a patch available for SQL Backup 3.2 which will hopefully solve the problem.

If I remember correctly, you must download

ftp://ftp.red-gate.com/old/SQL_Backup_R ... .2.0.5.zip

...and copy these files to the SQL Backup installation folder. Then there is an update to the extended stored procedure:

ftp://ftp.red-gate.com/old/SQL_Backup_R ... 2.0.10.zip

...this goes into the BINN folder of the SQL Server. You will want to run the query

DBCC xp_sqlbackup (FREE) to flush out the old xproc from memory and load the new one from disk.

Kiowa

Thankyou - will give it a try

Kiowa

Just a little stuck with the installation package. They all look like they need to go in the Redgate installation directory and not the MSSQL binn directory?

Brian Donahue

Yes -- only xp_sqlbackup.dll needs to go into BINN.