SqlBackup stopped restoring LogShipping jobs
vad2319
Posts: 40
Hello,
The SqlBackup 5 suddenly stopped restoring LogShipping jobs. The backups are being created on the back up server and are being transferred to the shared folder. But the restore server is not able to restore the jobs. It is stuck on a job that has failed to restore and gives the following error message.
Please advise what should I do next. Thanks!
***********************Begin****************************
Thread 0 error:
EOF#2 validation failed. Device 0. Expected 71, available 24.
Thread 1 error:
EOF#2 validation failed. Device 0. Expected 71, available 24.
Thread 2 error:
EOF#2 validation failed. Device 0. Expected 71, available 24.
SQL error 3013: RESTORE LOG is terminating abnormally.
SQL error 3203: Read on 'SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF01' failed, status = 995. See the SQL Server error log for more details.
SQL error 3738: Deleting database file 'D:\Program Files\Microsoft SQL Server\MSSQL\BACKUP\CTSY2K_TEST4\UNDO_CTSY2K_TEST4.dat'.
**************************End******************************
The Sql Server error log has the following information(This is just a subset of the Sql server error log):
Begin
2008-10-06 14:48:46.07 spid54 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 14:48:48.98 spid54 Recovery is checkpointing database 'CTSY2K_TEST4' (11)
2008-10-06 14:48:49.02 backup Log restored: Database: CTSY2K_TEST4, creation date(time): 2007/01/22(07:40:44), first LSN: 3298:288198:1, last LSN: 3298:378699:1, number of dump devices: 3, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {'SQLBACKUP_E0529899-AD02-4F52-822B-B1CCACD920EC', 'SQLBACKUP_E0529899-AD02-4F52-822B-B1CCACD920EC01', 'SQLBACKUP_E0529899-AD02-4F52-822B-B1CCACD920EC02'}).
2008-10-06 14:48:52.58 spid54 Starting up database 'CTSY2K_TEST4'.
2008-10-06 14:48:52.58 spid54 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 14:49:47.71 spid1 SQL Server has encountered 1 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [D:\Program Files\Microsoft SQL Server\MSSQL\Data\CTSY2K_TEST4\CTSY2K_TEST4.mdf] in database [CTSY2K_TEST4] (11). The OS file handle is 0x00000974. The offset of the latest long IO is: 0x0000024b76a000
2008-10-06 14:49:47.71 spid1 SQL Server has encountered 1 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [C:\Program Files\Microsoft SQL Server\MSSQL\data\tempdb.mdf] in database [tempdb] (2). The OS file handle is 0x00000518. The offset of the latest long IO is: 0x00000000620000
2008-10-06 14:51:41.15 spid54 Starting up database 'CTSY2K_TEST4'.
2008-10-06 14:51:41.15 spid54 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 14:51:42.73 spid54 Recovery is checkpointing database 'CTSY2K_TEST4' (11)
2008-10-06 14:51:42.76 backup Log restored: Database: CTSY2K_TEST4, creation date(time): 2007/01/22(07:40:44), first LSN: 3298:378699:1, last LSN: 3299:79030:1, number of dump devices: 3, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {'SQLBACKUP_046A6F78-3359-4F04-A87F-59C47616EAD7', 'SQLBACKUP_046A6F78-3359-4F04-A87F-59C47616EAD701', 'SQLBACKUP_046A6F78-3359-4F04-A87F-59C47616EAD702'}).
2008-10-06 14:51:42.76 spid54 SQL Server has encountered 1 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [C:\Program Files\Microsoft SQL Server\MSSQL\data\msdbdata.mdf] in database [msdb] (4). The OS file handle is 0x000004B0. The offset of the latest long IO is: 0x00000002084000
2008-10-06 14:51:47.49 spid54 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF01'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
2008-10-06 14:51:47.49 spid54 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF02'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
2008-10-06 14:51:47.49 spid54 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
2008-10-06 14:51:47.59 spid54 Internal I/O request 0x1AE3D440: Op: Read, pBuffer: 0x068D0000, Size: 65536, Position: 17235968, UMS: Internal: 0x140004, InternalHigh: 0x220019, Offset: 0x55470026, OffsetHigh: 0x4E495454, m_buf: 0x00163331, m_len: 70647808, m_actualBytes: 0, m_errcode: 995, BackupFile: SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF01
2008-10-06 14:51:47.59 spid54 Internal I/O request 0x1AE3CF90: Op: Read, pBuffer: 0x06890000, Size: 65536, Position: 17301504, UMS: Internal: 0x40000, InternalHigh: 0x1F0016, Offset: 0x2C0028, OffsetHigh: 0x54545547, m_buf: 0x34353345, m_len: 808465458, m_actualBytes: 0, m_errcode: 995, BackupFile: SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF02
2008-10-06 14:51:47.59 spid54 Internal I/O request 0x1AE3CE00: Op: Read, pBuffer: 0x06880000, Size: 65536, Position: 17301504, UMS: Internal: 0x424E414D, InternalHigh: 0x59545445, Offset: 0x37313735, OffsetHigh: 0x38313536, m_buf: 0x26001D00, m_len: 1191193088, m_actualBytes: 0, m_errcode: 995, BackupFile: SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF
2008-10-06 14:53:24.18 spid55 Starting up database 'CTSY2K_TEST4'.
2008-10-06 14:53:24.19 spid55 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 14:53:25.33 spid55 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_1DC5689D-4A6B-4C54-A2BA-81F5330FC07C02'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
2008-10-06 14:53:25.33 spid55 Internal I/O request 0x2EAFD2B0: Op: Read, pBuffer: 0x068C0000, Size: 65536, Position: 17301504, UMS: Internal: 0x45004D, InternalHigh: 0x42004D, Offset: 0x520045, OffsetHigh: 0x0, m_buf: 0x006D0064, m_len: 7209065, m_actualBytes: 0, m_errcode: 995, BackupFile: SQLBACKUP_1DC5689D-4A6B-4C54-A2BA-81F5330FC07C02
2008-10-06 15:03:23.89 spid57 Starting up database 'CTSY2K_TEST4'.
2008-10-06 15:03:23.90 spid57 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 15:03:24.40 spid57 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_C38CDA46-5DE4-4458-B1A3-21E00DB68A4401'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
End
The SqlBackup 5 suddenly stopped restoring LogShipping jobs. The backups are being created on the back up server and are being transferred to the shared folder. But the restore server is not able to restore the jobs. It is stuck on a job that has failed to restore and gives the following error message.
Please advise what should I do next. Thanks!
***********************Begin****************************
Thread 0 error:
EOF#2 validation failed. Device 0. Expected 71, available 24.
Thread 1 error:
EOF#2 validation failed. Device 0. Expected 71, available 24.
Thread 2 error:
EOF#2 validation failed. Device 0. Expected 71, available 24.
SQL error 3013: RESTORE LOG is terminating abnormally.
SQL error 3203: Read on 'SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF01' failed, status = 995. See the SQL Server error log for more details.
SQL error 3738: Deleting database file 'D:\Program Files\Microsoft SQL Server\MSSQL\BACKUP\CTSY2K_TEST4\UNDO_CTSY2K_TEST4.dat'.
**************************End******************************
The Sql Server error log has the following information(This is just a subset of the Sql server error log):
Begin
2008-10-06 14:48:46.07 spid54 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 14:48:48.98 spid54 Recovery is checkpointing database 'CTSY2K_TEST4' (11)
2008-10-06 14:48:49.02 backup Log restored: Database: CTSY2K_TEST4, creation date(time): 2007/01/22(07:40:44), first LSN: 3298:288198:1, last LSN: 3298:378699:1, number of dump devices: 3, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {'SQLBACKUP_E0529899-AD02-4F52-822B-B1CCACD920EC', 'SQLBACKUP_E0529899-AD02-4F52-822B-B1CCACD920EC01', 'SQLBACKUP_E0529899-AD02-4F52-822B-B1CCACD920EC02'}).
2008-10-06 14:48:52.58 spid54 Starting up database 'CTSY2K_TEST4'.
2008-10-06 14:48:52.58 spid54 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 14:49:47.71 spid1 SQL Server has encountered 1 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [D:\Program Files\Microsoft SQL Server\MSSQL\Data\CTSY2K_TEST4\CTSY2K_TEST4.mdf] in database [CTSY2K_TEST4] (11). The OS file handle is 0x00000974. The offset of the latest long IO is: 0x0000024b76a000
2008-10-06 14:49:47.71 spid1 SQL Server has encountered 1 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [C:\Program Files\Microsoft SQL Server\MSSQL\data\tempdb.mdf] in database [tempdb] (2). The OS file handle is 0x00000518. The offset of the latest long IO is: 0x00000000620000
2008-10-06 14:51:41.15 spid54 Starting up database 'CTSY2K_TEST4'.
2008-10-06 14:51:41.15 spid54 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 14:51:42.73 spid54 Recovery is checkpointing database 'CTSY2K_TEST4' (11)
2008-10-06 14:51:42.76 backup Log restored: Database: CTSY2K_TEST4, creation date(time): 2007/01/22(07:40:44), first LSN: 3298:378699:1, last LSN: 3299:79030:1, number of dump devices: 3, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {'SQLBACKUP_046A6F78-3359-4F04-A87F-59C47616EAD7', 'SQLBACKUP_046A6F78-3359-4F04-A87F-59C47616EAD701', 'SQLBACKUP_046A6F78-3359-4F04-A87F-59C47616EAD702'}).
2008-10-06 14:51:42.76 spid54 SQL Server has encountered 1 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [C:\Program Files\Microsoft SQL Server\MSSQL\data\msdbdata.mdf] in database [msdb] (4). The OS file handle is 0x000004B0. The offset of the latest long IO is: 0x00000002084000
2008-10-06 14:51:47.49 spid54 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF01'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
2008-10-06 14:51:47.49 spid54 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF02'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
2008-10-06 14:51:47.49 spid54 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
2008-10-06 14:51:47.59 spid54 Internal I/O request 0x1AE3D440: Op: Read, pBuffer: 0x068D0000, Size: 65536, Position: 17235968, UMS: Internal: 0x140004, InternalHigh: 0x220019, Offset: 0x55470026, OffsetHigh: 0x4E495454, m_buf: 0x00163331, m_len: 70647808, m_actualBytes: 0, m_errcode: 995, BackupFile: SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF01
2008-10-06 14:51:47.59 spid54 Internal I/O request 0x1AE3CF90: Op: Read, pBuffer: 0x06890000, Size: 65536, Position: 17301504, UMS: Internal: 0x40000, InternalHigh: 0x1F0016, Offset: 0x2C0028, OffsetHigh: 0x54545547, m_buf: 0x34353345, m_len: 808465458, m_actualBytes: 0, m_errcode: 995, BackupFile: SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF02
2008-10-06 14:51:47.59 spid54 Internal I/O request 0x1AE3CE00: Op: Read, pBuffer: 0x06880000, Size: 65536, Position: 17301504, UMS: Internal: 0x424E414D, InternalHigh: 0x59545445, Offset: 0x37313735, OffsetHigh: 0x38313536, m_buf: 0x26001D00, m_len: 1191193088, m_actualBytes: 0, m_errcode: 995, BackupFile: SQLBACKUP_7DD63A86-D656-42E7-AD98-CDEA155945AF
2008-10-06 14:53:24.18 spid55 Starting up database 'CTSY2K_TEST4'.
2008-10-06 14:53:24.19 spid55 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 14:53:25.33 spid55 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_1DC5689D-4A6B-4C54-A2BA-81F5330FC07C02'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
2008-10-06 14:53:25.33 spid55 Internal I/O request 0x2EAFD2B0: Op: Read, pBuffer: 0x068C0000, Size: 65536, Position: 17301504, UMS: Internal: 0x45004D, InternalHigh: 0x42004D, Offset: 0x520045, OffsetHigh: 0x0, m_buf: 0x006D0064, m_len: 7209065, m_actualBytes: 0, m_errcode: 995, BackupFile: SQLBACKUP_1DC5689D-4A6B-4C54-A2BA-81F5330FC07C02
2008-10-06 15:03:23.89 spid57 Starting up database 'CTSY2K_TEST4'.
2008-10-06 15:03:23.90 spid57 Bypassing recovery for database 'CTSY2K_TEST4' because it is marked IN LOAD.
2008-10-06 15:03:24.40 spid57 BackupMedium::ReportIoError: read failure on backup device 'SQLBACKUP_C38CDA46-5DE4-4458-B1A3-21E00DB68A4401'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
End
Comments
Could you please copy the file again from the primary server? To know which file to copy, look in the SQL Backup log file for the last restore process, for the name of the file. By default, the log files are stored in C:\Documents and Settings\All Users\Application Data\Red Gate\SQL Backup\Log\<instance name>.
SQL Backup Consultant Developer
Associate, Yohz Software
Beyond compression - SQL Backup goodies under the hood, updated for version 8