AWS - Working around 10000 part limit for SQLBackup uploads
ArnoldLieberman
Posts: 3 New member
in SQL Backup
We have just starting using S3 for our backups and have hit a situation where large files are failing to copy to S3.
Is there any way of adjusting the size of each "part" so that fewer parts are needed?
Using:
Windows Server 2008R2
SQL Server 2008R2 10.50.4339
SQL Backup 9.2.6.543
This is the (trimmed) error log:
2018-11-19 08:12:37,530 [14] INFO S3Client - Transferred 5242880/5242880...
2018-11-19 08:12:37,704 [17] ERROR S3Client - Error uploading file. Aborting upload...
2018-11-19 08:12:39,557 [16] ERROR S3Client - Error uploading file. Aborting upload... DONE
2018-11-19 08:12:39,561 [16] ERROR S3Client - An AmazonS3Exception was thrown: Part number must be an integer between 1 and 10000, inclusive
Amazon.S3.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)
at System.Net.WebRequest.<GetResponseAsync>b__78_1()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
...
2018-11-19 08:12:39,570 [16] INFO S3Client - Type = Amazon.S3.AmazonS3Exception
2018-11-19 08:12:39,572 [16] INFO S3Client - AmazonId2 = XMuBiNN8JKBMQAlp3onzTMH+sXpHS1ARR49z9NRFvQAm7VFor+1fykiBNV1Xzuk3hAO6f2o7h+M=
2018-11-19 08:12:39,573 [16] INFO S3Client - Message = Part number must be an integer between 1 and 10000, inclusive
2018-11-19 08:12:39,576 [16] INFO S3Client - ErrorType = Sender
2018-11-19 08:12:39,577 [16] INFO S3Client - ErrorCode = InvalidArgument
2018-11-19 08:12:39,579 [16] INFO S3Client - StatusCode = BadRequest
2018-11-19 08:12:39,588 [1] FATAL UploadClient - Upload failed. The backup file was not copied to hosted storage: FULL_(local)_xxxxxxxxxxx_20181115_094302.sqb
2018-11-19 08:12:39,595 [1] FATAL UploadClient - System.AggregateException: One or more errors occurred. ---> Amazon.S3.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)
at System.Net.WebRequest.<GetResponseAsync>b__78_1()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
...
2018-11-19 08:12:39,619 [1] ERROR Program - ExitCode UploadFailed
RedGate.SQLBackup.CloudBackup.App.Utils.ExitCodeException: Upload failed. The backup file was not copied to hosted storage: One or more errors occurred.
at RedGate.SQLBackup.CloudBackup.App.Upload.UploadClient.InternalUpload(String bucket, String uploadFolder, IDictionary`2 metadata, IDictionary`2 tags, IBackupFile backupFile, String existingUploadId, StorageClass storageClass)
at RedGate.SQLBackup.CloudBackup.App.Upload.UploadClient.Upload(IBackupFile backupFile, String s3Folder, String bucket, IDictionary`2 metadata, IDictionary`2 tags, String existingUploadId, StorageClass storageClass)
at RedGate.SQLBackup.CloudBackup.App.Upload.FileUploader.UploadFile(ICloudClientFactory cloudClientFactory, IFileFactory fileFactory, IAccountDetails accountDetails, ICommandLineArguments args)
at RedGate.SQLBackup.CloudBackup.App.Program.<>c__DisplayClass7_0.<UploadFile>b__0()
at RedGate.SQLBackup.CloudBackup.App.Utils.RetryLogic.Retry(Action action, Func`2 exceptionFilter, Int32 maxTries, BackOffAlgorithm backOffAlgorithm)
Is there any way of adjusting the size of each "part" so that fewer parts are needed?
Using:
Windows Server 2008R2
SQL Server 2008R2 10.50.4339
SQL Backup 9.2.6.543
This is the (trimmed) error log:
2018-11-19 08:12:37,530 [14] INFO S3Client - Transferred 5242880/5242880...
2018-11-19 08:12:37,704 [17] ERROR S3Client - Error uploading file. Aborting upload...
2018-11-19 08:12:39,557 [16] ERROR S3Client - Error uploading file. Aborting upload... DONE
2018-11-19 08:12:39,561 [16] ERROR S3Client - An AmazonS3Exception was thrown: Part number must be an integer between 1 and 10000, inclusive
Amazon.S3.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)
at System.Net.WebRequest.<GetResponseAsync>b__78_1()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
...
2018-11-19 08:12:39,570 [16] INFO S3Client - Type = Amazon.S3.AmazonS3Exception
2018-11-19 08:12:39,572 [16] INFO S3Client - AmazonId2 = XMuBiNN8JKBMQAlp3onzTMH+sXpHS1ARR49z9NRFvQAm7VFor+1fykiBNV1Xzuk3hAO6f2o7h+M=
2018-11-19 08:12:39,573 [16] INFO S3Client - Message = Part number must be an integer between 1 and 10000, inclusive
2018-11-19 08:12:39,576 [16] INFO S3Client - ErrorType = Sender
2018-11-19 08:12:39,577 [16] INFO S3Client - ErrorCode = InvalidArgument
2018-11-19 08:12:39,579 [16] INFO S3Client - StatusCode = BadRequest
2018-11-19 08:12:39,588 [1] FATAL UploadClient - Upload failed. The backup file was not copied to hosted storage: FULL_(local)_xxxxxxxxxxx_20181115_094302.sqb
2018-11-19 08:12:39,595 [1] FATAL UploadClient - System.AggregateException: One or more errors occurred. ---> Amazon.S3.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)
at System.Net.WebRequest.<GetResponseAsync>b__78_1()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
...
2018-11-19 08:12:39,619 [1] ERROR Program - ExitCode UploadFailed
RedGate.SQLBackup.CloudBackup.App.Utils.ExitCodeException: Upload failed. The backup file was not copied to hosted storage: One or more errors occurred.
at RedGate.SQLBackup.CloudBackup.App.Upload.UploadClient.InternalUpload(String bucket, String uploadFolder, IDictionary`2 metadata, IDictionary`2 tags, IBackupFile backupFile, String existingUploadId, StorageClass storageClass)
at RedGate.SQLBackup.CloudBackup.App.Upload.UploadClient.Upload(IBackupFile backupFile, String s3Folder, String bucket, IDictionary`2 metadata, IDictionary`2 tags, String existingUploadId, StorageClass storageClass)
at RedGate.SQLBackup.CloudBackup.App.Upload.FileUploader.UploadFile(ICloudClientFactory cloudClientFactory, IFileFactory fileFactory, IAccountDetails accountDetails, ICommandLineArguments args)
at RedGate.SQLBackup.CloudBackup.App.Program.<>c__DisplayClass7_0.<UploadFile>b__0()
at RedGate.SQLBackup.CloudBackup.App.Utils.RetryLogic.Retry(Action action, Func`2 exceptionFilter, Int32 maxTries, BackOffAlgorithm backOffAlgorithm)
Tagged:
Comments