Backup Copy to S3 Loses Access to Bucket

jcricegp

I am currently copying SQL Backups to S3 Storage using build 9.1.4.722. I have linked the Bucket using the interface and successfully tested the link. 

I have been experiencing an issue where the copy to S3 will get an 'Access Denied' error when attempting to copy the file. When the interface shows that the file copy error, I check the link in 'Service Options' and it shows that the Bucket is still linked. If I re-enter my Access, Secret Keys, and Bucket Name, the copy begins and is successful. This has been a problem for several of my Amazon EC2 instances. I have to check each instance to see if the error is occurring. Is there an alarm for this problem? 

I have upgraded this instance and several to 9.2.1.128 (Latest Build), in case the 9.1 build is the problem. 

I am including the log file of the error. 

Thanks, John R.

2018-04-10 05:20:12,326 [1] INFO  Program - SQL Backup upload client 9.1.4.722

2018-04-10 05:20:12,582 [1] INFO  Program - Copyright ©  Red Gate Software Ltd

2018-04-10 05:20:12,660 [1] INFO  Program - UseSignatureVersion4 was 'True'

2018-04-10 05:20:12,669 [1] INFO  Program - UseSignatureVersion4 is now 'True'

2018-04-10 05:20:12,707 [1] INFO  AccountDetails - AWS Keys are 'NOT' set'

2018-04-10 05:20:12,816 [1] INFO  S3UploadClientFactory - Setting Region to Default

2018-04-10 05:20:12,832 [1] INFO  S3UploadClientFactory - Defaulting EndPoint to 'US East (Virginia)'

2018-04-10 05:20:12,895 [1] INFO  ReadWriteRegistry - Instance name does not exist: (local)

2018-04-10 05:20:13,426 [1] ERROR S3UploadClientFactory - Failed to get Bucket '<S3 Bucket Name>' location 'Access Denied'

Amazon.S3.AmazonS3Exception: Access Denied ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (403) Forbidden. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.

   at System.Net.HttpWebRequest.GetResponse()

   at Amazon.Runtime.Internal.HttpRequest.GetResponse()

   --- End of inner exception stack trace ---

   at Amazon.Runtime.Internal.HttpRequest.GetResponse()

   at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)

   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)

   --- End of inner exception stack trace ---

   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)

   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)

   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext)

   at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller)

   at Amazon.S3.AmazonS3Client.GetBucketLocation(GetBucketLocationRequest request)

   at RedGate.SQLBackup.CloudBackup.App.S3.S3UploadClientFactory.GetS3RegionForBucket(String accessKeyId, String secretAccessKeyId, String bucket, RegionEndpoint defaultEndpoint)

2018-04-10 05:20:13,504 [1] ERROR Program - ExitCode  AuthorizationError

RedGate.SQLBackup.CloudBackup.App.Utils.ExitCodeException: Amazon Authorization Error: Access Denied

   at RedGate.SQLBackup.CloudBackup.App.S3.S3UploadClientFactory.GetS3RegionForBucket(String accessKeyId, String secretAccessKeyId, String bucket, RegionEndpoint defaultEndpoint)

   at RedGate.SQLBackup.CloudBackup.App.S3.S3UploadClientFactory.GetRegionForBucket(String region, String accessKeyId, String secretAccessKeyId, String bucket)

   at RedGate.SQLBackup.CloudBackup.App.S3.S3UploadClientFactory.CreateUploadClient(String instanceName, String s3Key, String s3SecretKey, String region, String bucket)

   at RedGate.SQLBackup.CloudBackup.App.Commands.Upload.FileUploader.UploadFile(IUploadClientFactory s3ClientFactory, IProgressEventSinkFactory progressSinkFactory, IFileFactory fileFactory, IAccountDetails accountDetails, ICommandLineArguments args)

   at RedGate.SQLBackup.CloudBackup.App.Program.<>c__DisplayClass6_0.<UploadFile>b__0()

   at RedGate.SQLBackup.CloudBackup.App.Utils.RetryLogic.Retry(Action action, Func`2 exceptionFilter, Int32 maxTries, BackOffAlgorithm backOffAlgorithm)

   at RedGate.SQLBackup.CloudBackup.App.Program.UploadFile(CommandLineArguments commandLineArguments)

   at RedGate.SQLBackup.CloudBackup.App.Program.Main(String[] args)

Eddie D

Hi

I suspect that we may already have an open support ticket with you on this problem.

We are are unable to replicate your problem.

We use components are from the Amazon S3 SDK to connect to the S3 server.

The AWS keys are stored in the CredentialManager on the relevant db server (under whichever user the SQB server components uses), under key "Redgate.SqlBackup.S3.{InstanceName}.{BucketName}". That may not be helpful if you are running the server components as Local System, but if you are using a loggable-in account then you could check them. 

You can view the Credentials Manager via Control Panel ->User Accounts ->Credentials Manager

Credentials in the Manager don't expire, so is it possible some other process is clearing them out occasionally?

Many Thanks
Eddie