Error: Timeout when trying to deploy

lee5i3

I have a project that has 14 steps (3 websites and rest are WCF web services)..

Deployment Manager is setup on a local box internally.. Agent is running on a Windows Azure VM

I did a health check and it passed with no problems.
It fails Stage 2 on "Upload packages to target machines"

Error says

The request channel timed out while waiting for a reply after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

Error Details:
2013-07-08 16:47:35 INFO Preparing to upload packages
2013-07-08 16:47:35 INFO Uploading packages...
2013-07-08 16:51:24 ERROR System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The request operation did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout.
at System.ServiceModel.Channels.ReliableRequestSessionChannel.SyncRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
--- End of inner exception stack trace ---

Server stack trace:
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at RedGate.Deploy.Shared.Contracts.IPackageService.UploadChunk(Byte[] data)
at CallSite.Target(Closure , CallSite , Object , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid2[T0,T1](CallSite site, T0 arg0, T1 arg1)
at CallSite.Target(Closure , CallSite , Object , Object )
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberAction(CallSite& callsite, Type binderType, LazyBinder binder, String_OR_InvokeMemberName name, Boolean staticContext, Type context, String[] argNames, Object target, Object[] args)
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberActionCallSite(Object target, String_OR_InvokeMemberName name, Object[] args, String[] tArgNames, Type tContext, Boolean tStaticContext, CallSite& callSite)
at ImpromptuInterface.Impromptu.InvokeMemberAction(Object target, String_OR_InvokeMemberName name, Object[] args)
at ImpromptuInterface.Dynamic.ImpromptuForwarder.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at RedGate.Deploy.Server.Proxies.WcfSecurityWrapper`1.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at CallSite.Target(Closure , CallSite , Object , Byte[] )
at ActLike_IPackageService_2d858676b1eb41aa9c0a21844e982fb8.UploadChunk(Byte[] data)
at RedGate.Deploy.Server.Tasks.Deploy.PackagePusher.UploadFile(IPackageService service, String fullPath, ILog log)
at RedGate.Deploy.Server.Tasks.Deploy.PackagePusher.Push(DownloadedPackage package, MachineEndpoint destination, ILog log)
at RedGate.Deploy.Server.Tasks.Deploy.UploadPackageActivity.<Execute>b__0()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()

Exception rethrown at [1]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at RedGate.Deploy.Server.Tasks.ActivityRuntime.<ExecuteChild>d__0.MoveNext()
2013-07-08 16:51:44 ERROR System.TimeoutException: The operation did not complete within the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.

Server stack trace:
at System.ServiceModel.Channels.InterruptibleWaitObject.Wait(TimeSpan timeout, Boolean throwTimeoutException)
at System.ServiceModel.Channels.ReliableOutputConnection.Close(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableRequestSessionChannel.OnClose(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnClose(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
at System.ServiceModel.Dispatcher.TerminatingOperationBehavior.AfterReply(ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at RedGate.Deploy.Shared.Contracts.IPackageService.FinalizeUpload()
at CallSite.Target(Closure , CallSite , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid1[T0](CallSite site, T0 arg0)
at CallSite.Target(Closure , CallSite , Object )
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberAction(CallSite& callsite, Type binderType, LazyBinder binder, String_OR_InvokeMemberName name, Boolean staticContext, Type context, String[] argNames, Object target, Object[] args)
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberActionCallSite(Object target, String_OR_InvokeMemberName name, Object[] args, String[] tArgNames, Type tContext, Boolean tStaticContext, CallSite& callSite)
at ImpromptuInterface.Impromptu.InvokeMemberAction(Object target, String_OR_InvokeMemberName name, Object[] args)
at ImpromptuInterface.Dynamic.ImpromptuForwarder.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at RedGate.Deploy.Server.Proxies.WcfSecurityWrapper`1.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at CallSite.Target(Closure , CallSite , Object )
at ActLike_IPackageService_2d858676b1eb41aa9c0a21844e982fb8.FinalizeUpload()
at RedGate.Deploy.Server.Tasks.Deploy.PackagePusher.UploadFile(IPackageService service, String fullPath, ILog log)
at RedGate.Deploy.Server.Tasks.Deploy.PackagePusher.Push(DownloadedPackage package, MachineEndpoint destination, ILog log)
at RedGate.Deploy.Server.Tasks.Deploy.UploadPackageActivity.<Execute>b__0()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()

Exception rethrown at [1]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at RedGate.Deploy.Server.Tasks.ActivityRuntime.<ExecuteChild>d__0.MoveNext()
2013-07-08 16:51:49 ERROR System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The request operation did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout.
at System.ServiceModel.Channels.ReliableRequestSessionChannel.SyncRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
--- End of inner exception stack trace ---

Server stack trace:
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at RedGate.Deploy.Shared.Contracts.IPackageService.FinalizeUpload()
at CallSite.Target(Closure , CallSite , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid1[T0](CallSite site, T0 arg0)
at CallSite.Target(Closure , CallSite , Object )
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberAction(CallSite& callsite, Type binderType, LazyBinder binder, String_OR_InvokeMemberName name, Boolean staticContext, Type context, String[] argNames, Object target, Object[] args)
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberActionCallSite(Object target, String_OR_InvokeMemberName name, Object[] args, String[] tArgNames, Type tContext, Boolean tStaticContext, CallSite& callSite)
at ImpromptuInterface.Impromptu.InvokeMemberAction(Object target, String_OR_InvokeMemberName name, Object[] args)
at ImpromptuInterface.Dynamic.ImpromptuForwarder.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at RedGate.Deploy.Server.Proxies.WcfSecurityWrapper`1.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at CallSite.Target(Closure , CallSite , Object )
at ActLike_IPackageService_2d858676b1eb41aa9c0a21844e982fb8.FinalizeUpload()
at RedGate.Deploy.Server.Tasks.Deploy.PackagePusher.UploadFile(IPackageService service, String fullPath, ILog log)
at RedGate.Deploy.Server.Tasks.Deploy.PackagePusher.Push(DownloadedPackage package, MachineEndpoint destination, ILog log)
at RedGate.Deploy.Server.Tasks.Deploy.UploadPackageActivity.<Execute>b__0()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()

Exception rethrown at [1]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at RedGate.Deploy.Server.Tasks.ActivityRuntime.<ExecuteChild>d__0.MoveNext()
2013-07-08 16:51:52 ERROR System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The request operation did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout.
at System.ServiceModel.Channels.ReliableRequestSessionChannel.SyncRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
--- End of inner exception stack trace ---

Server stack trace:
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at RedGate.Deploy.Shared.Contracts.IPackageService.FinalizeUpload()
at CallSite.Target(Closure , CallSite , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid1[T0](CallSite site, T0 arg0)
at CallSite.Target(Closure , CallSite , Object )
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberAction(CallSite& callsite, Type binderType, LazyBinder binder, String_OR_InvokeMemberName name, Boolean staticContext, Type context, String[] argNames, Object target, Object[] args)
at ImpromptuInterface.Optimization.InvokeHelper.InvokeMemberActionCallSite(Object target, String_OR_InvokeMemberName name, Object[] args, String[] tArgNames, Type tContext, Boolean tStaticContext, CallSite& callSite)
at ImpromptuInterface.Impromptu.InvokeMemberAction(Object target, String_OR_InvokeMemberName name, Object[] args)
at ImpromptuInterface.Dynamic.ImpromptuForwarder.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at RedGate.Deploy.Server.Proxies.WcfSecurityWrapper`1.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at CallSite.Target(Closure , CallSite , Object )
at ActLike_IPackageService_2d858676b1eb41aa9c0a21844e982fb8.FinalizeUpload()
at RedGate.Deploy.Server.Tasks.Deploy.PackagePusher.UploadFile(IPackageService service, String fullPath, ILog log)
at RedGate.Deploy.Server.Tasks.Deploy.PackagePusher.Push(DownloadedPackage package, MachineEndpoint destination, ILog log)
at RedGate.Deploy.Server.Tasks.Deploy.UploadPackageActivity.<Execute>b__0()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()

Exception rethrown at [1]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at RedGate.Deploy.Server.Tasks.ActivityRuntime.<ExecuteChild>d__0.MoveNext()

lee5i3

I got it to work..

Had to keep trying a few times until it did not time out..

Anyway to increase the timeout so it doesn't happen again?

chirayu

Hi

Thanks for reporting this issue. We have entered a bug in our system (RGD-765) and are looking at this right now.

I will update you once we have decided on the fix for this.

Thanks!

Chirayu

lee5i3

Thanks,

After trying to redeploy again, I am unable to deploy.
I tried half a dozen attempts, they all failed trying to upload packages.

I even tried unselected all except 1, but even though they are unselected, it still attempts to upload all packages.

Thankfully, our deployment schedule is every 3 weeks, so hopefully there will be a solution by then or I will need to deploy manually.

Emma A

This has been captured in an internal issue, bug number RGD-766 and has been addressed so should be available in the next release.
Thank you for the feedback.

lee5i3

Great!!

For now, I had to split up the services into multiple deployment manager projects, havent tried it yet, but hopefully if release isnt out before my next sprint, this should work..

Mike U

We've added a workaround for this problem (RGD-765) in the latest release (2.1.2.21). You can now configure the timeouts used in communications with a Deployment Agent. You need to edit the appSettings section of the Deployment Manager Server config file, C:\Program Files (x86)\Red Gate\Deployment Manager\Server\RedGate.Deploy.Server.exe.config.

Add a new line inside the <appSettings> element that reads:

<add key="Bindings.AgentSendTimeout" value="00:20:00" />

This will set the send timeout to 20 minutes, rather than the default 1 minute. Please note that this will also increase the time it takes for an Agent health check to fail if the Agent cannot be communicated with, which is why we don't use a very long timeout by default.

By the way, we've also fixed the bug where skipped steps would still have their packages sent to the agent, even though they weren't going to be deployed (bug reference RGD-766)

Hope that helps. Please let us know if you're still seeing timeout problems after changing that value.

lee5i3

So far so good, successfully deployed 15 steps.. however that was localhost, real test will be on my next sprint release which is scheduled on the 29th..

I'll let you know how that goes