EsentWriteConflictException forcing RGDM restart?

isme

Hey RG,

We had to restart RGDM today to regain stability for our deployments.

RGDM has written thousands of EsentWriteConflictExceptions to the Windows application event log. They look like this:

2014-06-19 18:09:15,083 [80] WARN  Raven.Storage.Esent.StorageActions.DocumentStorageActions [(null)] - Could not update the reduce key counter for index UniqueTargetsByCreationDate, key: http://target.cloudcorp.com:10301/. Ignoring this, multi step reduce promotion may be delayed for this value.
Microsoft.Isam.Esent.Interop.EsentWriteConflictException: Write lock failed due to outstanding write lock
   at Microsoft.Isam.Esent.Interop.Api.EscrowUpdate(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid, Int32 delta)
   at Raven.Storage.Esent.StorageActions.DocumentStorageActions.ExecuteOnReduceKey(String view, String reduceKey, Table table, IDictionary`2 columnids, Action updateAction, Action insertAction)
   at Raven.Storage.Esent.StorageActions.DocumentStorageActions.IncrementReduceKeyCounter(String view, String reduceKey, Int32 val)

We also restarted IIS to reduce its memory consumption from 8GB before restart to 2GB after restart.

We suspect RGDM is preventing IIS from freeing memory in quiet periods.

The symptoms I encountered personally include messages like "The package could not be downloaded from NuGet.", "Deployment on the agent failed.", and "(500) Internal Server Error".

The messages were intermittent and unpredictable. Restarting RGDM seems to have stopped them.

Unplanned restarts are disruptive for our development teams.

What can we do to avoid EsentWriteConflictException besides restarting the server?

Thanks,
Iain

chirayu

The error is thrown by the database we are using (RavenDb) and it is probably happening when the indexes are being updated.

Unfortunately, there is no good workaround for the problem.

You mentioned:

What can we do to avoid EsentWriteConflictException besides restarting the server?

You should not need to restart the web server though, only the Red Gate Deployment Manager service needs to be restarted.