Intermittent Timeouts while creating masked image
jamesgally
Posts: 3 New member
We use SQL Clone and Data Masker for generating our development databases on a weekly schedule
Our masking set has some range rules that occasionally timeout and cause the image creation to fail.
I've already reduced the commit frequency down to 25,000 records instead of the default 100k
Some weeks it'll work fine and others it might timeout once or twice before eventually creating which requires manual intervention over the weekend which is less than ideal and becoming unsustainable
Clone version 5.4.21.6541
Data Masker version 7.1.40.8606
Our masking set has some range rules that occasionally timeout and cause the image creation to fail.
I've already reduced the commit frequency down to 25,000 records instead of the default 100k
Some weeks it'll work fine and others it might timeout once or twice before eventually creating which requires manual intervention over the weekend which is less than ideal and becoming unsustainable
Clone version 5.4.21.6541
Data Masker version 7.1.40.8606
2024-03-17 04:45:06 [Warning] RuleWorker: HandleRuleWorkerLoopException: SQL Server exception happened. Checking for ERRTRAP {RuleId="20-0009-10", WorkerId="8", RunId="tmpC4EC::202403170422423538955"} 2024-03-17 04:45:06 [Error] RuleWorker: HandleRuleWorkerLoopException: ENDING on SQL Server error {RuleId="20-0009-10", WorkerId="8", RunId="tmpC4EC::202403170422423538955"} System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Operation cancelled by user. ---> System.ComponentModel.Win32Exception (258): The wait operation timed out. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest) at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest) at System.Data.SqlClient.SqlInternalTransaction.Commit() at System.Data.SqlClient.SqlTransaction.Commit() at System.Data.SqlClient.SqlBulkCopy.CommitTransaction() at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinuedOnSuccess(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source) at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinued(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source) at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsync(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source) at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestContinuedAsync(BulkCopySimpleResultSet internalResults, CancellationToken cts, TaskCompletionSource`1 source) at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource`1 source) at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken) at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken) at System.Data.SqlClient.SqlBulkCopy.WriteToServer(DataTable table, DataRowState rowState) at RedGate.DataMasking.SqlServer.BulkAlgorithm.Insertion.WriteToServer(IEnumerable`1 rows, SqlDataAdapter dataAdapter, DataTable dataTable, SqlBulkCopy bulkCopy) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\Insertion.cs:line 64 at RedGate.DataMasking.SqlServer.BulkAlgorithm.Insertion.<>c__DisplayClass4_0.<BulkInsert>b__0() in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\Insertion.cs:line 44 at System.Threading.Tasks.Task`1.InnerInvoke() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at RedGate.DataMasking.SqlServer.BulkAlgorithm.Insertion.BulkInsert(TempTable tempTable, IEnumerable`1 rows, IRuleProgress ruleProgress, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\Insertion.cs:line 47 at RedGate.DataMasking.SqlServer.BulkAlgorithm.Insertion.Insert(TempTable tempTable, IEnumerable`1 rows, IRuleProgress ruleProgress, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\Insertion.cs:line 27 at RedGate.DataMasking.SqlServer.BulkAlgorithm.BulkSubstitutionRule.Substitute(Table table, RowIdentifier rowIdentifier, IReadOnlyList`1 columnsToMask, String connectionString, MaskTuneOptions options, IRuleProgress ruleProgress, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\BulkSubstitutionRule.cs:line 94 at RedGate.DataMasking.SqlServer.RuleExecutors.SubstitutionRuleExecutor.ProcessSubstitutionRuleBulk(SqlConnection connection, DMSRule_Controller controller, DMSRule_Substitution rule, Int32 workerId, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleExecutors\SubstitutionRuleExecutor.cs:line 830 at RedGate.DataMasking.SqlServer.RuleExecutors.SubstitutionRuleExecutor.Execute(IController controller, DMSRule_Substitution rule, Int32 workerId, IRuleProgress ruleProgress, RuleProcessedEventBuilder processingEventBuilder, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleExecutors\SubstitutionRuleExecutor.cs:line 78 at RedGate.DataMasking.SqlServer.RuleWorker.RunTheRule(DMSRule ruleToRun) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleWorker.cs:line 751 at RedGate.DataMasking.SqlServer.RuleWorker.<ExecuteRule>b__72_0() in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleWorker.cs:line 613 at System.Threading.Tasks.Task`1.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at RedGate.DataMasking.SqlServer.RuleWorker.ExecuteRule() in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleWorker.cs:line 631 at RedGate.DataMasking.SqlServer.RuleWorker.RunLoadedRule() in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleWorker.cs:line 291
Tagged:
Comments
See excerpt of the log as requested with sensitive data redacted
Thanks
James
Please see excerpt of the log as requested with sensitive data redacted
James
A Redgate support ticket has been created for you, the ticket reference is #309451.
I will contact you via the Support Ticket, to request further information.
Many Thanks
Eddie
Senior Product Support Engineer
Redgate Software Ltd
Email: support@red-gate.com