Intermittent Timeouts while creating masked image

jamesgally · March 22, 2024 9:02AM

We use SQL Clone and Data Masker for generating our development databases on a weekly schedule

Our masking set has some range rules that occasionally timeout and cause the image creation to fail.
I've already reduced the commit frequency down to 25,000 records instead of the default 100k

Some weeks it'll work fine and others it might timeout once or twice before eventually creating which requires manual intervention over the weekend which is less than ideal and becoming unsustainable

Clone version 5.4.21.6541
Data Masker version 7.1.40.8606

2024-03-17 04:45:06 [Warning] RuleWorker: HandleRuleWorkerLoopException: SQL Server exception happened. Checking for ERRTRAP {RuleId="20-0009-10", WorkerId="8", RunId="tmpC4EC::202403170422423538955"}
2024-03-17 04:45:06 [Error] RuleWorker: HandleRuleWorkerLoopException: ENDING on SQL Server error {RuleId="20-0009-10", WorkerId="8", RunId="tmpC4EC::202403170422423538955"}
System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Operation cancelled by user.
 ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalTransaction.Commit()
   at System.Data.SqlClient.SqlTransaction.Commit()
   at System.Data.SqlClient.SqlBulkCopy.CommitTransaction()
   at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinuedOnSuccess(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinued(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsync(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestContinuedAsync(BulkCopySimpleResultSet internalResults, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken)
   at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServer(DataTable table, DataRowState rowState)
   at RedGate.DataMasking.SqlServer.BulkAlgorithm.Insertion.WriteToServer(IEnumerable`1 rows, SqlDataAdapter dataAdapter, DataTable dataTable, SqlBulkCopy bulkCopy) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\Insertion.cs:line 64
   at RedGate.DataMasking.SqlServer.BulkAlgorithm.Insertion.<>c__DisplayClass4_0.<BulkInsert>b__0() in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\Insertion.cs:line 44
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at RedGate.DataMasking.SqlServer.BulkAlgorithm.Insertion.BulkInsert(TempTable tempTable, IEnumerable`1 rows, IRuleProgress ruleProgress, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\Insertion.cs:line 47
   at RedGate.DataMasking.SqlServer.BulkAlgorithm.Insertion.Insert(TempTable tempTable, IEnumerable`1 rows, IRuleProgress ruleProgress, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\Insertion.cs:line 27
   at RedGate.DataMasking.SqlServer.BulkAlgorithm.BulkSubstitutionRule.Substitute(Table table, RowIdentifier rowIdentifier, IReadOnlyList`1 columnsToMask, String connectionString, MaskTuneOptions options, IRuleProgress ruleProgress, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\BulkAlgorithm\BulkSubstitutionRule.cs:line 94
   at RedGate.DataMasking.SqlServer.RuleExecutors.SubstitutionRuleExecutor.ProcessSubstitutionRuleBulk(SqlConnection connection, DMSRule_Controller controller, DMSRule_Substitution rule, Int32 workerId, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleExecutors\SubstitutionRuleExecutor.cs:line 830
   at RedGate.DataMasking.SqlServer.RuleExecutors.SubstitutionRuleExecutor.Execute(IController controller, DMSRule_Substitution rule, Int32 workerId, IRuleProgress ruleProgress, RuleProcessedEventBuilder processingEventBuilder, CancellationToken cancellationToken) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleExecutors\SubstitutionRuleExecutor.cs:line 78
   at RedGate.DataMasking.SqlServer.RuleWorker.RunTheRule(DMSRule ruleToRun) in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleWorker.cs:line 751
   at RedGate.DataMasking.SqlServer.RuleWorker.<ExecuteRule>b__72_0() in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleWorker.cs:line 613
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at RedGate.DataMasking.SqlServer.RuleWorker.ExecuteRule() in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleWorker.cs:line 631
   at RedGate.DataMasking.SqlServer.RuleWorker.RunLoadedRule() in \BuildAgentB\work\3d88336879780dc0\DMV6\RedGate.DataMasking.SqlServer\RuleWorker.cs:line 291

TheMaskedData · March 22, 2024 3:27PM

@jamesgally are you able to get the masking statistics (table and rule stats) that should be part of the log files when this job runs? The split ranges sound like they're running suuuuuuuper slow so I'm wondering if you check the results how many rows/s are getting masked.

jamesgally · March 22, 2024 4:20PM

Hi TheMaskedData

See excerpt of the log as requested with sensitive data redacted

Thanks
James

Data Masker for SQL Server

Masking Set Run Rule Statistics Report

Date of report: Saturday, March 16, 2024 1:36 PM

Data Masker Version: 7.1.40.8606

Masking Set Name: tmpA0E3

Masking Set Full Path and Name: <<REDACTED>>

This report was auto-generated: True

Masking Statistics

MaskingSetState=MASKING RUN ERRORED

TotalRules=180

RulesNotRun=65

RulesRun=115

RunTime=17 min. 48 sec.

RowOperations=68010608

RowOperationsPerSecond=63680

ColOperations=254121175

ColOperationsPerSecond=237941

TotalTables=831

TablesWithMaskeddRows=9

TablesWithNonZeroSrcRows=769

TablesWithRowsWithNoMaskingOperations=760

TotalSourceRowCount=2299634687

RuleID RuleType Stage RunState ExecutionNote Row Operations Rows/Sec Col Operations Cols/Sec Runtime Activations

---------- -------------- ------- -------- ------------------- -------------- -------- -------------- -------- -------------- -----------

00-0002 TriggerMgr(Dis Stage 1 Complete Processing Complete 0 sec. 1

00-0017 IndexMgr(Dis) Stage 1 Complete Processing Complete 0 sec. 1

00-0018 Cmd Stage 1 Complete Processing Complete 0 sec. 1

00-0037 IndexRefresh Stage 1 Complete Processing Complete 1 sec. 1

00-0038 RowCountRefres Stage 1 Complete Processing Complete 1 min. 32 sec. 1

00-0041 Cmd Stage 1 Complete Processing Complete 0 sec. 1

00-0043 Cmd Stage 1 Complete Processing Complete 11 sec. 1

00-0045 Cmd Stage 1 Complete Processing Complete 0 sec. 1

01-0004 RSplit Stage 1 Complete Processing Complete 12098290 42154 96786320 337234 4 min. 47 sec. 1

01-0010 RSplit Stage 1 Complete Processing Complete 10827300 130449 32481900 391348 1 min. 23 sec. 1

01-0012 RSplit Stage 1 Complete Processing Complete 1495284 8447 11962272 67583 2 min. 57 sec. 1

01-0013 RSplit Stage 1 Complete Processing Complete 874229 25712 3496916 102850 34 sec. 1

02-0007 RSplit Stage 1 Complete Processing Complete 12098290 201638 12098290 201638 1 min. 0 sec. 1

03-0008 RowInternal Stage 1 Complete Processing Complete 12098290 80121 12098290 80121 2 min. 31 sec. 1

03-0020 RSplit Stage 1 Complete Processing Complete 2589600 184971 2589600 184971 14 sec. 1

03-0021 RSplit Stage 1 Complete Processing Complete 4656296 26915 4656296 26915 2 min. 53 sec. 1

03-0022 Substitution Stage 1 Complete Processing Complete 41003 1952 123009 5857 21 sec. 1

03-0047 TableToTable Stage 1 Complete Processing Complete 198975 19897 596925 59692 10 sec. 1

04-0009 RSplit Stage 1 Errored Errored 11033051 28289 77231357 198029 6 min. 30 sec. 1

05-0014 RSplit Stage 1 Not Run 0 0 0 0 0 sec. 0

06-0015 RSplit Stage 1 Not Run 0 0 0 0 0 sec. 0

07-0026 RowInternal Stage 1 Not Run 0 0 0 0 0 sec. 0

07-0027 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

07-0028 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

07-0029 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

08-0036 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

09-0039 TableToTable Stage 1 Not Run 0 0 0 0 0 sec. 0

10-0004-01 Rule Group Stage 1 n/a 0 sec. 0

10-0007-01 Rule Group Stage 1 n/a 0 sec. 0

10-0009-01 Rule Group Stage 1 n/a 0 sec. 0

10-0010-01 Rule Group Stage 1 n/a 0 sec. 0

10-0012-01 Rule Group Stage 1 n/a 0 sec. 0

10-0013-01 Rule Group Stage 1 n/a 0 sec. 0

10-0014-01 Rule Group Stage 1 n/a 0 sec. 0

10-0015-01 Rule Group Stage 1 n/a 0 sec. 0

10-0020-01 Rule Group Stage 1 n/a 0 sec. 0

10-0021-01 Rule Group Stage 1 n/a 0 sec. 0

10-0040 RSplit Stage 1 Not Run 0 0 0 0 0 sec. 0

10-0040-01 Rule Group Stage 1 n/a 0 sec. 0

20-0004-02 Substitution Stage 1 Complete Processing Complete 1 0 8 1 5 sec. 1

20-0004-03 Substitution Stage 1 Complete Processing Complete 1059392 8683 8475136 69468 2 min. 2 sec. 1

20-0004-04 Substitution Stage 1 Complete Processing Complete 1196027 6873 9568216 54989 2 min. 54 sec. 1

20-0004-05 Substitution Stage 1 Complete Processing Complete 1203790 5166 9630320 41331 3 min. 53 sec. 1

20-0004-06 Substitution Stage 1 Complete Processing Complete 1209437 5281 9675496 42251 3 min. 49 sec. 1

20-0004-07 Substitution Stage 1 Complete Processing Complete 1214680 5125 9717440 41001 3 min. 57 sec. 1

20-0004-08 Substitution Stage 1 Complete Processing Complete 1212745 4949 9701960 39599 4 min. 5 sec. 1

20-0004-09 Substitution Stage 1 Complete Processing Complete 1211278 4903 9690224 39231 4 min. 7 sec. 1

20-0004-10 Substitution Stage 1 Complete Processing Complete 1203130 4930 9625040 39446 4 min. 4 sec. 1

20-0004-11 Substitution Stage 1 Complete Processing Complete 1201393 4923 9611144 39389 4 min. 4 sec. 1

20-0004-12 Substitution Stage 1 Complete Processing Complete 1172460 4864 9379680 38919 4 min. 1 sec. 1

20-0004-13 Substitution Stage 1 Complete Processing Complete 213957 4279 1711656 34233 50 sec. 1

20-0007-02 RowInternal Stage 1 Complete Processing Complete 1 1 1 1 0 sec. 1

20-0007-03 RowInternal Stage 1 Complete Processing Complete 1059392 19618 1059392 19618 54 sec. 1

20-0007-04 RowInternal Stage 1 Complete Processing Complete 1196027 20271 1196027 20271 59 sec. 1

20-0007-05 RowInternal Stage 1 Complete Processing Complete 1203790 20755 1203790 20755 58 sec. 1

20-0007-06 RowInternal Stage 1 Complete Processing Complete 1209437 20498 1209437 20498 59 sec. 1

20-0007-07 RowInternal Stage 1 Complete Processing Complete 1214680 20244 1214680 20244 1 min. 0 sec. 1

20-0007-08 RowInternal Stage 1 Complete Processing Complete 1212745 20909 1212745 20909 58 sec. 1

20-0007-09 RowInternal Stage 1 Complete Processing Complete 1211278 21250 1211278 21250 57 sec. 1

20-0007-10 RowInternal Stage 1 Complete Processing Complete 1203130 20392 1203130 20392 59 sec. 1

20-0007-11 RowInternal Stage 1 Complete Processing Complete 1201393 21453 1201393 21453 56 sec. 1

20-0007-12 RowInternal Stage 1 Complete Processing Complete 1172460 20936 1172460 20936 56 sec. 1

20-0007-13 RowInternal Stage 1 Complete Processing Complete 213957 19450 213957 19450 11 sec. 1

20-0009-02 Substitution Stage 1 Complete Processing Complete 1 1 7 7 0 sec. 1

20-0009-03 Substitution Stage 1 Complete Processing Complete 1018336 2986 7128352 20904 5 min. 41 sec. 1

20-0009-04 Substitution Stage 1 Complete Processing Complete 1082423 3066 7576961 21464 5 min. 53 sec. 1

20-0009-05 Substitution Stage 1 Complete Processing Complete 1096365 3205 7674555 22440 5 min. 42 sec. 1

20-0009-06 Substitution Stage 1 Errored Errored 1025000 2928 7175000 20500 5 min. 50 sec. 1

20-0009-07 Substitution Stage 1 Complete Processing Complete 1102627 3054 7718389 21380 6 min. 1 sec. 1

20-0009-08 Substitution Stage 1 Complete Processing Complete 1101715 3103 7712005 21723 5 min. 55 sec. 1

20-0009-09 Substitution Stage 1 Complete Processing Complete 1101841 3103 7712887 21726 5 min. 55 sec. 1

20-0009-10 Substitution Stage 1 Complete Processing Complete 1102810 3021 7719670 21149 6 min. 5 sec. 1

20-0009-11 Substitution Stage 1 Complete Processing Complete 1102208 3053 7715456 21372 6 min. 1 sec. 1

20-0009-12 Substitution Stage 1 Complete Processing Complete 1096398 2811 7674786 19678 6 min. 30 sec. 1

20-0009-13 Substitution Stage 1 Complete Processing Complete 203327 2033 1423289 14232 1 min. 40 sec. 1

20-0010-02 Substitution Stage 1 Complete Processing Complete 1 0 3 0 4 sec. 1

20-0010-03 Substitution Stage 1 Complete Processing Complete 948757 11859 2846271 35578 1 min. 20 sec. 1

20-0010-04 Substitution Stage 1 Complete Processing Complete 1029111 12705 3087333 38115 1 min. 21 sec. 1

20-0010-05 Substitution Stage 1 Complete Processing Complete 1064317 13822 3192951 41466 1 min. 17 sec. 1

20-0010-06 Substitution Stage 1 Complete Processing Complete 1077094 14361 3231282 43083 1 min. 15 sec. 1

20-0010-07 Substitution Stage 1 Complete Processing Complete 1083719 14449 3251157 43348 1 min. 15 sec. 1

20-0010-08 Substitution Stage 1 Complete Processing Complete 1080181 13335 3240543 40006 1 min. 21 sec. 1

20-0010-09 Substitution Stage 1 Complete Processing Complete 1083791 13057 3251373 39173 1 min. 23 sec. 1

20-0010-10 Substitution Stage 1 Complete Processing Complete 1086902 13586 3260706 40758 1 min. 20 sec. 1

20-0010-11 Substitution Stage 1 Complete Processing Complete 1086598 14111 3259794 42334 1 min. 17 sec. 1

20-0010-12 Substitution Stage 1 Complete Processing Complete 1083706 14259 3251118 42777 1 min. 16 sec. 1

20-0010-13 Substitution Stage 1 Complete Processing Complete 203123 13541 609369 40624 15 sec. 1

20-0012-02 Substitution Stage 1 Complete Processing Complete 1 0 8 0 14 sec. 1

20-0012-03 Substitution Stage 1 Complete Processing Complete 1 0 8 0 13 sec. 1

20-0012-04 Substitution Stage 1 Complete Processing Complete 0 0 0 0 14 sec. 1

20-0012-05 Substitution Stage 1 Complete Processing Complete 0 0 0 0 13 sec. 1

20-0012-06 Substitution Stage 1 Complete Processing Complete 0 0 0 0 14 sec. 1

20-0012-07 Substitution Stage 1 Complete Processing Complete 0 0 0 0 14 sec. 1

20-0012-08 Substitution Stage 1 Complete Processing Complete 0 0 0 0 14 sec. 1

20-0012-09 Substitution Stage 1 Complete Processing Complete 0 0 0 0 14 sec. 1

20-0012-10 Substitution Stage 1 Complete Processing Complete 0 0 0 0 14 sec. 1

20-0012-11 Substitution Stage 1 Complete Processing Complete 0 0 0 0 14 sec. 1

20-0012-12 Substitution Stage 1 Complete Processing Complete 1472762 15836 11782096 126689 1 min. 33 sec. 1

20-0012-13 Substitution Stage 1 Complete Processing Complete 22520 1501 180160 12010 15 sec. 1

20-0013-02 Substitution Stage 1 Complete Processing Complete 1 1 4 4 0 sec. 1

20-0013-03 Substitution Stage 1 Complete Processing Complete 87234 17446 348936 69787 5 sec. 1

20-0013-04 Substitution Stage 1 Complete Processing Complete 104812 20962 419248 83849 5 sec. 1

20-0013-05 Substitution Stage 1 Complete Processing Complete 117987 16855 471948 67421 7 sec. 1

20-0013-06 Substitution Stage 1 Complete Processing Complete 77781 19445 311124 77781 4 sec. 1

20-0013-07 Substitution Stage 1 Complete Processing Complete 92937 15489 371748 61958 6 sec. 1

20-0013-08 Substitution Stage 1 Complete Processing Complete 69422 17355 277688 69422 4 sec. 1

20-0013-09 Substitution Stage 1 Complete Processing Complete 80294 16058 321176 64235 5 sec. 1

20-0013-10 Substitution Stage 1 Complete Processing Complete 81526 16305 326104 65220 5 sec. 1

20-0013-11 Substitution Stage 1 Complete Processing Complete 76100 19025 304400 76100 4 sec. 1

20-0013-12 Substitution Stage 1 Complete Processing Complete 74881 14976 299524 59904 5 sec. 1

20-0013-13 Substitution Stage 1 Complete Processing Complete 11254 11254 45016 45016 1 sec. 1

20-0014-02 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-03 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-04 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-05 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-06 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-07 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-08 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-09 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-10 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-11 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-12 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0014-13 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-02 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-03 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-04 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-05 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-06 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-07 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-08 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-09 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-10 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-11 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-12 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0015-13 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0020-02 Substitution Stage 1 Complete Processing Complete 1 1 1 1 0 sec. 1

20-0020-03 Substitution Stage 1 Complete Processing Complete 219164 19924 219164 19924 11 sec. 1

20-0020-04 Substitution Stage 1 Complete Processing Complete 198846 22094 198846 22094 9 sec. 1

20-0020-05 Substitution Stage 1 Complete Processing Complete 249389 19183 249389 19183 13 sec. 1

20-0020-06 Substitution Stage 1 Complete Processing Complete 261645 18688 261645 18688 14 sec. 1

20-0020-07 Substitution Stage 1 Complete Processing Complete 249036 19156 249036 19156 13 sec. 1

20-0020-08 Substitution Stage 1 Complete Processing Complete 273173 19512 273173 19512 14 sec. 1

20-0020-09 Substitution Stage 1 Complete Processing Complete 269175 19226 269175 19226 14 sec. 1

20-0020-10 Substitution Stage 1 Complete Processing Complete 263065 18790 263065 18790 14 sec. 1

20-0020-11 Substitution Stage 1 Complete Processing Complete 262967 20228 262967 20228 13 sec. 1

20-0020-12 Substitution Stage 1 Complete Processing Complete 284520 20322 284520 20322 14 sec. 1

20-0020-13 Substitution Stage 1 Complete Processing Complete 58619 19539 58619 19539 3 sec. 1

20-0021-02 RowInternal Stage 1 Complete Processing Complete 1 1 1 1 0 sec. 1

20-0021-03 RowInternal Stage 1 Complete Processing Complete 276945 1909 276945 1909 2 min. 25 sec. 1

20-0021-04 RowInternal Stage 1 Complete Processing Complete 281852 1930 281852 1930 2 min. 26 sec. 1

20-0021-05 RowInternal Stage 1 Complete Processing Complete 294951 1992 294951 1992 2 min. 28 sec. 1

20-0021-06 RowInternal Stage 1 Complete Processing Complete 358669 2375 358669 2375 2 min. 31 sec. 1

20-0021-07 RowInternal Stage 1 Complete Processing Complete 477597 2984 477597 2984 2 min. 40 sec. 1

20-0021-08 RowInternal Stage 1 Complete Processing Complete 571755 3465 571755 3465 2 min. 45 sec. 1

20-0021-09 RowInternal Stage 1 Complete Processing Complete 557497 3318 557497 3318 2 min. 48 sec. 1

20-0021-10 RowInternal Stage 1 Complete Processing Complete 551552 3322 551552 3322 2 min. 46 sec. 1

20-0021-11 RowInternal Stage 1 Complete Processing Complete 560688 3377 560688 3377 2 min. 46 sec. 1

20-0021-12 RowInternal Stage 1 Complete Processing Complete 604160 3553 604160 3553 2 min. 50 sec. 1

20-0021-13 RowInternal Stage 1 Complete Processing Complete 120629 893 120629 893 2 min. 15 sec. 1

20-0030 Shuffle Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-02 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-03 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-04 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-05 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-06 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-07 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-08 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-09 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-10 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-11 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-12 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

20-0040-13 Substitution Stage 1 Not Run 0 0 0 0 0 sec. 0

21-0033 Shuffle Stage 1 Not Run 0 0 0 0 0 sec. 0

30-0034 Shuffle Stage 1 Not Run 0 0 0 0 0 sec. 0

99-0003 TriggerMgr(Ena Stage 1 Not Run 0 sec. 0

99-0016 IndexMgr(Ena) Stage 1 Not Run 0 sec. 0

99-0019 Cmd Stage 1 Not Run 0 sec. 0

99-0042 Cmd Stage 1 Not Run 0 sec. 0

99-0044 Cmd Stage 1 Not Run 0 sec. 0

99-0046 Cmd Stage 1 Not Run 0 sec. 0

jamesgally · March 22, 2024 4:25PM

Hi TheMaskedData

Please see excerpt of the log as requested with sensitive data redacted

James