Faulting appl/mod: w3wp.exe; ntdll.dll; TCP error code 10061

I'm trying to profile a Windows Workflow Service application hosted locally on IIS (Windows 7), but when I start the profiling I consequently get the following errors:

First error
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7c8f9
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0x1d34
Faulting application start time: 0x01cd1d5b6266cdc5
Faulting application path: C:\Windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 3f93e3da-894f-11e1-9c9a-0026187ad9e0

Second error
Timestamp: 18-4-2012 12:08:52
Message: HandlingInstanceID: c4a706f9-e6e7-4930-ae08-bb6eb1413ff5
An exception of type 'System.ServiceModel.EndpointNotFoundException' occurred and was caught.
---------------------------------------------------------------------------------------------
04/18/2012 14:08:52
Type : System.ServiceModel.EndpointNotFoundException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Message : Could not connect to http://localhost:81/CertigoEnterprise/Runtime/PolicyJobProcess/RenewalControllerJobProcess.svc. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:81. 
Source : mscorlib
Help link : 
Data : System.Collections.ListDictionaryInternal
TargetSite : Void HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)
Stack Trace : 
Server stack trace: 
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
   at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ClientReliableChannelBinder`1.RequestClientReliableChannelBinder`1.OnSend(TRequestChannel channel, Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ReliableChannelBinder`1.Send(Message message, TimeSpan timeout, MaskingMode maskingMode)
   at System.ServiceModel.Channels.ReliableChannelBinder`1.Send(Message message, TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.SendCloseMessage(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.CloseOutputSession(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.SecurityRequestSessionChannel.CloseOutputSession(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.CloseSession(TimeSpan timeout, Boolean& wasAborted)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnClose(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
   at System.ServiceModel.Channels.LayeredChannel`1.OnClose(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnClose(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)

When profiling is off, everything works perfectly.

Info
    Windows 7, x64, 4GB Windows Firewall turned off IIS Anonymous Authentication and App Pools run under NetworkService ANTS Performance Profiler 6.2, 6.3 Method-level timings; only methods with source Do not record SQL and file I/OI performance; Don't profile child processes Original port; Manually specified ASP.NET account

Comments

  • Quickest thing to try first: run _explicitly_ as administrator (right click on the ANTSGUI.exe). This kills 50% of bugs.
  • On my side there's only a RedGate.Profiler.UI.exe. Ran that as Admin.

    Unfortunately resulting in the same errors.
  • Try the same but withholding the ASP.NET account details
  • Same results.
  • OK, thanks. Can you send to us (support@red-gate.com)

    - Windows application event log (shows pertinent .Net errors)

    - ANTS log (Help->Open log folder)
  • Brian DonahueBrian Donahue Posts: 6,590 Bronze 1
    It's probably the same error - w3wp crashes because of heap corruption and then the website is unsurprisingly not available (10061 is a socket error which usually means a requested service is not answering the connection attempt).

    We would probably need a full memory dump to work out what was happening at the time of the heap corruption and I guess, try to prevent it.
  • I sent the files to the provided address.

    How do I create such a memory dump?

    A colleague of mine has the same problem. We can browse our application. Browse the .svc. But when a call goes out to one of the Workflow based services.... BOOM!

    We have profiled these services in the past... so it's interesting why this suddenly is happening.
  • Brian DonahueBrian Donahue Posts: 6,590 Bronze 1
    You can get a memory dump using adplus:
    ftp://support.red-gate.com/utilities/autodumper.zip
    Extract all files
    Open a command prompt and cd to the extracted files folder
    cscript adplus.vbs -crash -pn w3wp.exe
    Should make a huge .dmp file in a subfolder of adplus

    If you can actually get this to work, we can set up a private FTP to upload the dump to.

    Sometimes the cause of this is a corrupt PDB and sometimes assemblies that use app protection or obfucscation can cause it.

    I'd wager you can profile fine on "least detailed" mode, because all of the things that try to analyze the assembly so it can get line-level timings will not be executed.
  • Adplus ends with the following line:
    Starting to attach the debugger to each process
    Attaching to 1788 - W3WP.EXE
    

    No dumps are created...
    ADPlus Version: 6.03.002
    ADPlus Version Date: 02/21/2006
    OS Version: 6.1
    OS Build: 7601
    DateTimeStamp: Date_04-20-2012__Time_11-09-2828
    ADPlus was run on: MSDEV68
    Debuggers Installation Directory: C:\PD\Command\autodumper
     
    Command line arguments used were: 
    -crash -p 1788 
    Script engine Version: 5.8.16385
    Selected debugger: cdb.exe
    Output Directory: C:\PD\Command\autodumper\Crash_Mode__Date_04-20-2012__Time_11-09-2828
     
     
    g_Debug_Log = True
    g_CheckFreeSpace = True
     ------ Current Actions for Exceptions ----
      AccessViolation [av]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      InvalidHandle [ch]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      IllegalInstruction [ii]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      IntegerDivide [dz]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      FloatingDivide [c000008e]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      IntegerOverflow [iov]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      InvalidLockSequence [lsq]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      StackOverflow [sov]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      CPlusPlusEH [eh]       return: GN GN
          1st chance: Log;Time;Stack
          2nd chance: Log;Time;Stack;FullDump;EventLog
      UnknownException [*]       return: GN GN
          1st chance: Log;Time;Stack
          2nd chance: Log;Time;Stack;FullDump;EventLog
      NET_CLR [clr]       return: GN GN
          1st chance: VOID
          2nd chance: Log;Time;Stack;FullDump;EventLog
      CONTRL_C_OR_Debug_Break [bpe]       return: Q GN
          1st chance: Log;Time;EventLog;Stacks;LoadedModules;MatchingSymbols;MiniDump
          2nd chance: 
      Wake_Debugger [wkd]       return: Q GN
          1st chance: Log;Time;EventLog;Stacks;LoadedModules;MatchingSymbols;MiniDump
          2nd chance: 
      DLL_Load [ld]       return: GN GN
          1st chance: Log
          2nd chance: Log
      DLL_UnLoad [ud]       return: GN GN
          1st chance: Log
          2nd chance: Log
      Process_Shut_Down [epr]       return: VOID VOID
          1st chance: Log;Time;EventLog;Stacks;FullDump
          2nd chance: 
      Application_hang [aph]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      Ctl_C_Console_app [cce]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      Data_misaligned [dm]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      Guard_page_violation [gp]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      In_page_IO_error [ip]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      Invalid_system_call [isc]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
      Stack_buffer_overflow [sbo]       return: GN GN
          1st chance: Log;Time;Stack;MiniDump
          2nd chance: Log;Time;Stack;FullDump;EventLog
    Starting to attach the debugger to each process
    Attaching to 1788 - W3WP.EXE
    

    The least detailed profile does seem to create a more stable situation, but eventually, after a few minutes, the process randomly dies as well.
  • Brian DonahueBrian Donahue Posts: 6,590 Bronze 1
    Sorry but I don't think we will be able to troubleshoot this any further. I guess the .NET Framework is having some sort of issue?
Sign In or Register to comment.