Timed out waiting for port 8013, Object reference not set.

tableau

I'm getting this error when attempting to start a Memory Profiler against an asp.net app on IIS and using .NET 1.1 framework. I do have version 2.71 also installed. I'm profiling the app "http://10.19.44.5/policy"


Error messaage in app:
"There was a problem profiling the web application

Timed out waiting for port 8013 on 10.19.44.5 to become available."

Error in event log:
Unable to start profiler - exception details RedGate.Profiler.Engine.IIS.PortTimeoutException: Timed out waiting for port 8013 on 10.19.4.153 to become available

Server stack trace:
at RedGate.Profiler.Engine.IISUtilities.WaitForPort(IPAddress address, Int32 port, Int32 timeout)
at RedGate.Profiler.Engine.IISUtilities.WaitForPort(String hostname, Int32 port, Int32 timeout)
at RedGate.Profiler.Engine.ProfileIISController.StartProfilingIIS(String currentUserName, String& url, WebApplicationDefinition appDef)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

Exception rethrown at [0]:
at RedGate.Profiler.Engine.ProfilingSession.StartProfiler()
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [1]:
at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)
at System.Windows.Forms.MethodInvoker.EndInvoke(IAsyncResult result)
at _5._4(IAsyncResult )



If I change ANTS to profile "http://localhost/policy" I get a different error.

App error: "Object reference not set to an instance of an object."

Event log:

Unable to start profiler - exception details System.NullReferenceException: Object reference not set to an instance of an object.

Server stack trace:
at RedGate.Profiler.Engine.ProfileIISController._1(WebApplicationDefinition )
at RedGate.Profiler.Engine.ProfileIISController.StartProfilingIIS(String currentUserName, String& url, WebApplicationDefinition appDef)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

Exception rethrown at [0]:
at RedGate.Profiler.Engine.ProfilingSession.StartProfiler()
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [1]:
at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)
at System.Windows.Forms.MethodInvoker.EndInvoke(IAsyncResult result)
at _5._4(IAsyncResult )

Robert C

Hi,

Firstly, what OS and architecture are you running? I'm guessing Windows 2003, based on the fact that it's profiling on port 8013.

I wonder if you could check whether this registry key exists, and what its value is if so?

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\IISAdmin\ImagePath

The other thing you may wish to try is creating a directory named "logs" in your ANTS Profiler 3 installation directory. This should cause some debugging information to be written which may help us diagnose the problem.

Many thanks,
Robert

tableau

I'm running Server 2003 SP1, IIS6, 1.1 .NET Framework.

I made som eprogress -- localhost isn't a valid alias for this machine. I have to use the IP address. So the "object reference" issue is resolved.

Additionally, I don't have registry access to this machine and am waiting for one of our server admins to assist me on that. Perhaps that could be the root of the problem? Anyway, I'll see about that.

I created the logs subdirectory and re-ran the test. Two files get created. Here are their contents:

3/22/2007 8:44:06 AM ANTSProfilerLog initialized
3/22/2007 8:44:06 AM PID = 4860
3/22/2007 8:44:06 AM Log file name = ANTSProfilerLog4860.txt
3/22/2007 8:44:06 AM OS version = Microsoft Windows NT 5.2.3790 Service Pack 1
3/22/2007 8:44:06 AM Machine name = SCPOLP52
3/22/2007 8:44:06 AM User name = MBRUNER1
3/22/2007 8:44:06 AM PARAMHOST - Listening on port 8084
3/22/2007 8:44:06 AM PARAMHOST - Listening on port 8084, channel registered
3/22/2007 8:44:07 AM EditProjectClosingWizard start
3/22/2007 8:44:07 AM EditProjectClosingWizard start
3/22/2007 8:44:07 AM Couldn't find WebDev at: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\WebDev.WebServer.exe
3/22/2007 8:44:08 AM Using IIS Version 6
3/22/2007 8:44:08 AM About to get ASPNet username and password for IIS6
3/22/2007 8:44:08 AM About to get site for URL
3/22/2007 8:44:08 AM About to get IIS6 sites
3/22/2007 8:44:08 AM Got site 'PolicyPro' with identifier 595582558
3/22/2007 8:44:08 AM ENGINE - Couldn't find site for URL http://localhost/Policy
3/22/2007 8:44:08 AM About to show wizard
3/22/2007 8:44:19 AM Wizard dialog returned
3/22/2007 8:44:21 AM UI Start profiling
3/22/2007 8:44:21 AM ENGINE - Core2.dll found in regisry
3/22/2007 8:44:21 AM ENGINE Start profiling memory of ASP.NET web application http://10.19.4.153/Policy
3/22/2007 8:44:21 AM ENGINE The profiler is enabled
3/22/2007 8:44:21 AM TRANSFER - Waiting for pipe connection from core
3/22/2007 8:44:21 AM WaitForPort on address 127.0.0.1, port 8086
3/22/2007 8:44:22 AM ENGINE - Managed to hook into status updates from IIS Profile Host
3/22/2007 8:44:56 AM ENGINE - Closing IISProfileHost
3/22/2007 8:44:57 AM ENGINE - Caught Exception in ProfilingSession: Timed out waiting for port 8013 on 10.19.4.153 to become available
3/22/2007 8:47:49 AM ENGINE Closing profiling session





3/22/2007 8:44:22 AM IISProfileHostLog initialized
3/22/2007 8:44:22 AM PID = 8088
3/22/2007 8:44:22 AM Log file name = IISProfileHostLog8088.txt
3/22/2007 8:44:22 AM OS version = Microsoft Windows NT 5.2.3790 Service Pack 1
3/22/2007 8:44:22 AM Machine name = SCPOLP52
3/22/2007 8:44:22 AM User name = MBRUNER1
3/22/2007 8:44:22 AM ANTS Profiler IISProfileHost Start up
3/22/2007 8:44:22 AM Listening on port 8086
3/22/2007 8:44:22 AM StartProfilingIIS currentUserName PROG1\MBRUNER1, url http://10.19.4.153/Policy, .NET 2.0 OnePointOne, IIS6
3/22/2007 8:44:22 AM Parse IIS metabase
3/22/2007 8:44:22 AM InitServiceList
3/22/2007 8:44:23 AM StartIIS
3/22/2007 8:44:23 AM StartWebService for IIS 6
3/22/2007 8:44:23 AM WaitForPort on address 10.19.4.153, port 8013
3/22/2007 8:44:25 AM About to get site for URL
3/22/2007 8:44:25 AM About to get IIS6 sites
3/22/2007 8:44:25 AM Got site 'PolicyPro' with identifier 595582558
3/22/2007 8:44:25 AM About to get IIS6 x64 status
3/22/2007 8:44:25 AM WaitForPort start, url = http://10.19.4.153:8013/Policy
3/22/2007 8:44:25 AM WaitForPort on address 10.19.4.153, port 8013
3/22/2007 8:44:56 AM StartProfilingIIS - threw exception
3/22/2007 8:44:56 AM ENGINE - Received request to close IISProfileHost
3/22/2007 8:44:56 AM IISPROFILEHOST - Exiting due to closing: True and UI Exit: False
3/22/2007 8:44:56 AM Main() exiting

Robert C

Hi,

Thanks for the information.

You mention you don't have registry access: this does sound like it could be the problem. ANTS Profiler needs to be run as an administrator to profile IIS, I'm afraid. As well as reading the registry key I mentioned above, we also need to start a new IIS worker process (w3wp.exe), which I believe must be run as an administrator (though I could be wrong there).

If you are able to get registry access, it's probably worth giving it a go - it may work at that point.

If you're unable to run ANTS Profiler as an administrator, you might want to try using the ASP.NET Development Web Server. This is bundled with Visual Studio 2005, so it's .NET 2.0, but might help you to some degree.

(As an aside, the reason 2.7.1 didn't need to be run as an administrator was that it used a Windows Service to achieve privilege elevation. We decided to remove this in version 3, as the majority of developers do have administrative rights on their machines.)

Regards,
Robert

tableau

Registry access does not appear to be the issue. I've tried logging in under a different user account that definitely have registry access and am getting the same error.

Also, the reg key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\IISAdmin\ImagePath does exist.

You mention that "ANTS Profiler needs to be run as an administrator." Do you know what exact permissions are required? We profile our apps in a stress environment that is designed to mimic our production environment, including many of its security policies. I can request to be granted certain permissions and policies, but I need to know which ones. I don't think it is possible to be granted any sort of "God" access to the server; I need to be more specific than that.

In the meantime, I'm having some of the server admin guys attempt to start some ANTS Profiler sessions. If they can do it with their accounts, then I can be confident there is a security issue with my own User ID. If they can't, then perhaps the problem is not security related.

Robert C

Hi,

When profiling IIS6, we launch a w3wp.exe worker process. This process requires admin rights, although I'm afraid I'm unsure exactly which ones. I presume it needs to be able to impersonate the user that you've configured your application pool to run as.

We also need permissions to read from the IIS Metabase.

Hope that helps,
Rob

Sachin

Hi,

I am also undergoing the same problem. I did all the things suggested by you and other user group. But it is not working. Can we presume that red-gate profiler fails to profile Web Service!

Regards,
Sachin Bansal

Robert C

Hi Sachin,

Could you give me some more details about the problem you're having?

As I understand it, tableau's problems are as a result of not having administrator rights (which ones in particular we aren't yet certain).

Is this the same situation for you, or is ANTS failing to work even when you log on as a local administrator?

Regards,
Robert

Robert C

Sorry, I see you've posted more details on this thread. I'll make a reply there.

Sachin

hi Rob,

Thanks for your quick reply.
I am an administrator on my machine and I dont think there is any credential problem.

I have mentioned each detail with the topic

"Problem in profiling WebService! Exception thrownby Profiler

at this group only. The url is:
"http://www.red-gate.com/MessageBoard/viewtopic.php?p=16398#16398"

May i request you to look into it, and please try to help me out. I am also pasting the log enteries. In my case log files says:

/****************************************************/
File 1 ANTSProfilerLog17760.txt:

3/27/2007 1:50:20 PM ANTSProfilerLog initialized
3/27/2007 1:50:20 PM PID = 11788
3/27/2007 1:50:20 PM Log file name = ANTSProfilerLog11788.txt
3/27/2007 1:50:20 PM OS version = Microsoft Windows NT 5.2.3790 Service Pack 1
3/27/2007 1:50:20 PM Machine name = TCS031751
3/27/2007 1:50:20 PM User name = administrator
3/27/2007 1:50:20 PM PARAMHOST - Listening on port 8084
3/27/2007 1:50:20 PM PARAMHOST - Listening on port 8084, channel registered
3/27/2007 1:50:42 PM EditProjectClosingWizard start
3/27/2007 1:50:42 PM EditProjectClosingWizard start
3/27/2007 1:50:43 PM Using IIS Version 6
3/27/2007 1:50:43 PM About to get ASPNet username and password for IIS6
3/27/2007 1:50:43 PM About to get site for URL
3/27/2007 1:50:43 PM About to get IIS6 sites
3/27/2007 1:50:43 PM Got site 'Default Web Site' with identifier 1
3/27/2007 1:50:43 PM Got best VD match at path: [url=IIS://localhost/w3svc/1/root]IIS://localhost/w3svc/1/root[/url], URL /LM/W3SVC/1/ROOT
3/27/2007 1:50:43 PM AppPool for URL http://localhost/CheckProfiler is DefaultAppPool
3/27/2007 1:50:43 PM About to show wizard
3/27/2007 1:51:00 PM Wizard dialog returned
3/27/2007 1:51:02 PM UI Start profiling
3/27/2007 1:51:02 PM ENGINE - Core2.dll found in regisry
3/27/2007 1:51:02 PM ENGINE Start profiling performance of ASP.NET web application http://localhost/CheckProfiler/Service.asmx
3/27/2007 1:51:02 PM ENGINE The profiler is enabled
3/27/2007 1:51:02 PM TRANSFER - Waiting for pipe connection from core
3/27/2007 1:51:02 PM WaitForPort on address 127.0.0.1, port 8086
3/27/2007 1:51:03 PM ENGINE - Managed to hook into status updates from IIS Profile Host
3/27/2007 1:51:37 PM ENGINE - Closing IISProfileHost
3/27/2007 1:51:38 PM ENGINE - Caught Exception in ProfilingSession: Timed out waiting for port 8013 on 127.0.0.1 to become available

##############################################

File 2: IISProfileHostLog12584.txt

3/27/2007 2:33:54 PM IISProfileHostLog initialized
3/27/2007 2:33:54 PM PID = 12584
3/27/2007 2:33:54 PM Log file name = IISProfileHostLog12584.txt
3/27/2007 2:33:54 PM OS version = Microsoft Windows NT 5.2.3790 Service Pack 1
3/27/2007 2:33:54 PM Machine name = TCS031751
3/27/2007 2:33:54 PM User name = administrator
3/27/2007 2:33:54 PM ANTS Profiler IISProfileHost Start up
3/27/2007 2:33:54 PM Listening on port 8086
3/27/2007 2:33:55 PM StartProfilingIIS currentUserName SOLOMON\administrator, url http://localhost/CheckProfiler/Service.asmx, .NET 2.0 Two, IIS6
3/27/2007 2:33:55 PM Parse IIS metabase
3/27/2007 2:33:56 PM InitServiceList
3/27/2007 2:33:56 PM StartIIS
3/27/2007 2:33:56 PM StartWebService for IIS 6
3/27/2007 2:33:56 PM WaitForPort on address 127.0.0.1, port 8013
3/27/2007 2:33:58 PM About to get site for URL
3/27/2007 2:33:58 PM About to get IIS6 sites
3/27/2007 2:33:59 PM Got site 'Default Web Site' with identifier 1
3/27/2007 2:33:59 PM About to get IIS6 x64 status
3/27/2007 2:33:59 PM WaitForPort start, url = http://localhost:8013/CheckProfiler/Service.asmx
3/27/2007 2:33:59 PM WaitForPort on address 127.0.0.1, port 8013
3/27/2007 2:34:30 PM StartProfilingIIS - threw exception
3/27/2007 2:34:30 PM ENGINE - Received request to close IISProfileHost
3/27/2007 2:34:30 PM IISPROFILEHOST - Exiting due to closing: True and UI Exit: False
3/27/2007 2:34:30 PM Main() exiting
/****************************************************/


Please do let me know if you need any other information,
thaanks,
Sachin Bansal

tableau

OK, I'm making a little progress. I had several different types of users attempt to run ANTS and, finally, one of the top-level server administrators here was able to run it. We believe something in a user's group policy is defining a restriction that is preventing ANTS from profiling a web app. Now the question is exactly which restrictions are the problematic ones....

It may not be possible/feasible for my company to start granting the least restrictive group policy settings to anyone who will run ANTS Profiler. I may have to identify the specific restrictions that must be lifted. If that's the case, I may have some further questions. I have a feeling our security guys will want to know the minimum settings required to run Profiler.


From this experience I have a couple suggestions for Red Gate:

1) the error message I am getting is not very indicative of my issue. A "time out" waiting for the port seems to suggest something else is using that port. A better error message would be a real time saver.

2) Corporate environments often have rather complex security implementations where asking for "admin" access to a box is not a simple matter. More detailed security requirements for running Profiler would be very useful.

Robert C

Hi,

That's good news at least that there's some progress!

I understand that many companies do run locked down security policies, and this is obviously causing you problems here.

In previous versions of ANTS Profiler, we used a Windows Service running as a system account to achieve elevation. In this version, we decided to remove this as we had assumed most developers had full admin rights on development machines (and profiling on production servers is not really recommended!).

With regards to your suggestion, I certainly agree that a better error message would be good. It's a real shame that w3wp doesn't give us any errors - it doesn't make our life any easier in detecting what's gone wrong!

We'll try and improve this situation in future releases, and I'll try and look into exactly what permissions are required. If you make any discoveries in the mean time, please do post them on the forum.

Brian Donahue

Hi,

I remember this oldie-but-goodie, there are some file permissions that you, as the logged-in user, will need to be able to run w3wp.exe. Normally, w3wp running as Network Service has the necessary rights already. ANTS Profiler 2.x would do this job, by default, as LocalSystem, which pretty much has carte blanche on the system. You may want to have a look at this Microsoft article, of course replacing 'Network Service' with your logged-in user name:

http://support.microsoft.com/kb/833444

Brian Donahue

I did a bit more research into this. In addition to granting rights to the ASP .NET system folders for your login, http.sys will fail to service w3wp.exe without the following user rights:

-Bypass traverse checking
-Impersonate a client after authentication
-Logon as a batch job

If you get errors about getting information from the metabase, Microsoft make a nice utility called metaacl.vbs. You can download it from microsoft's website. It's an easy-to-use tool that will specifically grant or remove access to the IIS metabase for a certain account.

I've also found that DNS issues can also cause http.sys to fail to start. Http.sys is responsible for the communication layer of IIS. The idea is that if processing stops in w3wp, http.sys can continue answering requests while ASP .NET is still recovering from the fault.

The reason why ANTS Profiler does not show any errors is because it starts a separate process, the profileiiscontroller.exe, to start w3wp.exe. Therefore errors thrown by profileiiscontroller can't be caught and thrown to ANTS Profiler's UI, giving a clearer error message.

Brian Donahue

You may also want to try this outside of Profiler -- create a new application pool in IIS Manager and give it the same identity that you normally log in with. Assign the web application to that application pool. You may need to run iisreset.

Now open a web browser and see if you can see your web application. If so, all of the rights and permissions should be okay when you profile it.

Brian Donahue

Hi all,

Getting an application pool to run as a named user is one thing -- but I see there is another problem. When ANTS Profiler starts the application pool, it does so outside IIS using a Process Start:

%systemroot%\system32\inetsrv\w3wp.exe -debug -d http://foo:8013/bar/ -ap RedGateTestAppPool

Although you can set the application pool to run as any user inside of IIS (with a bit of effort), running this command will terminate with exit code -1 using an LUA user with ordinary privileges. (Just go ahead and try it!)

I realise that not everyone (and wisely so) has admin rights on their computers. So I tried a slew of things:

• Ran metaacl.vbs and granted full access to the IIS metabase to my user
• Added myself to the 'bypass traverse checking','Adjust memory quotas for a process', and 'Replace a process level token' user rights (for debugging reasons
• Audited failure for all privilege use (no hits there..)
• Allowed my account activation rights for IIS WAMREG DCOM component
• Used httpcfg (from the Windows 2003 CD\Support Tools) to create an ACL that allows my account to create a listener on port 8013

No matter what, I end up with an error in my application log from W3SVC-WP, event 2269, saying that http.sys couldn't be initialized. The only information from Microsoft about this is to grant my account the 'Bypass traverse checking' right.

My conclusion is that you need to run ANTS Profiler as a local administrator in order to profile ASP .NET on IIS 6 using the application pool model. I could contact Microsoft PSS this week for their answer, but based on what I've experienced, any workaround probably won't come without the requisite amount of pain.

You could also choose the 'debugging web server' option, which doesn't require administrator rights.

If anyone has had any success getting Profiler to work in IIS6 as a regular user, I'd love to know what the solution is. I would gladly send you the Red Gate promotional gear of your choice!