Large application analysis
efolkertsma
Posts: 3
Recently I have been optimizing our product using the ANTS profiler suite, but I am hitting a couple of issues that prevent me from doing a good job. The main purpose of this post is to hopefully get work around
suggestions for the problems I am having, as I am a beginner user with ANTS. It would also be interesting to hear if RedGate is having some of these issues on their radar for future resolution.
Summary:
1) Memory snapshots are limited to 4 GB
2) Not being able to attach post application launch/pause profiling
3) ANTS hangs the application when I have the application set to use the Server garbage collector
4) Detail of the information
Detailed:
1) Memory snapshots are limited to 4 GB
It seems that when I do a memory snapshot (in order to compare 2 to find leaks) it gives me a message that I cannot have snapshots of more than 4 GB for performance reasons. But I have 32 GB of memory and my application 'only' takes 6-8 GB of memory. I could live with a little less performance, even if the memory compare took say 8 hours over night it would at least help me forward. The way it is now prevents me from doing memory captures and pretty much renders the memory analyzer tool useless in 75% of my situations. The CLR memory tracker seems to work fine with 8+ GB, so I do not think there is a technical limitation on the GC side of things. I have spent a lot of time trying to build smaller versions of the tool, using a smaller data set, but in doing so it seems that some of the leakage disappears (probably due to the amount of resources I am using being lowered), so I would have to build multiple variants. This kind of limits the usefulness of the memory profiler in the drop-in sense of working as I have to make custom code/projects to profile.
Perhaps if I could select specifically which DLLs to track memory allocations from this would make it more feasible? (See item 4)
2) Not being able to attach post application launch/pause profiling
There is no attach to a running program (at least non that I could find.) This is causing some issues because for my situation launching the application through the profiler and then loading the project I need to load means I have to wait about 45 minutes before I can profile at all, which is a turn around time that is somewhat problematic for me. I have used a wide range of different profilers throughout my career and most allow you to attach to a program by parsing the PDB (or something similar.) I do not know the internals of how the ANTS profiler gets the information and I understand it may be using some code injection, so it cannot attach after the fact. It would be great if I could do the code injection, but then pause the profile information collection itself. I am currently experiencing a long wait time till I get to the point where I want the profiling information. For example without the profiler attached loading my project takes 45 seconds, with it it takes between 8 and 9 minutes. Loading the correct data without profiler attached takes about 20 seconds, with the profiler attached about 4-5 minutes. I have to go through a couple of these steps which takes a total of about 45 minutes (making code change, then compiling, starting profiler, doing capture, have capture analyzed.)
3) ANTS hangs the application when I have the application set to use the Server garbage collector
I made a change to our application to use the server mode garbage collector (<gcServer enabled="true" /> in the App.config), I also changed the latency mode to SustainedLowLatency (GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency;) and it helped improve performance of our application. However when I run the application through ANTS with both these settings enabled, it seems to 'hang' my application at the loading part, which does not happen when I do not run the program through the profiler. If I use the server GC, but don't change the latency mode the loading phase of the application works, but actually editting a resource results in a hang as well.
After attaching the debugger (Visual Studio) to it it seems like the application is deadlocked and one of the threads is stuck on ProfilingAPIAtachDetach::ProfilingAPIAttachThreadStart (NtWaitForSingleObject). I am hoping someone has a work around for this or maybe a way to look deeper into what is happening.
4) Detail of the information
This item is probably due to our overly complicated setup. Our project constist out of about 200 different visual studio projects (millions of lines of code in total, so it is not just all small projects), some of which are native C++, some are C#, some are CLI glue, some are auto generated code, etc. Then there are some DLLs we load at runtime and when the application runs we have about 35-45 threads active (it runs on a Dual CPU, 6 core per CPU, 12 HT, for a total of 24 hardware threads... so it is not 'that' crazy)
With this setup however I find that I have to use low-medium profiling overhead, because the highest overhead setting will just kill the profiler pretty much outright. Unfortunately after profiling with those settings I often find only 25% percent of the application time accounted for and 75% of 'hidden methods'/'insignificant methods'. However when we start fixing those '25%' we often see gains far beyond 25%, 2x performance gains. So I am not really sure how to interpret those results.
In any case, because we have too many projects, it would be great if we would specify which DLLs the application should profile, and which ones it should leave alone. There is a global setting for 'profile child processes', but ideally I would like to select on a per DLL basis whether or not it is instrumented.
suggestions for the problems I am having, as I am a beginner user with ANTS. It would also be interesting to hear if RedGate is having some of these issues on their radar for future resolution.
Summary:
1) Memory snapshots are limited to 4 GB
2) Not being able to attach post application launch/pause profiling
3) ANTS hangs the application when I have the application set to use the Server garbage collector
4) Detail of the information
Detailed:
1) Memory snapshots are limited to 4 GB
It seems that when I do a memory snapshot (in order to compare 2 to find leaks) it gives me a message that I cannot have snapshots of more than 4 GB for performance reasons. But I have 32 GB of memory and my application 'only' takes 6-8 GB of memory. I could live with a little less performance, even if the memory compare took say 8 hours over night it would at least help me forward. The way it is now prevents me from doing memory captures and pretty much renders the memory analyzer tool useless in 75% of my situations. The CLR memory tracker seems to work fine with 8+ GB, so I do not think there is a technical limitation on the GC side of things. I have spent a lot of time trying to build smaller versions of the tool, using a smaller data set, but in doing so it seems that some of the leakage disappears (probably due to the amount of resources I am using being lowered), so I would have to build multiple variants. This kind of limits the usefulness of the memory profiler in the drop-in sense of working as I have to make custom code/projects to profile.
Perhaps if I could select specifically which DLLs to track memory allocations from this would make it more feasible? (See item 4)
2) Not being able to attach post application launch/pause profiling
There is no attach to a running program (at least non that I could find.) This is causing some issues because for my situation launching the application through the profiler and then loading the project I need to load means I have to wait about 45 minutes before I can profile at all, which is a turn around time that is somewhat problematic for me. I have used a wide range of different profilers throughout my career and most allow you to attach to a program by parsing the PDB (or something similar.) I do not know the internals of how the ANTS profiler gets the information and I understand it may be using some code injection, so it cannot attach after the fact. It would be great if I could do the code injection, but then pause the profile information collection itself. I am currently experiencing a long wait time till I get to the point where I want the profiling information. For example without the profiler attached loading my project takes 45 seconds, with it it takes between 8 and 9 minutes. Loading the correct data without profiler attached takes about 20 seconds, with the profiler attached about 4-5 minutes. I have to go through a couple of these steps which takes a total of about 45 minutes (making code change, then compiling, starting profiler, doing capture, have capture analyzed.)
3) ANTS hangs the application when I have the application set to use the Server garbage collector
I made a change to our application to use the server mode garbage collector (<gcServer enabled="true" /> in the App.config), I also changed the latency mode to SustainedLowLatency (GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency;) and it helped improve performance of our application. However when I run the application through ANTS with both these settings enabled, it seems to 'hang' my application at the loading part, which does not happen when I do not run the program through the profiler. If I use the server GC, but don't change the latency mode the loading phase of the application works, but actually editting a resource results in a hang as well.
After attaching the debugger (Visual Studio) to it it seems like the application is deadlocked and one of the threads is stuck on ProfilingAPIAtachDetach::ProfilingAPIAttachThreadStart (NtWaitForSingleObject). I am hoping someone has a work around for this or maybe a way to look deeper into what is happening.
4) Detail of the information
This item is probably due to our overly complicated setup. Our project constist out of about 200 different visual studio projects (millions of lines of code in total, so it is not just all small projects), some of which are native C++, some are C#, some are CLI glue, some are auto generated code, etc. Then there are some DLLs we load at runtime and when the application runs we have about 35-45 threads active (it runs on a Dual CPU, 6 core per CPU, 12 HT, for a total of 24 hardware threads... so it is not 'that' crazy)
With this setup however I find that I have to use low-medium profiling overhead, because the highest overhead setting will just kill the profiler pretty much outright. Unfortunately after profiling with those settings I often find only 25% percent of the application time accounted for and 75% of 'hidden methods'/'insignificant methods'. However when we start fixing those '25%' we often see gains far beyond 25%, 2x performance gains. So I am not really sure how to interpret those results.
In any case, because we have too many projects, it would be great if we would specify which DLLs the application should profile, and which ones it should leave alone. There is a global setting for 'profile child processes', but ideally I would like to select on a per DLL basis whether or not it is instrumented.
Comments
Thanks for your post and sorry to hear you're experiencing some issues with the profilers! Regarding these issues...
1) Memory snapshots are limited to 4 GB
Apologies for this, I know the limitation can be quite frustrating, but 4gb snapshots is the largest the profiler can handle at the moment. I've gone ahead and created a request for support of larger snapshots though (with internal reference MP-2325).
2) Not being able to attach post application launch/pause profiling
Both profilers do actually have an option to attach to processes (though a .NET limitation makes attaching possible only for .NET 4+ applications)
Kindly see here -
http://documentation.red-gate.com/displ ... +a+process
http://documentation.red-gate.com/displ ... +4+process
(Please note that for the memory profiler, you do need to disable both server and concurrent garbage collection in order to attach.)
As far as pausing though, I'm sorry to say there are some technical reasons why the developers have decided not to provide this feature. Nonetheless, we do have a feature request (logged internally with reference PP-3496) and I've +1'd this on your behalf. Many thanks for the feedback!
3) ANTS hangs the application when I have the application set to use the Server garbage collector
I don't think we've seen this before so I'm not immediately sure of a workaround, sorry to say. Just to clarify--is the hanging happening with the Memory or Performance profiler (or both)? Does enabling/disabling the different profiling options make any difference?
4) Detail of the information
To cut down the overhead from the performance profiler, you can remove pdb files for any of the assemblies you're not interested in and then use one of the "Only methods with source" profiling modes (the profiler determines if something is a "method with source" if there is a pdb file available). Another thing that can help is if you want to use line-level timings but don't need line-level timings for certain assemblies --you can then add those to the line-level black list in %LocalAppData%Red GateANTS Performance Profiler 8LineLevelBlacklist.xml -- hope that might help!
Then to show the 'hidden methods'/'insignificant methods' you can profile with Tools>Advanced Options>Avoid profiling extremely trivial methods disabled. And when you see "x collapsed insignificant methods" in the call tree, you can right-click to reveal them. If you want to see both methods with and without source, make sure to also choose "All methods" from the Display Options.
(Just a note: we don't have timelines set for any of the requests mentioned above, I'm afraid, but I'll let you know of any updates!)
Jessica Ramos | Product Support Engineer | Redgate Software
Have you visited our Help Center?