Noisy Output

Paul Wardle

This tool looks very promising. My company is implementing an exception handling strategy. Your tool looks like a great way to check its effectiveness.

I would like the tool to check whether possible exceptions are handled - but within reason!

Currently the output is very noisy, and for me makes the tool practically useless.

Example: If you call any method that allocates objects it says it could generate an OutOfMemoryException. Of course this is correct, but I shouldn't be catching an exception of this type in my code.

Example: This code indicates that an ArgumentOutOfRangeException could be thrown:

if (string.IsNullOrEmpty(s))
    return string.Empty;

return s.Substring(0, s.Length - 1);

Well, no it will never be thrown.

Example: This constructor could throw these exceptions???

System.ArgumentNullException
System.ArgumentOutOfRangeException
System.FormatException
System.OutOfMemoryException

public MessageEventArgs(MessageStatus status, string msg, params object[] formatArgs)
{
   if (msg == null)
      throw new ArgumentNullException("msg");
   if (formatArgs == null)
      throw new ArgumentNullException("formatArgs");

   _msg = string.Format(CultureInfo.CurrentUICulture, msg, formatArgs);
}

The ArgumentOutOfRangeException is generated by a call deep inside String, and is actually generated by a StringBuilder ctor. The parameters are setup by the string class and will probably never be wrong.


Example: Guess how many exceptions this snippet could throw?

public MyException(string msg, bool sentMsg)
			: base(msg)
{
   Trace.WriteLine(string.Format(CultureInfo.InvariantCulture, "MyException: {0}:{1}", msg, sentMsg));
   _sentMsg = sentMsg;
}

:shock: Its this little list:

System.Configuration.ConfigurationException
System.ArgumentException
System.ArgumentNullException
System.ArgumentOutOfRangeException
System.Configuration.ConfigurationErrorsException
System.ExecutionEngineException
System.FormatException
System.IndexOutOfRangeException
System.InvalidCastException
System.InvalidOperationException
System.IO.PathTooLongException
System.MemberAccessException
System.MissingMethodException
System.NotSupportedException
System.NullReferenceException
System.OutOfMemoryException
System.Reflection.AmbiguousMatchException
System.Reflection.CustomAttributeFormatException
System.Security.SecurityException
System.TypeLoadException

Wow.

These are a few simple examples of the output that I get. With a real program there are so many false negatives that its no use to me.

FYI: I am running the tool from the command line and specify the following options:

hunt /a:"assembly.dll" /mr:"report.html" /all /icc:on /sd:1

Do you plan to improve the behaviour of the tool - i.e. make it detect such cases?

I appreciate that strictly speaking, the tool is doing its job However I need to detect unhandled exceptions that could be thrown in the "real world"; the tool should not detect a potential coding mistake when I have actually screened out the failure cases.

I see that your feature list on the web site mentions filters. Is available now or is this going to be implemented soon? Do you have any suggestions of how this would work?

Its a good first effort, and appreciated!

James Moore

Hi Paul,

Thanks for the feedback.

The .NET Framework certainly can throw a large number of exceptions for fairly innocuous snippets of code, we have certainly had quite a lot of discussions about how to deal with number of exceptions generated.

We do want to improve the output you get from the tool! In this v1 release we decided to put the core functionality out and limit the choices we make about what to include/exclude from analysis by default and try and work with users to find the best way of doing this (we have removed some classes which are involved in one off initialization and have large code paths).

We have discussed (and the functionality is in the engine) allowing users to ignore certain types of exception (for example OutOfMemoryExceptions) or exceptions from specific locations, this did not make it into the public interfaces for v1 of the product. Similar functionality should be fairly easy to achieve with the XML output and a little bit of XSLT (I will see what I can put together today and post it up on the site later).

Your code snippet with the s.Substring call in it is a more tricky case – we would need to determine that s.Length (infact String.get_Length) is deterministic and immutable I will talk to the developers and see if we are likely to be able to do this in a future release, however I would worry it would increase analysis time even further (and many of these methods are not implemented in IL but are externally implemented by the CLR).

Additionally the if(String.IsNullOrEmpty(s)) presents some quite interesting inference problems. This simple case should be fairly easy to support but supporting this sort of inference across functions (which can possibly have side-effects and invalidate logical conclusions reached so far) in the general case is going to be incredibly difficult.

We are looking at many different ways to improve the output so that developers write the exception handlers they need, and no more than that, during development rather than in reaction to crashes during QA or out in the field.

Please let me know if you have any other thoughts on how we could improve the situation,

I appreciate the time you have taken giving us feedback,

Regards,

James

Paul Wardle

Thanks for your prompt response!

Have you thought about using a technique similar to the SuppressMessage attribute used by FXCop? I could then suppress the messages for exceptions that I know could never be thrown.

James Moore

Thats a good idea and could certainly work for user's code. I would like to try and find somthing which would work for code inside the framework as well..

James

Paul Wardle

You need to consider the mission statement for why someone wants to detect the exceptions that may occur. Although some of the boundary cases could occur on a theoretical basis, following the 80:20 rule, most people would want to know of the obvious exceptional cases that have been missed.

Certainly, most of the time I would like to know when one of the exceptions that are documented against the .Net framework are not handled.

Have you thought about considering the xml documentation that is generated alongside an assembly? If I call another component that may throw an exception and document that I may throw that exception too, then IMHO it really isn't an issue as far as robustness goes.

James Moore

Paul,

I have altered our XSLT stylesheet to support a very simple ruleset which allows you to exclude certain exceptions.

The first thing you need to do is to create a file in your working directory called ruleset.xml which looks somthing like:

---

ruleset.xml

---

<?xml version="1.0" encoding="utf-8"?>
<RULESET xmlns="http://www.red-gate.com/CommandLine/ExceptionHunter/ExceptionHunterResultsRuleset.xsd">
	<!-- Ignore all System.ArgumentOutOfRangeException -->
	<IGNOREEXCEPTION class="System.ArgumentOutOfRangeException" />
	<!-- Ignore all System.OutOfMemoryException -->
	<IGNOREEXCEPTION class="System.OutOfMemoryException" />
</RULESET>

---

The second file you need to create is Summary.XSLT

---

summary.xslt

---

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" 
	xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
	xmlns:eh="http://www.red-gate.com/CommandLine/ExceptionHunter/ExceptionHunterResults.xsd" 
	xmlns:rs="http://www.red-gate.com/CommandLine/ExceptionHunter/ExceptionHunterResultsRuleset.xsd">
	
	<xsl:output method="html" version="4.0" encoding="iso-8859-1" indent="yes"/>
	<xsl:param name="css">ExceptionHunter.css</xsl:param>

	<xsl:template match="/">
		<html>
			<head>
				<title>Exception Hunter Report</title>	
				<link type="text/css" rel="stylesheet"><xsl:attribute name="href"><xsl:value-of select="$css"/></xsl:attribute></link>
			</head>
			<body>	
			 	<xsl:apply-templates select="*" />
			</body>
		</html>
	</xsl:template>

	<xsl:template match="eh:ExceptionHunter">
		<h1>Exception Hunter Results</h1>
		<!-- Output Toc -->
		<div class="classtoc">
			<xsl:apply-templates select="." mode="classtoc" />
		</div>
		
		<!-- Loop through each class and output class -->
		<xsl:for-each select="eh:Result/eh:Method[not(@class=preceding::eh:Result/eh:Method/@class)]">
			<xsl:sort case-order="upper-first" order="ascending" select="@class"/>
			<xsl:call-template name="process-class">
				<xsl:with-param name="className" select="@class" />
			</xsl:call-template>
		</xsl:for-each>
		
		
		<xsl:apply-templates select="*" />
	</xsl:template>

	<xsl:template match="eh:ExceptionHunter" mode="classtoc">
		<ul>
		<xsl:for-each select="eh:Result/eh:Method[not(@class=preceding::eh:Result/eh:Method/@class)]">
			<xsl:sort case-order="upper-first" order="ascending" select="@class"/>
			<li><a><xsl:attribute name="href"><xsl:text>#</xsl:text><xsl:value-of select="@class" /></xsl:attribute><xsl:value-of select="@class" /></a></li>	
		</xsl:for-each>
		</ul>
	</xsl:template>

	<xsl:template name="process-class">
		<xsl:param name="className"/>
		<a><xsl:attribute name="name"><xsl:value-of select="$className"/></xsl:attribute></a>
		<H2><xsl:value-of select="$className" /></H2>
		
		<!-- Loop through each Result for this class -->
		<xsl:for-each select="//eh:Result[eh:Method/@class = $className]">
			<H3><xsl:value-of select="eh:Method/@name" /><xsl:apply-templates select="eh:Method" mode="parameterList"/></H3>
			<div class="ExceptionList">
					<xsl:choose>
						<xsl:when test="count(eh:Exceptions/eh:Exception) > 0">
							<!-- BUG: If we then exclude all exceptions through the ruleset.xml we do not output No Exceptions text -->
							<ul>
									<xsl:apply-templates select="eh:Exceptions/eh:Exception" mode="exceptionlist"/>
							</ul>
						</xsl:when>
						<xsl:otherwise>
							<div class="noExceptionText">					
								No Exceptions
							</div>
						</xsl:otherwise>
					</xsl:choose>
			</div>
		</xsl:for-each>
		
	</xsl:template>
 
 	<xsl:template match="eh:Exception" mode="exceptionlist">
 		<xsl:variable name="exclass" select="@class"/>

		<xsl:choose> 		
			<xsl:when test="count(document('ruleset.xml')//rs:IGNOREEXCEPTION[@class=$exclass]) = 0">
				<li><xsl:value-of select="@class"/></li>
			</xsl:when>		
		</xsl:choose>
 	</xsl:template>
 	
<xsl:template match="eh:Method" mode="parameterList">
 		<xsl:text>(</xsl:text>
		<xsl:for-each select="eh:Parameter">
			<xsl:value-of select="@class" />
			<xsl:choose>
				<xsl:when test="count(following-sibling::eh:Parameter) = 0"/>
				<xsl:otherwise>
					<xsl:text>, </xsl:text>
				</xsl:otherwise>
			</xsl:choose>
		</xsl:for-each>
		<xsl:text>)</xsl:text> 		
 	</xsl:template> 
</xsl:stylesheet>

---

Once you have these two files you need to run Exception Hunter in the following manner:

hunt /assembly:Mono.Cecil.dll /all /xml:output.xml

You then need to run the XSLT transform:

msxsl test.xml Summary.xslt > output.html

If this sort of thing would be useful I can extend it to deal with more fine grained rulesets.

James

James Moore

Paul,

Your idea about documented framework/component exceptions is a good one - this will require a few changes to the engine to work.

As for generating documentation for your assembly and checking the documentation is correct we should be able to do that with some more XSLT (perhaps in conjunction with the first change).

I will have a look into this as soon as I get a chance to.

James

Paul Wardle

Thanks James! I will try these exclusions when I next have chance. Have a Happy Xmas...!