I’m So Sick of Testing and Sorting Through Logs by Hand

Not that kind of logs, silly!

Software testing is a very important part of releasing any product. After all, no one wants a big buggy product. (Especially when it can ruin your whole mission.) On the other hand though, testing software is boring. I’d much rather be writing software than testing it. (Besides, my code never has bugs! *sarcasm*)

When I’m working on a project for school or work, I usually spend a lot more time testing and tracking down bugs rather than coding (80% of effort on 20% of work kind, of thing). I usually try to give my code pretty good test coverage, but its tedious to run through a large set of tests, especially just remembering them all.

To help with this, I wrote a Python tool to run my tests and then display the results visually using HTML, rather than a log file or something similar.

Let me just start off saying that while I’ve heard of JUnit and EUnit, I’ve never really taken the time to learn to use them. As such, if you read my post and think this or another framework would be better than my tool, leave me a comment and let me know!

Anyways, I was browsing the internet today and came along the Python doctest module, which “searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown.” Essentially, it will do that 20 minutes of typing tests for me and (AND!) without the typos that I inevitably make. This was pretty exciting, so I fired up a Python shell, copied the example file, ran it and… nothing.

If all the tests pass, the tool just shuts up! How refreshing. (I’m not being sarcastic here, I really like this approach. Very close to the Unix Rule of Repair.) That’s good, but what about when the tests fail. I made a few changes to the expected output to guarantee that a few tests fail and this is the resulting output:

**********************************************************************
File "tester.py", line 6, in __main__
Failed example:
 factorial(5)
Expected:
 12
Got:
 120
**********************************************************************
File "tester.py", line 18, in __main__.factorial
Failed example:
 [factorial(long(n)) for n in range(6)]
Expected:
 [0, 1, 2, 6, 24, 120]
Got:
 [1, 1, 2, 6, 24, 120]
**********************************************************************
File "tester.py", line 20, in __main__.factorial
Failed example:
 factorial(30)
Expected:
 26552859812191058636308480000000L
Got:
 265252859812191058636308480000000L
**********************************************************************
File "tester.py", line 30, in __main__.factorial
Failed example:
 factorial(30.1)
Exception raised:
 Traceback (most recent call last):
 File "C:\Python26\lib\doctest.py", line 1241, in __run
 compileflags, 1) in test.globs
 File "<doctest __main__.factorial[5]>", line 1, in <module>
 factorial(30.1)
 File "tester.py", line 48, in factorial
 raise ValueError("n must be exact integer")
 ValueError: n must be exact integer
**********************************************************************
2 items had failures:
 1 of   1 in __main__
 3 of   8 in __main__.factorial
***Test Failed*** 4 failures.

Pretty informative right? Little bit of an eyesore though, especially for exceptions. This would get tedious to read if you had a few dozen failed tests. Because of this, I started thinking about ways that I could parse the data and display it as a website for easier viewing. I fought a long and hard battle with the Unix shell, sed, awk, and regular expressions trying to get them to do what I wanted, but to no avail. Then I noticed that I could just provide my own implementation of a few key functions to do what I wanted, namely the DocTestRunner class. I was definately making things harder than necessary.

The DocTestRunner class… Oh boy was that confusing. If it wasn’t for a blog entry by André Roberge with some sample code on how to do this, I don’t think I could have figured it out. Below you can find the code that I came up with on how to generate the data set to make the HTML page.

class MyDocTestRunner(doctest.DocTestRunner):

	def report_success(self, out, test, example, got):
		self.Successes += 1

	def report_failure(self, out, test, example, got):
                self.Failures += 1
		failureReport = [str(test.filename), example.source, example.lineno, example.want, got]
		self.failureReport += [failureReport]

	def summarize(self, verbose=False):
		print "Summary:"
		print "Successes: " + str(self.Successes)
		print "Failures: " + str(self.Failures)

if __name__ == "__main__":
	doctest.DocTestRunner = MyDocTestRunner

	doctest.DocTestRunner.Failures = 0
	doctest.DocTestRunner.Successes = 0
	doctest.DocTestRunner.failureReport = []

	doctest.testmod(example, verbose=False, exclude_empty=False)

The code above overrides the various reporting messages that are called once testmod() is invoked. The way I approached it, I created a global list containing the relevant data for each test that failed. As tests are run, new data is added. This list is then passed on to the next stage of the program to generate the HTML document.

Note that references to the example module is where I stored tests at. In the future, I plan to allow the module to be supplied on the command line instead of always using example.

The code to generate the HTML is:

def printHTMLHeader(header):
 print """
<html>
<head>
	<link>
	<link href="styles.css" rel="stylesheet" type="text/css">
</link>
<title>
 """
 print header
 print """
</title>
</head>
<body>
 """

def printHTMLFooter():
 print """
</body>
</html>
 """

printHTMLHeader("Failure Report")
doctest.testmod(example, verbose=False, exclude_empty=False)
for failure in doctest.DocTestRunner.failureReport:
 print """
<table border=\"1px\">
<tr>
<td>
<table width=\"100%\" border=\"1px\">
<tr style="font-weight:bold;">
<td>Filename</td>
<td>Source</td>
<td>Line Number</td>
</tr>
</td>
"""

 print "
<tr>"
 for i in range(3):
    print "
<td>" + string.strip(str(failure[i])) + "</td>
"
    print """</tr>
</table>
</td>
</tr>
<tr>
<td>
    """

 print "
<table width=\"100%\" border=\"1px\">"
 print """
<tr style="font-weight:bold;">
<td>Expected</td>
<td>Actual</td>
</tr>
<tr>
 """
 for i in range(2):
   print "
<td>" + string.strip(str(failure[i+3])) + "</td>
"

 print """</tr>
</table>
</td>
</tr>
</table>
"""

print "Done"

printHTMLFooter()

You can view the output file here.

Here is a screenshot of the auto-generated web page

Now I won’t sit here and pretend that the above HTML is pretty, but I had a lot harder job doing the parsing than I expected I would, so a simple HTML table seemed fine. Later, I might try to improve the appearance of it. A little color would go a long way.

doctest is a cool module, but it only supports Python code by default. However, most of my projects use something other than Python. To run tests on programs not written in Python, I can use a Python snippet I wrote a few posts ago. It runs a command from the shell and captures stdout and stderr. Here it is again:

# Runs a system command from the command line
# Captures and returns both stdout and stderr as an array, in that respective order
def doSystemCommand(commandText):
   p = subprocess.Popen(commandText, shell=True, bufsize=1024, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
   p.wait()
   stdout = p.stdout
   stderr = p.stderr
   return [stdout.read(),stderr.read()]

Full code for this tool and the example.py file (which contains the test) are available here.

Despite how unattractive the web page is, I find it to be a lot more pleasant than reading a bunch of command line output. Hopefully this tool will help me speed up development and become better at testing my programs.