I've recently blogged about CoverMonkey, a code coverage tool for Mozilla's SpiderMonkey JavaScript interpreter and my JavaScript HTML parser.
One of the really useful things you can do with a code coverage tool is check whether your test suites are actually exercising all of the code you're trying to test. (A hat tip here to my colleague Donovan Preston who insisted on good tests for dom.js-CoverMonkey was his idea.) There is already a very good test suite for HTML parsers and I was able to get my HTML parser to pass those tests.
That test suite is large and comprehensive, but when I ran CoverMonkey over the tests I found that there about 100 lines (out of some 5000) in my parser that did not get run by the tests. These were for pretty obscure things. Like what happens if you put a <!DOCTYPE> after <body>, or what happens if you have a NUL (\u0000) character in a <script> tag or what if you use ≂̸ character reference, which expands to two 16-bit characters rather than just one) in RCDATA state (like in a <xmp> element) ?
I ended up writing 65 test cases to improve code coverage for my parser up to almost 100%. (There is still some untested code for handling <noscript> tags when scripting is disabled because the way the test suite is set up does not allow testing with scripting disabled). Anne van Kesteren was kind enough to grant me commit rights to the html5lib repo, and my tests are now part of the suite: here and here.
I'm pretty pleased with this. It is a small thing, and there are only a handful of developers who will ever need to run these particular tests, but being able to use CoverMonkey to find and fix these few remaining holes in an otherwise excellent set of tests was pretty cool.
(If you are one of those handful of developers who might actually run these tests, note that my parser does not do validation and does not report parse errors. My tokenizer test cases include parse errors where I think they should be reported, but I can't actually verify with my own parser that there are supposed to be errors in those spots. So the tokenizer tests might have errors related to error reporting. The parser test cases also do not include any errors, but I doubt that will break things for anyone, since the errors in the parser tests aren't standardized in any way and can't really be compared.)
So, the moral of the story is: validating your test suites with a code coverage tool really works!
@__DavidFlanagan
This is a really great post. :-)
The thing I like the most about using a coverage tool to increase coverage is how fun it is trying to figure out how to write a simple program to trigger some rare case and cover some weird branch. The feedback loop of modifying a test and seeing the coverage change gives a deep intuitive sense of how the code works.
Sweet!