• Vespucci Unit Testing Proposal

last modified February 2, 2008 by tcoulter

// UNDER CONSTRUCTION // 

Motivation

From a code perspective, Vespucci is getting big enough to warrant a unit testing framework.The disagreement in which framework to choose -- and how to implement that framework within our codebase -- is the motivation for this proposal.  

Current Implementation / Problem

From my interactions with OpenLayers and TOPP developers, it seems there's a large reliance on Test.AnotherWay for unit testing the OpenLayers codebase. By induction, then, the default action when choosing a unit testing framework for Vespucci would be to use what OpenLayers is currently using. Although I think OpenLayers' current framework -- Test.AnotherWay -- solves many of the problems a unit testing framework is expected to solve, I think it is deficient in ways that make it unsuitable as a long term solution.

In the red corner... 

Although this proposal is a proposal to simply move away from Test.AnotherWay, it should be noted that this is also a proposal to move toward JSUnit, another javascript unit testing framework. Since it appears as though these are the only two frameworks to catch the interest of those involved, only these two frameworks will be evaluated in this proposal.

Benefits

The benefits of each framework are highlighted below.

Test.AnotherWay... 

  • Is the current framework used by OpenLayers. OpenLayers developers would be familiar with it if it were used for Vespucci.
  • Provides an in-browser test runner to make sure tests pass in multiple browsers (IE, FF, Safari, etc.).
  • Provides batch-running of multiple tests without human intervention.
  • Provides jailed test execution of test files so names of test functions won't conflict across files.
  • Provides basic assertion methods for testing: ok (boolean assertion method), fail (automatic failure), eq (equality). See here.
  • Provides a "planning" method for telling the framework that it should notify failure if a specified number of assertions were not made.
  • Provides simulation of mouse input.
     

JSUnit...  

Note: The links provided for JSUnit's documentation are direct links to frames contained within the full documentation page. You can find the full page here. Click on "Documentation" in the upper right.

  • Is a popular framework in the style of JUnit and other xUnit frameworks. Developers who have used these frameworks will be familiar with JSUnit if it were to be used for Vespucci.
  • Provides an in-browser test runner to make sure tests pass in multiple browsers (IE, FF, Safari, etc.).
  • Provides batch-running of multiple tests without human intervention.
  • Provides jailed test execution of test files so names of test functions won't conflict across files.
  • Provides basic assertion methods for testing: assert (boolean assertion method), fail (automatic failure), assertEquals (equality). See here.
  • Provides extra assertion methods that increase readability of test methods. For instance, assertFalse: This is the same as saying assert(!condition), however, it explicitly states that the condition must be false.
  • Provides setUp() and tearDown() methods to reduce duplication in test files.
  • Provides the concept of test suites, where test files can be grouped together in logical sets. Click here. View the source for an example.
  • Provides ways to run the same tests with multiple page configurations by including the test functions in a separate file.
  • Provides unit testing support for OS/browser combinations that don't support automatic test-detection.
  • Provides a concise overview of test results when tests are finished running.
  • Provides Eclipse integration.
  • Runs unit tests in a random order, so tests wont inadvertently (and inconspicuously) rely on previous tests that were run before it.
  • Has active Development. See the SourceForge page: http://sourceforge.net/projects/jsunit/
  • Contains unit tests for itself to make sure it is functioning correctly. From a high-level perspective this may superficial (who says the tests are good tests?), but it bodes well for continued use of JSUnit.

Disadvantanges

The disadvantages of each framework are highlighted below.

Test.AnotherWay...

  • Does not provide setUp() or tearDown() methods, or provide clear and explicit assertion functions. Both of these disadvantages encourage the writing of tests files with large amounts duplicated code, as well as less clear test functions. For example, notice the large amount of duplication between tests for OpenLayers' Util class. Also, count how many times the map is initialized in tests for OpenLayers' DragFeature control.
  • Does not provide test suites. Instead, test files must simply co-exist within the same list -- list-files.html -- which becomes unwieldy as the number of test files increases.
  • Only supports the most popular OS/browser combinations (IE 5.5 and 6.0, Mozilla/Firefox, Opera 7.5 and 8). Although this usually suffices, if support for other OS/browser combinations are desired in the future, our tests run the risk of needing to be rewritten in light of new requirements.
  • Does not run tests in random order. This increases the risk of bugs in our test code (ie., when our tests are wrong, and not the program's code), where tests are written under the assumption that they are self contained, when in reality they rely on an external factor that is congruent with the order in which the tests were run.
  • Is convoluted, and apparently, is wholly contained within the run-tests.html file. This prevents any -- or, almost all -- changes to the test framework that may be desired by Vespucci developers.
  • Does not appear to be in active development.

JSUnit...

  • Does not provide a planning method.
  • Does not provide simulation of mouse input.
  • To my knowledge, has not been used by OpenLayers developers.

Compare and Contrast

From a simple "bullet count" of points above, it would appear that JSUnit has more benefits than Test.AnotherWay, as well as less disadvantages. However, it's not the number of each that matter, but the qualitative aspects of each that must be considered.

As already mentioned above, Test.AnotherWay provides the basic necessities of a javascript unit testing framework: It has assertion methods, it has an in-browser test runner, and in general, it sees test files as concise, separate sets of test functions. However, its disadvantages -- and even some of its advantages -- encourage tests that are rigid, and are hard to maintain over the lifespan of the project.

Point (1): Test.AnotherWay's lack of setUp() and tearDown() methods encourage code duplication.

This point has likely already been communicated in Test.AnotherWay's list of disadvantages; however, I wanted analyse what a developer must do if she wishes to remove duplication using Test.AnotherWay as a test framework.

In OpenLayers DragFeatureControl tests, the map variable and the layer variable are initialized at the beginning of almost every function. This is duplication. There can arguably be worse duplication, but this will suffice as an example. Say, to remove this duplication, we pull the lines of code into their own setUp() method. We'd get code that looks like this: ­

var map;
var layer;

function setUp() {
	map = new OpenLayers.Map("map");
	layer = new OpenLayers.Layer.Vector();
}

function test_Control_DragFeature_activate(t) {
	setUp();
	// The actual test code...

// More test functions that look like the one above...

Although this looks good, and appears to remove most of the duplication, it has an important code smell: more duplication! The call to setUp() has to be duplicated at the beginning of every test function. Although it isn't horrible for this one small instance, as the number of tests in this file grows -- and, as the number of files that use this paradigm grow -- the more this call to setUp gets duplicated all throughout the test code. If I had control of all the code, including the code in the unit testing framework, this would be smell enough for me to add functionality that calls the setUp() method from the framework level. This way, it'd be optional, and there'd be no chance of "forgetting to call it" in test files that contain a setUp() method.

Point (2): Multiple, explicit assertion methods make test code more readable.

This one I think is best explained by example. In the box below, the text in the middle is a set of assertions written for Test.AnotherWay, and the text on the right is assertions written for JSUnit. A description in English of what both of these mean is contained on the left.

Assert condition is true. t.ok(condition) assert(condition)
Assert condition is true.  t.ok(condition) assertTrue(condition)
Assert condition is false. t.ok(!condition) assertFalse(condition)
Assert items are equal. t.eq(item1, item2) assertEquals(item1, item2)
Assert items are not equal. t.ok(item1 != item2) assertNotEquals(item1, item2)
Assert item is null. t.ok(item == null) assertNull(item)
Assert item is not null. t.ok(item != null) assertNotNull(item)
Assert item is undefined. t.ok(!item) ??? assertUndefined(item)
Assert item is not undefined. t.ok(!!item) ??? assertNotUndefined(item) 
Assert item is "not a number". ??? assertNaN(item)
Assert item is not "not a number". ??? assertNotNaN(item)
Mark this test as failed. t.fail() fail()
­
From the code above, is it apparent which set of functions is more readable? Aside from JSUnit having two ways to "assert a condition is true", it appears as though JSUnit wins out.


Point (3): Test.AnotherWay's plan() method is actually detrimental to writing reusable test code.

A quote from the Test.AnotherWay documentation:

If you can't tell in advance how many assertions your test function is going to make, or if you are too lazy
to count, you may omit the plan method call. This weakens your tests, so such tests are marked with "no plan"
warning in the results pane.­

The quote above assumes that "not being able to tell how many assertions your code makes" is some inherent  weakness inside a   . It assumes that the number of assertions that a test makes

­