Testing Best Practices
This document will cover important things to keep in mind when writing unit tests. Since it's in the context of opencore, it will be biased towards writing doctests with zope, since that's most of what we do here. However, many of the concepts should be generally applicable.
So what should unit tests do?
Unit tests should exercise a piece of code. Ideally, each function written will have tests that cover all possible inputs, and expected responses. This includes the normal successful case, but arguably more importantly, the failure, or edge cases. Bugs are usually caused by functions that get unexpected inputs, or give unexpected results to certain inputs. Therefore it's important to verify that the code works as expected given the range of inputs it can receive.
Now given that python is such a dynamic language, it is virtually impossible to cover the range of inputs that a function can receive. However, there is a sweet spot that will give you most bang for your buck. Testing each range or edge case should give reasonable confidence with the function.
An example should help clear this up. Let's say we wanted to write a function that will find the average of a list of numbers. We write:
def average(lst): return sum(lst) / len(lst)Now when writing tests for this function, we should consider the possible cases, and expected behaviors. Good inputs are range(1, 6), or [1, 3, 5]. It is important to include tests for the failure case as well. In this example, passing an empty list is one edge case. What should happen? Maybe it's alright that a ZeroDivisionError gets raised. Or maybe it should return -1. Or None. The important thing however, is that this case is illustrated in the tests. That way the expected behavior is known and well defined. Other good cases to consider are when the numbers don't come out to an even average. Or if the numbers in the list themselves are floats. This is an overly simplistic example, but it should illustrate the type of thinking that should go on when writing tests.
Code coverageCode coverage is a simple way to tell if you don't have enough tests. Functions can contain many branches of execution, and it's important to try and hit as many of those as possible. In some cases it may be difficult to simulate the environment to trigger a particular path. This can hint at a design problem with the code. If it's difficult to test a particular piece of code, that can also mean that it's difficult to use, or reuse in different contexts. In any case, the important thing is hitting each branch of execution.
100% code coverage is also by no means proof that code works as expected. Just because each branch is getting hit, doesn't mean that the function is fully tested. In the example above, calling the function with any input will result in 100% coverage. That doesn't mean that there is adequate test coverage. But if you see blocks of code that don't have any coverage at all, that's usually a sign that there isn't enough.
Zope has a built-in coverage tool. However, the percentages don't seem to come out right. That doesn't mean that you shouldn't run it however. After writing a suite of tests, you should verify that you have decent coverage. This report can be generated with the --coverage option to the zope test runner.
zopectl> test -s opencore.whatever --coverage /path/to/where/reports/are/stored
BugsDespite valiant testing efforts, bugs still manage to creep in. When a new bug is reported, usually the best course of action is to write a unit test that causes the bug. (Make sure a trac ticket exists first!) This helps determine when a bug is fixed, and prevents it from creeping in again in the future. After a while you develop a nice regression suite that gives you reasonable confidence with the state of your system.
ProcessWhile there is some debate as to whether tests should be written before or after the code, what's more important is that the tests get written. Sometimes writing tests first helps focus the actual development. Sometimes though, it doesn't make as much sense because you can go on and on developing some cool new api that you think may be useful, but is not actually needed. For the kind of work that we do, it may be easier to first write an initial stub implementation to see what kind of information a view or page template needs. Essentially this step involves figuring out what the api should be. Then once you can define an api for what you need, you can start writing tests for it. Ideally the flow is write the failing test, verify the test fails, write the code, run the tests and see a passing test. This is helpful because there's less of a disconnect between what you think the code does and what it actually does. In practice however, I write my tests in large batches because it takes zope so long to run the tests. The good news is that if there's a problem, you can always slow down.
The code is not done unless there are tests for it!
ConsistencyIt's also important for us as an organization to be consistent. This will make it easier to read someone else's tests, and figure out what's going on. It's also important to test the various zope patterns in a similar way (adapters, utilities, views, ...)
- Your doctests should tell a story. This makes it easier to read, and serves as great documentation for the code.
- Try to be as explicit as possible. If there are weird side effects that you are counting on, mention it. If there is some special setup that is going on, say why you have to do it, and who's going to do it in "reality". The description/comments around the test should make it easy to see what's going on, and how the various situations are triggered.
- If the whole test consists of calling one method with lots of different variables, ie calling a complicated view by setting up lots of different request variables each time, that usually means that there is some refactoring that should go on, usually extract method. For example, the view __call__ method can be broken down into a simple dispatcher, with the actual work done by other methods. These other methods can then be tested in isolation. This makes is easier to pinpoint problems, manage complexity, and reuse code. Listen to the tests. In general, if it's difficult to test all the various cases that an object handles, it's a signal that it should be broken up into smaller pieces(see below).
- When testing an adapter/utility/view, instantiate it directly first. This makes it easier to follow what's going on. It also makes it easier to find the implementation, because you don't have to look anything up in the zcml. However, you must verify that going through the zope machinery gives you back what you expect. For a view, this means traversing to it. For adapters, it means performing the adaptation. And for utilities, it means calling getUtility.
- This note is more generally applicable, but above all, use common sense. We write tests because there are many ways we benefit from them. Additionally, we should be thinking about ways to improve our process overall, testing or otherwise.
Write Code that Can be Easily TestedHow we design our code greatly effect how effective our testing will be at exposing bugs or debugging issues.
- break long code sequences into functions, objects and methods
Anything inside a block that can not be called directly might be easier to test if it was in it's own callable.
- isolate behavior so you test it without alot of setup. Consider this simple view method example:
def view_method(self): val1 = self.request['key'] val2 = self.context.getVal2() newval = self.do_something_intense(val1, val2) ...
'do_something_intense' can be tested without a request or a particular context.
While we are calling the zope doctests we write unit tests, purists will call them integration or functional tests. Any time zcml is involved, and the setup is not the test itself, it becomes a functional test. The reason is because it depends on other external factors being set up properly before the test can be written.
Mocks and Stubs
These can both provide ways to help isolate the part of the system under test and reduce dependencies on context and other systems.
Both are used to replace objects or systems that you want to decouple during a test. There are several other kinds of test implementations:
... excerpted from Mocks Aren't Stubs , an interesting article (with a bias in favor of mocking)
- Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
- Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
- Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.
- Mocks are ... objects pre-programmed with expectations which form a specification of the calls they are expected to receive.
Don't test the Zope Component Architecture when you don't mean to
When you want to get a view to test, don't do this:
>>> view = self.portal.people.unrestrictedTraverse('@@view')
we don't need to be testing the component architecture framework in our zopectl tests... functional tests are the place where we can ensure that the pages are responding to the correct URLs. when testing components in the zopectl tests, please just instantiate the class directly, and set its acquisition context:
>>> view = path.to.view.class.ViewClass(self.portal.people, request) >>> view = view.__of__(self.portal.people)
...excerpted from component lookup in zopectl tests considered harmful (on opencore-dev)