Hello there, tech readers.
I have a question to ask, and I hope one of you out there can give me a good answer to it.
So I have recently been introduced to Unit Testing as a development process, as part of the Software Engineering course I just took at Penn. Now, firstly we didn’t go into very detailed technical aspects of how Unit tests are done, but I did get a chance to write tests while I wrote code, and I found it to be incredibly useful. During the course I also wrote PHPUnit tests for the CMI/IARCS online judge, and, while I think PHPUnit is better than nothing, it still sucks for data-driven applications like the CMI judge.
Now for one thing, PHPUnit is buggy. But I think the major issue is that the entire model (which I think, is called the xUnit model) is crappy for certain types of applications, especially Web Applications.
I also think, that the model is pretty useless for event driven applications like most desktop applications written in C, since it’s going to be ridiculously messy to wait on events and callbacks to proceed the test. On the other hand, most Java programs would probably have a thread for doing anything that would block, and while I would’ve called that absolutely stupid until a month back, I can see why that’s so useful for writing unit tests.
In fact, Java makes unit tests bloody easy even without a framework. We wrote a fairly complex Java program last Spring as part of the Software Systems course, and without knowing anything about Unit testing, writing tests were obvious every time I wrote a new component: public static void main!
I’ve been thinking about how I would like to implement a Unit testing framework for the OPC judge, and I did come up with the following framework, and I do believe that it’s a fairly obvious way of going about it, and that one of the hundreds of unit testing frameworks should provide something like this, and that one you readers can point me to it.
So, first I’d like to argue that we should be able to define the state of the system well. In unit testing terminology, this might be called “fixtures”. There are two parts of the fixture, one is the static data stored by the program before it started, and then a series of actions that changed the system to its present state.
For example in PHP, the state would be [database] x [$_SESSION], and here the $_SESSION is usually small enough. These are the only two things that are remembered across two calls to the Apache server. (In this situation, you can imagine a second user changing the database at the same time, but for the purpose of testing, let’s skip these situations.)
In event driven, single threaded programs, the state is actually very easily defined. Each time you’re in the event loop, waiting for the next event, we have a state. I wouldn’t be concerned much (with some exceptions) about intermediate states. It’s these states where the program is doing nothing that has to be tested.
(In multi-threaded programs, talking of state is pretty useless, so in that case, we should stick to the xUnit style of testing.)
Once you have the state, you can define actions. Actions may or may not change the state. Each unit test would be of the following form: set up the fixture, run a few actions to bring the system state to a state in which we want to test the system, and finally test the action we’re interested in.
Unfortunately, as a developer, I find myself not liking the idea of building the fixture for a particular test. It’s painful, you have to hand compute the results, errors, and any small change to the dataset changes the answers, and so on. So the first thing I want of my testing framework, is that I should be able to write tests without fully knowing how the fixture looks like. But the actions depend on the fixture itself. So e.g., I want to test that the problems/ page works on the IARCS judge. depends on the data. I would should be able to write something like:
$problemname = get_value_for_situation ("problem-name", "Any problem name in the database, maybe one that uses images, so that we can test it");
< sanity checks go here >
Now this is independent of the fixture. So if a user wants to create a new test based on this for a new fixture (or for a new value of problem-name), he might do something like:
$ create-test test-problems-page.php <fixturedir>
and now, the testing framework asks the user to input a value for “problem-name”, and stores this answer in a testing database permanently (which should not be part of the code repository, but you could imagine a central repository, where testers can push their tests to).
This can go on. Answers can depend on the fixture too, for instance if in the above case an Exception is thrown while opening “problems/$problemname”, then the framework should ask the tester:
An exception [ContestNotStartedException] was thrown, is this the expected behavior [y/n] ?
..and the answer goes into the database.
It could be more complex. The tester could write specific tests for his fixture:
The test passed with no assertions failing. Do you want to test anything else [y/n]? y
> assert ($output->xpathQuery ("//img[@href="image1.jpg]").count != 0)
.. and this code goes into the database.
Event driven C programs
The second thing I want in a framework is specific to event driven programs, and very specific to the libraries/languages used. There are some unit tests that I would actually like to write for my Soc 2009 code for Pidgin.
A simple concept that we learnt in Software Engineering was “Design to Test,” which makes lots of sense. Therefore, I believe it makes sense for a C program to emit signals specifically to indicate that some action has completed successfully or changed the state, even if that signal is not intended to be used by other code.
For instance, if I had to rewrite my GoogleSource implementation in Liferea, I should be able to write unit tests that look like:
<initialize the GoogleSource subscription>
subscription_update (subscription); /* asynchronous network interactions */
g_wait_on_signal (subscription, "update-complete", &xml);
<validate the XML >
[technical detail: This scheme may not be easy to implement. One way is to keep continue the event loop within the wait_on_signal function. I’m not sure about the details of this, and I might be wrong. The other is to keep this entire code in another thread, and “subscription_update” should would not be an internal API call, but a “top-level” “action”.]
This should make it possible to write linear tests within the event driven model, while keeping the test code as an observer to the state of the system.
I stop here. I’m getting lazy, and I still have work to do on my Pidgin Webkit plugin, and so I’m not proof reading this. Hopefully, even though most of you didn’t reach till this paragraph, this documents some of the ideas I had for how a unit test system should behave. And even better, if you know some unit test framework that I can modify to suit my needs, do let me know.