Hello there, tech readers.
I have a question to ask, and I hope one of you out there can give me a good answer to it.
So I have recently been introduced to Unit Testing as a development process, as part of the Software Engineering course I just took at Penn. Now, firstly we didn’t go into very detailed technical aspects of how Unit tests are done, but I did get a chance to write tests while I wrote code, and I found it to be incredibly useful. During the course I also wrote PHPUnit tests for the CMI/IARCS online judge, and, while I think PHPUnit is better than nothing, it still sucks for data-driven applications like the CMI judge.
Now for one thing, PHPUnit is buggy. But I think the major issue is that the entire model (which I think, is called the xUnit model) is crappy for certain types of applications, especially Web Applications.
I also think, that the model is pretty useless for event driven applications like most desktop applications written in C, since it’s going to be ridiculously messy to wait on events and callbacks to proceed the test. On the other hand, most Java programs would probably have a thread for doing anything that would block, and while I would’ve called that absolutely stupid until a month back, I can see why that’s so useful for writing unit tests.
In fact, Java makes unit tests bloody easy even without a framework. We wrote a fairly complex Java program last Spring as part of the Software Systems course, and without knowing anything about Unit testing, writing tests were obvious every time I wrote a new component: public static void main!
I’ve been thinking about how I would like to implement a Unit testing framework for the OPC judge, and I did come up with the following framework, and I do believe that it’s a fairly obvious way of going about it, and that one of the hundreds of unit testing frameworks should provide something like this, and that one you readers can point me to it.
So, first I’d like to argue that we should be able to define the state of the system well. In unit testing terminology, this might be called “fixtures”. There are two parts of the fixture, one is the static data stored by the program before it started, and then a series of actions that changed the system to its present state.
For example in PHP, the state would be [database] x [$_SESSION], and here the $_SESSION is usually small enough. These are the only two things that are remembered across two calls to the Apache server. (In this situation, you can imagine a second user changing the database at the same time, but for the purpose of testing, let’s skip these situations.)
In event driven, single threaded programs, the state is actually very easily defined. Each time you’re in the event loop, waiting for the next event, we have a state. I wouldn’t be concerned much (with some exceptions) about intermediate states. It’s these states where the program is doing nothing that has to be tested.
(In multi-threaded programs, talking of state is pretty useless, so in that case, we should stick to the xUnit style of testing.)
Once you have the state, you can define actions. Actions may or may not change the state. Each unit test would be of the following form: set up the fixture, run a few actions to bring the system state to a state in which we want to test the system, and finally test the action we’re interested in.
Unfortunately, as a developer, I find myself not liking the idea of building the fixture for a particular test. It’s painful, you have to hand compute the results, errors, and any small change to the dataset changes the answers, and so on. So the first thing I want of my testing framework, is that I should be able to write tests without fully knowing how the fixture looks like. But the actions depend on the fixture itself. So e.g., I want to test that the problems/ page works on the IARCS judge. depends on the data. I would should be able to write something like:
$problemname = get_value_for_situation ("problem-name", "Any problem name in the database, maybe one that uses images, so that we can test it");
route ("problems/$problemname");
< sanity checks go here >
Now this is independent of the fixture. So if a user wants to create a new test based on this for a new fixture (or for a new value of problem-name), he might do something like:
$ create-test test-problems-page.php <fixturedir>
and now, the testing framework asks the user to input a value for “problem-name”, and stores this answer in a testing database permanently (which should not be part of the code repository, but you could imagine a central repository, where testers can push their tests to).
This can go on. Answers can depend on the fixture too, for instance if in the above case an Exception is thrown while opening “problems/$problemname”, then the framework should ask the tester:
An exception [ContestNotStartedException] was thrown, is this the expected behavior [y/n] ?
..and the answer goes into the database.
It could be more complex. The tester could write specific tests for his fixture:
The test passed with no assertions failing. Do you want to test anything else [y/n]? y
> assert ($output->xpathQuery ("//img[@href="image1.jpg]").count != 0)
.. and this code goes into the database.
Event driven C programs
The second thing I want in a framework is specific to event driven programs, and very specific to the libraries/languages used. There are some unit tests that I would actually like to write for my Soc 2009 code for Pidgin.
A simple concept that we learnt in Software Engineering was “Design to Test,” which makes lots of sense. Therefore, I believe it makes sense for a C program to emit signals specifically to indicate that some action has completed successfully or changed the state, even if that signal is not intended to be used by other code.
For instance, if I had to rewrite my GoogleSource implementation in Liferea, I should be able to write unit tests that look like:
testParseSuccess (){
<initialize the GoogleSource subscription>
subscription_update (subscription); /* asynchronous network interactions */
g_wait_on_signal (subscription, "update-complete", &xml);
<validate the XML >
}
[technical detail: This scheme may not be easy to implement. One way is to keep continue the event loop within the wait_on_signal function. I’m not sure about the details of this, and I might be wrong. The other is to keep this entire code in another thread, and “subscription_update” should would not be an internal API call, but a “top-level” “action”.]
This should make it possible to write linear tests within the event driven model, while keeping the test code as an observer to the state of the system.
I stop here. I’m getting lazy, and I still have work to do on my Pidgin Webkit plugin, and so I’m not proof reading this. Hopefully, even though most of you didn’t reach till this paragraph, this documents some of the ideas I had for how a unit test system should behave. And even better, if you know some unit test framework that I can modify to suit my needs, do let me know.
Arnie Almighty 
As far as writing unit tests for web apps go, how about using selenium or watir? I write my unit tests in firewatir (ruby/linux) and they are merging into watir. I have not yet used phpUnit but I always thought they would be more useful if i were writing php libraries rather than web apps. The paradigm here is to do the unit test from the other side of the browser.
I really didn’t follow what you want to go in the database in your contest code example but this is sort of what my unit tests look like
require 'firewatir' include FireWatir ff=Firefox.new ff.goto(url) assert( ff.text_field( :id, '1' ).exists? ) ff.text_field(:name,"login").set("User_Name") ff.text_field(:name,"passwd").set("Password") ff.button(:value,"Sign In").click assert( ff.pre( :id, 'login_success' ).exists? ) ff.link(:text, "Sign Out").click ff.closeYou could put whatever in or out of whatever database you want along the way.
Oh and I ma of course not implying that I am not using the ruby unit test framework. I mean all that goes inside a
Ruby sort of makes unit test writing way too simple to leave out.
> ff.text_field(:name,”login”).set(“User_Name”)
So, one of my issues is, where does the “User_Name” and “Password” come from? Do you hardcode it? If you hardcode it, then you have to set up the database for the test into a known state, which I find rather messy. Also, maybe you might want to write tests that tries logging in with Usernames in different languages for example, and logins which are meant to fail. Do you hardcode each one of them?
I haven’t used Selenium, but I saw it in action during my course by another team, and I think it’s rather cool.
(Btw, from your example, I like the fact that the code is linear, although the ff.button(:value, “Sign In”).click is not something that happens immediately. This is one of the things I wanted in Event driven C programs.)
Ummm.. data fixtures should be the responsibility of the model. My current favourite model framework is php doctrine. Any respectable model framework should support data fixtures.
I personally like setting up my tests differently. My first tests perform a clean deployment of the app to the local test server thus testing the app deploy scripts. Then I test the CRUD in that very order – create, retrieve, update, delete.
For instance in the above example, I would first test the user registration process which would create the user/password in question.
Umm… you lost me there. What do you mean by immediately?
The model? But you might want a different fixture for a different test case. Otherwise you have to generate one-single-one-size-fits-all fixture right in the beginning, which is impossible since you won’t know when you start writing your code as to what kind of tests you are going to write. For the CMI judge I’ve been lucky since I didn’t start writing the tests from day 1: I just took all the real world data, and am using that as my fixture. (But again, it’s a large fixture, so setting up the fixture for each and every test case is slow.) Tests can be much more than CRUD. The test has to be permanently reproducible, and should be repeatable if anybody changes your component. The general xUnit format is the following: for each and every test case, a setUp () is called which sets up the fixture into a known state, then the test is called, and finally a tearDown is called to restore the fixture if required.
Meaning, when you do a .click, it does network requests and stuff in the background. At what point does the function .click return? Only after the page is loaded right? Or does it return immediately, and you need to write an additional function saying “wait_for_page_to_load ()”. From your example it seems to wait, and that’s what I like about it.
The same thing happens in most C GUI programs, especially ones that deal with network like Pidgin and Liferea. So when you make a “request” to the application, it might not be able to process it immediately, the event loop will go on running. One of the points I was making in my blog post, was that there should be a more standardized way of knowing when an action was completed, which in C would probably be done with a signal-emit-connect mechanism.