Krzysztof Koźmic's blog

I am a believer in the value of test­ing. How­ever not all tests are equal, and actu­ally not all tests pro­vide value at all. Raise your hand if you’ve ever seen (unit) tests that tested every cor­ner case of triv­ial piece of code that’s used once in a blue moon in an obscure part of the sys­tem. Raise your other hand if that test code was not writ­ten by human but generated.

As with any type of code, test code is a lia­bil­ity. It takes time to write it, and then it takes even more time to read it and main­tain it. Con­sid­er­ing time is money, rather then blindly unit test­ing every­thing we need to con­stantly ask our­selves how do we get the best value for the money – what’s the best way to spend time writ­ing code, to write the least amount of it, to best cover the widest range of pos­si­ble fail­ures in the most main­tain­able fashion.

Notice we’re opti­mis­ing quite a few vari­ables here. We don’t want to blindly write plenty of code, we don’t want to write sloppy code, and we want the test code to prop­erly ful­fil its role as our safety net, alarm­ing us early when things are about to go belly up.

Test­ing conventions

What many peo­ple seem to find chal­leng­ing to test is con­ven­tions in their code. When all you have is a ham­mer (unit test­ing) it’s hard to hit a nail, that not only isn’t really a nail, but isn’t really explic­itly there to being with. To make mat­ters worse the com­piler is not going to help you really either. How would it know that Login­Con­troller not imple­ment­ing ICon­troller is a prob­lem? How would it know that the new depen­dency you intro­duced onto the con­troller is not reg­is­tered in your IoC con­tainer? How would it know that the pub­lic method on your NHiber­nate entity needs to be virtual?

In some cases the tool you’re using will pro­vide some level of val­i­da­tion itself. NHiber­nate knows the meth­ods ought to be vir­tual and will give you quite good excep­tion mes­sage when you set it up. You can ver­ify that quite eas­ily in a sim­ple test. Not every­thing is so black and white how­ever. One of diag­nos­tics pro­vided by Cas­tle Wind­sor is called “Poten­tially mis­con­fig­ured com­po­nents”. Notice the vague­ness of the first word. They might be mis­con­fig­ured, but not nec­es­sar­ily are – it all depends on how you’re using them and the tool itself can­not know that. How do you test that efficiently?

Enter approval testing

One pos­si­ble solu­tion to that, which we’ve been quite suc­cess­fully using on my cur­rent project is approval test­ing. The con­cept is very sim­ple. You write a test that runs pro­duc­ing an out­put. Then the out­put is reviewed by some­one, and assum­ing it’s cor­rect, it’s marked as approved and com­mit­ted to the VCS repos­i­tory. On sub­se­quent runs the out­put is gen­er­ated again, and com­pared against approved ver­sion. If they are dif­fer­ent the test fails, at which point some­one needs to review the change and either mark the new ver­sion as approved (when the change is legit­i­mate) or fix the code, if the change is a bug.

If the expla­na­tion above seems dry and abstract let’s go through an exam­ple. Wind­sor 3 intro­duced way to pro­gram­mat­i­cally access its diag­nos­tics. We can there­fore write a test look­ing through the poten­tially mis­con­fig­ured com­po­nents, so that we get noti­fied if some­thing on the list changes. I’ll be using Approval­Tests library for that.

[Test]
pub­lic void Approved_potentially_misconfigured_components()
{
    var con­tainer = new Wind­sor­Con­tainer();
    container.Install(FromAssembly.Containing<HomeController>());

    var han­dlers = GetPotentiallyMisconfiguredComponents(container);
    var mes­sage = new String­Builder();
    var inspec­tor = new DependencyInspector(message);
    fore­ach (IEx­poseDe­pen­den­cy­Info han­dler in han­dlers)
    {
        handler.ObtainDependencyDetails(inspector);
    }
    Approvals.Approve(message.ToString());
}

pri­vate sta­tic IHan­dler[] GetPotentiallyMisconfiguredComponents(WindsorContainer con­tainer)
{
    var host = container.Kernel.GetSubSystem(SubSystemConstants.DiagnosticsKey) as IDi­ag­nos­tic­sHost;
    var diag­nos­tic = host.GetDiagnostic<IPotentiallyMisconfiguredComponentsDiagnostic>();
    var han­dlers = diagnostic.Inspect();
    return han­dlers;
}

What’s impor­tant here is we’re set­ting up the con­tainer, get­ting the mis­con­fig­ured com­po­nents out of it, pro­duce read­able out­put from the list and pass­ing it down to the approval frame­work to do the rest of the job.

Now if you’ve set up the frame­work to pup-up a diff tool when the approval fails you will be greeted with some­thing like this:

You have all the power of your diff tool to inspect the change. In this case we have one new mis­con­fig­ured com­po­nent (Home­Con­troller) which has a new para­me­ter, appro­pri­ately named miss­ing­Pa­ra­me­ter that the con­tainer doesn’t know how to pro­vide to it. Now you either slap your­self in the fore­head and fix the issue, if that really is an issue, or approve that depen­dency, by copy­ing the diff chunk from the left pane to the right, approved pane. By doing the lat­ter you’re noti­fy­ing the test­ing frame­work and your team­mates that you do know what’s going on and you know it’s not an issue the way things are going to work. Cou­pled with a sen­si­ble com­mit mes­sage explain­ing why you chose to approve this dif­fer­ence you get a pretty good trail of excep­tion to the rule and rea­sons behind them.

That’s quite an ele­gant approach to a quite hard prob­lem. We’re using it for quite a few things, and it’s been giv­ing us really good value for lit­tle effort it took to write those tests, and main­tain them as we keep devel­op­ing the app, and the approved files change.

So there you have it, a new, use­ful tool in your toolbelt.