Ask HN: What you think about differential unit testing?

5 points

12 years ago

Our website currently has little/no unit tests (we have selenium testing and manual testing). We would like to start testing more, but we're running into a quandary as how to start. Much of our code is not written in a way that lends itself to unit testing so to be able to start testing, we have to refactor, which might break everything.

One approach that's come up is called "differential unit testing" where I don't need to go in a write the tests - but I measure the inputs and outputs of different functions and I monitor if they changed after my code changes (at least that's how I understand it.) There's a paper about it at http://web.engr.illinois.edu/~taoxie/publications/ast07-diffut.pdf. I first encountered a version of the idea at http://zachholman.com/talk/move-fast-break-nothing/ (their case they are running both in parallel so it's easier to check each case - but the concept would be the same if I had common input cases and output cases.)

Now it's question how I implement such an approach. One way might be put in certain waypoints in my code where I can run a stack trace and save it - then use those inputs to different functions as my test cases. I save a "baseline" and then as I change my code, I run the tests, looking for changes.

I can see several issues with the approach. It might test too many versions - I don't know which are important changes to test and which are insignificant (without manual intervention.) I might actually have a broken implementation to start - and I'm comparing agains that. maybe others..

I'm not try to suggest this approach would replace more traditional unit testing - but as a form of "canary" to detect when something is not how it use to be at a more granular and earlier level than doing full integration testing.