Find each bug once

Despite our best efforts, every now and then a bug slips through. Sometimes it is caught early in our own testing, sometimes it goes all the way into production.

What happens when the bug is found? Hopefully, it is fixed, the fix is verified, and a new version is released. This happens all the time. In some organizations it is a formal process with multiple gates, in some it is a very informal one. The part which I find interesting is, if for some reason this bug was to appear again, how long time would it take to find and diagnose it?

A bug should only be found by a human once #

If a human is involved in catching a regression, I believe you have failed! In my mind, finding the reappearance of a known bug should be a completely automated process. That means, for every bug you fix there should be an automated test which will fail if the bug reappears. It doesn’t matter if that test is a unit test, an integration test, a system test, or any other kind of test. The important part is that it is automated.

I don’t know how many times I’ve had the feeling of “didn’t we have this problem a few months ago?” only to later find out “yes, we did have this problem a few months ago.”

So, my suggestion is rather simple.

For each bug you fix, write an automated test to prove it!

If you do not do this, you either waste a lot of time by doing manual regression testing, or you are unprofessional enough to let your users be your testers.

A nice side effect of this practice is that over time you build a rather robust regression test suite for known, actual bugs. The tests also give you a form of documentation for all the special case handling that is often the mark of a battle-hardened system.

Updates #

2024-04-24: Republished this post which was originally written for my previous blog.