Regression Driven Tests
After years of reading authors preaching about the best tools and techniques for test driven development; and smug community leaders rolling their eyes at those who dared to write feature code without having written 3 times the amount of test code beforehand, TDD is apparently dead!
What now? No more tests?
Not so fast!
Tests vs TDD
A few years back, every software blog I read was boasting the virtues of TDD. The message was so loud that no one would have dared to question it. If you were not doing it, you were doing it wrong.
To me, TDD made testing seem like a huge time investment, but the benefits looked so great that it must have been worth it.
I read books, shopped for the best test framework and read some more. As is usual in open source software, every framework had it’s strongly opinionated following. In Ruby, RSpec seemed to have the strongest community and better documentation. A handful of resources also explained how well it integrated to Cucumber. Being familiar with Agile methodology, adopting TDD and RSpec seemed perfect.
After setting the whole thing up and writing unit tests for my first few use cases… I started pushing tests down the priority lane… I quickly stopped writing tests before writing the code and eventually gave up testing completely.
I had these fancy test suites and methodologies but wasn’t using them. Worst of all this was so complicated that writing tests did not feel natural; it was breaking my development flow.
While these tools and development methods are great for organized teams with a product that is going through the maturity phase, they can encumber the feature rollout process quite significantly for a startup team.
Tests before TDD
Before TDD there was testing. Writing no test code means having to manually test and potentially debug your code before every deployment. With limited resources, you have limited time to test.
The amount of time you put into writing test code is up to you, but you should test. No matter how good you are, if you don’t do it, you will introduce regression bugs.
Regression driven tests
If you have no tests written and don’t know where to start, starting with regression tests is a good idea.
A regression bug is introduced when you unknowingly break some previously developed functionality while developing a new feature or doing maintenance work. It does not necessarily mean that the developer is careless or that the software is poorly designed. Untested code is very prone to regression bugs.
You might know what you’re doing today and think the code you’re writing is so simple it can never break, but a year from now when you don’t recognize the code you’ve written yourself, you might tell a different story. You will eventually cause a regression and wonder what idiot implemented the now broken functionality and then have an oh… moment where you realize that you did.
When regression bugs occur, it is important to make sure that they’re taken care of for good.
They’re particularly painful because…
- They occur when you’re focused on something else
- They make you feel like you’re going backwards
- They can make your product look unreliable
- They tend to be back with a vengeance
When left untested, the odds of regression bugs coming back to haunt you are high. If you don’t reintroduce them yourself, someone else might. Extra pain points when a client experience them… repetitively.
Solution
Testing is expensive and is a hard sell to management. But once they see how destructive regression bugs can be they will agree to budget some test code time. (Or you could just go ahead unapproved, it depends on what type of shoes you’re in.)
Introducing writing test code as a disciplinary measure to those introducing a regression bug is a nice way to bring test culture to your team. You can even go as far as integrating Hip-Chat to your CI server and have a bot publicly shame those who dare to break the build!
This way, you make sure that your whole team has their test suite running locally and that they’re familiar with writing test code.
You have to be careful not to turn this into bullying, but gamifying the process can make the adoption easier.
Then what?
Should I TDD, just test or regression test?
As with every business decisions it should come down to a costs/benefits analysis. The greater the benefits of a feature being test covered (high impact in case of failure) and the lower cost of writing the tests (in resources), the higher the incentive to write tests for a feature.
In some cases writing test code before writing the feature code can help in better defining the feature. In this case go ahead and TDD the hell out of this feature.
A few questions you can ask yourself when wondering how you should test a feature :
How core is this?
Is this feature the bread and butter of your business model or is it just a nice-to-have?
If an eventual failure in this feature results in your clients inability to use your product, you must take all the necessary precautions to prevent that failure.
How deep is this?
How deep is this feature in your application stack?
If you raised an exception at different places in your application, how often would you see a stack trace line related to this feature?
i.e. In a Rails application, a rack middleware is deeper than a controller action.
What work load could this bring?
If this feature were to fail, how much time would you or a member of your team have to spend correcting the situation?
If your data gets corrupted someone is obviously going to have to spend some time through the server logs or database backups. That someone will not like it.
Keep in mind that the consequences of a bug are not necessarily equal to its gravity. Something as seemingly stupid as a button disappearing could mean several hours spent in support ticket responses.
How bad could this look?
A professional image is important; having UI glitches here and there can quickly tarnish the image your clients have of your product. You want and need your clients to be confident that your product is reliable.
Start simple
RSpec or minitest?
RSpec is great, but has a high learning curve when compared to simpler testing solutions like the Ruby Standard Library included minitest.
The more hurdles you set, the more chances you have of seeing your testing initiative being rejected. RSpec is a great testing framework but you have to learn it’s DSL before you can be confident that you’re writing your tests properly. On the other hand minitest tests are just plain Ruby classes. Rails core developer Aaron Patterson wrote a nice biased but balanced post on his blog comparing both.
Conclusion
How you test is a journey on which you should learn through experience. Try the different tools and methods and see what best fits you and your team.
Regression driven testing is a good place to start when you’ve been developing without tests for a while or when you want to introduce tests to a team that has little to no experience with them.
Whether or not you test is more a question of whether or not you want to be warned before walking on a banana peel and end up flat on your face. Some might fancy doing haute voltige without a security net, but I don’t think you should be doing this at home.