Sureness, or how many tests do we need?

Sureness, or how many tests do we need?

Imagine that you got a task after your colleague. He implemented it just before leaving for a vacation. Now it is your job to finish and release it. It might practically any task solution.

Example: To make this example more practical, lets say that the task is to count reactions that are stored in pageReactionsRepository. This is a function he implemented: (this code is used for reactions for instance here)

suspend fun getReactionsForPage(pageKey: String, userUuid: String?): PageReactionsResponse =
        .let { reactions ->
                pageKey = pageKey,
                reactionsCount = emptyReactionsMap + reactions.groupingBy { it.reaction }.eachCount(),
                userChoice = userUuid?.let { reactions.find { it.userUuid == userUuid } }?.reaction

Is this solution correct? Think about it for a minute. After analysing above code you might conclude that this solution seems correct, but are you sure about it? Even a better question would be: what is your degree of sureness that this solution is correct? There is one thought experiment that help us establish that. How much money would you bet on this function correctness?

Releasing is always a bet

Think of a release as a bet. You need to put money on correctness and if you are wrong you are loosing it all. It won’t be a fair bet though. If you are right, you win nothing. If you are wrong, you need to pay. A third alternative is to spend more of your precious time and investigate.

So now, would you bet 10 cents on this solution? Yeah, why not. Would you bet 10 dollars? Some readers probably would, but I wouldn’t and from my observations most experienced developers wouldn’t neither as we’ve seen enough good looking code with surprising outcome (we sometimes collect such snippets as puzzlers). Would you bet 1000 dollars on this solution working absolutely correctly? I think no one sane would.

This game design is no coincidence. It simulates you from your employers perspective. Assuming there are no other verification mechanisms this is how this would work - in one hand you can spend more time which is valuable, on the other a bug on production can be costly. Sure, in most cases there are some testers on the way what complicates the picture. But depending too heavily on testers is dangerous and the fact that we passed a bug is already a risk. We will talk about it later.

The stake is different in different products and functionalities

Most mistakes can be fixed. The biggest cost are:

  • Users’ discontent that can negatively influence their behaviour and your company press.
  • Users’ inability to use your product, so to leave money

This cost is much smaller for small companies or start-ups. It is also much smaller for internal products (for our own employees). This is why they care less about proper testing. On the other hand, big products known publicly are under much higher pressure. In Allegro, where I work, some mistakes might cost the company thousands or even millions of dollars. This is why we treat testing seriously.

We build sureness via different forms of checking

So let’s assume this is a 100 dollars bet. What would you do to make sure it works fine? This is where we start the process of checking.

Note: Checking and testing is not the same, but those 2 terms are often confused. People often call testing what should be called checking. I will not play language nazi here, and I stick with the “software testing” as a subcategory of checking.

The first intuitive step is to check it out manually. The steps are:

  1. Use this functionality and see how does it behave
  2. If it seems correct, think about possible problems and check them out. So we try to emulate a situation that might not be handled well and we see what happens. It anything catches our attention we stop and try to fix it. If not we feel more confident that our functionality works fine.

This is what many programmers do, but it works well only for short term. The problem is that it does not scale well.

We can still loose the bet in the future - we need automatic testing

When I was still a student, a friend of mine, who worked in a big corporation, complained to me that other people write some components and they do not leave any tests. “This is extremely immature” he said “Later I need to touch it to introduce some change, and I have no idea if I broke something or not”. This system couldn’t be tested manually. It was (and is) too big and I am not sure if there is a single person who understands it all. Even if it could, it would take preposterous amount of time for each small change.

This is the answer to the question why hiring many manual testers is not a good long-term solution (note: Even though it is practiced by many companies in Poland where you can cheaply hire many students.). It does not scale well. The bigger your system the more manual testers you need to maintain.

We need automatic testing - testing we can run again and again and checks all main functionalities.

It is not free as it needs to be written, and it needs to be maintained (when we need logic in the system, we need to update tests as well). In the end if your project is big and changes over time, it will surely pay off.

Lack of tests leads to anxious developers and terrible code

Having properly tested code is very important for us, developers. Without it we feel anxious when operating on it. And we should be, a small change might break who knows what. This have further consequences. When developers are worried to touch the code, and the project is too big for them to comprehend and test it all, they will do minimal amount of changes possible. They will not refactor the legacy code. As a consequence:

  • Components look worse and worse over time
  • We will soon have outdates system design, not reflecting how our system works now
  • Code will be less and less understand by developers

Testing takes time and skills

Those are, I believe, the most important consequences of testing on developers and code. They are not the only one. Some are not so great, like:

  • Proper testing takes a lot of time
  • Testing requires adjustments in architecture, what is both positive (it is more clean) and negative (it is more complicated)
  • Testing well is hard and needs to be learned - For me unit testing is the hardest part of the development. It is easy to write tests that test little and constrain our code a lot. It is hard to do the other way around.

It is also said, that if we first think about tests before implementation, we design both tests and components differently. Probably in a better way, but who knows for sure.

Testing is not easy, but it is worth to learn it. It takes time, but it will pay off later. The thing is that we need to learn to write proper tests in a proper amount so they serve us well.

We need to stay in touch with tests

Another trick teams do when developers do not want to write code is hiring a tester to write automatic tests. It works… to a certain degree. He might be very helpful by writing end-to-end tests for you, but they are not enough as we will see later. Also we should also not fully resign from testing ourselves.

As creators, we need to know our product and how it is used well. If we don’t, we loose contact with it. This should never happen for many reasons (decision making, system design, operating on the project). Testing is actually one of the best ways to stay in touch with the system.

Do not send unchecked functionality to a tester

One terrible mistake I see some developers doing is sending unchecked code to testers without even running it. It is nearly sure it will be back with some mistakes. The time developer saved on not checking will be wasted on getting back to the task and starting it again. He also wasted another person’s time. And increased a risk of a problem release to production. Tester should be the second person to check something out. Not the only one. This should never happen.

The differences between testing system and testing components - We need to unit test components ourselves

Tester can write end-to-end tests on the system. Having it well tested we can have some degree of sureness our system behaves correctly. Though it does not mean that its components work correctly.

A few times I had a funny situation where mistakes in 2 components cancelled each other and whole system worked fine. After fixing one of them unexpected behaviour revealed itself. A much common problem is that component is not correctly implemented but the way we use it do not reveal it. Although if we use this component somewhere else we might have a problem. This causes a cascade of problems. Refactorization then might be much harder.

Another thing is that when we test components we know better what extreme situations might break them.

Finally, unit tests are generally much faster, and they are a perfect tool we can use when we develop a solution to check out what is the status.


(for summary read headlines)

I hope you have some idea how many tests do you need.

The most important question you should ask yourself is: “How much money would I bet on this solution working correctly”. This is your amount of sureness that this test works now.

You should also ask yourself “How much money would I bet on this solution not breaking in the future”. This is your degree of sureness that it is properly automated tested.

We use tests to answer important questions for now and for the future. We trade time for answers. So automated tests should cover everything that is important for us to know. Do not depend on “code looks fine”. The code might change. We need enough automated tests to confidently make changes in your code, and be able to quickly verify that it still works fine. Although in some cases it might be reasonable to not answer all the questions. Writing good tests is an are of making decisions, keeping in mind your employer best interest.

What do you think about the article?
Marcin The author of the Effective Kotlin and Android Development in Kotlin books, founder of the Kt. Academy and Learning-Driven, programming trainer, speaker at international conferences, experienced developer.
comments powered by Disqus