Konubinix' opinionated web of thoughts

Code Coverage

Fleeting

Code coverage is the indicator that tells how much lines of code have been seen in the tests.

Code coverage is often seen in testing, and whether you run dedicated programs to test your code (like pytest) or you run your application (like ./myapp.py) don’t matter, provided you end up with confidence that you tried corner cases of your application and that lines got executed in those situations.

Reading a code coverage is solving an inverse problem. You don’t care about the code coverage per se. You are interested in the quality of the code, and the code coverage, although not very good, is the best indicator we could find to find out whether we considered all the code in the tests.

Therefore, I would discourage saying “I need to fix the code coverage” or “I need to have a code coverage to X%”, because that would put you in the state of mind of making an objective out of the measure (Goodhart’s law). I would rather say “I need to test the code” or “I need to test all those parts of the code”. See also what implies 80% of coverage in sonar?

coverage says something about the code more than about the tests

When people realize that the code coverage has a value X%, their first intuition is to add tests to cover the code to reach (X+something)%.

I think this is a wrong intuition. Those should think about two aspects of the relation between code and tests.

  1. if the tests cover all the promises => remove the untested code,
  2. if you don’t want to remove the uncovered code => find out what implicit promises it is about and make it an actual promise => add tests

People tend to go through the “add test” conclusion without making this reasoning and end up testing dead code, because of the coverage.

unintended consequences of code coverage

I believe that we should consider tests as being an instrumental objective. But having an indicator gives incentive to make it green.

Yet, in the case of code coverage, this is what is likely to happen

  1. you write some code, part of it fulfil your promise, part of it is mostly future dead code,
  2. you cover the whole code because you want to reach 100% of code coverage,
  3. you end up testing your dead code,
  4. after some time, a test of the dead code fails,
  5. you spend some time fixing it.

If you focused of the coverage of your promise, you would have identified the dead code faster.

Notes linking here