Which tests executed this line of code?

Leverage coverage.py’s dynamic contexts and find out!

Photo by Joshua Lawrence on Unsplash

Why?

You’re staring at a mysterious line of code in a huge codebase, wondering which test executes it. Running the entire test suite multiple times takes too long. What can you do?

How?

The answer is : coverage.py with dynamic contexts. This will enable us to, after having executed the test suite once, quickly find out which test(s) executed any given line of code in any file. The documentation is quite sparse, so here we’ll work through a complete end-to-end example.

Example

Setup

Let’s start by creating a new directory. Within it, we’re going to create 3 files:

• t.py , where we’ll define two functions:

def sum(left, right):
    return left + right

def multiply(left, right):
    return left * right

• test_t.py , which will have tests which run t.py :

import t

def test_sum():
    result = t.sum(1, 2)
    expected = 3
    assert result == expected


def test_multiply():
    result = t.multiply(1, 2)
    expected = 2
    assert result == expected

• .coveragerc , which contains:

[run]
dynamic_context = test_function

Next, create a Python3.8+ virtual environment, and make sure you have installed coverageand pytest .

Then, we’ll run out test suite, which will save a sqlite database to a file called .coverage :

$ coverage run -m pytest
================================================= test session starts ==================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/marcogorelli/mcve-coverage
collected 2 items

test_t.py ..                                                                                                     [100%]

================================================== 2 passed in 0.01s ===================================================

As we’ll soon see, we can query the .coverage file using sqlite.

The .coverage sqlite database schema

If we consult the coverage.py documentation, we can see that the following tables will be of use to us:

• context , which will give us the test file names;
• file, which will give us the file name;
• line_bits, which will give us the line numbers.

The docs state that in line_bits, the line numbers are stored as numbits, so we’re going to have to use some helper functions from coverage.py to convert them to integers.

Querying .coverage

We’re now ready to query the database to figure out which tests executed a given line of code. For this example, we’ll find the test(s) which executed line 2 from t.py. We can do this by making a file query.py with

import os
import pprint
import sqlite3

from coverage.numbits import register_sqlite_functions

conn = sqlite3.connect(".coverage")
register_sqlite_functions(conn)
c = conn.cursor()
result = c.execute(
    "select context.context "
    "from line_bits, context, file "
    "where line_bits.context_id = context.id "
    "and line_bits.file_id = file.id "
    "and num_in_numbits(?, numbits) "
    "and file.path = ?",
    (2, os.path.abspath("t.py")),
)
pprint.pprint(result.fetchall())

If we execute it, we see:

$ python query.py
[('test_t.test_sum',)]

So, test_sum from the file test_t.py is the test we’re looking for. Nice!

What if I use branch coverage?

If you tried running the above on your own project in which you have enabled branch coverage, you may have noticed that it doesn’t quite work. That’s because if you’re using branch coverage, then you should use the arc table instead of the line_bits , and change the line

and num_in_numbits(?, numbits)

to

and arc.tono = ?

Conclusion

We learned how to find out which test executes a given line of code. This is useful when working with large projects where the test suite takes a long time to run and it’s not feasible to run the whole thing to figure out which test(s) execute each line of code. By leveraging coverage.py and dynamic contexts, however, we can just run the test suite once, and then query the generate sqlite database.

You can find the example from this article at https://github.com/MarcoGorelli/which-tests-executed-this-code.