Leverage coverage.py’s dynamic contexts and find out!
You’re staring at a mysterious line of code in a huge codebase, wondering which test executes it. Running the entire test suite multiple times takes too long. What can you do?
The answer is : coverage.py with dynamic contexts. This will enable us to, after having executed the test suite once, quickly find out which test(s) executed any given line of code in any file. The documentation is quite sparse, so here we’ll work through a complete end-to-end example.
Let’s start by creating a new directory. Within it, we’re going to create 3 files:
t.py, where we’ll define two functions:
def sum(left, right):
return left + right
def multiply(left, right):
return left * right
test_t.py, which will have tests which run
result = t.sum(1, 2)
expected = 3
assert result == expected
result = t.multiply(1, 2)
expected = 2
assert result == expected
.coveragerc, which contains:
dynamic_context = test_function
Next, create a Python3.8+ virtual environment, and make sure you have installed
Then, we’ll run out test suite, which will save a
sqlite database to a file called
$ coverage run -m pytest
================================================= test session starts ==================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
collected 2 items
test_t.py .. [100%]
================================================== 2 passed in 0.01s ===================================================
As we’ll soon see, we can query the
.coverage file using sqlite.
The .coverage sqlite database schema
If we consult the coverage.py documentation, we can see that the following tables will be of use to us:
context, which will give us the test file names;
file, which will give us the file name;
line_bits, which will give us the line numbers.
The docs state that in
line_bits, the line numbers are stored as numbits, so we’re going to have to use some helper functions from
coverage.py to convert them to integers.
We’re now ready to query the database to figure out which tests executed a given line of code. For this example, we’ll find the test(s) which executed line
t.py. We can do this by making a file
from coverage.numbits import register_sqlite_functions
conn = sqlite3.connect(".coverage")
c = conn.cursor()
result = c.execute(
"select context.context "
"from line_bits, context, file "
"where line_bits.context_id = context.id "
"and line_bits.file_id = file.id "
"and num_in_numbits(?, numbits) "
"and file.path = ?",
If we execute it, we see:
$ python query.py
test_sum from the file
test_t.py is the test we’re looking for. Nice!
What if I use branch coverage?
If you tried running the above on your own project in which you have enabled branch coverage, you may have noticed that it doesn’t quite work. That’s because if you’re using branch coverage, then you should use the
arc table instead of the
line_bits , and change the line
and num_in_numbits(?, numbits)
and arc.tono = ?
We learned how to find out which test executes a given line of code. This is useful when working with large projects where the test suite takes a long time to run and it’s not feasible to run the whole thing to figure out which test(s) execute each line of code. By leveraging
coverage.py and dynamic contexts, however, we can just run the test suite once, and then query the generate
You can find the example from this article at https://github.com/MarcoGorelli/which-tests-executed-this-code.