Testing with pytest

A test is a function that calls your code with known inputs and uses assert to say what should come back. You run all of them with one command, and the command fails if any assertion fails. That is the whole idea; pytest just makes it comfortable.

pytest finds tests by name: files called test_*.py (or *_test.py) and functions inside them that start with test_. You install it with pip install -U pytest into a virtual environment and run pytest in the folder. A plain assert x == y is enough: when it fails, pytest shows both values.

The function under test is auto_fare in fare.py, an auto rickshaw fare with a minimum charge and a per-km rate (illustrative numbers):

# fare.py
def auto_fare(distance_km: float, night: bool = False) -> int:
    """Return the fare in rupees for an auto ride."""
    if distance_km < 0:
        raise ValueError("distance cannot be negative")
    fare = 30
    if distance_km > 1.8:
        fare += (distance_km - 1.8) * 15
    if night:
        fare *= 1.5
    return round(fare)

Press Run to step through test_fare.py while every test passes. Then switch Try it to the version where someone changed the per-km rate to 10 and see what the same tests report.

# Tests for fare.py. Run from this folder with: pytest -q
import pytest
from fare import auto_fare


@pytest.fixture
def short_trip():
    return 1.5  # km, inside the minimum fare


def test_minimum_fare(short_trip):
    assert auto_fare(short_trip) == 30


def test_longer_trip_adds_per_km():
    # 5 km: 30 + (5 - 1.8) * 15 = 78
    assert auto_fare(5) == 78


def test_night_fare_is_one_and_half():
    assert auto_fare(5, night=True) == 117


def test_negative_distance_is_rejected():
    with pytest.raises(ValueError):
        auto_fare(-2)

What to notice

1. You run pytest, not the file. pytest -q collects the tests and runs each one in isolation. -q keeps the output short.
2. A fixture is setup by name. @pytest.fixture on short_trip and a parameter called short_trip in a test is all the wiring. pytest calls the fixture and passes in its return value. Put fixtures in conftest.py to share them across files.
3. State the expected value. assert auto_fare(5) == 78 writes down the answer. A test that recomputes 30 + (5 - 1.8) * 15 would pass even if the rule was wrong in both places.
4. Failures explain themselves. assert 62 == 78 and where 62 = auto_fare(5) came from pytest’s assertion rewriting. No print statements needed.
5. Test the error path too. with pytest.raises(ValueError): passes only if the block raises. Silent bad input is a bug you want caught.
6. The exit code is the contract. 0 when all pass, 1 when any fail. CI systems read that number.

Here is the real terminal output of the two runs, copied from pytest 9.2.0:

$ pytest -q                     # fare.py correct
....                                                                     [100%]
4 passed in 0.00s

$ pytest -q                     # fare.py with the per-km rate typed as 10
.FF.                                                                     [100%]
=================================== FAILURES ===================================
_________________________ test_longer_trip_adds_per_km _________________________

    def test_longer_trip_adds_per_km():
        # 5 km: 30 + (5 - 1.8) * 15 = 78
>       assert auto_fare(5) == 78
E       assert 62 == 78
E        +  where 62 = auto_fare(5)

test_fare.py:17: AssertionError
...
FAILED test_fare.py::test_longer_trip_adds_per_km - assert 62 == 78
FAILED test_fare.py::test_night_fare_is_one_and_half - assert 93 == 117
2 failed, 2 passed in 0.01s

Why this matters for AI work

Agent evaluations are tests with a softer assertion. In ADK, AgentEvaluator runs an eval set inside an ordinary test_ function, and a score below the threshold fails the test, which fails the job. Everything here carries over: the file naming, fixtures for shared setup, pytest -q in CI, exit code 1 as the stop signal. See Running evals: web UI, CLI and pytest.