Problem

I'm using the "factory as fixture" pattern in pytest, but I'm running into a frustrating issue with error reporting. When an error occurs inside the factory function, pytest reports it as a test failure rather than a fixture setup error.

Real-world example with pytest "factory as fixture" pattern

I'm using the official "factory as fixture" pattern described in the pytest documentation:

@pytest.fixture
def make_customer_record():
    def _make_customer_record(name):
        # If an error happens here, it's reported as a test failure
        record = models.Customer(name=name, orders=[])
        return record

    return _make_customer_record


def test_customer_records(make_customer_record):
    # The error would be reported here
    customer_1 = make_customer_record("Lisa")
    customer_2 = make_customer_record("Mike")
    customer_3 = make_customer_record("Meredith")

Source: pytest documentation - Factories as fixtures

When an exception is raised inside _make_customer_record, the error is reported in the test at the line where the factory is called, not as a fixture setup error.

Question

Is there a way to make errors raised by factory functions (returned from fixtures) report as fixture errors rather than test failures? Something like a hypothetical pytest.error() function would be ideal.

Specifically, I want the test report to show:

ERROR tests/test_hello_world.py::test_hello_world

Instead of what I currently get:

FAILED tests/test_hello_world.py::test_hello_world

I want errors in factory functions to be reported exactly like exceptions raised directly in fixtures - as setup errors rather than test failures. This would make test reports much clearer by distinguishing between actual test failures and infrastructure/setup problems.

The documentation explains how to use factory fixtures but doesn't address this error reporting issue.

Problem

I'm using the "factory as fixture" pattern in pytest, but I'm running into a frustrating issue with error reporting. When an error occurs inside the factory function, pytest reports it as a test failure rather than a fixture setup error.

Real-world example with pytest "factory as fixture" pattern

I'm using the official "factory as fixture" pattern described in the pytest documentation:

@pytest.fixture
def make_customer_record():
    def _make_customer_record(name):
        # If an error happens here, it's reported as a test failure
        record = models.Customer(name=name, orders=[])
        return record

    return _make_customer_record


def test_customer_records(make_customer_record):
    # The error would be reported here
    customer_1 = make_customer_record("Lisa")
    customer_2 = make_customer_record("Mike")
    customer_3 = make_customer_record("Meredith")

Source: pytest documentation - Factories as fixtures

When an exception is raised inside _make_customer_record, the error is reported in the test at the line where the factory is called, not as a fixture setup error.

Question

Is there a way to make errors raised by factory functions (returned from fixtures) report as fixture errors rather than test failures? Something like a hypothetical pytest.error() function would be ideal.

Specifically, I want the test report to show:

ERROR tests/test_hello_world.py::test_hello_world

Instead of what I currently get:

FAILED tests/test_hello_world.py::test_hello_world

I want errors in factory functions to be reported exactly like exceptions raised directly in fixtures - as setup errors rather than test failures. This would make test reports much clearer by distinguishing between actual test failures and infrastructure/setup problems.

The documentation explains how to use factory fixtures but doesn't address this error reporting issue.

• pytest

Share Improve this question Follow asked Mar 26 at 19:37 rceprercepre 32322 silver badges1515 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

There is a conceptual problem:

@pytest.fixture
def get_age():
    ages = {"Alice": 30, "Bob": 31}
    def _get_age(name):
        return ages[name]
    return _get_age

def test_foo(get_age):
    age = get_age("Peter")

This leads to a KeyError, but whose fault is it? The test or the factory function?

From a technical perspective, the error occured during test execution, not fixture setup or teardown. Also, from the call stack, pytest cannot tell that the error occured in a factory function that was returned from a fixture.

Pytest is highly customizable and allows you to hook into almost any step of the test procedure and manipulate how results are reported along the way.

But first, we need to wrap the factory function (or parts of it) to be able to determine that the error originated in that function. That information must then be conveyed to pytest so we can alter reporting in one of the pytest hooks.

There are numerous ways to achieve this, this is just an example:

# conftest.py

from pytest import TestReport, fixture

class FactoryFixtureExceptionTracker:
    def __init__(self):
        # pytest's builtin "record_property" fixture. Must be updated for each test.
        self.record_property = None

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_value:
            self.record_property("factory_fixture_failure", True)

    def __call__(self, func):
        """ To use it as a decorator. """
        def wrapper(*args, **kwargs):
            with self:
                return func(*args, **kwargs)
        return wrapper


@fixture(scope="session")  # can be used in any fixture, no matter its scope
def blame_fixture():
    return FactoryFixtureExceptionTracker()

# updates test context for each test
@fixture(scope="function", autouse=True)
def _update_blame_fixture(blame_fixture: FactoryFixtureExceptionTracker, record_property):
    blame_fixture.record_property = record_property

def pytest_report_teststatus(report: TestReport):
    if report.when == "call":
        if any(key == "factory_fixture_failure" for key, _ in report.user_properties):
            # Change from "FAILED" to "ERROR".
            # Second and third values can be freely chosen.
            return "error", "e", "ERROR"

The blame_fixture object is then used to mark your factory functions (as a decorator) or parts of it (as a context manager) like this:

# test_fixtures.py

import pytest
AGES = {"Alice": 30, "Bob": 31}

@pytest.fixture
def get_age_decorator(blame_fixture):
    @blame_fixture
    def _get_age(name):
        return AGES[name]
    return _get_age

@pytest.fixture
def get_age_context_manager(blame_fixture):
    def _get_age(name):
        assert isinstance(name, str)  # this is certainly the test's fault
        with blame_fixture:
            return AGES[name]
    return _get_age

@pytest.fixture
def failing_fixture():
    assert False

def test_decorator(get_age_decorator):
    get_age_decorator("Peter")

@pytest.mark.parametrize("name", ("Peter", 1))
def test_context_manager(get_age_context_manager, name):
    get_age_context_manager(name)

def test_failing_fixture(failing_fixture): ...

def test_failing(): assert False

def test_success(): ...

The test summary now looks like this:

=========================== short test summary info ============================
FAILED test_fixtures.py::test_context_manager[1] - assert False
FAILED test_fixtures.py::test_failing - assert False
ERROR test_fixtures.py::test_decorator - KeyError: 'Peter'
ERROR test_fixtures.py::test_context_manager[Peter] - KeyError: 'Peter'
ERROR test_fixtures.py::test_failing_fixture - assert False
==================== 2 failed, 1 passed, 3 errors in 0.02s =====================

Notice the two test_context_manager tests: the first is reported as test failure, because providing an integer as a name is clearly the test's fault. The second is reported similar to "test_failing_fixture", where there actually was an error during fixture setup.

If you want to automatically wrap any callable returned from fixtures, you can have a look at pytest's hook documentation. I'm almost certain there is one that allows you to manipulate the return values of fixtures.