Real Objects, Safer Tests: PyPI’s Migration from pretend

TL;DR: Thank you, Alex

This post is a thank-you to Alex Gaynor for pretend, “a library for stubbing in Python” that fits in one file, last released in April 2018, and never needed another release. PyPI’s test suite leaned on it from February 2015 until November 10, 2026, when the last import pretend left the codebase (#20956). Eleven years and nine months is a good run for anything in software.

It is also the story of how I stopped using it: 53 pull requests fitted into the gaps between other work over five months, 11,006 call-sites down to zero, six production bugs that the stubs had been masking, and a test suite that grew by a third without getting a third slower.

Underneath, this is about three things at once. Safety: bugs caught by a test instead of by a user. Security: no check loosened on the way, and a few added. And the day-to-day experience of working in this codebase, which got simpler.

History class

Back in February 2015, Donald Stufft was rebuilding PyPI from scratch as Warehouse, and the very first test commit was titled “Start testing with something simple”: thirty lines testing the WSGI entry point, with a pretend.stub() for the app and a call_recorder for the configure step. That was the first of a great many.

If you haven’t used it, pretend gives you three tools, and between them they cover most of what a test wants from a fake:

import pretend

# an object with exactly the attributes you name
request = pretend.stub(method="POST", POST={"name": "foo"}, db=session)

# a callable that records how it was called
send = pretend.call_recorder(lambda *a, **kw: None)

# a callable that raises
service.loads = pretend.raiser(TokenExpired)

# and the assertion half
assert send.calls == [pretend.call(request, user, "message")]

No magic attributes, no auto-generated children, no spec= to forget. Ask a stub for an attribute nobody declared and you get an AttributeError, which is exactly what you want from a fake. It was the right choice in 2015, and no criticism is intended toward anyone who wrote a stub since: it was the house style, it was fast, and it kept the suite at 100% line and branch coverage, a bar PyPI still holds today. By this June the suite held 11,006 call-sites across 163 files: 5,137 pretend.stub(...) and 2,986 pretend.call_recorder(...) among them.

The shape a stub can’t see

A stub is shape-blind by design. pretend.stub(store=lambda path, filepath, meta: None) accepts any call, and so does the production method it stands in for, right up until someone renames filepath to file_path and makes meta keyword-only. The test keeps passing. Production does not.

That’s not a security hole, but it’s the same family of risk. PyPI’s job is to be boring and correct for the millions of installs a day that depend on it. A test that can’t fail doesn’t remove the failure. It moves it to the people running pip install.

I think of this as safety work, next to the work that gets called security. The two pull in the same direction. A check no test reaches can drift without anyone noticing. A suite that runs the real request, the real form and the real row is what keeps a rate limit or an ownership check enforced through the next refactor. The more of the codebase the suite exercises, the more I can trust a green run, and the faster I can land a fix knowing it broke nothing next to it. Reliability for users and speed for maintainers come from the same place.

The case that pushed me was a trusted publishing bug in October 2025. #18814 had added an issuer_url column to GitLab trusted publishers. The view that registers a GitLab publisher didn’t fill it in, and its tests couldn’t notice, because they never touched the database. In production the insert failed, users got a 503 when adding a publisher (#18879), and the fix was one line (#18881). The same week, a reviewer on a related change quoted one line from its tests:

assert service.store_jwt_identifier(pretend.stub(), pretend.stub()) is None

That opened #18880, “Tests: Over-mocking leads to errors surfaced in production”, and a plan: replace stubs with real objects wherever a real object exists.

The rules are relatively simple. Use a real object whenever one exists, and reach for a fake only when there is no real type to instantiate. In priority order:

1. Real services. Warehouse already ships Null* and Dummy* implementations of its service interfaces for development (NullMetrics, DummyRateLimiter, NullEmailBreachedService, …). Register the real one and spy on the one method the test asserts on with mocker.spy. The real code runs, with real return values, and the spy records the call.
2. Real models from factories. PyPI’s tests already used factory_boy for database rows. SomeFactory.build() gives a transient instance with no database I/O (about 18x cheaper than .create(), and it doesn’t slow down as the session fills), so a “fake project” is now a real Project.
3. The real request. Pyramid’s DummyRequest from the pyramid_request and db_request fixtures, with the one attribute the view reads set on it.
4. mocker.create_autospec(RealClass, instance=True) when a class exists but no instance is handy. Autospec checks signatures, which is the part a stub skipped.
5. mocker.patch(..., autospec=True) for constructors and module functions; side_effect=SomeError replaces pretend.raiser.
6. mocker.sentinel for an opaque value that’s only ever compared with is.
7. types.SimpleNamespace, last, for a shape with no real type behind it. It still raises on an undeclared attribute, which a bare MagicMock() does not.

The assertion side maps almost one to one: assert x.calls == [pretend.call(a)] becomes x.assert_called_once_with(a), an exact list becomes x.call_args_list == [...], and assert x.calls == [] becomes x.assert_not_called(). I avoided assert_has_calls: it checks a subsequence, so extra calls pass silently.

Here is one test as it stood in October, the “email not found” case for deleting an address. It passed, and it had passed for years. Can you spot the testing bug?

def test_delete_email_not_found(self, monkeypatch):
    email = pretend.stub()

    def raise_no_result():
        raise NoResultFound

    request = pretend.stub(
        POST={"delete_email_id": "999999999999"},
        user=pretend.stub(id=pretend.stub(), emails=[email], name=pretend.stub()),
        db=pretend.stub(
            query=lambda a: pretend.stub(
                filter=lambda *a: pretend.stub(one=raise_no_result)
            )
        ),
        find_service=lambda *a, **kw: pretend.stub(),
        session=pretend.stub(flash=pretend.call_recorder(lambda *a, **kw: None)),
    )
    monkeypatch.setattr(
        views.ManageVerifiedAccountViews, "default_response", {"_": pretend.stub()}
    )
    view = views.ManageVerifiedAccountViews(request)

    assert view.delete_email() == view.default_response
    assert request.session.flash.calls == [
        pretend.call("Email address not found", queue="error")
    ]
    assert request.user.emails == [email]

The db stub is the bug. The test constructs its own query chain whose only job is to raise NoResultFound, so the view’s real lookup never runs, and the id in the POST body never reaches Postgres. Which is a shame, because 999999999999 is larger than an integer column holds. Send that id to the real database and Postgres raises before the view’s “not found” branch can run, and the user gets a 500 instead of a flash message. The test could never have told you, because the part that would have failed was the part it replaced.

Here is the same test after #20857:

@pytest.mark.parametrize("email_id", ["9999", "999999999999", "nope"])
def test_delete_email_not_found(self, verified_request, default_response, email_id):
    primary = verified_request.user.primary_email
    verified_request.POST = MultiDict({"delete_email_id": email_id})

    view = views.ManageVerifiedAccountViews(verified_request)

    assert view.delete_email() == default_response
    verified_request.session.flash.assert_called_once_with(
        "Email address not found", queue="error"
    )
    assert verified_request.user.emails == [primary]

A real user from a fixture, a real session, and the id goes to the database. Half the lines, three cases instead of one, and the assertion on flash is signature-checked now. The bug was found by deleting the fake, and the fix is a _parse_email_id helper that turns anything an integer column can’t hold into “not found”.

Before and after, across tests/:

In 53 pieces

Eleven thousand call-sites don’t move in one pull request, and I didn’t have a free month to try. The first PR replaced the shared fixtures in conftest.py with a small shim: a wrapper around a real NullMetrics method that answered both metrics.increment.calls (the pretend API) and metrics.increment.assert_called_once_with(...) (the mock API). That let the unconverted files keep passing while each package moved on its own, easiest first, one PR at a time. The shim came out in the last PR, along with the dependency.

That shape mattered, because this was never a project with a deadline. Each slice was an hour or an afternoon between incident work, reviews and the rest of the queue, put down again with main green.

The chart starts in August 2025, when a single commit converted two storage test files and set the pattern the rest would follow. Then nothing moved for ten months. Between opening the issue in October and merging the first conversion PR in June, the count went up, from 10,332 to 11,007, because every other PR kept writing tests the way the surrounding file did (as you would, and as I did). That is the case for going through every file on purpose, even slowly. Left to “convert as you touch”, the old idiom spreads faster than it gets cleaned up.

The flat stretch from mid-July to mid-September 2026 is two months when nothing else left room for it. The steep part in October is admin, email, accounts and manage, the four packages that held two thirds of the call-sites.

A few numbers, added up across all of those pieces:

• 53 pull requests, 133 commits, first commit June 3, last merge November 10 (159 days).
• Around 65 hours in total, spread over 27 separate days. (Narrator: he did not keep a timesheet. That number is reconstructed from session logs, so treat it as a rough one.)
• Test code: 25,475 lines added, 34,420 removed, a net 8,945 lines fewer.
• Production code: 369 lines added and 166 removed, all of it in the six bug fixes below.
• Median time from opening a PR to merging it: 23 hours. Thanks to Jacob Coffee, Alc-Alc and Ee Durbin for the reviews.

Bugs the real objects found

This is the part that justifies the hours. Every item below is a production bug that sat under an existing test. The test could never have caught it, because the part that would have failed was a stub. Each shipped as its own fix, with the converted tests that exposed it.

• A malformed account-association id returned a 500 instead of a form error (#20701). WTForms still runs the lookup validator after the UUID validator rejects the value, so the raw string reached Postgres. The stubbed user service had accepted anything.
• The limit on pending organization requests was never enforced (#20705). The view passed max_apps= to a form that read max_applications=. The page hid the form at the limit, but a direct POST went straight through. A stub form had no keyword to get wrong.
• Organization trusted publishers skipped three guards the account page had (#20743): the verified-email check, the cap on pending publishers, and both registration rate limits.
• Revoking or re-sending an organization invitation with a bad token returned a 500 (#20744). Only TokenExpired was caught; a token that failed its signature check, for example after a secret rotation, could be neither revoked nor replaced. The old test had replaced the token service’s loads with a lambda.
• Three email management views returned a 500 on an unusable id (#20857): non-numeric, outside the integer range, or a file part instead of a string.
• Seven project, organization and team views returned a 500 on a non-UUID id (#20871). Comparing "nope" against a UUID column raises in Postgres before the view’s own “not found” branch can run. The tests had posted uuid.UUID objects, which no browser form ever sends.

None of these is exploitable in any meaningful way. They are the kind of thing a user hits, files a support ticket about, and waits on. One of them, the organization publishing view, is a security control, and it now carries the same email verification, pending-publisher cap and rate limits as the account page it was copied from. No check got looser anywhere. That was a rule for every conversion: keep the assertion it replaced, or tighten it.

The conversions also turned up tests that could not fail, which I fixed as I went: an assert_not_called() on a recorder that was never patched in, a patch on views.SQLAlchemyORMPage while the view imported its own copy, a rate limiter registered under two names so the view could look up the wrong one and pass, a db.get() that returned a row pending deletion from the identity map, and a token service stub that returned a constant, so every invitation assertion was vacuous. Autospec caught signature drift in stubs that had declared the wrong parameter names for years.

Guardrails, so it doesn’t creep back

• A Ruff banned-api entry refuses import pretend with a pointer at #18880.
• The pyramid_request fixture rejects any form value a browser couldn’t have sent: only str, bytes and file parts get through (#20922). Posting a UUID object is how the id bugs above stayed hidden.
• An autouse fixture fails any test that mutates a unittest.mock.sentinel, because sentinels are process-wide singletons and a stray write leaks into whichever test reads that name next (#20274).
• Since May, every warning is an error. More on that below.

Finishing the job, and the stopwatch

In May 2025, Trail of Bits wrote about making PyPI’s test suite 81% faster, from 163 seconds to 30 for about 4,700 tests, mostly through pytest-xdist, sys.monitoring coverage and tighter test discovery. That work is why I could run the full suite before every commit of these conversions.

The suite is at 7,559 tests now. Going in, I expected the conversions to make it slower. Every stub that became a real object is a little more work per test: a real NullMetrics with ten spied methods takes about 2ms to build, and most unit tests pull it in through the service registry. Hundreds of tests that used to hand a view a pretend.stub(db=...) now run against a real transaction on a real Postgres, with factory rows instead of attributes on a stub.

That is largely the whole point – tests that touch the data layer are the ones that catch data-layer bugs. The question is what it cost, and the answer is almost nothing. The Tests job on main stayed between 200 and 260 seconds from April through October, while the suite grew by about 700 tests and a large share of them moved onto the database. The conversions also removed nearly 9,000 lines of test code. Most of that was request, session and service setup repeated in every test, where a fixture or a factory already had the real thing.

The drop in November is a separate change that fell out of this work. The coverage configuration recorded per-test contexts on every run. That blocks coverage’s sys.monitoring core, so every run, in CI and locally, had fallen back to the slower tracer despite the Trail of Bits change asking for the fast one. Making contexts opt-in (#20899) took a local make tests from 89 seconds to 43, and CI from about 220 seconds to about 140.

To separate the two effects, I checked out the tree from the day before the first conversion PR, built a Docker image from that commit, and ran its suite and today’s on the same laptop, three times each.

Per test, the June tree on sys.monitoring costs 5.3 ms and today’s costs 5.0 ms. That is the number that says real objects and real rows cost nothing: the suite is a third bigger, far more of it reaches the database, and each test got slightly cheaper. The coverage change paid for the growth, so the wall clock reads the same 38 seconds as in June, for 1,936 more tests.

Working in the suite now

The part that doesn’t show up in a chart is what it’s like to write a test today.

• Call a service with the wrong parameter name and the test fails on that line with a TypeError that names it, because the double is autospecced. Before, it passed, and the mistake surfaced in review if someone read closely.
• Post a value no browser could send and the test fails at the assignment, with the value in the message, instead of passing by database coincidence.
• A view test starts from db_request and a factory row. The request, session and service setup that opened every test is now one fixture per module.
• A new warning from a dependency fails the suite in the PR that introduces it.
• The whole thing runs in 38 seconds on a laptop, so “run everything before every commit” is a habit rather than a sacrifice.

Fewer lines to write, clearer failures when I get it wrong, and a fast loop. That is most of what developer experience means on a project like this.

pytest 9.1 and --max-warnings

Part of keeping a suite honest is refusing to let warnings pile up. Simon Willison’s TIL on treating warnings as errors is the two-line version: put filterwarnings = error in your pytest config, and a deprecation that would have scrolled past becomes a failing test with a traceback you can --pdb into.

Warehouse has run that way since May (#20101). Every warning the suite emits is either an error or on a short, commented allowlist with the upstream issue it’s waiting on. When a dependency starts deprecating something, I find out in the pull request that bumps it, not in a log nobody reads.

Not every project can get to error in one step. A suite with a few hundred warnings needs a ratchet: fail if the count goes up, let people pay it down over time. In 2024 I wrote a small plugin for that, pytest-max-warnings, and this spring I proposed the same thing upstream. It landed as --max-warnings (and a max_warnings ini option) in pytest 9.1, released in June, with its own exit code so CI can tell “tests failed” from “too many warnings”. The plugin is archived with a note pointing at the built-in option. It did its job, and now it’s one fewer dependency for everyone who used it.

What have we learned, Mike?

That a test suite is a safety system, and like any safety system it is only as good as the parts it actually exercises. That the cheapest time to find a bug is when the fake becomes real, in a pull request, with a reviewer, and not in a support ticket. And that a big change doesn’t need a big block of time, only a shape that lets you stop at any point with main green.

If you maintain a project with a test suite of its own, here is a small thing to try: pick one test that stubs a collaborator, swap in the real one, and see whether it still passes. Sometimes it won’t, and that is the whole point.

To Alex Gaynor, for a library that did one thing in a single file and needed no changes for eight years while the project around it grew up. To Donald Stufft, for picking it in 2015. To the reviewers who read fifty-three test refactors without once asking “is this done yet”.

PyPI benefited from pretend for nearly twelve years, until it was no longer needed. You were, for all of them, the great pretender. We salute you.

My work on PyPI is funded by Alpha-Omega, a fund and catalyst for sustainable open source security improvements and an associated project of the OpenSSF. That support is what let me put the spare hours of five months into these conversions, the six production fixes they surfaced, and a suite I trust more than I did in June.