Engineering beyond
the happy path.
Field notes on building Python systems that hold up in production.
The decisions, the trade-offs, and the code behind them.
Explore all articles below. Browse the archive →
A few good places to start
Follow your curiosityThe articles
Notes from the infrastructure layerAI code review should not be fully autonomous
The obvious way to build an AI code reviewer is a webhook: a merge request opens, a model reads the diff, the comments appear.
Circuit breakers should be per origin, not per client
A circuit breaker is the simplest reliability pattern to explain and the easiest to key wrong.
Exactly-once is a lie; exactly-once effects are not
Kafka cannot make your database update exactly once. Nothing can, because the database and the broker are two systems with two commits and no transaction that spans them, and…
Graceful Kafka consumer shutdown in Kubernetes
A Kafka consumer under Kubernetes is redeployed several times a day, and every redeploy sends it SIGTERM in the middle of a batch.
Graceful shutdown in Kubernetes is a protocol, not a signal handler
Every web framework handles SIGTERM. It stops accepting connections, lets the requests already in flight finish, and exits cleanly.
gRPC channels should not be pooled by address alone
A gRPC channel is expensive to open and cheap to keep, so every service that talks to more than one gRPC backend grows a channel pool, and the first pool is always a dictionary…
How I start a production-grade Python library in 2026
Every library in this series started the same way: one command, forty-one files, and a green quality gate about three seconds later.
How to partition an existing PostgreSQL table without rewriting your application
ALTER TABLE ... PARTITION BY does not exist. Turning a live table into a partitioned one means making a new parent, adopting the old table as its DEFAULT partition, and draining…
Idempotency across a chain of microservices
One idempotency key in one service is a solved problem. A chain is not, because the retry that matters happens at the top and the effect that matters happens at the bottom, with…
Idempotency for background jobs and Kafka consumers
The Idempotency-Key header gets the attention because it has a name and a spec, but the same problem arrives at every worker that takes jobs from a queue, and it arrives more…
Idempotency keys: the part everyone gets wrong
An Idempotency-Key header is the most widely copied idea in payment APIs, and the most widely misimplemented.
Mapping Python exceptions to gRPC status codes without leaking internals
gRPC has sixteen status codes and your service has a hundred exception types, so somebody has to write the map.
Migrating from pg_partman to application-managed partitions
pg_partman is the default answer for PostgreSQL partition maintenance, and it is a good one when you can install extensions and your team is comfortable operating inside the…
One lifecycle for HTTP, gRPC, workers and cron jobs
An HTTP API, a gRPC server, a Kafka consumer and a nightly job are one application with four ways for work to enter it.
Partition retention is not DROP TABLE
The retention job is the one line of the partitioning setup that nobody reviews. Find the partitions older than the window, drop them, run it nightly.
PgBouncer transaction mode and async SQLAlchemy: the production setup nobody documents enough
Your SQLAlchemy configuration works perfectly against PostgreSQL, and then someone puts PgBouncer in front of the database in transaction mode, which is the only mode that solves…
Publishing to PyPI without API tokens: Trusted Publishing end to end
A PyPI API token in a repository secret is a password with no expiry, no scope beyond the project it was minted for, and no way to tell who used it.
Redis health checks: PING is not the whole story
A health check that answers True for a server that cannot take a write is worse than no health check, because something acts on it.
Reliability is not retry=3
retry=3 is the first thing anybody adds to an HTTP client and the last thing anybody revisits.
Retries can make an outage worse: designing a retry budget
Three retries at every hop of a five-service call is not resilience. It is a multiplier, and it multiplies hardest exactly when the bottom service is failing, which is the one…
Retry-After, backoff and jitter: what a production HTTP client actually does
The retry loop every codebase has is four lines: try, catch, sleep, try again. A production HTTP client's retry policy is a checklist of about eight decisions that the four lines…
Safe gRPC retries: which status codes you should actually retry
max_attempts=3 is the most common line in a gRPC client configuration and the least examined.
Should your application create Kafka topics on startup?
Somebody has to create the topic. The three candidates are the broker, doing it automatically the first time anyone mentions a name; the application, doing it at startup; and a…
Stop passing AsyncSession everywhere
Every async SQLAlchemy codebase I have worked on has the same signature, repeated at every level:
Testing database migrations with Testcontainers: up, down and up again
The five migration tests post made the case; this one is the setup, step by step, from an empty tests/ directory to a green job in CI that walks every revision forward, back and…
The anatomy of a production Python gRPC server
A grpc.aio server is six lines. A grpc.aio server you can put behind a load balancer, roll out three times a day and hand to an on-call rota is a different object, and the…
The five migration tests every Python project should run in CI
We test application code until the coverage badge is green, and then we deploy database migrations that have been run exactly once, on a laptop, in one direction.
The production checklist for aiokafka
aiokafka is a good client with defaults chosen for a library, not for your service, and the gap between the two is where the incidents live.
The Unit of Work pattern in SQLAlchemy 2
Every repository I have ever seen written for the first time has a commit() in it. It is there so that the id comes back, so that the next repository can use it, so that the test…
Transactional inbox: the other half of the outbox pattern
The outbox gets the attention because it solves the dramatic problem, the event that never left.
Transport-independent errors: one domain error, HTTP and gRPC responses
A service that speaks HTTP to the outside and gRPC to its neighbours has two answers for every failure, and the two drift.
Twelve libraries, one engineering standard, no monorepo
Bedrock Python is sixteen repositories: twelve libraries, two tools, a template and this site.
UUIDv7 as a PostgreSQL partition key
Range-partitioning a table by time normally costs you the primary key: PostgreSQL requires every unique constraint to contain the partition column, so PRIMARY KEY (id) becomes…
Warmup, readiness and liveness are three different things
Most services answer all three questions with one handler, usually one that pings the database. Each conflation has its own outage.
We started writing documentation for AI coding agents
In 2024 I wrote documentation for developers. Somewhere in 2026 I noticed that a good share of the readers were not people.
What every production Python microservice reimplements
Open the repository of any backend service that has been in production for a year and look for the code that is not the product.
What happens when Kafka is down for an hour?
Not "is down for a second, and the retry catches it". An hour: a broker rolling badly, a disk full on every node, a network partition between availability zones.
What to monitor in a SQLAlchemy connection pool
The dashboard everyone builds first shows connections in use, and it is the least useful of the numbers available.
When should Redis fail open?
Redis is down. It is your cache, your rate limiter and the store behind your idempotency keys, and every request that arrives now has to decide what to do without it.
Why application lifecycle should not belong to FastAPI
FastAPI's lifespan is a good API. It is an async context manager: whatever you set up before the yield is the startup, whatever you do after it is the shutdown, and it runs…
Why gRPC interceptors break on streaming RPCs
An interceptor that times a call, counts its errors and binds a request id is twenty lines, and it works.
Why I stopped wrapping HTTP clients
Every company I have worked at eventually wrote its own HTTP client wrapper. It starts as a retry helper, grows a config class, learns to emit metrics, and ends as class…
Your models and your schema have drifted. Would CI notice?
The migration ran, the deploy went out, and the models and the database now say different things.
Zero-dependency cores: why optional dependencies matter in infrastructure libraries
An infrastructure library is one that ends up in every service, and every dependency it declares ends up there too.
Managing PostgreSQL partitions, one failure at a time
There are two ways a partitioned table gets you out of bed. The first is an INSERT at 03:00 that PostgreSQL rejects because nobody created next month's partition.
Timeouts are not deadlines: how latency budgets break across microservices
Every service I have run had a timeout on every outgoing call, and every one of them still managed to take longer than any number in its config.
Welcome to the Bedrock Python Blog
This is the home of the Bedrock Python ecosystem — what it is, why it exists, and the thinking behind it.
Introducing mr-review: AI-powered merge request reviews
Code review is one of the highest-leverage activities in a software team, and also one of the most inconsistent. Reviewers get tired, context-switch mid-review, miss things.
The Transactional Outbox pattern in Python: omni-box
Distributed systems have a classic problem: you want to update your database and publish an event to Kafka in the same operation, but there is no cross-system transaction.
No articles found. A different angle?
Try a broader term like “retries” or “PostgreSQL”, or clear a filter.