The code that gets you promoted is a feature with a nice UI. The code that saved our weekends was four unglamorous invariants nobody ever noticed. Here are the four we got burned by, each with the specific failure that taught it to us. None of them sounded important until they cost us.
Money is integer cents, never float
A nightly pricing job summed line items in floating point. 0.1 + 0.2 is 0.30000000000000004, and across the run the rounding drifted. The reconciliation report came out 3 cents short one morning. Three cents is nothing until finance can't close the books because the number doesn't tie, and then it's a half-day of two people tracing where it went.
Floats can't represent most decimal fractions exactly, and money is decimal by definition, so every sum quietly accumulates error:
# wrong — drifts
total = sum(item.price for item in items) # floats
# right — exact
total_cents = sum(item.price_cents for item in items) # ints
Store cents as integers, or use a Decimal. Never a float. The one that bit us wasn't the first cent — it was the aggregate over ~120k rows that no test covered, because every test used round numbers that happen to sum cleanly.
The subtler version showed up later at the boundary between our system and a payment processor that spoke floats. We stored integer cents, correctly, then serialized to 19.99 for the API and parsed the float back on the response. float("19.99") is 19.989999999999998, and rounding that back to cents was fine, until a division-based split — a refund allocated across three line items — landed one cent off because the float division didn't distribute evenly. The rule tightened: money is integer cents everywhere inside the system, and the only float conversion happens at the exact serialization boundary, with an explicit round, and the split is computed on the integers before conversion so the cents always sum to the whole:
# split 1000 cents across 3 lines: 334 + 333 + 333, not 3 x 333.33
def split_cents(total_cents: int, n: int) -> list[int]:
base, extra = divmod(total_cents, n)
return [base + 1 if i < extra else base for i in range(n)]
divmod distributes the remainder deterministically so the parts sum back to the total. A float split loses or gains a cent on the remainder, and over a month of refunds that cent shows up in the reconciliation report as the same 3-cents-short problem, one processor removed.
Idempotency keys on every retryable write
A payment webhook timed out at the network layer after the write had already committed. The sender saw the timeout and retried. We charged the card twice, and both writes succeeded because nothing said the second one was a duplicate. One customer, two identical charges, a refund and an apology.
Any write that can be retried will be retried — webhooks, queue consumers, client calls over a flaky connection. Networks guarantee at-least-once, not exactly-once. The exactly-once part is yours to build:
def apply_payment(idempotency_key, amount_cents):
if store.seen(idempotency_key):
return store.result_for(idempotency_key) # replay, don't re-apply
result = charge(amount_cents)
store.record(idempotency_key, result)
return result
The key comes from the caller — the webhook's event ID, a client-generated UUID. Same key, same outcome, no matter how many times it arrives. Without it, "we'll just retry on failure" is a plan to double-charge people, which is exactly what it did.
The seen check has to be atomic with the write, and our first version wasn't. We checked seen, saw nothing, then charged, then recorded — three separate steps. Two copies of the same webhook arrived a few milliseconds apart, both passed the seen check before either recorded, and both charged. The race re-created the exact double-charge the key was supposed to prevent. The fix is to make the key a unique constraint in the database and let the insert be the check:
def apply_payment(idempotency_key, amount_cents):
try:
store.reserve(idempotency_key) # unique-constrained insert
except UniqueViolation:
return store.result_for(idempotency_key) # someone else has it
result = charge(amount_cents)
store.record_result(idempotency_key, result)
return result
The database's unique constraint is the one thing that's actually atomic under concurrency. The second insert fails at the constraint, so the second caller reads the first's result instead of charging. An idempotency check built out of a read-then-write in application code is not idempotent under a race, and payment webhooks race by design because the sender retries aggressively.
tenant_id on every query in a multi-tenant system
A support tool ran a lookup that filtered by user email but not by tenant. Two customers each had a user with the same email address. One customer's support agent opened the tool and saw the other customer's records. That's a data breach, and the fix was one WHERE clause that should have been there from the first commit:
# a query without tenant_id is a bug, full stop
rows = db.query(Order).filter(
Order.tenant_id == ctx.tenant_id,
Order.status == "open",
)
In a shared database the tenant boundary lives in the query. Forget it once and rows leak across customers. The scary part is it looks fine in dev, where you have one tenant and everything "works." We don't rely on remembering it anymore — the repository layer injects tenant_id, and row-level security sits under that as a backstop so a forgotten clause fails closed instead of leaking. The invariant is "no query without a tenant filter," and an invariant that matters gets enforced by a gate, not by discipline.
The repository injection looks like this: every query goes through a scoped session that already knows the tenant, so a developer physically can't write a query without the filter — the filter is applied for them:
class TenantRepository:
def __init__(self, session, tenant_id):
self._session = session
self._tenant_id = tenant_id
def query(self, model):
return self._session.query(model).filter(
model.tenant_id == self._tenant_id, # applied every time
)
Then row-level security in the database enforces the same boundary a layer down, keyed off a session variable set at connection time. If someone bypasses the repository and hits the session directly, RLS still filters the rows. Two layers, and they fail closed: a query that somehow reaches the database without a tenant context returns nothing, rather than everything. We also wrote a test that asserts tenant A's context can never read tenant B's rows across every model — seed two tenants, query as A, assert zero B rows — and it runs on every build. The leak that started this was one missing WHERE clause; the defense is two enforcement layers and a test, because "remember the clause" is exactly the discipline that failed the first time.
Validate at the boundary, fail fast
An endpoint trusted a request body and passed it straight to the domain. A field we expected to be a positive number arrived negative, flowed through, and set an account's credit balance to a nonsense value. The bad data was three tables deep before anyone noticed it, and unwinding it meant a manual correction across all three.
The boundary is the one place you can still reject garbage cheaply. Once it's past validation it spreads, and every downstream system treats it as real:
class GrantCredits(BaseModel):
amount: PositiveInt
account_id: str = Field(min_length=1)
# raw request never touches the domain — this DTO does
Parse into an explicit DTO, reject at the door with a clear error, never hand a raw request body to domain logic and hope the shape is right.
Why these specifically
None of these are clever, which is the point. Clever code gets attention and review. Boring invariants get skipped because they're obvious — right up until the aggregate drifts 3 cents, the retry double-charges a card, the tenant filter leaks a customer's data, or the unchecked field poisons a balance table. Write them first, before the feature. They're cheap on day 1 and unaffordable in the incident channel at 2am.