SQL & data access, section 3 of 3, 3h

Writing data safely

By the end: You can write to a database so that retries, crashes and concurrent requests do not corrupt it.

  • Transactions
  • Constraints as the last line of defence
  • Upserts
  • Idempotent writes
  • Two requests at once

New to sql & data access? Most of the world's useful information sits in databases: tables of rows and columns, like very large spreadsheets that many programs share at once. SQL is the language you use to ask a database questions. You describe what you want — which columns, from which table, matching which conditions — and the database works out how to fetch it. Why it matters

Reading from a database is forgiving. Writing to it is not, because requests retry, processes crash halfway, and two of them arrive at the same instant. This section is the four mechanisms that keep a table correct through all of that: transactions, constraints, upserts and row locks. What they prevent is the quiet kind of failure, a double charge or a balance that lost an update, which careful application code cannot catch on its own because the check and the write are two separate moments.

All or nothing#

A transaction is a group of statements that either all take effect or none do. Without one, a process that crashes between two statements leaves the database in a state that neither statement intended: credits deducted with no usage recorded, or an order with no line items. Wrap anything that must stay consistent together, and note that a transaction is also the unit of isolation, so another connection sees nothing until it commits.

BEGIN;
UPDATE accounts SET credits = credits - 10 WHERE id = 7;
INSERT INTO usage (account_id, credits) VALUES (7, 10);
COMMIT;   -- both happen, or on error neither does

-- ROLLBACK undoes everything since BEGIN. Most client libraries do it for you
-- when the block raises:
-- with conn.transaction():
--     ...            # an exception here rolls back and re-raises
Four properties get quoted at this point: atomicity (all or none), consistency (constraints hold), isolation (concurrent transactions do not see each other's half-finished work), durability (a commit survives the power going out).

Keep a transaction short, and never hold one open across a model call

An open transaction holds locks and pins the rows other transactions need. A handler that starts a transaction, calls a model that takes eleven seconds, then writes, blocks every other request touching those rows for eleven seconds, and under load that becomes a connection pool exhausted error somewhere unrelated. Do the slow call first, then open the transaction to write.

Let the database refuse bad data#

CREATE TABLE usage (
  id          bigserial PRIMARY KEY,
  account_id  bigint NOT NULL REFERENCES accounts(id),
  credits     int    NOT NULL CHECK (credits > 0),
  request_id  text   NOT NULL UNIQUE,     -- one row per request, however many retries
  created_at  timestamptz NOT NULL DEFAULT now()
);

Each of those does a specific job. PRIMARY KEY makes the row addressable and unique. REFERENCES accounts(id) is a foreign key: the database refuses a usage row for an account that does not exist, and refuses to delete an account that still has usage, which is what stops the orphan rows that break every later join. NOT NULL says the column must have a value. CHECK encodes a rule the type cannot, here that a charge is positive. UNIQUE is the one doing the real work below.

Application code has bugs, deploys race, and two services write the same table. Constraints do not forget, and they hold for every writer including the migration somebody runs by hand at midnight. A unique request id means a retried request cannot charge twice, and it means that whatever the application logic does.

Insert or update#

-- Retry is harmless: the second attempt inserts nothing and raises nothing.
INSERT INTO usage (account_id, credits, request_id)
VALUES (7, 10, 'req_8f2c')
ON CONFLICT (request_id) DO NOTHING;

-- Upsert: insert, or update what is already there. The excluded pseudo-table
-- holds the row you tried to insert.
INSERT INTO daily_cost (day, model, cost_usd)
VALUES ('2026-03-11', 'sonnet', 12.40)
ON CONFLICT (day, model) DO UPDATE
SET cost_usd = daily_cost.cost_usd + excluded.cost_usd;

-- Return what happened, so the caller knows without a second query.
INSERT INTO usage (account_id, credits, request_id)
VALUES (7, 10, 'req_8f2c')
ON CONFLICT (request_id) DO NOTHING
RETURNING id;    -- no row returned means it was already there
ON CONFLICT needs a unique constraint on those columns to conflict against. Without one it is a syntax error, which is a useful reminder that the constraint is the mechanism.

Two requests at once#

Two requests both read a balance of 10, both subtract 10, both write 0, and one charge disappears. Nothing errored and no code was wrong on its own; the problem is that reading and writing were two separate moments and something happened in between. This is the lost update, and there are three standard fixes.

time   request A                     request B
 1     SELECT credits -> 10
 2                                   SELECT credits -> 10
 3     UPDATE SET credits = 0
 4                                   UPDATE SET credits = 0
       one of the two charges has vanished, and nothing raised
-- 1. Do the arithmetic in the database. One statement, no gap.
UPDATE accounts SET credits = credits - 10
WHERE id = 7 AND credits >= 10
RETURNING credits;      -- no row returned means there were not enough credits

-- 2. Lock the row first, if you must read, decide, then write.
BEGIN;
SELECT credits FROM accounts WHERE id = 7 FOR UPDATE;  -- B blocks here until A commits
-- ... decide ...
UPDATE accounts SET credits = credits - 10 WHERE id = 7;
COMMIT;

-- 3. Optimistic: carry a version and fail if it moved.
UPDATE accounts SET credits = 0, version = version + 1
WHERE id = 7 AND version = 4;     -- 0 rows updated means someone else got there first
Prefer the first. The WHERE credits >= 10 is what makes it safe: the check and the write are the same statement, so nothing can happen between them.

Locking works and costs you concurrency, so take locks in a consistent order across your codebase. Two transactions that lock account 7 then account 9, and account 9 then account 7, will deadlock; Postgres detects it and kills one with a deadlock detected error, which the application should retry rather than treat as a failure.

Read committed is the default, and it is weaker than people assume

At Postgres's default isolation level, each statement sees a fresh snapshot, so two statements inside one transaction can see different data, and a row you read can be changed by someone else before you write it. Everything above exists because of that. Raising the level to serializable makes the database detect the conflict and abort one transaction, which is correct and means your code must handle a serialization failure by retrying.

Where people go wrong

Checking 'does this row exist?' in code and then inserting. Two requests pass the check at the same moment. Let a unique constraint decide.

If this comes up in an interview

A client retries a request after a timeout. How do you make sure it is not processed twice?

The client sends a unique request id. The server stores it with a unique constraint as part of the same transaction as the work, and on conflict returns the original result instead of repeating the work.

Then they push further

What if the work includes calling an external API that is not idempotent?

Record the intent with the request id before calling, and on retry check that record and ask the external system what happened rather than calling again.

What gets marked up. Marked up for enforcing it in the database with a constraint rather than only checking in application code, which races.

What is a race condition in a read-modify-write, and how do you prevent it?

Two transactions read the same value, each modifies it, and one write overwrites the other. Prevent it by doing the update atomically in one statement, locking the row before reading, or using optimistic concurrency with a version column.

Build this: Build a credits table and an endpoint that deducts credits per model call. Fire twenty concurrent requests with duplicate request ids and confirm the balance and usage rows are exactly right.

This is one section of SQL & data access, 14 hours across 3 sections. Read it in full to get the whole topic in order.

Stop 9 of 50 on the AI Engineer route

0 of 50 topics done · 0h behind you, 603h to go

Next on this routeSampling & decodingTemperature, top-p, and why zero is not deterministic.9h

What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.