Data pipelines interview questions
7 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Moving data reliably
Read the section that teaches thisIf this comes up in an interview
How do you make a data pipeline safe to rerun?
Idempotency: write to a partition keyed by the period being processed and replace it rather than appending, so a rerun produces the same result. Parameterise by date range so backfills use the same code path.
How do you handle an upstream schema change?
Validate incoming data against an expected schema and fail loudly rather than writing whatever arrived. Adding nullable columns is safe; removals and type changes need a versioned migration with both supported during the transition.
What quality checks would you put in a pipeline?
Row count against expectation, uniqueness of keys, null checks on required fields, range checks on numerics, and a volume check. The volume check is what catches an upstream failure that still produces structurally valid but nearly empty data.
Orchestration, streaming and lineage
Read the section that teaches thisIf this comes up in an interview
When would you choose streaming over batch?
Only when freshness changes a decision, such as fraud detection or alerting. Streaming is much harder to operate, so the test is whether anyone would act differently if the data were an hour old.
How do you handle late-arriving data?
Decide explicitly how late an event may be and still be included, using a watermark, and define what happens to events after that. Assuming ordered arrival produces subtly wrong aggregates.
A daily pipeline you can rerun safely
Read the section that teaches thisIf this comes up in an interview
A daily job failed halfway and was restarted. Now some rows appear twice. How do you prevent this?
Make each run idempotent: process a well-defined partition such as one day, write to a temporary location, validate, then replace the whole partition atomically. Reruns then overwrite rather than append, and a failed run publishes nothing.
Then they push further
The target is a database table, not files. Same idea?
Yes: delete and insert the day's rows inside one transaction, or use a merge on a natural key. Either way a rerun leaves the same final state.
Data for a day can keep arriving for a week. How do you handle that?
Reprocess a trailing window, for example the last seven days every run. Idempotent partition replacement makes that safe.
What gets marked up. Marked up for idempotency via partition replacement, atomic publishing, and handling late data by reprocessing a window.
Pipelines for AI: keeping a RAG index in sync
Read the section that teaches thisIf this comes up in an interview
How do you keep a RAG index up to date as documents change?
Run an incremental sync: hash each document's content, and permissions, to skip unchanged ones, re-chunk and re-embed changed ones by replacing all their chunks, delete chunks for documents that disappeared from the source, and quarantine documents that fail to parse. Record the embedding model with each vector.
Then they push further
A user complains the assistant cited a policy withdrawn last month. What failed?
Deletions were not propagated. The sync handled additions and updates but never removed documents missing from the source. Add a set difference between source ids and index ids to every run, and a test for it.
What gets marked up. Strong answers include deletions and permissions without being prompted.
These are the questions for one topic. The whole question bank covers every topic on the site.