Asked about Moving data reliably
How do you make a data pipeline safe to rerun?
An answer that works
Idempotency: write to a partition keyed by the period being processed and replace it rather than appending, so a rerun produces the same result. Parameterise by date range so backfills use the same code path.
Reading an answer is not being able to give one. The section that teaches this is Moving data reliably, 5 hours of Data pipelines.
Learn this properlyAlso asked about Data pipelines
How do you handle an upstream schema change?What quality checks would you put in a pipeline?When would you choose streaming over batch?How do you handle late-arriving data?A daily job failed halfway and was restarted. Now some rows appear twice. How do you prevent this?How do you keep a RAG index up to date as documents change?