Asked about A daily pipeline you can rerun safely

A daily job failed halfway and was restarted. Now some rows appear twice. How do you prevent this?

An answer that works

Make each run idempotent: process a well-defined partition such as one day, write to a temporary location, validate, then replace the whole partition atomically. Reruns then overwrite rather than append, and a failed run publishes nothing.

Then they push further

The target is a database table, not files. Same idea?

Yes: delete and insert the day's rows inside one transaction, or use a merge on a natural key. Either way a rerun leaves the same final state.

Data for a day can keep arriving for a week. How do you handle that?

Reprocess a trailing window, for example the last seven days every run. Idempotent partition replacement makes that safe.

What gets marked up

Marked up for idempotency via partition replacement, atomic publishing, and handling late data by reprocessing a window.

Reading an answer is not being able to give one. The section that teaches this is A daily pipeline you can rerun safely, 5 hours of Data pipelines.

Learn this properly

Also asked about Data pipelines

How do you make a data pipeline safe to rerun?How do you handle an upstream schema change?What quality checks would you put in a pipeline?When would you choose streaming over batch?How do you handle late-arriving data?How do you keep a RAG index up to date as documents change?