Asked about Orchestration, streaming and lineage
When would you choose streaming over batch?
An answer that works
Only when freshness changes a decision, such as fraud detection or alerting. Streaming is much harder to operate, so the test is whether anyone would act differently if the data were an hour old.
Reading an answer is not being able to give one. The section that teaches this is Orchestration, streaming and lineage, 4 hours of Data pipelines.
Learn this properlyAlso asked about Data pipelines
How do you make a data pipeline safe to rerun?How do you handle an upstream schema change?What quality checks would you put in a pipeline?How do you handle late-arriving data?A daily job failed halfway and was restarted. Now some rows appear twice. How do you prevent this?How do you keep a RAG index up to date as documents change?