Asked about Pipelines for AI: keeping a RAG index in sync

How do you keep a RAG index up to date as documents change?

An answer that works

Run an incremental sync: hash each document's content, and permissions, to skip unchanged ones, re-chunk and re-embed changed ones by replacing all their chunks, delete chunks for documents that disappeared from the source, and quarantine documents that fail to parse. Record the embedding model with each vector.

Then they push further

A user complains the assistant cited a policy withdrawn last month. What failed?

Deletions were not propagated. The sync handled additions and updates but never removed documents missing from the source. Add a set difference between source ids and index ids to every run, and a test for it.

What gets marked up

Strong answers include deletions and permissions without being prompted.

Reading an answer is not being able to give one. The section that teaches this is Pipelines for AI: keeping a RAG index in sync, 4 hours of Data pipelines.

Learn this properly

Also asked about Data pipelines

How do you make a data pipeline safe to rerun?How do you handle an upstream schema change?What quality checks would you put in a pipeline?When would you choose streaming over batch?How do you handle late-arriving data?A daily job failed halfway and was restarted. Now some rows appear twice. How do you prevent this?