Asked about When pandas is slow: vectorising, Polars and DuckDB

A pandas job on a 20 GB CSV runs out of memory. What are your options?

An answer that works

Read only needed columns with compact dtypes, process in chunks, convert to Parquet, or use a tool that streams and parallelises such as Polars in lazy mode or DuckDB, which can query files larger than memory. Move to a distributed engine only if a single machine truly cannot cope.

Reading an answer is not being able to give one. The section that teaches this is When pandas is slow: vectorising, Polars and DuckDB, 3 hours of NumPy & pandas.

Learn this properly

Also asked about NumPy & pandas

Why is vectorised NumPy faster than a Python loop?What does SettingWithCopyWarning mean?How would you handle missing values?What is the difference between a view and a copy in NumPy?How do you reduce the memory a DataFrame uses?After a join, your total revenue doubled. What happened?