Memory, views and larger-than-memory data

This is where array code goes quietly wrong. Slicing usually returns a view onto the same memory rather than a copy, so writing to what you thought was a separate array changes the original — no error, wrong numbers. This section covers views and copies, choosing data types so a table fits in memory at all, what to do when it does not, and how to recognise the point where pandas has stopped being the right tool for the size of data in front of you.

Views and copies#

An array is a block of memory plus a description of how to walk it. A view is a second description of the same block, which is why slicing costs nothing regardless of how large the array is, and why writing through a slice writes into the original. This is efficient and it is the source of confusing bugs, because nothing in the syntax tells you which you have.

a = np.arange(10)
b = a[:5]         # a view
b[0] = 99
a[0]              # 99  -- the original changed

c = a[:5].copy()  # an independent copy

The rule is short. Basic indexing — a slice, a step, a transpose, a reshape — returns a view. Advanced indexing — a boolean mask, a list or array of indices — returns a copy, because the selected elements are not evenly spaced and cannot be described by strides. Assignment through either does write to the original, which is the part that makes the rule feel inconsistent until you see why.

np.shares_memory(a, a[2:7])      # True   slice: a view
np.shares_memory(a, a[::2])      # True   step: still a view
np.shares_memory(a, a.T)         # True   transpose: a view
np.shares_memory(a, a[a > 3])    # False  mask: a copy
np.shares_memory(a, a[[0, 2]])   # False  list of indices: a copy

a[2:7].flags["OWNDATA"]          # False  it is borrowing
a[2:7].base is a                 # True   this is who it borrowed from

a[a > 3] = 0                     # but assignment through a mask DOES write back

pandas has the same question and has answered it differently over time. Under the copy-on-write behaviour that became the default in pandas 3, every operation behaves as though it returned a copy, and the copy is only actually made if you write to it. That removes the ambiguity: chained assignment now raises ChainedAssignmentError rather than the old SettingWithCopyWarning, and .loc is the way to assign in both.

Data types#

Memory scales directly with the data type, and the default is generous. A column of ages stored as the default 64-bit integer uses eight bytes per value to hold a number that fits in one. On a million rows that is 8 MB against 1 MB, and on a table with thirty such columns it is the difference between fitting in memory and not.

dtype      bytes   range or precision              use it for
int8         1      -128 to 127                     small counts, ages, flags
int16        2      -32,768 to 32,767               most counts
int32        4      about +/- 2.1 billion           ids, unless they exceed that
int64        8      the default                     large ids, timestamps in ns
float32      4      about 7 significant digits      embeddings, activations, images
float64      8      about 16 significant digits     the default; money, accumulators
bool         1      True or False
category   varies   stores each distinct string once  repeated strings in pandas

df.memory_usage(deep=True).sum()     # 70,500,000 on a 1M-row frame

df["age"] = df["age"].astype("int8")
df["country"] = df["country"].astype("category")

df.memory_usage(deep=True).sum()     # 2,000,000 -- 35x smaller

Categorical types for repeated strings are frequently the largest single saving, because a column of a million country names becomes a million small integers plus a lookup table of the handful of distinct values. Use it when the number of distinct values is small relative to the rows, and not when nearly every value is unique, where it adds overhead instead.

np.array([120, 100], dtype="int8") + np.int8(10)    # [-126  110], no warning
np.int8(120) + np.int8(10)                          # -126, with a RuntimeWarning

np.float32(1e8) + np.float32(1.0) == np.float32(1e8)    # True

arr = np.arange(5)                  # int64
arr += 0.5
# UFuncTypeError: Cannot cast ufunc 'add' output from dtype('float64')
#                 to dtype('int64') with casting rule 'same_kind'
arr = arr + 0.5                     # this works, and returns a new float64 array

Where the memory actually goes#

A MemoryError almost never happens at the line that allocated the data. It happens at some intermediate result: every arithmetic expression allocates a new array of the same size, so a chain of four operations on a 6 GB matrix briefly needs several copies of it. The fixes are in-place operators, which write into the existing buffer, and doing the work in chunks.

X = X - mean          # allocates a whole new X
X /= std              # in place: no allocation

np.subtract(X, mean, out=X)        # explicit form of the same thing

# Chunking, when even one pass will not fit:
total = 0.0
for chunk in pd.read_csv("events.csv", chunksize=500_000):
    total += chunk["cost_usd"].sum(dtype="float64")

Data larger than memory#

• Read in chunks and process each, so memory stays bounded. pandas read_csv takes chunksize and returns an iterator of frames.
• Use a columnar format such as Parquet and read only the columns you need. A twenty-column CSV read for two columns parses all twenty; Parquet reads two.
• Give read_csv the dtypes rather than letting it infer them. Inference reads the file to guess, then stores the widest type it saw, and a single stray string turns a numeric column into object.
• For larger analytical work, an engine like DuckDB or Polars handles data beyond memory efficiently and is often far faster than pandas on a single machine.
• The point at which pandas is the wrong tool is not a fixed size. It is when one machine's memory cannot hold a few copies of the data, which in practice is somewhere around a few gigabytes on a laptop and tens of gigabytes on a server.