How to handle large datasets in Python even if you are a newbie

Introduction

Working with large datasets in Python raises a familiar problem: a file is loaded with pandas, and the program slows to a crawl or crashes outright. The cause is usually not the data’s size itself but how it is loaded — everything pulled into memory at once. With a handful of practical techniques, Python can comfortably handle datasets far larger than available RAM. This guide covers seven of them, starting simple and working upward, so the right approach for each situation becomes clear. A sample-data generator script and code snippets are available on GitHub for readers who want to follow along.

1. Read the data in chunks

The most beginner-friendly approach is processing data in smaller pieces rather than loading it whole. Consider computing total revenue over a large sales dataset:

import pandas as pd

# Define chunk size (number of rows per chunk)
chunk_size = 100000
total_revenue = 0

# Read and process the file in chunks
for chunk in pd.read_csv('large_sales_data.csv', chunksize=chunk_size):
    # Process each chunk
    total_revenue += chunk('revenue').sum()

print(f"Total Revenue: ${total_revenue:,.2f}")

Instead of loading all 10 million rows at once, the code loads 100,000 rows at a time, sums each chunk, and accumulates the total — so only 100,000 rows ever sit in RAM regardless of file size. Use it for aggregations (sum, count, average) or filtering over large files.

2. Load only the columns you need

Datasets often carry far more columns than an analysis requires. Loading only the necessary ones cuts memory sharply — for example, keeping just the customer age and purchase amount from a wide file:

import pandas as pd

# Only load the columns you actually need
columns_to_use = ('customer_id', 'age', 'purchase_amount')

df = pd.read_csv('customers.csv', usecols=columns_to_use)

# Now work with a much lighter dataframe
average_purchase = df.groupby('age')('purchase_amount').mean()
print(average_purchase)

With usecols, pandas loads only the specified columns; if the original file has 50 columns and only a few are kept, memory usage falls by roughly 90 per cent or more. Use it whenever the needed columns are known before loading.

3. Right-size the data types

By default pandas often chooses generous types — int64 where int8, int16, or int32 would do, or float64 where float32 is sufficient. For a dataset of product ratings (1-5 stars) and user IDs:

import pandas as pd

# First, let's see the default memory usage
df = pd.read_csv('ratings.csv')
print("Default memory usage:")
print(df.memory_usage(deep=True))

# Now optimize the data types
df('rating') = df('rating').astype('int8')  # Ratings are 1-5, so int8 is enough
df('user_id') = df('user_id').astype('int32')  # Assuming user IDs fit in int32

print("nOptimized memory usage:")
print(df.memory_usage(deep=True))

Choosing types that match the actual value ranges can cut memory usage several-fold with no change in behaviour. Common conversions: int64 to a smaller integer type depending on range, float64 to float32 when extreme precision is unnecessary, and object to category for repeated values.

4. Use the category dtype for repeated text

When a column contains repeated text values — country names, product categories — pandas stores each occurrence separately. The category dtype stores unique values once and references them with compact codes. For an inventory file whose Category column holds only 20 unique values repeated across millions of rows:

import pandas as pd

df = pd.read_csv('products.csv')

# Check memory before conversion
print(f"Before: {df('category').memory_usage(deep=True) / 1024**2:.2f} MB")

# Convert to category
df('category') = df('category').astype('category')

# Check memory after conversion
print(f"After: {df('category').memory_usage(deep=True) / 1024**2:.2f} MB")

# It still works like normal text
print(df('category').value_counts())

The conversion substantially reduces memory for low-cardinality columns, and the column still filters, groups, and sorts as usual. Use it for any text column with many repeats: categories, states, countries, departments.

5. Filter while reading

When only a subset of rows matters, filtering can happen during loading rather than after. For transactions from 2024 only:

import pandas as pd

# Read in chunks and filter
chunk_size = 100000
filtered_chunks = ()

for chunk in pd.read_csv('transactions.csv', chunksize=chunk_size):
    # Filter each chunk before storing it
    filtered = chunk(chunk('year') == 2024)
    filtered_chunks.append(filtered)

# Combine the filtered chunks
df_2024 = pd.concat(filtered_chunks, ignore_index=True)

print(f"Loaded {len(df_2024)} rows from 2024")

This combines chunking with filtering: each chunk is filtered before being kept, so the full dataset never resides in memory — only the wanted rows. Use it when a row subset is defined by a condition.

6. Use Dask for parallel processing

For truly huge datasets, Dask provides a pandas-like API that handles chunking and parallelism automatically. Computing a column mean over a massive file looks like this:

import dask.dataframe as dd

# Read with Dask (it handles chunking automatically)
df = dd.read_csv('huge_dataset.csv')

# Operations look just like pandas
result = df('sales').mean()

# Dask is lazy - compute() actually executes the calculation
average_sales = result.compute()

print(f"Average Sales: ${average_sales:,.2f}")

Dask does not load the file into memory; it builds an execution plan and runs it when .compute() is called, using multiple CPU cores where possible. Use it when data outgrows pandas even with chunking, or when parallel processing is wanted without complex code.

7. Sample during development

Exploration and testing rarely need the full dataset. When building a preprocessing pipeline for a machine-learning model, a sample suffices:

import pandas as pd

# Read just the first 50,000 rows
df_sample = pd.read_csv('huge_dataset.csv', nrows=50000)

# Or read a random sample using skiprows
import random
skip_rows = lambda x: x > 0 and random.random() > 0.01  # Keep ~1% of rows

df_random_sample = pd.read_csv('huge_dataset.csv', skiprows=skip_rows)

print(f"Sample size: {len(df_random_sample)} rows")

Loading the first N rows is fastest for quick exploration; random sampling across the whole file is better for statistical work or when the file is sorted in a way that makes the top rows unrepresentative. Use sampling during development, testing, and exploratory analysis before full runs.

Conclusion and practical caveats

Handling large datasets does not require expert-level skills — the key is knowing the data and the task. Most of the time, chunking combined with smart column selection covers 90 per cent of cases; as needs grow, tools like Dask or a switch to columnar formats such as Parquet are the natural next step.

Two caveats worth remembering: downsizing numeric types silently truncates or overflows values that exceed the smaller type’s range, so ranges should be checked first; and chunked or sampled computations can differ subtly from whole-dataset results for non-additive statistics (medians, quantiles), which need dedicated approaches. The pandas scaling guide covers these patterns in depth. Related reading on this site: probability concepts for data science and DIY Python functions for JSON parsing.