Why is Pandas So Slow? 4 Fixes to Make Pandas 100x Faster

The same Pandas job, written seven ways over a million rows: the slowest takes 8.5 seconds, the fastest takes 0.02, and no new library is involved. Most advice for slow Pandas points somewhere outside your file: switch to Polars, try DuckDB, rent a GPU. This video goes through the four places time actually goes, in the order you should check them. First, a 90-second cProfile diagnosis (one million multiplications costing 175 million function calls). Then the four fixes: vectorizing away the per-row loop (around 100x), tightening dtypes with categoricals and downcasting (7.6x less memory than object columns and a 4x faster function), switching engines to Polars or DuckDB (8-25x over well-written Pandas when they own the Parquet read), and GPU acceleration with RAPIDS cuDF's cudf.pandas mode. Every number comes from a benchmark repo you can clone, and every implementation is checked against the others before any timing prints. ⏱️ Chapters 0:00 - Same job, seven ways: 8.5s vs 0.02s 0:52 - The benchmark setup 1:45 - The four bottlenecks, in order 2:45 - Start with a profiler: reading cProfile 4:24 - Fix 1: stop looping, vectorize (~100x) 6:11 - Fix 2: dtypes and categoricals 8:28 - Fix 3: switch engines to Polars or DuckDB 10:14 - Fix 4: GPU acceleration with RAPIDS cuDF 12:08 - Which fix matches your bottleneck 🔗 Links pandas: https://pandas.pydata.org Polars: https://pola.rs DuckDB: https://duckdb.org RAPIDS cuDF (cudf.pandas): https://docs.rapids.ai/api/cudf/stable/cudf_pandas/ #pandas #python #dataengineering #pandas #pandasperformance #python #Polars #DuckDB #cuDF #RAPIDS #vectorization #iterrows #cProfile #dataengineering #datascience #Parquet #pythonperformance #dataframe