Blazing Fast pandas Without Changing a Line

‍‍

WarpSpeed completely rewrote pandas’ kernels from scratch, specialised for modern hardware.

The result is a drop-in replacement version of pandas that runs 38.4x faster on a standard benchmark; reducing 10 minutes of work down to merely 17 seconds. All while retaining the same API on the surface, same semantics underneath, and remaining 100% compliant.

Read on to see how.

‍

The Benchmark

The standard benchmark behind that number is PDS-H: a suite of 22 queries over a mock wholesale company, the kind of business pipeline that runs on pandas daily. Each query starts life as an everyday business question. Let’s take a simple example. What were the top 5 product categories by revenue last quarter? Here is the pandas it turns into:

The real PDS-H queries get more involved than this, but their commonality is that they are all ordinary business questions, expressed as a handful of SQL-like operations.

We run exactly these queries from the suite maintained by the team behind Polars, keeping the same data and the same pandas queries and swapping in only WarpSpeed underneath, without changing a single line of code.

A worked example: merge

So, where does the speedup actually come from? And why it is so hard to attain without changing the semantics of pandas at all? To help answer this, let’s walk through one operation all the way down. merge is a good candidate, since it appears in nearly every PDS-H query and exercises both halves of the problem at once.

A merge (or a join, in database terms) combines two tables on a column they share.

You have a table of orders. Each row names the customer who placed it, but only by ID; the name, address, and signup date live in a separate customer table, one row per ID. A merge looks up each order's customer and copies those columns in, so every order row comes out carrying the customer's name.

In the figure above, the customer table is on the left, where the ID is unique: one row per customer, so it is the side you look into. Orders sits on the right and reuse those IDs freely. So Alice, customer 1 with two orders, shows up as ID 1 twice, and both of her rows indeed come out as saying "Alice."

Why pandas is slow

pandas runs a hash join. Under the hood that requires several separate passes over the data:

1. Factorize the keys. The key column on each side is mapped to dense integer codes (a hash, for string keys). Every key on both sides is touched.
2. Build the hash table (group in the figure below). One side's codes go into a hash table mapping each key to the row positions where it occurs, keeping a list of positions since a key can repeat.
3. Probe it (match in the figure below). Each key on the other side is looked up, producing a pair of integer arrays: for every output row, which left row and which right row it came from.
4. Gather the columns. Those two index arrays are applied to every column of both frames, one "take" pass per column, copying the selected rows into freshly allocated output arrays.

None of this runs in parallel: every step is confined to a single core. Worse, most of the work is memory-bound rather than compute-bound. The build and probe scatter through the hash table at unpredictable addresses, and the gather reads source rows in whatever order the index arrays happen to point, so the core spends more of its time waiting on cache misses than doing arithmetic. The passes don't fuse either; each one writes out an intermediate that the next reads back in. In effect we pay for a single core, stalling on memory, several times over.

…and how we speed it up

WarpSpeed fixes both. It fuses the four passes — factorize, build, probe, gather — into a single sweep, so each row is touched once instead of written out and read back, and it spreads the work across cores by radix hash partitioning the keys.

Dispatching on the data

WarpSpeed goes one step further. Rather than fixing an algorithm up front, each kernel inspects its input cheaply at runtime, and when a cheap check says a shortcut is safe, it takes it, sometimes relying on conditions that a human author would rarely think to test.¹

A small case makes this concrete. The pandas call query.isin(subject) tests each element of query for membership in subject. By default pyarrow builds a hash set from subject and probes it. A perfectly reasonable choice, and algorithmically optimal in the general case.

But when subject is small and its values are densely packed, so that max(subject) - min(subject) is close to len(subject), a bitset does better: allocate one bit for each value across subject's range, set the bits that actually occur, and every membership test becomes a single indexed read, with no hashing and fully sequential access. WarpSpeed checks the range at runtime and takes that path whenever it turns out cheaper.

The same idea returns in merge. Suppose we merge orders against customers, but after cutting the customer side down to a handful of rows, say every customer from one small town. Now the expected output is tiny, and most orders will find no match, so the probe mostly fails and the kernel turns compute-bound rather than memory-bound.

Here a Bloom filter on the integer key wins. A Bloom filter is one-sided: it can return false positives but never false negatives, so a negative answer is definitive (if it says a key is absent, it truly is). Since almost every order misses, almost every probe gets that definitive negative and is rejected in a few instructions, with no hash-table lookup at all. The rare positives (the real matches, plus the occasional false positive) still fall through to the exact hash-table check, so the result stays correct; the filter only saves the lookup on the common (rejecting) case. This pays off precisely when the bitmap fits in L2 cache, which the small key set makes possible.

What it takes to replace a library

pandas is a scientific Python library, built the way most of them are: friendly Python objects on top, compiled code doing the heavy computation underneath.

The object you work with is the DataFrame, a table with named columns and numbered rows. Each column holds values of a single type: one column of integers, another of dates.

When you call something like df.merge(...), the Python layer does the bookkeeping: which columns to line up, what the result’s columns should be called, how the rows are labeled. But the values themselves live beneath that surface as buffers of raw memory, and the number-crunching is handed down to a kernel: a small compiled routine that loops over those buffers directly, far faster than Python could.²

Verification is tricky

Building a faithful replacement is hard for two different reasons. The first is the kernels themselves: low-level C++ where every subtlety, from how floating-point rounds to how missing values behave, has to come out bit-for-bit identical to pandas. The second is matching pandas’ semantics around them: the same column names, the same rules for when data is copied or shared. The two are hard in different ways and go wrong in different ways, so verification splits the same way, into a buffer layer for the numbers and an integration layer for the semantics.

The buffer layer

The buffer layer is the raw numbers in memory. To match pandas you have to reproduce every decision baked into the data itself, however odd.

Take missing values, for instance. Pandas writes "no value" as NaN, which follows some surprising rules: NaN isn't even equal to itself, and a column of integers that picks up a single missing value is silently coerced into floats. Another example is a plain sum over a long column of float64 values: IEEE-754 addition isn't associative, so the moment you fan the reduction across many cores the values accumulate in a different order than pandas’ single serial loop, and the result drifts. Matching pandas bit-exactly means carrying Kahan compensated summation rather than settling for a faster float32 accumulation. To keep ourselves honest, our verifiers deliberately seed columns with extreme outliers, so any kernel that cuts that corner drifts past tolerance and fails.

The integration layer

The integration layer is everything about how those buffers get wired into a DataFrame: the column names and row labels, the copy-versus-view rules, and which columns secretly share memory.

What happens if both tables you're merging have a column called price? pandas keeps both and renames them price_x and price_y, and you have to rename in exactly the same cases. And what if two columns of a single frame are quietly the same block of memory, as they can be, since pandas groups same-type columns together and ordinary assignments can leave them aliased? Then a kernel that treats each column as its own private buffer can write to one and silently corrupt the other.

Testing it end-to-end

None of this is tested on toy frames conjured in memory. Every check runs end-to-end, on data built the way real pipelines build it — through read_csv, read_parquet, and after a concat — so the buffers under test are laid out in memory exactly as pandas would lay them out. That exposes two independent axes of adversarial input, one aimed at each layer:

Results

The WarpSpeed accelerated version of pandas is not only a drop-in replacement and over 38x faster on average, but in fact, it is strictly faster on every single query in the benchmark.

Could Claude do it?

Claude Code is a strong baseline for any type of software engineering work. So we asked a straightforward question: if you point Claude at pandas and ask it to make things faster while staying correct, how far does it get?

We ran the strongest baseline we could think of, Claude Code in “ultracode” mode. This allows it to use numerous subagents, set up dynamic workflows, and apply its strongest “machinery” in order to get the job done. We tested both Opus and Fable (separately). They each ran hundreds of subagents and totaled more than a thousand dollars in API credits.

Both runs completed successfully. However, their performance gains were extremely modest compared to WarpSpeed. Both Opus and Fable attained only about 1.3× geomean speedup. This is compared to WarpSpeed’s average of 38×. Opus only performed surface-level Python changes. In contrast, Fable attempted only slightly more ambitious native-code changes, but in the process, regressed on about 10% of the queries, which ended up running slower than the original pandas.

Why was this the case? Well, to stay within pandas’ strict semantics, and since they were lacking strong verification mechanisms, the Claude agents mostly relegated themselves to surface-level refactors, mostly centered around rewriting Cython kernels, rather than perform sweeping changes, like those enacted by WarpSpeed. This underscores why strong verification is necessary, doubly so when dealing with complex code bases — without strong verifiers it’s hard to muster the “courage” needed to make sweeping algorithmic changes.

Afterword

Legacy code doesn't scale. It has to stay backward compatible and is rarely rebuilt, falling further behind with every hardware generation.

Rewriting the code takes an expert; a person proficient in the library's internals with deep knowledge of the hardware, working from scratch. Those experts are the bottleneck. Since migration is technically daunting and risks correctness, by default the code stays, and execution remains slow.

At doubleAI, we're automating and scaling expertise. We first applied our system WarpSpeed to GPU kernels. Now, we have done the same to pandas: the default tool of data science and the layer under a huge share of ML and business-logic pipelines.

‍

‍