GB300 speed on B200 hardware

Maxwell for inference

Maxwell is our code execution model. So far, we've used it to speed up kernels, data pipelines and video model inference. This time we pointed it at an entire LLM serving stack.

Nowadays we all use agents. Agentic workloads are multi-turn: we call the model, it outputs tool calls, those tool calls run, and then go back into the model for its next turn. Typically, much of this time is spent waiting for the model to finish its response. This means that end-to-end latency, i.e., the time from sending a request until the last token arrives, is what decides how fast an agent gets its work done. Like everyone, we run a lot of agents, so this is the number we care about most. There’s another metric which measures “decode speed” alone, i.e., that leaves out the wait before the first token. Since every agent turn starts with that first-token wait, under load this can take up half the turn. Therefore, we prefer the end-to-end metric as a true measure of how long an agentic task really takes.

Agents turn out to be hard to serve. How many concurrent sessions are there? What are they doing? How often do they come back for more? SemiAnalysis built AgentX to measure exactly this.

AgentX is a benchmark that replays real Claude Code sessions against an inference server, including their subagents and the pauses while tools run, and measures both throughput per GPU and how fast each user gets served.

We had Maxwell optimise DeepSeek-v4.1-Flash focusing on end-to-end latency, and measured it using AgentX.

The fastest serving on B200

Maxwell's engine is not only extremely fast, but it is also incredibly consistent.

State-of-the-art engines, like vLLM and SGLang, do well in some regimes and badly in others. Unlike them, Maxwell's engine remains close to the hardware maximum, whether it is serving a handful of agents, or several hundred. Maxwell's engine beats both across the whole range, and pulls furthest ahead when the server is busiest. With over a hundred agents running, each one gets its answers about twice as fast as with vLLM, and the GPUs still produce more tokens overall.

At low load, vLLM is the strongest of the published setups. Maxwell's engine still beats it at every load.

How close is that to what the hardware can do (”the roofline”)? For each load, we estimated the most output a B200 could possibly produce, given its memory bandwidth and compute and the time agents spend waiting on their tools. From 64 to 256 concurrent sessions, Maxwell's engine gets within about 10% of that maximum, and at 64 sessions it's right at it.

GB300 speed on B200

GPU for GPU, how close can B200 get to GB300, NVIDIA's newer and much stronger chip?

To find out, we compared Maxwell's engine with every GB300 run published on InferenceX on up to 4 GPUs — which is the number of B200 GPUs that Maxwell’s optimisations were done on.¹

Agents wait less

End-to-end latency has two components: the wait for the first token, and the time it takes to stream the rest. Agents feel the first component on every turn, because each turn adds new context that has to be read before the model can start answering. Under load, vLLM and SGLang can keep an agent waiting more than a second for it.

Maxwell's engine keeps that wait short. From 1 to 128 concurrent sessions, nine out of ten requests get their first token within about 0.2s. At 128 sessions, that's several times faster than vLLM and SGLang.

Maxwell's engine also handles far more agents at once. Published vLLM and SGLang results on B200 stop at 128 and 256 concurrent sessions. The reason for this is that they struggle to handle high concurrency. For instance, vLLM crashes at 512 sessions and above. Maxwell's engine supports these high concurrencies with ease: at 1,024 sessions on a single node, it produces nearly 5,000 tokens per second per GPU, while still streaming answers at around 40 tokens per second to each agent.

Maxwell adapts dynamically

Published AgentX results are tuned one load at a time. The vLLM and SGLang lines in the charts above aren't really lines. Each point is a separate configuration of the engine, launched and tuned for that one load.

Maxwell's engine is a single engine that serves every load. It adapts as the load changes, from switching its topology, to swapping the underlying kernels, to changing how it schedules requests, all while it serves.

There's much more to this. How do you make sure a faster engine still gives the same answers? What do you do when the optimiser finds a speedup that turns out to be a bug? And how much of this carries over to other models and other hardware? We'll expand on these in future posts.

Coming to OpenRouter

We'll be serving Maxwell-optimised models on OpenRouter soon. Stay tuned…