NVIDIA Groq 3 LPX Hits 3,431 Tokens/sec on Long Context Benchmarks

NVIDIA's Groq 3 LPX, the interactive inference accelerator for the Vera Rubin platform, has posted its first third-party benchmark: 3,431 output tokens per second on Artificial Analysis's 100K context benchmark using the Gemma 4 31B model. This result targets the "high-interactivity" serving tier—the kind of responsiveness needed for multi-turn agentic sessions where context grows into the hundreds of thousands of tokens.

Why Long Context + Interactivity Is Hard

Agentic workloads are multiturn: each turn appends output to the context, and later turns must re-process everything that came before. At 100K+ input tokens, serving at 3,000+ tokens per second per user while managing a large KV cache is a systems challenge. The standard trick—tensor parallelism (TP)—splits computation across chips, but at the very small batch sizes required for interactive latency, the fixed cost of coordinating (collective ops) can eat the speedup.

The bottleneck is first-bit latency, not bandwidth. As the article states, with small tensors the transfer time is dominated by latency (A) rather than the amount of data (N) divided by bandwidth (B).

Groq 3 LPX's Approach

Groq 3 LPX solves this with deterministic, compiler-scheduled workload planning. The compiler pre-schedules when every tensor leaves and arrives, overlapping computation and communication at fine granularity. Preplanned chip-to-chip networking minimizes first-bit latency, making TP effective even at batch size 1.

On the SPEED-Bench, which measures agentic and coding-specific tasks, Groq 3 LPX hit a median of 4,767 output tokens per second. The system also supports multiple deployment configurations with Vera Rubin NVL72:

• Prefill-decode disaggregation
• Attention-FFN disaggregation
• Speculative external-drafter decoding

These can scale to multi-trillion parameter models, pairing Groq 3 LPX's interactivity with Vera Rubin's throughput for large-scale AI factories.

Who This Is For

Developers building agentic systems that demand high interactivity with long context—especially those running multi-agent workflows on 2T+ parameter models—will find this benchmark relevant to infrastructure planning.