Qwen3.5 on Ironwood: What Google’s 4.7x Gain Proves

Article Brief

Key Takeaways

4 Points24s Read

1. 01Measured gain–Google reports 4.7x higher prefill-heavy and 3.1x higher decode-heavy throughput at concurrency 512.
2. 02Real baseline–The comparison is Google’s April-to-June Ironwood software progress, not a new hardware generation.
3. 03Test topology–One host used four physical Ironwood chips exposing eight logical devices.
4. 04Open question–No matched Nvidia GPU test was published, so cross-vendor leadership remains unproven.

A 4.7x inference speedup is the kind of number that can overpower the details around it. Google says its engineers achieved roughly that throughput gain on prefill-heavy Qwen3.5-397B workloads between April and June 2026, while decode-heavy throughput improved about 3.1x. Those are meaningful results for one of the largest open-weight mixture-of-experts language backbones available.

They are not evidence that Google’s Ironwood TPU is 4.7 times faster than an Nvidia GPU.

Google did not publish that comparison. Its figures describe progress from an internal April software baseline to a more heavily optimized June stack on the same accelerator platform. The useful conclusion is more specific: modern AI inference performance depends on how precisely the model, compiler, kernels, memory system and interconnect are made to work together.

The silicon matters. So does everything required to keep it busy.

What Google’s 4.7x number actually measures

Google’s systems engineering report evaluates the FP8 Qwen3.5-397B-A17B checkpoint in text-only mode on one host with four physical Ironwood chips. Those chips expose eight logical devices, which is why Google labels the configuration v7x-8. Throughput is reported as total input and output tokens processed per second, divided by the physical chip count.

The team used two deliberately different workload shapes. Its prefill-heavy test paired an 8,000-token input with a 1,000-token output. That pattern emphasizes the parallel computation needed to absorb a long prompt. Its decode-heavy test reversed the balance, using a 1,000-token input and an 8,000-token output, which puts more pressure on repeated token generation and memory bandwidth.

Both workloads were tested at 64, 128, 256 and 512 concurrent requests. Google’s headline improvements—about 4.7x for prefill-heavy traffic and 3.1x for decode-heavy traffic—refer specifically to the 512-concurrency tier and compare the company’s April and June implementations.

That baseline distinction is essential. The result measures the value of Google’s optimization work over two months. It does not compare Ironwood with an H100, B200, B300, GB200 or any other GPU. It also does not establish a universal speedup for every prompt length, batch size, serving engine or latency target.

TECHi’s earlier look at the broader AI infrastructure war described the contest in terms of chips, power and capital. Google’s Qwen work adds another competitive surface: how quickly a platform can turn an awkward new model architecture into an efficient production workload.

Why Qwen3.5 is awkward to serve

The Qwen model card identifies 397 billion parameters in the language backbone but about 17 billion activated for a forward pass. That sparse mixture-of-experts design avoids using every expert for every token, yet it creates a different systems problem: the serving stack must route tokens to the correct experts without wasting memory or communication bandwidth.

Qwen3.5 also mixes two attention systems across 60 language layers. Fifteen repeating groups each contain three Gated DeltaNet layers and one gated-attention layer, all paired with mixture-of-experts feed-forward blocks. The attention layers have only two key-value heads, while the expert layers contain 512 routed experts and select ten of them alongside one shared expert.

Those numbers do not fit neatly into a default eight-way tensor-parallel layout. Google explains that two key-value heads cannot be uniformly divided across eight logical devices. Replicating those heads on every device would work, but it would duplicate the key-value cache and consume memory needed to sustain higher concurrency.

This is where the result becomes more interesting than a raw accelerator specification. Google did not solve the problem by applying one universal “TPU optimization” switch. It changed the way this particular model was divided, moved and executed. That is a different chapter from Alibaba’s earlier Qwen 3 launch: model capability starts the competition, but serving efficiency determines what operators can afford to put in front of users.

The speedup came from coordinated fixes

Google’s central change was a hybrid parallelism scheme. Attention work used eight-way data parallelism, allowing each logical device to retain Qwen’s two complete key-value heads and process its local request batch. The mixture-of-experts layers switched to eight-way expert parallelism, distributing the 512 routed experts evenly so that each device held 64.

Moving between those two layouts requires token exchange. Google chose an all-gather and reduce-scatter sequence instead of dynamic all-to-all routing. The approach replicates more token data locally, but Google says it produces more deterministic serving latency. Engineers then fused and pipelined communication so that transfers could overlap with useful computation.

The team also wrote model-aware kernels in Pallas, JAX’s custom-kernel language. Those kernels addressed three separate execution paths: attention, expert routing and Qwen’s Gated DeltaNet recurrent state.

One attention fix changed the key-value page size from 16 tokens to 256. On Google’s concurrency-512 decode test, this reduced a reported kernel step from 428 microseconds to 283 microseconds. Larger pages can waste more space, but on Ironwood they also reduce indexing work that was stalling the Vector Processing Unit. It is a clean example of a setting that cannot be judged independently of the target hardware.

The expert-routing path was assigned to Ironwood’s SparseCore, which is designed for irregular addressing and embedding-style operations. Rather than creating padded intermediate tensors in high-bandwidth memory, Google’s custom gather and reduction kernels moved routed token data into contiguous buffers. The report says a fused unpermutation path cut the relevant HBM reads from 20 to 10 and writes from 15 to five.

For Gated DeltaNet, Google fused the causal convolution with the recurrent state update. Keeping intermediate values in registers avoided six HBM round trips. The team also rearranged part of the linear-attention calculation, used lower-precision state variables where its scoped checks supported that choice, and introduced layouts for variable-length sequences that reduced padding.

None of these changes alone explains 4.7x. The gain emerged because sharding, communication, memory layout and kernels were tuned as one system. That same memory-first instinct appears in TurboQuant’s memory-efficiency approach, although the two projects solve different bottlenecks and should not be treated as comparable benchmarks.

Google’s broader Ironwood architecture description makes the co-design philosophy explicit. Each chip has 192 GiB of high-bandwidth memory and a stated 7.4 TB/s of peak HBM bandwidth. The company’s 42.5-exaflops FP8 figure applies to a complete 9,216-chip system, not the four-chip host used for this Qwen test. XLA supplies broad compiler optimization; Pallas permits targeted control when a model’s bottleneck falls outside the general path.

The roofline results clarify the achievement

Google also compared its measured throughput with its own estimated hardware limits at concurrency 64. These numbers are separate from the 4.7x and 3.1x improvements reported at concurrency 512.

For the 8K-input, 1K-output workload, the company estimated a discounted prefill ceiling of 4,500 tokens per second per chip. Its optimized stack reached 3,707, or 82.4% of that discounted roofline. For the 1K-input, 8K-output workload, it estimated a discounted 850-token decode ceiling and measured 677, or 79.6%.

Those percentages are not shares of an absolute theoretical maximum. Google’s undiscounted estimates were higher—5,170 for prefill and 978 for decode—before engineers applied an efficiency discount to account for real execution. The measured figures were about 71.7% and 69.2% of those undiscounted estimates, respectively.

A roofline model asks what should be possible once the dominant physical constraint is identified. Prefill is largely compute-bound because many prompt tokens can be processed in parallel. Decode is more often memory-bound because weights and state must be moved repeatedly while tokens are produced sequentially.

The result shows that Google closed much of the gap between a functional Qwen implementation and the performance its engineers believe Ironwood can deliver. It does not show that the same roofline assumptions apply to another vendor’s hardware.

Google Cloud’s Ironwood performance guide reinforces that point: teams are told to identify whether a workload is limited by compute, memory or interconnect traffic before choosing sharding and kernel strategies. More parallelism is not automatically better when the extra communication becomes the bottleneck.

Why this is not an Nvidia comparison

A defensible accelerator comparison would have to control far more than the model name.

The model checkpoint and numerical precision must match. Input and output lengths, concurrency, batching policy and maximum context must be equivalent. The serving engines need comparably mature implementations. Measurements should expose time to first token, output-token latency, aggregate throughput and service-level constraints, not only the most flattering aggregate number.

Hardware count is not enough either. Accelerators differ in memory capacity, bandwidth, power envelope, networking and price. Production buyers ultimately need cost per successful request or cost per useful token at their required latency and quality—not an isolated tokens-per-second result under a vendor-specific configuration.

Google’s report provides no controlled Nvidia baseline with those conditions. It also does not claim one. Using the 4.7x figure as proof that Ironwood beats Nvidia would turn a solid engineering result into a claim the evidence cannot carry.

The reverse assumption would be just as careless. The absence of a GPU comparison does not diminish Google’s internal improvement or establish that a GPU would win. It simply leaves the cross-vendor question unanswered.

This distinction matters because Qwen3.5 is available through an open model ecosystem, with framework support documented in the Hugging Face Transformers guide. Different infrastructure teams can optimize the same architecture in different ways. The eventual fit for a deployment may be the stack that best matches its prompt mix, latency target, budget, regional capacity and existing software.

What infrastructure buyers should take from the result

Google’s work is a practical warning against benchmarking a new model too early and treating the first number as permanent.

The April implementation ran on Ironwood, but it did not yet reflect everything the hardware could do for Qwen’s unusual attention and expert-routing design. By June, model-specific sharding and kernels had substantially changed throughput without requiring a new chip.

A team comparing infrastructure should ask how mature each serving path is. Is the model running through generic kernels or tuned ones? Can the engine distribute its experts without excessive communication? Does the attention layout preserve enough memory for the required concurrency? Are variable-length requests causing padding waste? Which optimizations are upstream, reproducible and supported outside a vendor’s internal environment?

Correctness needs equal weight. Google says it tested one lower-precision gating path against a Float32 reference and observed zero deviation in the audited gating weights. That is useful evidence for the narrow calculation it measured. It is not proof of identical end-to-end answers or unchanged quality across every task, checkpoint and prompt.

Availability and operating cost remain open questions in the report. So do performance under low concurrency, strict interactive-latency requirements and different prompt distributions. A batch-heavy agent platform and a consumer chatbot may reach different conclusions even when both serve Qwen3.5. Those gaps belong beside Google’s AI infrastructure spending when buyers evaluate what impressive engineering means for production economics.

The competitive advantage is optimization speed

Google frames parts of its work as reusable modules rather than a collection of one-off patches. That is strategically important. Open models are changing too quickly for an infrastructure team to spend months rebuilding every serving path from scratch.

Reusable attention, grouped-matrix, routing and communication components shorten the distance between “the model runs” and “the model runs efficiently.” Model-specific work does not disappear, but engineers can concentrate it on genuinely new components such as Qwen’s Gated DeltaNet path.

That is the most consequential claim in the report. An accelerator platform competes not only through peak FLOPs or memory bandwidth, but through how quickly its software team can adapt those resources to an unfamiliar model.

Google’s 4.7x result shows how far an internal baseline can move when the full stack is tuned around one model. It does not crown a universal hardware winner. It demonstrates that, for frontier inference, the benchmark is increasingly a property of the entire system.