llama.cpp’s 4.26× Intel Gain Has a Narrow Catch

The 4.26× number is the part of llama.cpp b10016 that will travel fastest. It is also the part most likely to lose its denominator.

The result comes from a contributor-run test on one Intel Arc Pro B70. It measures prompt prefill at an 80,000-token context, using a new oneDNN-backed SYCL attention path designed to send eligible work through Intel’s XMX engines. The comparison is against llama.cpp’s previous Flash Attention path with fa=1. It is not a 4.26× improvement in token generation, an end-to-end chatbot benchmark, or a universal result for Intel Battlemage GPUs.

That distinction does not make the work unimportant. It makes it useful.

The implementation landed through llama.cpp pull request #25222, which GitHub records as merged on July 15 at 07:26 UTC. The b10016 release followed at 09:10 UTC and points to the same merge commit. This is live code, not a proposal waiting in a branch. It is also a fast-moving llama.cpp build tag, not a conventional stable release with months of field testing behind it.

What users have now is a promising new path, a narrow benchmark, and a reason to test their own workloads carefully. It also adds a useful data point to the wider push toward memory-efficient local inference, where the decisive bottleneck is often the movement and storage of context rather than a model’s headline parameter count.

Article Brief

What the benchmark actually shows

4 Points24s Read

1. 0180K prefill–The reported 4.26× gain compares the new XMX path with llama.cpp’s prior fa=1 path on one Arc Pro B70.
2. 02Shorter prompt–At 512 tokens, the reported gain was 1.21× versus the old FA path and about 1.17× versus fa=0.
3. 03Not generation–The published test measured prompt prefill with generation disabled; it does not prove faster token decode.
4. 04Narrow eligibility–Model shape, tensor type, oneDNN build support, sequence count, and prompt length decide whether the new path is used.

What changed inside llama.cpp

Local LLM performance is usually reduced to a tokens-per-second number, but inference has different phases with different bottlenecks.

Prefill is the work done when the model reads the prompt already in front of it. That prompt might be a short question, a long document, a code repository, or tens of thousands of tokens retrieved by an agent. Decode is what happens afterward, as the model generates new tokens one at a time.

The new b10016 result is about prefill.

The merged implementation adds a oneDNN-backed attention path to llama.cpp’s SYCL backend. When the operation meets the required conditions, the path can dispatch attention work to Intel’s XMX matrix engines. When those conditions are not met, llama.cpp falls back to another implementation.

Those conditions matter. According to the merged code and its accompanying notes, eligible work requires f16 key and value tensors, supported attention-head dimensions, matching key and value head sizes, compatible grouped-query attention, a single sequence, and a prefill length of at least 32 tokens. Models using attention sinks or softcap behavior fall back. oneDNN support must also be present at compile time, and the implementation includes a runtime kill switch.

In plainer terms, installing b10016 does not guarantee that every prompt will run through the new kernel. The model architecture, tensor layout, build configuration, and runtime request all help decide whether it qualifies.

The pull request also adds multi-device synchronization work. That should not be mistaken for proof of multi-GPU scaling. No multi-GPU performance result was supplied with the benchmark.

This is exactly where headline language tends to outrun protocol proof. The code contains a faster route for eligible prefill shapes. It does not replace every attention operation on every Intel GPU.

Where 4.26× comes from

The strongest reported result used Qwen3.6-27B-Q8_0 on an Intel Arc Pro B70. The benchmark ran prefill only, with generation disabled, and used two repeats on one machine.

At an 80,000-token prompt, the new XMX path reported 730.20 tokens per second. llama.cpp’s prior Flash Attention path reported 171.22 tokens per second. Divide the first result by the second and the reported gain is 4.26×.

That is a substantial improvement in the measured operation. A developer feeding a very large repository, transcript, or retrieved document set into a local model could care far more about prefill time than a user asking a two-line question. Long-context work is not a synthetic curiosity anymore. It is one reason the practical boundary between edge and cloud AI keeps shifting as local hardware becomes more capable.

Still, the boundary of the test is unusually important. There is no fa=0 result at 80,000 tokens because that path had already run out of memory. The new XMX path then ran out of memory at 82,000 tokens.

The 4.26× figure therefore sits close to the tested memory edge. It compares two Flash Attention paths that behaved very differently at long context; it does not compare the new path with a successful non-FA baseline at the same 80,000-token length.

That does not invalidate the result. It tells us what the result answers: on the contributor’s B70, for this model and eligible long-context prefill, the oneDNN/XMX route was much faster than llama.cpp’s previous fa=1 route.

It does not answer how fast the model chats afterward.

The 512-token result is less dramatic—and more representative

At a 512-token prompt, the new path reported 948.53 tokens per second. The prior fa=1 path produced 783.01 tokens per second, making the gain about 1.21×.

The available fa=0 result at that same length was 813.96 tokens per second. Against that baseline, the new path was approximately 1.17× faster.

That is the calmer number readers should keep beside 4.26×.

A 17% to 21% prefill improvement can still be meaningful, particularly when local agents repeatedly rebuild context. But it is not the kind of number that supports a claim that Intel LLM inference has suddenly become four times faster across ordinary use.

The widening gap at 80,000 tokens suggests the new path handles this particular long-context workload far better than the previous FA implementation. It does not establish a fixed scaling curve for other models, quantizations, or Battlemage cards.

There is another missing comparison: the pull request does not provide a controlled SYCL-versus-Vulkan benchmark across the same contexts. Community reports contain useful observations about both backends, but they come from different machines, drivers, models, and command lines. Mixing them into one performance claim would manufacture certainty that the evidence does not contain.

Why the Arc Pro B70 is a plausible beneficiary

Intel lists the Arc Pro B70 with 32 Xe cores, 256 XMX engines, 32GB of GDDR6 memory, and 608GB per second of memory bandwidth. It is a large Battlemage workstation card with enough memory to attempt contexts that smaller consumer GPUs cannot hold.

The new llama.cpp path is designed to use the part of that hardware intended for matrix-heavy work. That makes the direction technically credible. It does not make the result portable by default.

Intel’s Arc B580, Arc Pro B60, and Arc Pro B70 share a broader architecture family, but they do not share the same memory capacity, engine count, bandwidth, or driver environment. A B70 result should remain a B70 result until other cards reproduce it.

The oneDNN 3.11 documentation describes scaled dot-product attention graph recognition and Intel GPU optimizations using f16 or bf16 inputs, f32 intermediates, supported head dimensions up to 512, and XMX acceleration. It also makes clear that graph recognition is not a promise of speed in every case. Its reference primitive can retain an O(S²) intermediate.

That warning matters because the name “Flash Attention” carries its own expectations. The original FlashAttention paper describes an exact, IO-aware tiled algorithm that reduces traffic between high-bandwidth memory and on-chip SRAM while avoiding materialization of the full attention matrix.

The paper explains why attention can be reorganized to save memory movement. It does not independently validate llama.cpp’s Intel implementation, its memory behavior, or its benchmark. The precise description here is llama.cpp’s oneDNN-backed Flash Attention path, not a claim that the original implementation has simply been transplanted onto Battlemage.

The correctness check is encouraging, not conclusive

Performance work is only valuable if the output remains trustworthy.

The contributor included a Gemma-4-12B-Q8_0 perplexity comparison. The reported values were 698.19 ± 10.44 for the new path and 695.79 ± 10.40 for the comparison, alongside a shorter wall-clock time.

Those uncertainty ranges overlap closely. That is useful evidence that the tested path did not produce an obvious quality collapse in this limited run.

It is not evidence of zero accuracy loss. The release summary describes a 0.11% difference, while the displayed central values differ by roughly 0.34%. Neither figure justifies a sweeping quality claim. Perplexity on one model is also not a substitute for checking structured output, tool calls, retrieval fidelity, or numerical stability across architectures.

The responsible reading is narrow: the contributor’s limited correctness result is reassuring enough to justify further testing. Independent reproduction is still missing.

Who should test b10016 now

The strongest use case is a workload where prompt ingestion is already visible as a delay.

That includes local coding agents reading large repositories, retrieval systems assembling long evidence packets, document-analysis tools, and sessions that repeatedly rebuild large context windows. In those cases, prefill is not a footnote. It can dominate the wait before the first generated token arrives. The same latency question increasingly shapes consumer on-device AI, even though the hardware and software stacks are different.

Users running short prompts followed by long answers should expect a different balance. A faster prefill path may improve time to first token while leaving most of the generation phase unchanged. The published benchmark does not establish a decode speedup.

Testing should preserve the denominator. Use the same llama.cpp commit, driver, model, quantization, context length, batch settings, and backend when comparing the new path with the old one. Record prompt-processing speed separately from token-generation speed. Watch the logs to confirm that the request actually takes the oneDNN path rather than silently falling back.

Context length should be swept rather than sampled once. A useful run might begin at 512 or 4,096 tokens, then move through 32K and the largest context the card can hold. Memory consumption, time to first token, prompt throughput, generation throughput, and output correctness all belong in the result.

If the workload uses attention sinks, softcap behavior, an unsupported head layout, or multiple sequences, the new route may not apply. If the card is a B580 or B60, the B70 number is a hypothesis to test, not a result to inherit.

There is also a release-management decision. b10016 is evidence that the code has landed, but llama.cpp build tags move quickly. Anyone using it for production work should pin the tested commit, preserve the previous working build, and verify representative prompts before switching traffic. The project’s SYCL backend documentation is the starting point for supported hardware, build flags, and runtime behavior.

The comparison Battlemage users still need

llama.cpp’s current documentation explains how to build and run the SYCL backend. Release notes and community tests offer isolated hardware results, but they do not answer the decision that matters for a buyer or operator.

Should an Arc user choose SYCL or Vulkan, turn Flash Attention on or off, and what changes at short, medium, and very long context? Answering that requires the same commit, driver, model, quantization, command line, and hardware across every run.

b10016 supplies one important piece of that answer. It shows that an eligible oneDNN/XMX prefill path can materially outperform llama.cpp’s prior FA route on one B70, with the advantage becoming striking at 80,000 tokens.

It does not settle the backend choice, the consumer-card result, or the generation story.

A useful gain, kept in its proper box

The 4.26× figure is real within the benchmark that produced it. That box contains one Arc Pro B70, one Qwen model, prefill-only execution, two repeats, eligible oneDNN/XMX shapes, and an 80,000-token comparison against the prior fa=1 path.

At 512 tokens, the same work produced a smaller 1.21× gain over the old FA path and about 1.17× over fa=0. Those numbers are not disappointing. They are the context that makes the long-context result believable without turning it into marketing.

Intel’s Battlemage hardware has gained a more capable route through llama.cpp. Now the result needs reproduction across cards, models, drivers, contexts, and real applications. Until then, b10016 is a strong prefill optimization—not a fourfold rewrite of local inference.