Got Qwen3.8-Next-Flash ngram SSD offload working in llama.cpp!

Enhancing Llama.cpp Performance with Qwen3.8-Next-Flash: SSD Offloading of Ngram Tables via mmap

In the world of large language models, optimizing memory usage and performance is a continual pursuit. Recently, exciting developments around the Qwen4 architecture have highlighted innovative methods for managing sizeable ngram tables—which store indexable facts and specific data—more efficiently. This post explores a practical approach to offloading these ngram tensors to SSD storage using mmap, thereby substantially reducing RAM requirements while maintaining performance.

The Significance of Ngram Tables in Qwen4 Architecture

Qwen4 introduces a notable design feature: exporting indexable facts into dedicated ngram lookup tables rather than embedding them directly within the model’s weights. This design choice allows:

Increased Parameter Space: Freed-up parameters enhance reasoning and problem-solving capabilities, especially beneficial for models with a 6B parameter size.
Faster Information Access: Since the ngram data is stored separately, it obviates the need for computationally expensive matmul operations for lookup, enabling more efficient retrieval.

For example, a 3.8-Next-Flash model with 125 billion parameters and an additional 51 billion ngrams totals approximately 176 billion parameters. Remarkably, these ngrams constitute about 30% of the total model footprint, suggesting significant memory savings potential.

Leveraging mmap for SSD Offloading

Llama.cpp offers a feature—--mmap—which allows tensors to be memory-mapped to disk storage. When properly configured, this enables the model to access parts of its data from SSD, freeing up RAM without compromising latency or throughput.

Practical Scenario:
On a system like a 128GB MacBook Pro with a 128GB RAM limit, running large models such as Q4_K_XL (~112GB in quantized form) or even Q5 scales becomes feasible by offloading less frequently accessed data to SSD. The --mmap feature makes it possible to handle the entire ngram table—potentially exceeding 100GB—by paging it in as needed.

Technical Challenges and Solutions

During initial attempts, a notable challenge emerged: the interleaved layout of the quantization shards causes the engram tensors to get tangled with other data, leading to memory constraints. To address this:

Quant Layout Adjustment: By remapping or scrambling the internal structure of the quantization, it’s possible to isolate engram data to be stored separately, facilitating mmap offloading.
Scripted Remapping: Using a custom script (provided below), the model’s quant layout can be modified to segregate engram tensors, allowing successful SSD offloading.

Once this adjustment is made, the benefits are clear:

Performance Impact

Crucially, offloading the ngram data in this manner does not degrade performance. Tests indicate consistent throughput around 36 tokens/sec (without MTP), matching in-memory operation. The ability to offload large portions of the model means that models like Q2_K_XL (requiring about 64GB) or Q4_K_XL (about 96GB) can run smoothly on machines with memory constraints, provided the GPU’s wired limit is adequately configured.

Implementation Guide

No special flags are required to enable mmap—it’s the default behavior on current builds of llama.cpp (including the Qwen3.8-Next-Flash PR #27742). However, to adapt your model for ngram offloading:

Use the provided script to adjust your model’s quant layout.
Ensure --mmap is enabled (which it should be by default).

Download the script here: Quant Layout Remapper

Note: Always verify your model’s layout after modification, and tailor the script if working with different quantization sizes or architectures.

Additional Resources and Community Input

For further tuning, some community members have shared flag configurations to avoid the need for layout remapping, such as:
--load-mode mmap --override-tensor per_layer_token_embd.weight=CPU
Unsloth, the developer behind the model, has incorporated similar layout adjustments in recent releases, leaving engram data in separate shards, which further optimizes memory management.

Final Thoughts
Offloading sizable ngram tables to SSD via mmap, combined with strategic quant layout modifications, unlocks new possibilities for running large models efficiently on limited hardware. This approach significantly reduces RAM footprint without sacrificing inference speed, making advanced AI models more accessible to a wider range of systems.

For developers and enthusiasts aiming to implement this technique, the key steps are:

Adjust quant layout to detach ngram tensors.
Use the provided remapping script to modify your model’s quantized weights.
Enable --mmap in llama.cpp to activate on-disk tensor loading.
Tune your system’s wired GPU memory limit to accommodate larger models.

Feel free to explore the script and community flags mentioned above, and share your experiences. The ongoing evolution of model optimization continues to break down hardware barriers, bringing powerful AI capabilities within reach.