Innovative Storage Format Boosts Large Language Model Inference Speeds Up to 2x
Introduction
As the deployment and utilization of Large Language Models (LLMs) continue to expand across diverse applications, optimizing their inference performance remains a critical area of research and development. Traditional storage formats like GGUF and safetensors excel at efficient model storage and reproducibility but are not inherently designed with runtime execution speed in mind. Addressing this gap, recent developments have introduced a novel approach—an optimized storage format that facilitates near-instantaneous model loading and inference acceleration.
This article details the creation and benchmarking of ThinTensor, an innovative storage and runtime format that can deliver up to double the inference speed of existing solutions under specific conditions.
The Concept Behind ThinTensor
The motivation for ThinTensor emerged from observations during unsupported model experimentation. While existing formats effectively store model weights, their design doesn’t optimize for fast parsing and execution. Therefore, the core idea was to encode models in a way that aligns the storage structure directly with runtime parsing workflows, minimizing loading and preprocessing overhead.
Key Technical Design
At the heart of ThinTensor is a structured archive format featuring a comprehensive header that simplifies and accelerates model loading. The header includes:
- Length and version information
- Offsets to vital components like the manifest, page table, and raw data
- An archive hash for integrity verification
Sample SDK structure (conceptually):
rust
pub struct Header {
pub header_len: u32,
pub version: u32,
pub manifest_off: u64,
pub manifest_len: u64,
pub page_table_off: u64,
pub page_count: u64,
pub data_off: u64,
pub archive_hash: [u8; 32],
}
This setup allows rapid access: the loader reads the header, locates the manifest (a JSON blob containing model metadata), and maps raw pages directly into memory. Consequently, the inference engine can parse and utilize model weights with minimal delay, bypassing traditional, time-consuming loading procedures.
Additionally, the format uses the extension .thin to denote these archives, emphasizing their streamlined nature.
Performance Benchmarks
Preliminary benchmarking results demonstrate substantial performance gains over conventional formats, especially when comparing inference throughput with Hugging Face’s BF16 implementations:
| Model | ThinTensor tok/sec | HF tok/sec | Speedup | Cosine Similarity |
|——————————|——————–|————|———|——————-|
| SmolLM3-3B | 94.00 | 47.92 | ~2.0x | 0.99937 |
| Qwen2.5-3B-Instruct | 93.08 | 48.18 | ~1.93x | 0.99889 |
| Phi-4-mini-instruct | 53.80 | 39.09 | ~1.38x | 0.99990 |
| TinyLlama-1.1B-Chat | 180.28 | 132.36 | ~1.36x | 0.99778 |
| Qwen3.5-0.8B | 115.91 | 64.33 | ~1.80x | 0.99981 |
| Gemma-4-E2B | 43.60 | 1.62 (VRAM bottleneck) | ~27x (approximate) | 0.99967 |
| OLMoE-1B-7B-0924-Instruct | 33.70 | 9.22 | ~3.65x | 0.99921 |
Note: The cosine similarity metric indicates how closely the model’s output responses match the original safetensor or similar benchmarked models, serving as an important indicator of output fidelity.
Enhancing Inference Efficiency via Role-Aware Quantization
A significant innovation within ThinTensor is its role-aware quantization approach. Instead of a one-size-fits-all quantization (e.g., uniformly applying 8-bit or 4-bit quantization), the system dynamically assigns different quantization precisions to various tensor roles—such as attention layers, LM heads, or KV caches.
For example:
- Less sensitive layers (like feed-forward layers) might be selectively quantized to MXFP4 or FP8.
- Sensitive layers (like attention mechanisms) retain higher precision (BF16 or float).
This strategic quantization maintains output quality and cosine similarity, ensuring the model responses remain faithful while benefiting from increased inference speed.
Comparison and Advantages
Compared to optimized runtime solutions like TensorRT, ThinTensor offers notable portability and rapid deployment advantages:
- Portability: The .thin files can run across similar hardware setups without recompilation.
- Speed: Model loading and inference are significantly faster—initial benchmarks show over twice the throughput in some cases.
- Resource Efficiency: Compatible with limited VRAM environments due to layer-streaming options, enabling large models to run on modest hardware.
Limitations and Future Outlook
While promising, the project is still in early stages, with several caveats:
- Currently optimized primarily for NVIDIA CUDA architectures (especially Blackwell-Hopper GPUs).
- Not yet a comprehensive replacement for tools like llama.cpp.
- Further model support, larger prompt handling, and more extensive benchmarking remain in development.
- Some results are preliminary, potentially affected by hardware limitations (e.g., 8GB VRAM on a laptop GPU).
The underlying codebase is also a work in progress—developed with input from AI-assisted optimization techniques. As such, continuous refinement is expected.
Conclusion and Call for Collaboration
ThinTensor exemplifies how thoughtful data structuring and selective quantization can significantly enhance large model inference performance. By aligning storage formats closely with runtime execution, it is possible to achieve near-halving inference times without sacrificing output quality.
Researchers and developers working on inference engines, quantization techniques, or model serialization are encouraged to review the open-source repository and contribute feedback:
This initiative aims to foster a new paradigm where model deployment becomes faster, more portable, and resource-efficient—facilitating broader AI adoption across platforms and devices.
Disclaimer: The project employs AI assistance for optimization strategies; the core concepts and initial implementations are experimental. Further validation and development are ongoing to mature this approach for production use.
In summary, ThinTensor showcases a promising avenue for accelerating LLM inference by combining innovative storage formats with intelligent quantization—paving the way for more efficient and accessible AI deployments.
