Compressing RAG Embeddings with TurboQuant

This post is a figure-driven reading of turboquant-embed, a local reference implementation of TurboQuant for embedding compression and retrieval. The question is not whether TurboQuant looks elegant on paper. The question is whether it buys anything tangible for RAG once we compare it against FP32, scalar quantization, product quantization, and hybrid retrieval pipelines. For the full mathematical theory (proofs, distortion bounds, and the QJL/PolarQuant lineage), see the companion article on data-oblivious vector quantization.

Why TurboQuant Exists

A corpus of $N$ vectors in dimension $d$, stored in float32, costs $4Nd$ bytes. For 100K documents embedded with a 1536-dimensional model, that is roughly 614 MB before any vector-store overhead, metadata, replication, or ANN index structures. This is manageable on a server. It is much less manageable on a laptop, across per-tenant indexes, or when the corpus changes faster than a codebook can be retrained.

Google Research introduced TurboQuant ¹ as part of a broader compression story covering both vector search and LLM KV-cache efficiency. The official blog post frames the ambition clearly: high-dimensional vectors are memory-hungry, traditional quantizers carry nontrivial codebook overhead, and TurboQuant is presented as a theoretically grounded, data-oblivious alternative for extreme compression.

The paper makes the claim precise. TurboQuant is an online, data-oblivious quantization scheme that aims at near-optimal distortion rates for both mean-squared error and inner-product distortion. The recipe has three steps:

1. Random rotation. Apply a random orthogonal matrix $\mathbf{\Pi}$ to the vector. This induces a Beta distribution on each coordinate, a known, universal shape that does not depend on the input data.
2. Lloyd-Max quantization. Quantize each rotated coordinate independently using scalar codebooks matched to the Beta density. Because the density is the same for every coordinate and every input, a single precomputed codebook suffices.
3. QJL residual correction. The MSE quantizer introduces inner-product bias. A 1-bit QJL sketch ² on the residual $\mathbf{r} = \mathbf{x} - \tilde{\mathbf{x}}_\text{mse}$ corrects this: the resulting estimator is provably unbiased with concentration guarantees.

The critical word is data-oblivious. There is no k-means fitting on the corpus. The rotation is deterministic once the seed is fixed, and the scalar codebooks are universal. A new corpus can be compressed immediately, without a training pass. The theoretical guarantee is that MSE distortion is at most $\frac{\sqrt{3}\pi}{2} \cdot 4^{-b}$, within a factor of 2.7 of the information-theoretic lower bound.

That is the context for turboquant-embed. The repository does not try to reproduce the full systems story of the Google blog. It focuses on the embedding and retrieval side of TurboQuant, with a local, CPU, NumPy-first implementation that is useful for RAG experiments, benchmark reproduction, and integration prototyping. The question it asks is not “Is TurboQuant theoretically interesting?”; the paper answers that. The question is:

For RAG, when does TurboQuant beat the usual alternatives, and what exactly do the current figures prove?

What turboquant-embed Implements

At the API level, the implementation is deliberately simple:

from turboquant_embed import TurboQuantEmbedCompressor, CompressedEmbeddings

compressor = TurboQuantEmbedCompressor(dim=1536, bits=4)
compressed = compressor.compress(embeddings)

scores, top_idx = compressor.topk_inner_product(compressed, query_vector, k=10)

compressed.save("index.npz")
loaded = CompressedEmbeddings.load("index.npz")

Under the hood, the repo tracks a few distinctions that matter when discussing RAG systems seriously. First, it separates serialized bytes (packed on-disk or transferable size) from resident bytes (actual in-memory NumPy buffers). The blog and paper claims should be tied to serialized storage when comparing compression. Second, it exposes three rotation backends: qr for the canonical dense Haar-style baseline, hadamard for an engineering-oriented $O(d \log d)$ approximation at larger dimensions, and auto to select between them. Third, it distinguishes MSE-only scoring from QJL-corrected scoring, because pure reconstruction quality and unbiased inner-product estimation are not the same problem.

The asymmetric inner-product estimator used by the product variant is:

where $r = x - \hat{x}_{\mathrm{MSE}}$ is the residual, $\gamma = \lVert r \rVert_{2}$, and $k$ is the sketch dimension. The first term comes from the MSE quantizer; the second is the QJL bias correction. The $\sqrt{\pi/2}$ factor compensates for the information lost in the sign quantization: it arises because $\mathbb{E}[|g|] = \sqrt{2/\pi}$ for $g \sim \mathcal{N}(0,1)$, and the estimator must cancel this scaling to remain unbiased. The derivation and complete proof are in the theoretical companion.

How to Read the Figure Slate

The benchmark directory mixes two kinds of figures. Some are primary comparative evidence for RAG: plots that actually answer whether TurboQuant is useful against meaningful baselines on labeled retrieval benchmarks. Others are internal or synthetic diagnostics: useful for building intuition about the quantizer’s behavior, but not the main evidence for deployment claims.

Storage and Operational Motivation

Before asking whether TurboQuant retrieves well, it is worth establishing that the compression itself is real, stable, and operationally interesting. That is what the first three figures do.

Packed Memory Scaling

The first figure plots serialized storage in megabytes for FP32, TQ-2bit, TQ-3bit, and TQ-4bit as the corpus grows from 10K to 100K vectors at $d = 1536$. The compression ratios are large and remarkably stable:

This is a storage figure, not a retrieval figure. It establishes that packed bytes scale linearly with corpus size (the ratio does not degrade) but says nothing about whether indexing overhead, latency, or retrieval quality follow the same pattern.

Equal-Memory Retrieval

The natural follow-up question is: if memory is the actual constraint, does compression buy retrieval quality by fitting more vectors into the same budget? On synthetic data, the answer is yes, dramatically so at small budgets:

At 10 MB, FP32 can barely fit any vectors and recall is nearly zero, while TQ-2bit already reaches 0.34. The effect saturates (TQ-2bit plateaus around 50% recall and TQ-4bit around 84%), so this should be read as a budget intuition plot rather than a public retrieval-effectiveness claim. But the intuition is sound: under tight memory constraints, compression is not just a storage optimization; it is a retrieval enabler.

Encoding Speed

The training-free nature of TurboQuant is not just a conceptual convenience. It translates into concrete encoding speed differences relative to PQ-style pipelines that need to fit codebooks on the corpus:

This is the weakest operational figure in the slate: the PQ timings are noisy and likely sensitive to hyperparameters and implementation choices. It supports the training-free story, but I would not use it as the lead public performance claim without a tighter experimental protocol.

What the Quantizer Itself Is Doing

With the storage motivation established, the next step is to understand the quantizer’s behavior before trusting it on real retrieval data. The figures in this section are all synthetic: controlled experiments designed to isolate specific properties of the method. They are not publication-grade retrieval results, but they are necessary to build confidence that the quantizer behaves predictably.

Fidelity vs Compression

The most basic diagnostic question is: how does retrieval fidelity degrade as the bitrate drops? The figure below traces Recall@10 on synthetic data as a function of compression ratio for the pure MSE variant and the QJL-corrected product variant.

The MSE variant degrades gracefully: 0.984 at 8-bit, 0.851 at 4-bit, 0.520 at 2-bit. The product variant buys more aggressive compression but pays for it steeply: at 2-bit, recall collapses to 0.046. That collapse is the most important negative result visible in the entire benchmark suite. The product/QJL path is not magic at very low bitrates.

Dimension Robustness

A practical concern for any quantizer intended for real embedding stacks is dimensional robustness. Models in production span from 128-dimensional MiniLM to 3072-dimensional BGE-M3 and beyond. After random rotation, TurboQuant stays remarkably stable:

The TQ column barely moves. SQ4 drifts slightly downward at higher dimensions. PQ is essentially flat but at a much lower level; this is synthetic data, and the PQ baseline is not especially strong in this setup. The figure supports the intuition that TurboQuant is dimension-stable, but it should not be oversold as a superiority proof by itself.

Top-1 Recovery

A different angle on quantization quality: how often is the true top-1 result recovered inside the approximate top- $k$? This matters for applications where the best answer must not be lost entirely, even if the approximate ranking is imperfect.

For $d = 1536$ at 2-bit:

Both methods recover the true top-1 with high probability once the reranking depth reaches 16 or so. The metric is relatively forgiving (it only asks whether the best result appears somewhere in the top- $k$, not whether the full ranking is preserved), but it confirms that TurboQuant does not catastrophically lose the best answer.

Scaling with Corpus Size

The last diagnostic question before turning to real retrieval benchmarks: does the compression ratio hold as the corpus grows? The answer is yes: the ratio stays fixed at 7.8x from 1K to 100K vectors:

The recall values here are pessimistic synthetic diagnostics; the absolute numbers are not meaningful for real retrieval claims. The reliable takeaway is the linear packed-storage scaling: the compression ratio does not erode with corpus size.

Comparative Evidence on BeIR

Everything so far has been either operational motivation or synthetic diagnostics. The figures in this section are different: they measure TurboQuant against meaningful baselines on labeled retrieval benchmarks (BeIR), with real embedding models, and in pipeline configurations that resemble actual RAG systems. This is where the claims must be the most careful, and where the results are the most interesting.

Hybrid Retrieval Under Compression

This is the most directly RAG-relevant figure in the entire repository. In practice, most RAG systems do not rely on dense retrieval alone; they combine a sparse leg (BM25 or similar) with a dense leg, fusing the results with reciprocal rank fusion or a learned combiner. The deployment question is concrete: if we compress the dense leg, do we lose the hybrid lift?

The figure shows paired query-wise deltas for dense-only and hybrid (BM25 + dense) retrieval, comparing TQ-4bit against FP32 on BeIR scifact and nfcorpus:

The hybrid deltas are negligible; confidence intervals comfortably include zero on both datasets. The sparse leg absorbs whatever small perturbations the quantizer introduces in the dense ranking. It is worth noting that the sparse leg here is a local bm25s reference with Snowball stemming, not a production search server. The right claim is therefore not that TurboQuant improves hybrid retrieval, but that it does not materially damage it in this setup.

Status Quo Against Standard Quantizers

This is the central comparative figure, the one that answers the question a RAG engineer would actually ask: why should I care about TurboQuant instead of just using the standard FAISS quantizers?

The figure plots NDCG@10 against actual bytes per vector for FP32, TurboQuant, SQ, PQ16, and OPQ16, on two BeIR datasets (SciFact and NFCorpus) with two embedding models (MiniLM and mpnet). For SciFact with MiniLM:

Two results stand out. First, in the ultra-compact regime, TQ-2b is materially stronger than PQ16; the gap is not small. Second, around the ~8x compression point, TQ-4b is essentially tied with SQ4. The advantage over scalar quantization is not dramatic at 4-bit; the honest result is narrower and still useful: TurboQuant clearly beats PQ/OPQ at low byte counts, and remains competitive with SQ at moderate compression.

Retention Relative to FP32

The same story told differently: NDCG@10 normalized as retention relative to FP32, which makes the practical message easier to read at a glance.

TurboQuant stays in the near-lossless band across both bitrates, while PQ sits visibly below it. Retention above 100% is not a real quality gain; it is evaluation noise and finite-sample variance. The right reading is “essentially equal to FP32,” not “better than FP32.”

Synthesis

Taken together, the figures tell a coherent story with clear boundaries.

The storage and scaling figures establish that TurboQuant delivers 8x–16x packed compression with strictly linear scaling; the ratio does not erode with corpus size. The encoding speed figure supports the claim that the compression path is materially lighter than PQ-like fitting, even if the exact timings need a tighter protocol. The synthetic diagnostic figures confirm that the quantizer degrades gracefully with bitrate, remains stable across dimensions, and does not catastrophically lose the best results, with the important exception that the product/QJL path collapses at 2-bit.

The RAG-facing figures are the real evidence. Hybrid retrieval is preserved: compressing the dense leg with TQ-4bit does not damage the BM25+dense lift on the BeIR datasets tested. Against standard FAISS quantizers, TurboQuant materially outperforms PQ and OPQ in the ultra-compact regime and stays competitive with scalar quantization at 4-bit. The advantage over SQ is not huge, but TurboQuant achieves it without any corpus-specific training.

The honest limits are equally clear. Several figures are synthetic diagnostics, not publication-grade retrieval benchmarks. The product/QJL variant is not magic at very low bitrates. The advantage over scalar quantization is narrow at 4-bit. And the repository measures a reference CPU/NumPy implementation, not a production ANN system. These figures say something real about compression quality and retrieval preservation, but they do not prove end-to-end serving superiority.

What This Means for RAG

For RAG, the strongest argument for TurboQuant is not that it is mathematically elegant. The strongest argument is this: it reduces embedding storage by roughly one order of magnitude, stays competitive with the quantization baselines people actually use, preserves hybrid retrieval quality, and does all of this without fitting a corpus-specific quantizer.

That is a meaningful niche. It is especially relevant for local RAG, per-tenant indexes, fast-moving corpora, and experiments where you want a compression layer without turning the pipeline into a codebook-training project.

For a deeper understanding of why these compression-versus-quality tradeoffs look the way they do (why the product variant collapses at 2-bit, why scalar quantization is competitive at 4-bit, and what the information-theoretic limits actually are), see the theoretical article on data-oblivious vector quantization.

The thing it does not justify is a broad systems claim:

TurboQuant is already a better production vector store than the usual ANN stack.

That is a different claim, and this repository is not trying to make it.

Reproducibility

Everything discussed here is tied to local artifacts and scripts:

1. implementation: turboquant-embed
2. figure analysis guide: benchmarks/FIGURES_ANALYSIS.md
3. status-quo benchmark: benchmarks/status_quo_quantization_bench.py
4. retrieval benchmark slate: benchmarks/rag_benchmarks.py
5. official paper: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
6. official Google Research post: TurboQuant: Redefining AI efficiency with extreme compression

References

@misc{laabsi2026turboquantrag,
  title        = {Compressing RAG Embeddings with TurboQuant},
  author       = {Laabsi, Zakaria},
  year         = {2026},
  month        = {Apr},
  howpublished = {\url{https://zlaabsi.github.io/posts/turboquant-embed/}},
  note         = {Blog post}
}