Product Quantization Is a Recall–Memory Contract

At large vector counts, storing every coordinate as a 32-bit float becomes an architecture decision. One billion 768-dimensional vectors require about 3.1 TB for the raw coordinates alone, before IDs, indexes, replicas, and allocator overhead.

Product quantization (PQ) changes that equation by replacing each full vector with a compact learned code. The price is approximation error. The system is no longer buying “a smaller index”; it is choosing a recall–memory contract.

The mechanism #

Split a d-dimensional vector into M sub-vectors. Train a small codebook for each subspace. Store the nearest codeword identifier rather than every floating-point component.

If every sub-quantizer uses 256 centroids, each identifier fits in one byte. With M = 96, a vector can be represented by a 96-byte PQ code rather than 3,072 bytes of float32 data. This comparison excludes surrounding index state, but shows why the technique matters.

At query time, the engine builds distance lookup tables between query sub-vectors and codewords, then sums approximate distances for candidate codes.

query
  -> coarse centroid search
  -> probe selected inverted lists
  -> approximate distance over PQ codes
  -> optional exact rerank
  -> top-k

Faiss IndexIVFPQ combines an inverted file (IVF) with PQ encoding of residual vectors. That introduces two distinct ways to miss a relevant result:

1. The correct vector is in a list that was not probed.
2. Quantization changes the relative order of candidates.

Do not compress those into a single tuning knob.

Measure against an exact baseline #

Create a representative, versioned evaluation set of queries. Compute exact top-k neighbors on a manageable corpus, then compare approximate results.

def recall_at_k(exact_ids, approx_ids, k):
    exact = set(exact_ids[:k])
    approx = set(approx_ids[:k])
    return len(exact & approx) / k

Measure at least:

• recall@k by query cohort;
• p50, p95, and p99 search latency;
• resident memory and index build time;
• candidates scanned and lists probed;
• quality after filters are applied;
• degradation after embedding-distribution changes.

Average recall can conceal failure for short queries, rare languages, or a high-value tenant. Product metrics should identify which misses actually change user outcomes.

Tune the whole retrieval path #

Increasing nprobe explores more coarse lists and usually improves candidate coverage at a latency cost. Increasing PQ code size reduces compression and can reduce distance error. Reranking a larger shortlist with original vectors improves final ordering but requires access to those vectors and more bandwidth.

Common mistakes #

• Training codebooks on an unrepresentative sample.
• Reporting only index memory while keeping a full-vector copy on every replica.
• Tuning on random vectors instead of production query–document pairs.
• Changing the embedding model without retraining and re-evaluating the index.
• Treating top-k recall as identical to answer quality in a RAG system.

Decision #

Use PQ when exact-vector memory or bandwidth is the binding constraint and the product can define an acceptable measured loss. Keep exact search when the corpus is small enough, errors are extremely expensive, or operational simplicity is worth the RAM.

Compression is not free capacity. It is an explicit exchange of representation fidelity for memory, evaluated at the product boundary.

Further reading #

• Faiss IndexIVFPQ reference
• Billion-scale similarity search with GPUs
• Filtered vector search is a recall budget