Retrieval-augmented generation (RAG) systems face a persistent tension: every query stuffs thousands of tokens into the context window, while the retriever and the generator are still optimized as two separate, disconnected systems. CLaRa (Continuous Latent Reasoning), a framework from researchers at Apple and the University of Edinburgh, tackles both problems at once. Described in a paper on arXiv, CLaRa compresses documents into a small set of continuous “memory tokens” and then performs both retrieval and generation in that shared latent space. The goal: minimize context, avoid double encoding, and let the generator teach the retriever what actually matters for downstream answers.
From raw documents to persistent memory tokens
CLaRa starts with a semantic compressor (SCP) that appends a small number of learned memory tokens to each document. The base model is a Mistral-7B-style transformer with a LoRA adapter that switches between compression and generation roles. During compressor pretraining, the model learns to reconstruct what matters in a document from its memory tokens alone, using question answering and paraphrase supervision to keep the compressed vectors semantically rich and retrievable.
An auxiliary mean-squared-error objective aligns the average latent state of document tokens with the average latent state of memory tokens. The paper reports that this alignment adds a small but consistent gain of roughly 0.3 to 0.6 F1 points at compression ratios of 32 and 128, and keeps compressed and original representations in the same semantic region.
Joint retrieval and generation in a shared space
After offline compression, each document is represented only by its memory tokens. CLaRa then trains a query reasoner and an answer generator on the same basis. The query reasoner is another LoRA adapter that maps an input query into the same number of memory tokens used for documents, so retrieval becomes pure embedding search: cosine similarity between the query embedding and each candidate document embedding.
The best-scoring compressed documents are combined with the query and fed to the generator adapter. Training uses only a standard next-token prediction loss on the final answer — there are no explicit relevance labels. Gradients flow through both retrieval and generation via a differentiable top-k estimator, which is what lets answer quality supervise the retriever directly.
Compression quality and QA accuracy
The compressor is evaluated on four question-answering datasets: Natural Questions, HotpotQA, MuSiQue and 2WikiMultihopQA. In the standard setting, retrieving the top five Wikipedia documents per query, SCP-Mistral-7B reaches an average F1 of 39.86 at 4x compression — 5.37 points above the hard-compression baseline LLMLingua-2 and 1.13 points above the strongest soft-compression baseline, PISCO. In the oracle setting, where the gold document is guaranteed to be among the candidates, it reaches an average F1 of 66.76 at 4x compression, 17.31 points above LLMLingua-2 and 5.35 above PISCO.
Notably, the compressed representation can even outperform a BGE-based text retriever paired with a full-document generator on average F1. In other words, well-trained soft compression can exceed full-text RAG while shrinking the reference text by factors of 4 to 128.
Performance does degrade at very high compression ratios (above 32) in the oracle setting, though the drop stays moderate under realistic retrieval conditions. The research team’s explanation: weak document relevance limits the system before compression quality does.
End-to-end results and reranking
For end-to-end QA, CLaRa uses 20 candidate documents per query at compression ratios of 4, 16 and 32. In the standard setting, CLaRa-Mistral-7B reaches an F1 of 50.89 on Natural Questions and 44.66 on 2WikiMultihopQA at 16x compression — comparable to DRO-Mistral-7B, which reads full uncompressed text, while using a document representation sixteen times smaller. On some datasets CLaRa at 16x compression actually improves on DRO, for example moving from 43.65 to 47.18 F1 on 2WikiMultihopQA.
On the retrieval side, CLaRa used as a reranker posts strong recall figures. With pretraining initialization at 4x compression on HotpotQA, the paper reports recall@5 of 96.21, beating a supervised BGE reranker baseline at 85.93 by more than ten points.
What Apple released
Apple’s research team released three models on Hugging Face — CLaRa-7B-Base, CLaRa-7B-Instruct and CLaRa-7B-E2E — along with full training pipelines in the apple/ml-clara repository on GitHub.
Why this matters in plain language
Most RAG systems today work like a librarian who photocopies entire chapters and hands them to a reader with a stopwatch running. CLaRa instead teaches the model to keep a dense, meaning-preserving summary of every document in its own internal “shorthand,” and to search and answer directly in that shorthand. The practical promise is lower cost and latency — contexts shrink by 16x or more — plus a simpler stack, since retrieval and generation stop being separately tuned systems. For teams running RAG at scale, compression-native retrieval is a direction worth tracking, much as running small models locally is explored in the guide to fully local multi-agent orchestration on this site.
Limitations and what to watch
CLaRa is a research release, not a production system. Results are reported on 7B-parameter models and Wikipedia-based QA benchmarks; behaviour on enterprise documents, other languages and larger models is untested in the paper. The oracle-setting numbers assume the right document is already in the candidate pool, which real deployments cannot guarantee, and quality visibly degrades at the most aggressive compression ratios. Worth watching: whether the approach scales past 7B, whether independent replications confirm the reported gains, and whether compression-native retrieval appears in commercial RAG stacks.
