VarRate Boosts Long-Context LLM Performance Without Training

Researchers developed VarRate, a new method to compress key-value caches in large language models (LLMs) without requiring training. The key-value cache presents a major memory bottleneck during LLM inference, especially with long contexts. Existing methods either irreversibly discard tokens or distribute resources equally, both of which limit performance.

VarRate assigns each token a variable amount of low-rank budget based on how relevant it is to the query. This approach avoids the accuracy collapse seen in token-selection methods when importance signals weaken. It also improves upon uniform low-rank coding by allocating resources more efficiently.

Testing on Llama-3.1-8B and Qwen2.5-7B models, VarRate maintained performance within 0.8 points of uncompressed models while using 20% of the memory. The team reports VarRate outperforms existing compression techniques like KVzip in several scenarios and reduces prefill overhead.