The rising measurement of Massive Language Fashions (LLMs) makes environment friendly inference difficult, primarily as a result of reminiscence calls for of the autoregressive Key-Worth (KV) cache. Present eviction or compression strategies scale back price however depend on heuristics, akin to recency or previous consideration scores, which serve solely as oblique proxies for a token’s future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement studying (RL) drawback: studying to rank tokens by their predicted usefulness for future decoding. To this finish, we introduce KV Coverage (KVP), a framework of light-weight per-head RL brokers educated on pre-computed technology traces utilizing solely key and worth vectors. Every agent learns a specialised eviction coverage guided by future utility, which evaluates the standard of the rating throughout all cache budgets, requiring no modifications to the underlying LLM or extra inference. Evaluated throughout two totally different mannequin households on the long-context benchmark RULER and the multi-turn dialogue benchmark OASST2-4k, KVP considerably outperforms baselines. Moreover, zero-shot checks on normal downstream duties (e.g., LongBench, BOOLQ, ARC) point out that KVP generalizes effectively past its coaching distribution and to longer context lengths. These outcomes reveal that studying to foretell future token utility is a strong and scalable paradigm for adaptive KV cache administration.
