Part 3. TANREN — Patching the Evolved Cache Policy into a Real vLLM and Measuring It

Left: vLLM’s default eviction. Right: the TANREN evolved policy. A replay of measured TTFT from real conversations (ShareGPT) on a real vLLM. Watch the tail — the slow side of the response times.

This is Part 3 of the TANREN series. The project overview is here, Part 1 (Atari Pong) is here, and Part 2 (cache eviction) is here.

TANREN in 30 Seconds

TANREN does not train LLM weights at enormous cost. Instead, it asks a cheap, frozen LLM to write candidate code, then evolves those programs under an automated scorer. The goal is to approach results that large-scale compute reached through hundreds of millions of frames of training — using one desktop PC, a few days of search, and a few dollars of API cost.

In Part 2, an evolved cache policy went 11-1 against twenty years of classics on replayed Twitter traces — that is, in simulation. The obvious next question is: does a win in the simulator survive on a real server? This time I left the simulator, patched the code into a live vLLM, and measured on hardware. To say it up front: I lost first. The diagnosis is part of the story, so I write it as it happened.

What I Did

An LLM server saves computation by caching the results (the KV cache) for the shared beginnings of conversations — a mechanism called prefix caching, built into vLLM. A cache hit makes a large difference: in a separate reference measurement, a 2,048-word prompt went from 1,500 ms TTFT to 177 ms on a hit (−88%). But memory is finite, so when it runs out, something must decide which blocks to evict. It is the same “what to throw away” problem as in Part 2 — this time inside a real engine.

I did three things:

1. Trained no model weights at all; instead, evolved a readable eviction policy of a few dozen lines.
2. Patched it into a single file of vLLM (a one-file patch; one line of change rolls it back).
3. Replayed exactly the same sequence of real conversations (ShareGPT, from a segment never used during evolution) against both the default eviction and the evolved policy, A/B style.

Hardware: NVIDIA GB10, vLLM 0.19.1, Llama-3.2-1B. Cache pressure is reproduced deterministically with --num-gpu-blocks-override.

The Results — As Measured

At the load level where memory just runs short (where cache contention actually happens), the evolved policy performed as follows against the default:

Metric	Effect
Tail latency (p99 TTFT)	13-16% faster
Cache hit rate	+about 1 point

TTFT is the time until the first character comes back; p99 is the value for the slowest 1% of responses — the number SLOs are written against. In plain terms: under load, the responses that wait longest start coming back 13-16% sooner. Here are the measurements per block count and seed:

Cache size (blocks) / seed	p99 TTFT	Hit rate
2000 / s1	74ms → 69ms (−7.9%)	42.6% → 44.4% (+1.8pt)
2000 / s2	71ms → 68ms (−3.9%)	44.4% → 44.7% (+0.4pt)
2500 / s1	72ms → 62ms (−13.4%)	48.4% → 50.4% (+2.0pt)
2500 / s2	72ms → 61ms (−14.2%)	50.3% → 50.7% (+0.4pt)
3000 / s1	72ms → 61ms (−15.4%)	53.1% → 54.4% (+1.3pt)
3000 / s2	71ms → 60ms (−15.4%)	54.0% → 55.1% (+1.1pt)

The effect peaks right where memory starts to run out (blocks 2500-3000 — the “knee” of the load curve) and shrinks below that point (2000). The data itself shows the structure: the benefit depends on the load.

Some honest costs, stated up front:

• It only helps under memory pressure. On an idle server there is no difference.
• The prototype is plain Python, so the median (ordinary) response gets 1-2 ms slower — a small compute cost paid on every request in exchange for removing the worst-case spikes. A production C++ implementation should eliminate this, but I have not measured that, so I will not claim it.
• The baseline is vLLM’s default eviction, not the strongest cache algorithm in the world.

What I Most Want to Write About: Losing First

At first, this experiment looked like a complete failure. In the simulator, the policy beat the default by +6-8% hit rate. Patched into the real machine: almost zero difference, at every pressure level I tried.

I nearly concluded the simulated win had been an illusion. But the patch was demonstrably working. So I turned TANREN’s core habit — measure the moment of failure — on my own experiment.

flowchart TD
    A["Simulator: +6-8%"] --> B["Real machine: no difference"]
    B --> C["Diagnose: measure the failure itself"]
    C --> D["Mismatch 1: where I measured<br/>pressure steps were coarse and<br/>stepped right over the 'knee'"]
    C --> E["Mismatch 2: the verifier's physics<br/>vLLM's default is smarter than plain LRU,<br/>so my simulated 'default' wasn't the real default"]
    D --> F["Re-find the knee on hardware;<br/>switch the metric from mean to tail (p99)"]
    E --> F
    F --> G["The win appears: p99 −13-16%"]
    style B fill:#fee2e2,stroke:#b91c1c
    style G fill:#d1fae5,stroke:#059669

The diagnosis found two quiet but decisive mismatches:

1. I was measuring in the wrong place. This policy’s advantage concentrates sharply at the boundary where the cache just runs short — the knee. My load steps were coarse enough to step right over it.
2. The simulator’s behavior differed subtly from the real thing. My simulator assumed simple eviction, but vLLM’s actual default is already smarter than plain LRU — it is deliberate about the order in which old blocks are dropped. The baseline I had been calling “default-equivalent” was not the real default.

Once I located the knee on real hardware and switched the metric from the mean to the tail, the win appeared. The policy had been working all along; I had simply been wrong about where and what to measure.

The lesson: when moving a policy to a new environment, copying the formula is not enough. Mirror the target engine’s actual eviction mechanics in your verifier, align your measurements to the operating point where the benefit lives, and look at the tail of the distribution, not just the mean. This lesson pays off directly in Part 4.

What It Is Good For

Conversational LLM services are the most common form of LLM in production today. On a busy server, cutting tail latency by more than a tenth and reducing GPU recomputation translates directly into machine count and cost. And the artifact is a few dozen readable lines of code, with no model changes and a one-line rollback — an improvement that can be added safely.

The distribution path is also realistic: not a from-scratch engine, but a plugin that inserts only the eviction logic into existing OSS (vLLM, LMCache, and the like) — building on the ecosystem rather than competing with it.

Limitations

• This is not a universal speedup. It touches one lever among many in LLM efficiency — prefix-cache eviction — and a secondary one at that. This is not “your LLM gets several times faster.”
• The effect is conditional: it helps the tail latency of conversational servers under memory pressure, and does nothing on idle servers.
• It does not help RAG. I confirmed this on real data (RAGPulse): RAG combines different documents on every request, so the requests do not land on the prefix cache. That said, I also measured that a different scheme — caching each document independently — has roughly twice the theoretical ceiling. That is a topic for later.
• The baseline is the default eviction. Hardware comparisons against the ARC/LFU family and recent document-cache methods are still to come.

Win where you can, and where you cannot, say so with measurements — that is the policy of this series.

Protocol (for Fact-Checking)

• Hardware: NVIDIA GB10 / vLLM 0.19.1 / Llama-3.2-1B. Pressure reproduced deterministically via --num-gpu-blocks-override.
• Data: real ShareGPT conversations from a held-out segment unused during evolution, replayed as an identical sequence to both configurations.
• Knee (blocks 2500-3000) × seeds: hit rate +0.4-2.0pt, p99 TTFT −13-16%. Cost: median TTFT worsens by 1-2 ms due to the prototype’s Python overhead.
• The champions and verifiers from Parts 1-2 are public on GitHub.

---

Next in the series: the same system is applied to the heart of LLM serving — the scheduler that decides the order requests are processed in — halving tail latency under heavy load. Back to the project overview.