Part 5. TANREN — Discovering Breakout's Tunnel Strategy on Its Own, and Measuring Exactly Why the Rest Is Out of Reach

The AI on the right side of the screen is a few dozen lines of readable Python. Nobody told it to dig a tunnel. It opens a hole at the edge of the bricks, sends the ball behind the ceiling, and destroys dozens of bricks in one rally — rediscovering Breakout’s classic tunnel strategy mid-evolution, on its own.

This is Part 5 of the TANREN series. The project overview is here, and Part 1 (Atari Pong 21–0) is here. After the live-vLLM episodes (Part 3, Part 4), the series returns to the games.

TANREN in 30 Seconds

TANREN does not train LLM weights at enormous cost. Instead, it asks a cheap, frozen LLM to write candidate code, then evolves those programs under an automated scorer. The goal is to approach results that large-scale compute reached through hundreds of millions of frames of training — using one desktop PC, a few days of search, and a few dollars of API cost.

In Part 1, this system reached Pong’s theoretical ceiling (21–0 in all 40 games). This time, the same system takes on Breakout — two days, about $13. But the substance of this article is not what it achieved. It is that the reason for what it could not achieve was pinned down by measurement.

The Strategies Evolution Acquired on Its Own

Tracing the lineage of the evolution, you can watch it acquire strategy in stages, taught by no one:

flowchart LR
    A["1. Tracking<br/>follow the ball, survive"] --> B["2. Serve management<br/>the ball speeds up around the 4th hit<br/>of each life — adapting to the environment's rule"]
    B --> C["3. Aiming at the edges<br/>the beginnings of burst play"]
    C --> D["4. The tunnel strategy<br/>'realizing' that behind-the-ceiling rallies<br/>are where the points live (36-hit bursts = 300+)"]
    style D fill:#d1fae5,stroke:#059669

Even more interesting: along the way, the system reverse-engineered the Breakout environment itself. Which RAM byte holds the ball’s position, when the speed-up happens, and the fact that every tunnel “ignition” (the start of a behind-the-ceiling rally) falls inside a narrow critical window of 35-46 cumulative hits — the physics of the environment was extracted in a form no neural network can ever show you: numbers and rules. One finding of this experiment is that LLM-driven evolution also works as a debugger for the environment.

The Result — What Worked and What Did Not

It reached about 6.1× the human benchmark. It did not reach DQN, which was trained on 200 million frames. Rather than stopping there, the real work of this experiment was establishing by measurement why it could not.

Why It Fell Short — Measured, Not Guessed

The diagnosis concluded: the remaining points live in millions of micro-adjustments — final-2-pixel corrections against the fast ball (vy=13), across every situation, under 5% noise. That mass does not physically fit into a few dozen readable lines of if-else.

The decisive evidence came from a clone-state experiment. I took 30 identical states just before tunnel ignition and branched the replay with different actions — and the ignition rate did not move no matter which action was chosen (median spread 0.00). In other words, there is no room for a smarter decision at that point. Beyond here it is not a judgment problem but a reflex problem, and that is neural network territory.

The Takeaway — Where This Method Applies

TANREN builds rules of judgment; it does not replace perception and reflexes. Where the value concentrates in a few structural insights (Pong’s kill-shot, caching, scheduling), a few dozen lines of code can reach the top. Where the value spreads across countless micro-adjustments (fast-ball reflexes, joint control), neural networks have the advantage. In real systems the two can coexist in layers, and TANREN’s role is building the readable judgment layer that sits on top.

Reporting “reached 6.1×” and “stopped at 47%” with the same precision, and making the boundary itself the deliverable, is the long way around as far as promotion goes. But it is exactly this boundary that made Parts 2-4 (cache and scheduling — problems governed by rules of judgment, not reflexes) safe choices for production work.

Honest Notes

• Input is RAM (128 bytes of internal state), not pixels. One dedicated program per game. The opponent is the standard built-in AI.
• The DQN 401.2 comparison carries protocol differences (theirs: 30 episodes, no-op starts, ε=0.05, no sticky actions; ours: sticky p=0.05, 500 held-out episodes). This note accompanies the numbers wherever they appear.
• “Max 430” is the maximum observed across all evaluation episodes.
• Sticky actions at p=0.05 (each action has a 5% chance of repeating the previous one) is a custom anti-memorization protocol, distinct from ALE’s standard 0.25.

Next in the series: Atari Skiing — the game deep RL struggled with for twenty years, where readable Python beat every published RL agent. Back to the project overview.