Part 8. TANREN — The Number I Almost Claimed and Then Withdrew: Atari Freeway and Measurement Honesty

Freeway is the game where a chicken crosses a highway — how many crossings can you complete in 2 minutes 16 seconds. Left: human-pro level (the same code calibrated with a post-crossing hesitation; measured 29.2 ≈ the published human benchmark 29.6). Right: TANREN, tracking the speeds of all ten cars and predicting arrival times two lanes ahead, crossing without hesitation (measured 31-32). Inside: about 60 lines of readable Python.

This is Part 8 of the TANREN series. The project overview is here, and the previous part (Bowling — the lost fight that built the entry gate) is here. The subject this time is not the score. It is measurement honesty.

TANREN in 30 Seconds

TANREN does not train LLM weights at enormous cost. Instead, it asks a cheap, frozen LLM to write candidate code, then evolves those programs under an automated scorer. The goal is to approach results that large-scale compute reached through hundreds of millions of frames of training — using one desktop PC, a few days of search, and a few dollars of API cost.

This series has repeatedly published numbers of the form “rivaled large-scale RL in days, for dollars.” Which is exactly why the evaluation conditions behind those numbers are the lifeline of the whole system. This is the story of getting those conditions wrong once, catching it myself, and withdrawing the number.

What Happened: 31.9 Came from a Mismatched Measurement

Evolution produced 31.9 without much drama. Past DQN’s 30.3, past the human benchmark’s 29.6. A number worth publishing — or so it looked.

The pre-publication cross-check found a protocol mismatch. The verifier’s default start condition had been carried over from the preceding Breakout experiment (sticky actions). The published values — DQN, MuZero, Agent57 — were measured under “no-op start variation.” Re-measured under the same conditions, 31.9 dropped to 28.6 — short of DQN and short of the human benchmark. Digging further, I also found a duplicated episode-execution path inside the verifier, with one side missing the start variation entirely — meaning selection had effectively run without variation.

Three actions followed:

1. Withdraw the number. No figure from the 31.9 family is used for publication (this article mentions it only as a mis-protocol value).
2. Repair the verifier. Protocol elements (start variation, sticky actions) were consolidated into one function, the protocol name is stamped into every judgment output, and a check was added that flags identical scores across seeds as suspicious.
3. Re-run the evolution under the correct protocol. Runs were rebuilt with no-op variation included, reaching 31.2 in the final held-out judgment.

The word “surpassed” can only be used after matching the comparison paper’s evaluation conditions against your own settings, line by line — this lesson became a permanent rule of the system.

The Result — with Every Bar Shown

The human benchmark and DQN — trained on 200 million frames — were passed with about 60 readable lines of code and a few dollars. R2D2 at 34.0, MuZero at 33.03, and Agent57 at 32.59 remain above, and that fact is published alongside the result. Freeway’s score effectively saturates around 34 (R2D2’s variance is ±0.00); the remaining ~3 crossings live in the territory of a perfect rhythm that never waits for a single car. I believe this number sits close to the edge of what readable reactive code can do.

Inside the Code

What the final code (~60 effective lines) does:

• Speed tracking for all ten cars: per-step differences of the car coordinates in RAM (screen wraparound handled with circular differences).
• Two-lane lookahead: before advancing, predict the time-to-arrival of cars one and two lanes ahead and pick the crossing moment.
• Stall escape: if no crossing has happened for a while, step back once and rebuild the rhythm.

Which RAM bytes hold the cars, and which lanes run fast — the system identified that too, by measurement.

Honest Notes

• Input is RAM (128 bytes of internal state), not pixels. One dedicated program per game.
• The comparison protocol matches the published side (no-op start variation, deterministic, 27,000-step cap). Bars verified against the primary source, arXiv:2003.13350 Table H.4.
• Past figures like 31.9/32.0 are mis-protocol values (a sticky-actions setting carried over) and are not used for publication.
• This is not a general-purpose agent — it is a readable algorithm dedicated to this one game.

Closing the Atari Episodes — the Contrast in Full

With this article, the Atari side of the series stands at five games. Against RL built on large-scale compute (DQN: 200 million training frames; Agent57: roughly 78 billion), TANREN entered every one of them with one desktop, days at most, and a few dollars of API cost.

It is not a story of total victory. But every game — won, lost, or never entered — stands on the same verifier discipline and the same primary-source cross-checking. How a number was measured matters more than the number itself. That is one of this series’ conclusions.

This closes the Atari episodes for now. Back to the project overview. The Pong and cache champions and verifiers are public on GitHub.