Part 6. TANREN — On Atari Skiing, Deep RL's Hardest Slope, Readable Python Beats Every Published RL Agent

Left: the same code with dulled reactions, calibrated to the published human benchmark (measured −4266 ≈ human −4337). Right: TANREN, threading 20/20 slalom gates while leaving the human pace behind. What runs on the right is not a neural network but about 130 lines of readable Python (the core is a few dozen lines of gate tracking).

This is Part 6 of the TANREN series. The project overview is here, and the previous part (Breakout — where the method’s boundary was drawn) is here. This is the centerpiece of the Atari episodes: a clean sweep on the game that deep RL struggled with for twenty years.

TANREN in 30 Seconds

TANREN does not train LLM weights at enormous cost. Instead, it asks a cheap, frozen LLM to write candidate code, then evolves those programs under an automated scorer. The goal is to approach results that large-scale compute reached through hundreds of millions of frames of training — using one desktop PC, a few days of search, and a few dollars of API cost.

Nowhere is that contrast sharper than here. On Skiing — where DeepMind’s Agent57 needed roughly 78 billion training frames to surpass the human benchmark — readable Python got past every published RL agent and the human benchmark with a 2-hour gate measurement, 2.5 hours of evolution, and a few dollars of API cost.

Why Skiing — the Game That Frustrated RL for Twenty Years

Skiing (slalom) has a difficult score structure: score = −(elapsed time + missed gates × a 5-second penalty), and the penalty is settled in one lump at the finish line. That makes it nearly impossible to work backwards from the reward to “which gate did I get wrong” — a textbook case of delayed credit assignment, deep RL’s weakest spot.

The published numbers split dramatically. MuZero, which plays Breakout to its 864-point ceiling, collapses to −29,968 on Skiing. Agent57’s paper is built around surpassing humans on all 57 games — and Skiing was one of its last walls. In other words, this is an unusual game: easy for humans, hardest for deep RL. I will say openly that the game was chosen for exactly that reason — it is a place where readable rules of judgment (see the gate, look ahead, steer) can beat neural-network scale.

The Result — a Clean Sweep of Published RL and the Human Benchmark

Let me state the contrast plainly. Agent57 spent roughly 78 billion training frames on a large-scale compute platform to pass the human benchmark. TANREN sits above it — and above every published RL agent — after less than a day on one desktop and a few dollars of API cost. This does not mean TANREN is strong at everything. It means this game, where the value condenses into a few rules of judgment, was deliberately chosen — using the pre-measurement described next.

Estimating the Odds Before Running — the Entry Gate

Starting with this experiment, TANREN requires an entry gate: before any evolution run, a numeric search with no LLM involved (skeleton code plus parameter search, 2 hours, nearly free) measures the ceiling of what this class of code can reach, and evolution is launched only if that ceiling can plausibly reach the published best.

• Skiing’s gate measurement: proportional steering with just 5 parameters already scored −3619 — past Agent57 and the human benchmark before evolution even started. GO.
• The same gate concluded for Boxing and Tennis that the ceiling falls far short of the published values, and both were abandoned without a single evolution run (how this gate came to be is the subject of the next article, on Bowling).

What Evolution Added

Starting from the gate’s seed (proportional steering, −3619), 60 generations over 2.5 hours improved the score to −3258 (confirmation run). Reading the final code, you find techniques nobody taught it:

• Multi-gate lookahead: not just the next gate, but a weighted interpolation through gates 2-3 ahead, forming a racing line.
• Velocity smoothing and arrival prediction: estimating its own lateral velocity with an EMA and steering toward the predicted position at gate arrival.
• Bang-bang control over a 16-direction heading: left/right taps driven purely by the difference to the target heading — a readable steering law.

Which RAM byte holds the gates and which holds the heading — the system identified that too, by experiment (including the reverse-engineering discovery that the remaining-gate counter is stored in BCD).

Honest Notes

• Input is RAM (128 bytes of internal state), not pixels. One dedicated program per game.
• The comparison protocol matches the published side (no-op start variation, deterministic, 27,000-step cap). Scores were verified against the primary source, arXiv:2003.13350 Table H.4 (secondary sources often call Agent57 the strongest; the table’s strongest is R2D2(Bandit) at −3851.44).
• “~78 billion frames” is Agent57’s published training volume. TANREN trains no weights (evolution happens in code), so the numbers are not on the same axis — the contrast is offered as a sense of cost scale.
• This is not a general-purpose agent. TANREN produces a readable algorithm dedicated to this one task.

Next in the series: the lost campaign that built the entry gate behind this sweep — Atari Bowling. Back to the project overview.