Part 1. VOI and KOI: Metrics for the Token Economy — Maximizing AI's Result per Dollar

As companies put large language models (LLMs) into internal work and products, many CIOs and CFOs are hitting the same problem: infrastructure cost that is hard to keep under control.

Sending simple email summaries or document searches that no one ever re-reads to the top-tier, most expensive frontier model has become common enough to earn a name: Tokenmaxxing. The habit of treating AI usage itself as a productivity metric looks a lot like an earlier chapter in IT, when engineers were judged by lines of code (LOC) and the result was more verbose code, not better software.

In this piece I use recent news and real cases to look at the management risk Tokenmaxxing creates, the rise of “model routers” as a first line of defense, and the outcome-driven optimization that should come after them.

1. What Tokenmaxxing actually looks like

Fewer than 10% of companies are thought to have an accurate picture of their AI cost and usage today. On the ground, budget overruns and gaming of internal rules are already happening.

Case	What happened	In numbers
Meta’s “Claudeonomics” shutdown	An employee-built leaderboard ranked “who used external models the most.” To score points, people had AI agents generate large numbers of tiny edits. After it was reported in the press, the dashboard was shut down.	About 60–73.7 trillion tokens used company-wide in 30 days. The top individual used about $20 million worth in 30 days.
Uber’s budget overrun	Use of autonomous coding tools spiked and burned through the full-year AI budget in the first few months. Left-running agents kept re-running queries and quietly consumed tokens.	AI agents grew to autonomously handle 11% of backend code changes.
A venture capital case	A VC gave five staff top-tier accounts. They kept using the top model for simple work like email replies and summaries.	Just five people generated billing on a pace above roughly $100K per month.

Meta’s numbers are hard to picture, so it helps to lay them out.

73.7T

tokens used (30 days, company-wide)

~$20M

top individual’s spend (30 days)

<10%

firms that know their AI cost

This is the same pattern as when LOC or commit count became a productivity metric: people optimized for line count and commit count instead of quality. When a measure becomes a target, the measure breaks. This is Goodhart’s law.

The sharp turn toward “Tokenminimizing”

Out of concern for usage-based billing, companies have started to set strict limits and access caps (governance).

Company / sector	Measure taken
AT&T / Zscaler	Temporarily cut off access to certain developer tools over budget pressure. Applied role-based governance, giving legal and marketing only cheaper, lightweight models.
A U.S. aerospace and defense firm	Introduced a hard cap of $250 per employee per month. Power users used up their quota in the first four days, so the system was set to force access to the top model off.
A major airline	Tied token allocation directly to specific “projects” and their “expected revenue,” managed by the finance team as infrastructure cost.

Some, like pharmaceutical maker Novo Nordisk, have moved back toward a practical setup. After analyzing past clinical trial data with an LLM, they found that plain Microsoft Excel, with no AI features, was cheaper and more reliable — “if Excel gives a better result, you should stay in Excel.”

2. The first line of defense: the rise of model routers

The mechanism now spreading to hold this financial risk down is the model router.

Instead of letting users pick a model, a classifier evaluates the complexity and type of the input (prompt) in real time and automatically routes the request to the smallest, cheapest model that can handle it.

flowchart LR
    U["Request<br/>(prompt)"] --> C{"Classifier<br/>rates complexity / type"}
    C -->|simple task| S["light, low-cost model"]
    C -->|medium task| M["mid-tier model"]
    C -->|hard task| L["top model"]
    classDef s fill:#dcfce7,stroke:#16a34a,stroke-width:2px;
    classDef m fill:#fef9c3,stroke:#ca8a04,stroke-width:2px;
    classDef l fill:#fee2e2,stroke:#dc2626,stroke-width:2px;
    class S s;
    class M m;
    class L l;

Routing methods and results of the main players

Company / service	How the routing works	Cost / operations stats
Palantir (Evolve)	Works across multiple providers. Beyond switching routers, it also automates prompt tuning to avoid duplicate token use.	Auto-switches from heavy models to a small Nano model, cutting compute cost by 97%. At a customer construction firm, cut overall token use by 60%.
OpenRouter / Not Diamond	Lets you set a 1–10 dial between cost-first and quality-first, using Not Diamond’s prediction engine underneath.	In real dynamic-selection stats, cheaper Gemini Flash–class models made up about 33%, while top-model calls were held to 10%.
Sakana AI	A system that coordinates and switches between multiple AI models in real time.	Routes math to one model and science to Google-class models, matching top frontier-model performance at low cost.
Databricks (Unity AI Gateway)	A gateway for budget monitoring and access control.	CEO Ali Ghodsi has said it is “very popular among large companies about to burn through their budget.”
Cognition (Devin)	Has a built-in router that assigns work to the best AI agent by task difficulty.	For simple work like bug checks, it applies an in-house model that runs at about 1/100 the cost of the top model, cutting cost by 35%.

Lining up the reduction rates each company reports gives a sense of how large the routing effect can be.

Cognition (to in-house model)35%

Palantir customer (token use)60%

Palantir (heavy → Nano, compute cost)97%

Switch to Chinese OSS models (AI budget)99%

* Targets and measurement conditions differ by company, so these rates are not directly comparable. Read them only as a sense of scale.

The shift to Chinese open-source models and the birth of an “AI finance market”

Lately, more teams are avoiding the high cost of U.S. closed models and shifting requests to cost-efficient Chinese open-source models (Alibaba’s Qwen, DeepSeek, and others), compressing their AI budget to 1/100 (a 99% cut).

Riding this move, Together AI, which hosts open-source AI, saw its monthly processed token count grow sharply over a few months, with annualized revenue passing $1 billion. Infrastructure companies like Fireworks AI and Runpod are also drawing investment.

There is even new financial plumbing forming to make sense of AI providers’ hard-to-read bills, by making infrastructure visible and hedgeable. Examples include the launch of the “Ornn Token Price Indices,” which track major model prices weighted by real trading volume, and the development of a “Compute Futures” market by exchanges such as the Chicago Mercantile Exchange (CME). The financialization of AI infrastructure is moving quickly.

3. The blind spot in current routing

Here is where current mainstream model routers have an important oversight for business operations.

They optimize only for the shape of the prompt (the input), not for the value of the outcome (KPI) that the output produces.

Input complexity and business value do not always line up. Laying them out on these two axes shows the area current routers miss.

Simple prompt

Complex prompt

High value

Case A
A short summary that drives a decision.
The router sends it to a light model, but it really needs the top model or multi-model checking.

Complex and important.
The top model is the right call.
Router and value agree.

Low value

Simple and low-value.
A light model is enough.
Router and value agree.

Case B
Long, but a memo no one re-reads.
The router sends it to an expensive model, but a light model or local SLM is enough.

On one diagonal (lower-left and upper-right), the router’s judgment and business value agree. On the other diagonal, they clash.

• Case A: a simple prompt, but a task that cannot fail. “Summarize this progress data in one line” is a short prompt. But if that one line feeds a decision on a project worth hundreds of millions, no small error is acceptable. It is a case where you should spend on the top model or on multi-model verification. Yet the current router sees “simple” and sends it to the cheapest model.

• Case B: a complex, long prompt, but a low-value task. A complex query over a large pile of internal documents can still be “spell-checking an internal memo no one re-reads.” However long it is, a light model or a local SLM is enough. But the current router decides “this needs advanced reasoning” and sends it to an expensive model.

As long as the only basis is input complexity (token length and prompt difficulty), you cannot maximize AI’s return on investment (ROI). I think we need to move to routing based on how much a task contributes to business outcomes (KPI).

4. From “input-driven” to “outcome-driven”

Today’s model routers mostly think like this:

flowchart TB
    subgraph IN["Input-driven (current)"]
      direction TB
      A1["Is this prompt hard?"] --> A2["Which model can handle it?"] --> A3["Which is cheapest with enough quality?"]
    end
    subgraph OUT["Outcome-driven (next stage)"]
      direction TB
      B1["Which KPI does this task affect?"] --> B2["How much quality does that KPI need?"] --> B3["Within that quality, which model gives the most result per dollar?"]
    end
    IN -.evolve.-> OUT
    classDef in fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px;
    classDef out fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px;
    class A1,A2,A3 in;
    class B1,B2,B3 out;

Input-driven is the right direction. But at the next stage, the question itself has to change.

What matters here is not cost reduction alone. If cutting AI cost becomes the goal, quality can fall with it. But if you chase quality alone, everything flows to expensive models and you are back to Tokenmaxxing.

What you should watch is this ratio:

$$\text{Result} \div \text{Cost}$$

How much quality or KPI improvement did each dollar of AI spend produce? Without that ratio, you cannot measure AI’s real return on investment.

5. VOI, KOI, and Nishiki

For this problem I propose two ideas: VOI (Value on Investment) and KOI (KPI on Investment). The tool that implements them is Nishiki, an open-source project.

flowchart LR
    VOI["VOI (what to aim for)<br/>measure AI investment<br/>in value, not money"]
    KOI["KOI (how to measure)<br/>KPI ÷ cost"]
    NISHIKI["Nishiki (what measures it)<br/>measure model candidates<br/>on real data and pick<br/>the best path"]
    VOI --> KOI --> NISHIKI
    classDef a fill:#e0e7ff,stroke:#4f46e5,stroke-width:2px;
    classDef b fill:#fef9c3,stroke:#ca8a04,stroke-width:2px;
    classDef c fill:#dcfce7,stroke:#16a34a,stroke-width:2px;
    class VOI a;
    class KOI b;
    class NISHIKI c;

VOI: Value on Investment

VOI is the idea of measuring AI investment in value, not as pure cost.

Using one million tokens does not make it valuable. Using an expensive model does not make it valuable. What matters is what the AI output produced. Did accuracy go up? Did human review time go down? Did support get faster? How did it move real KPIs like revenue, retention, error reduction, review speed, or reconciliation accuracy? What you need is not a token count, but a way to measure value.

KOI: KPI on Investment

KOI is VOI put into a form you can actually measure in practice.

$$\text{KOI} = \frac{\text{KPI}}{\text{Cost}}$$

It measures how much KPI you achieved against the cost the AI spent. Where ROI expresses “return on investment,” KOI puts KPI in place of Return.

But the point is not to blindly pick the cheapest model. In Nishiki, there is always a quality floor (a minimum quality line).

Task	Example quality floor
Financial document reconciliation	Drop any candidate below a set agreement rate with the human’s final judgment
Contract extraction	Do not use a model whose F1 score falls under a set line
Inquiry classification	Exclude a model whose misclassification rate exceeds the allowed range

Above that floor, it picks the path with the highest KOI. This lets you maximize result per dollar while protecting quality.

6. What Nishiki aims for: outcome-linked model routing

Nishiki is an OSS that reads the source code of an AI / LLM agent and measures and optimizes “quality and cost-effectiveness per dollar” based on real data.

For an existing AI agent, Nishiki answers questions like these:

• Is the current model setup really cost-effective?
• How far can you swap in cheaper models without dropping quality?
• Which model produces the most KPI per dollar?
• Which tasks need a high-end model, and which are fine with a light one?

Measured examples

Here are two measurements made with Nishiki (see the Nishiki project page for details).

Target	Current state	Path Nishiki found
Extraction task (lightweight, reproducible in minutes)	The frontier model was the quality champion (F1 0.94) but the worst value per dollar	DeepSeek delivered 84% of that quality at about 1/117 of the cost — roughly 100x the quality per dollar.
Financial reconciliation agent (real operation, Bedrock, 50 items)	The current multi-model setup sat at KOI 28	Kept quality at or above the current setup while cutting cumulative cost by 73% and improving KOI by 3.8–4.3x.

−73%

cumulative cost (quality at or above current)

3.8–4.3x

KOI (result per dollar) improvement

Today’s model routers mostly pick a model by looking at the prompt. What Nishiki aims for is the next stage: not picking the best model for the prompt, but picking the best model for the KPI.

Look at contribution to outcomes, not input difficulty. Look at KPI per dollar, not token count. Look at cost per unit of result, not cost reduction. I think this is where the next layer of AI infrastructure is heading in the Tokenmaxxing era.

Conclusion

The real problem with Tokenmaxxing is this: we can measure how much AI was used, but not the value that AI produced.

Model routers are an important first step. By not using an over-expensive model and routing each task to the smallest one that fits, you can hold down runaway AI cost. But that alone is not enough.

What we need next is routing based on the outcome the output produces, not on input complexity. How much value did each dollar of AI cost create? How much KPI improvement? To answer that, I think we need the ideas of VOI, KOI, and Nishiki.

The answer to Tokenmaxxing is not to stop using AI, and not to simply run to cheaper models. What we need is to use AI correctly, against the outcome.

I am building Nishiki as an experimental OSS for exactly that. If you are interested, please see the GitHub repository (matu79go/nishiki).