Measure AI investment by successful work, not tokens

When evaluating AI tools, it is easy to watch the wrong numbers: tokens consumed, seconds per answer, benchmark scores, or tasks opened per month. Those numbers describe system activity, but they do not tell us whether useful work was actually completed.

OpenAI's July 17 essay, 'A scorecard for the AI age,' offers a more useful frame. It uses four dimensions: useful work, cost per successful task, dependability, and value as usage scales. This is especially relevant to small teams because it forces measurement back onto product outcomes without requiring a heavy analytics program.

1. Did useful work increase?

The first question is about output, not usage. How many tasks cleared the quality bar? In a content workflow that might be a publishable draft; in software, a change that passes tests and matches intent; in support, a correctly resolved request. Without a measurable finish line, AI can generate activity without generating outcomes.

2. What did each successful task really cost?

The API bill is not the full cost. Human review, rewriting, failed runs, and recovery from mistakes also count. A cheap model that needs three attempts and a long review can cost more than an expensive model that passes once. The right denominator is not tokens but successfully completed tasks.

3. How dependable is the result?

A system that is excellent sometimes and dangerous at other times is weak in production even if its average score looks high. Dependability can be tracked through correction rates, retries, rollbacks, and the points where a human remains mandatory. One of my most useful questions is not 'can I ship without checking?' but 'is the time needed to check going down?'

4. Does each dollar buy more work at scale?

Tracking the same workflow over time matters here. If completed work grows faster than total cost while quality holds, the economics are improving. Model routing, better prompts, caching, delegating to smaller models, and clearer acceptance criteria can all improve that ratio.

This completes the operational side of my earlier note on token pricing starting to crack: what a team buys should be a dependable outcome, not raw computation.

A practical start is simple: choose one repeated workflow, write down its quality bar, and track completed tasks, total human time, and AI cost for a month. Then compare tools by which one produces the same quality at lower total cost, not by which one looks smartest. For a small product, a real AI scorecard does not need to be more complicated than that.