The predictive engine
Every other cost tool reads the bill after the money is gone. Meter reads the prompt and forecasts the bill first, in about three hundredths of a millisecond, with no network and no database. That forecast is what a budget ceiling reserves against, which is the only reason the ceiling can hold.
Where it sits
Every request through Meter runs the same lifecycle. Steps 1 to 5 have to finish before a single byte reaches the provider, so the estimate cannot afford a database round trip. It does not make one. The engine is pure computation over the prompt already in hand.
predictor/refresh.py reads the ledger on a 120 second timer, fits on the older 75% of rows, scores against the newest 25%, and installs only the per-feature factors that beat their own held-out rows. It writes into memory that predict() reads. It never queries anything during a request.
Step 4 reserves real budget against it before the call goes out. Reserving after the fact is the bug: a thousand simultaneous requests all read the same healthy balance and all proceed.
A background task refits from the ledger every 120 seconds into an in-memory dict. The request path only ever reads that dict, so the engine gets smarter without getting slower.
Why this is genuinely hard
The tempting formula is output ≈ ratio × input_tokens. Longer prompt, longer answer. Toggle it on and watch the fit fall apart against real calls.
Output length is set by the scope of the work requested, not by the length of the request.
Build me a CRM is four tokens and produces five thousand. Fix this typo, followed by an 800-token file, produces twelve. Any model keyed on input length has it exactly backwards.
The structural trick
Conflating these is why the first version got both wrong. A forecast wants to be accurate. A ceiling check wants to be safe. Those are different objectives and one number cannot serve both.
What will this probably cost?
hover →Dashboard figures, Treasurer runway, cost per outcome. Accuracy is the job, so it carries no safety padding at all.
What could it cost at worst?
hover →The hard ceiling reserves against this one. With max_tokens set, output cannot exceed it, so the guarantee is structural rather than statistical.
The forecast used to carry a 1.30 safety buffer. It was removed on purpose. The buffer and the history factor are both fitted as actual over scope, so applying both computed the correction twice and median error rose from 77% to 204% as the loop learned. Safety now lives entirely in the bound, where it cannot corrupt the forecast.
The pipeline, end to end
Stages 0 to 2 are a deterministic waterfall: if hard evidence exists, take it and leave. Only when none does do the soft signals compound, and each of those terms is bounded so a keyword-stuffed prompt cannot run away. Scroll, and the readout on the left carries one real prompt through every stage.
The key covers the payload, the model, max_tokens, and the attribution tags. Tags belong in it because they select the history correction seven stages later. Leave them out and one team gets served another team's calibrated answer, silently, and worse the better the correction gets.
key = sha256(payload, model, max_tokens, project, feature, actor)A JSON-schema response is bounded by its own structure. The input term stays in, because pulling entities out of a 10k-token document is not the same size job as describing one user, and a flat constant would repeat the mistake this whole engine exists to fix.
scope = 100 + input_tokens x 0.1If the user stated it, we do arithmetic instead of guessing, and this is the most accurate rule in the system. Coverage is the highest-value detail in the engine: one missed hyphenated form, two-sentence against in two sentences, moved total error from 84% to 192%. More than half the error came from a single hyphen.
"in three sentences" -> 3 x 35 = 105A prompt is rarely one task. Summarize this AND write the query is both, so scopes add rather than compete. Five multipliers then read intent: how forceful the verb is, whether reasoning is being demanded, terse against exhaustive, how much of the message is instruction rather than pasted material, and whether it opens with a generation verb.
scope = min(80 + sum(tasks), 1500)
x verb x cot x qualitative x instruction x imperativeA fitted per-bucket multiplier, searched offline against held-out calls and loaded from data/fitted.json at startup. This is the number written to the ledger as scope_tokens, and it has to be, because it is the fixed baseline the learner fits its correction against. Fit against anything downstream and each refresh divides by the last one and oscillates forever.
scope_tokens = scope x factor[bucket] (code = 4.03)Teams are consistent, because their prompts come from one template written by one person. Inside a single template, observed output length varies by only 1.0 to 1.4x, against 10x across templates. That regularity is the entire mechanism. The factor is a shrunk median of actual over predicted, looked up down a ladder from most specific to least.
factor = exp( n x ln(median(actual/scope)) / (n + 1) )
(project,feature,actor) -> (project,feature) -> (project)
-> (feature) -> (bucket,model) -> (bucket) -> 1.0max_tokens clamps the forecast, it never replaces it. Letting it short-circuit the pipeline measured 594% error against 192% for clamping, because max_tokens is a safety valve most SDKs set by default, not a statement of intent. A team with 4096 boilerplate would otherwise get one identical prediction for every prompt they ever send.
predicted = min(max(15, scope x factor), bound)
bound = max_tokens, else p95[bucket] x 1.2, else 4096All of it, on one line
Everything above collapses to this. Read it inside out: build a scope from the prompt, scale it by what this kind of task normally does, scale again by what this specific team normally does, then hold it under a ceiling it structurally cannot cross.
predicted_cost_usd = price(input_tokens, predicted, model)Watch it compute
This is not a mock-up. These are the numbers predict() returns for that prompt on the shipped engine.
Had the prompt said "in three sentences", stage 2 fires first and short-circuits all of it: 3 × 35 = 105, then straight to the bucket factor. A stated length is the most accurate rule we have, so we never guess past it.
Does it actually work
A prediction engine that quotes one accuracy number is hiding something, because accuracy depends entirely on the shape of the traffic. Here is every shape we tested, including the one we are worst at.
Median absolute percentage error. We do not quote MAPE: a handful of 20-token answers dominate it permanently and it flatters nobody honestly.
Templated production traffic, which is what agent workloads actually are. 24 of 32 feature tags sit at or under 15% median error. Best is entity-tag at 0.9%.
Open-ended prompts from strangers sit at 49% and that is near a hard ceiling. Output spread is std(log) = 1.16 and our features explain R² of 0.28, where reaching 30% error would need roughly 0.88.
A brand new feature tag starts around 80% error and needs about 20 calls of its own. History does not generalise between features, and we measured that rather than assuming it.
What is measured, what is fitted
A number you cannot trace is a number you cannot trust. These are the values the engine loads at startup, not the ones in the design doc. Note the two marked tuned to neutral: we hand wrote those cues, an offline search over real traffic found they earned nothing, and we shipped the search result instead of our own idea.
The optimizer went further than was comfortable. On real traffic our task keywords fire on 17.4% of prompts and chain-of-thought cues on 1.7%. Predicting a flat constant per bucket and ignoring the prompt entirely scores 51.4% against 44.2% for the full machinery, so the scope extraction is worth about seven points, not the bulk of the estimate. That is written down here rather than quietly dropped.
Why this is a first
Pre-flight cost estimators exist. The ones we evaluated stopped at the guess: they never fed real usage back, so their learning tier was permanently inert. Closing that loop is the whole contribution here. At capture, the actual token count lands in the ledger beside the prediction that reserved for it, and a gated background fit turns those pairs into a per-feature correction.
The gate is per key, not all or nothing. An earlier version scored the whole candidate batch on a pooled median, so one bad feature vetoed four good ones and the loop silently installed nothing for days. It also scored a shrunk candidate and then installed the raw one, which means it validated an object that never existed. Both are fixed and pinned by a test, and both are the kind of bug that only appears when you boot the real thing against a real ledger.
See the predictions land against their actuals, live, on the dashboard.
Open dashboard →