Skip to content

Cost per accepted diff

Quality is one axis; the price of the diff is the other. A candidate that adds +0.10 for five times the tokens can beat one that adds +0.09 for 1.1x if the gate only reads the held-out number. This page is the second ledger: what one diff cost, what that buys, and when a quality gain is refused because its efficiency regressed.

1. What a diff costs

Cost is the per-diff price block. Every field is optional on purpose:

field meaning where it comes from
calls model calls spent producing the diff agents.Usage.calls
prompt_tokens / completion_tokens the provider's real split agents.Usage
tokens total when only a total is known (an evidence card records one) EvidenceCard.cost_tokens
wallclock seconds spent Usage.seconds / EvidenceCard.cost_wallclock
usd a dollar estimate the caller measured caller-supplied
from agentdescent import Cost, cost_efficiency, cost_status

cost = Cost.from_usage(usage, usd=0.03)      # real counts, caller's price
print(cost.tokens, cost.usd)
print(cost_efficiency(0.10, cost))           # held-out delta per 1k tokens
print(cost_status(cost_efficiency(0.10, cost), 0.05))   # improved/neutral/…

A missing number is None, never 0. Zero is a measurement ("this diff was free"); an unmeasured field is not. Cost().known is False, and every derived value (cost_efficiency, the report's per-accepted fields) stays None rather than pretending the diff was free. Cost.from_cards sums what the contributing evidence cards record for one working diff — and in-tree that is nothing, since no producer writes a cost onto a card, so it returns Cost() as well. The acceptance option has the fallback and what require_cost does with it.

usd is whatever the caller measured. There is no provider price table and no billing in this repository: an unpriced run leaves usd=None and still reports tokens.

2. The acceptance option

CostPolicy is a wrapper, not a replacement. It runs the inner gate first and returns its rejection untouched — a cheap candidate the quality gate refused is still refused. It only prices candidates that were about to commit:

from agentdescent import CostPolicy
from agentdescent.defaults import DefaultAcceptance

acceptance = CostPolicy(
    DefaultAcceptance(),
    reference_quality_per_cost=0.05,   # per 1k tokens, from an earlier accepted diff
    tolerance=0.05,                    # a 5% wobble is neutral, not a verdict
    require_cost=True,                 # no measurement -> defer, not "free"
)
field what it decides
reference_quality_per_cost refuse when the measured quality_per_cost regressed against this reference, even though raw quality rose
min_gain_per_1k_tokens refuse when efficiency is below an absolute floor
tolerance dead band around the reference, so noise-sized wobbles are neutral
require_cost with no measured cost, defer (category="cost-unknown") instead of pricing it at zero

A refusal is AcceptDecision(accept=False, category="cost-regression") and names the measured efficiency, the reference and the gain in detail; the cost block (with status) is on provenance["cost"]. The decision that a cost regressed is a refusal, not a rejection of the candidate's quality: the quality gain is real, it just did not pay for itself.

The cost the policy reads is MergeContext.diff_cost when the caller measured one. Otherwise it is Cost.from_cards over the cards that justify the diff, and in-tree that fallback is always unknown: no producer writes cost_tokens/cost_wallclock onto an EvidenceCard — evolution.py and async_evolve.py build cards without them — so the sum is empty and from_cards returns Cost(), not a free diff. A caller that wants the gate to price a diff passes the measurement itself:

from agentdescent import Cost

ctx.diff_cost = Cost.from_usage(usage)   # the caller's own metered Usage

require_cost=True on a run whose cards carry no cost

Every candidate the inner gate accepted is then deferred, with category="cost-unknown": that run cannot commit until the caller prices the diff. The reference_quality_per_cost and min_gain_per_1k_tokens branches are read only from a measured cost, so neither fires on the card fallback.

3. The report

cost_report turns decision records into the two numbers a review needs — cost per accepted diff and cost per point of gain — and counts missing cost instead of zeroing it:

from agentdescent import cost_report

report = cost_report(decisions, reference_tokens_per_point=20_000)
print(report.tokens_per_accepted, report.usd_per_accepted)
print(report.tokens_per_point, report.cost_status)   # improved/neutral/regressed/unknown

Each record is a mapping (or object) with accepted: bool, an optional gain (held-out delta) and an optional cost. Spend covers every decision, so tokens_per_accepted is the amortised price of one accepted diff; the per-point fields divide by the accepted positive gains. missing_cost counts accepted diffs that carried no usable cost, and cost_status is unknown without a reference — there is no scale to call anything better or worse against.

The averages are per unit, and each one is None unless every decision measured that unit: usd is optional by design and an evidence card records tokens and wallclock but never calls, so a row priced in one unit and not another is normal. Dividing a tokens-only numerator by all accepted diffs would report a dollar figure cheaper than the evidence, which is the same failure as counting an unpriced diff as free — one field down. The partial totals (tokens, usd, wallclock) and missing_cost stay visible so the gap is still countable.

4. The promotion bar

PromotionRules(require_cost=True) is the fail-closed bar on the promotion side. A PromotionRequest without a measured cost (or with an all-None Cost) defers with cost in missing; it is never treated as zero and never a rejection. With the default require_cost=False, requests without cost keep exactly the decision and hashes they had before this feature.

The promotion bar and CostPolicy(require_cost=True) are not the same test, and the difference is deliberate: promotion needs a measurement (Cost.known), so a wallclock-only cost passes it, while CostPolicy needs a measurement in the unit it divides by, so the same cost defers there. A caller who wants the stricter reading on the promotion side should price the diff in the unit it gates on.

5. What is not here

Provider billing, a price table, and a claim that tokens are the only cost axis.

agentdescent/pilot.py is shipped, not planned — it landed in #1121 — and its PilotReport carries the run's cost mapping in cost, beside the scores, evidence and manifest of one evolution run. CostReport remains the cost-only surface this page describes: cost_report turns decision records into the two averages above and never reads a PilotReport. The pilot CLI and its own page are #937 step 3.