A model’s price per million tokens does not tell a buyer what a multi-step AI agent costs to finish a job. In a benchmark supplied to TECHi by AI/ML API, six models each attempted 12 customer-support refund cases three times: 216 runs in all. At the October 2, 2026 rates used by the submitter, reported average cost ranged from $0.00184 to $0.04139 per run, roughly a 22.5-fold spread. The submission’s underlying run logs and full test harness were not supplied, so these are the provider’s reported observations, not an independent TECHi benchmark. TECHi recalculated the table’s average token-cost arithmetic from the figures provided; those calculations agree within rounding.

Four effects explain the gap:

  • Context resending. The agent sends the whole conversation again at every step, and each step is billed for it.
  • Retries. Some calls fail and must be repeated, and every repeat carries the full context.
  • Latency. The steps run one after another, so each step adds waiting time.
  • Wrong results. Some runs end incorrectly, so the real cost of a task is total spend divided by the number of tasks done correctly.

The sections below measure each effect.

Every step sends the whole conversation again

In the chat-completions loop used for this test, the application resent the system prompt, tool definitions, user request, earlier model messages and tool results at later steps. Repeated input was billable, although a cache hit could lower its rate. Other API designs can manage conversation state differently; the finding describes this implementation.

The submitted trace follows a refund for one item from a three-item order, handled by Claude Sonnet 5.5 without an injected failure. It shows five calls with 1,230, 2,102, 2,193, 2,351 and 2,554 input tokens. Those add to 10,430. The final request contained 2,554 tokens, so 7,876 input tokens across the five requests were earlier context resent in the loop. That is about 75.5% of the total. This calculation assumes the growing requests retained the earlier prefix, as the submission says.

The same trace reportedly produced 614 output tokens. Using the submitter’s October 2 prices of $2.60 per million input and $13 per million output tokens, the reconstructed input cost is $0.0271 and output cost $0.0080. Input is about 77% of this example’s token bill despite the higher output rate. The prices are historical inputs to the comparison, not current quotes.

Across the submitted 216 runs, input reportedly made up between 30% and 82% of the modeled token bill, depending on the model. Tokenizers also differ. The submitter says the first request for the same task counted 470 input tokens on GPT-4.1 mini and 1,230 on Claude Sonnet 5.5. That makes cross-provider token totals a weak efficiency measure by themselves; compare completed tasks and money spent instead.

What one task costs on six models

Each model attempted the same 12 refund cases three times. The agent had five tools and a written refund policy; a run ended when the model stopped calling tools or the loop hit its step limit. The submitted table uses AI/ML API’s claimed prices as of October 2, 2026. The current AI/ML API pricing page is the source for today’s rates; they have changed for several models, so the historical cost column must not be used as a live buying quote. Token counts and cache shares are averages reported by the submitter, not independently audited usage logs.

Submitted model averages at October 2, 2026 rates

  • DeepSeek V4 Flash: 9,574 input and 731 output tokens per task; 83% of input served from cache; $0.39 input, $0.0078 cached input and $1.56 output per million tokens; $0.00184 per run.
  • GPT-4.1 mini: 6,714 input and 258 output tokens per task; 41% of input served from cache; $0.52 input, $0.13 cached input and $2.08 output per million tokens; $0.00295 per run.
  • GPT-5 mini: 6,361 input and 1,089 output tokens per task; 47% of input served from cache; $0.325 input, $0.0325 cached input and $2.60 output per million tokens; $0.00403 per run.
  • Claude Haiku 4.5: 9,462 input and 714 output tokens per task; 0% of input served from cache; $1.3754 input, $0.1375 cached input and $6.877 output per million tokens; $0.01792 per run.
  • Gemini 3.6 Flash: 8,327 input and 1,247 output tokens per task; 0% of input served from cache; $1.95 input, $0.195 cached input and $9.75 output per million tokens; $0.02840 per run.
  • Claude Sonnet 5.5: 12,362 input and 711 output tokens per task; 0% of input served from cache; $2.60 input, $0.26 cached input and $13.00 output per million tokens; $0.04139 per run.

The cost per run spans 22 times, from $0.00184 to $0.04139. Caching, reasoning tokens and input volume explain most of it.

Cache treatment is a large part of this comparison. The submission says DeepSeek V4 Flash served 83% of input tokens from cache, cutting its modeled cost per run by 62% against the same run without a discounted cache rate. It reports savings of 27% for GPT-4.1 mini and 18% for GPT-5 mini. The Claude and Gemini runs were uncached in this test. Their cost rows therefore reflect a different cache setup, which limits a direct model-to-model price ranking.

The submission reports that GPT-5 mini produced an average 1,089 output tokens per task, 702 of them internal reasoning tokens, and Gemini 3.6 Flash produced 1,247, including 982 such tokens. If these usage fields include billed reasoning tokens as described, visible answer length alone would understate output cost. The submitted GPT-5 mini row assigns about 70% of its bill to output.

The submission reports between 6,361 and 12,362 input tokens per task and 4.7 to 5.9 calls per task across models. Those figures combine different tokenizers, step counts and cache conditions. A buyer comparing current providers should use an identical task set, matching cache configuration and the live model catalog, then calculate cost per accepted result.

A failed call adds about a fifth to the cost

The submitted test deliberately introduced an unusually high failure rate. The first calls to get_order, get_customer and send_email could each return a simulated 503 error with 25% probability. At least one such failure appeared in 130 of the 216 reported runs. This is a stress test, not a claim about normal production reliability.

In this loop, a model-visible retry was another API call carrying accumulated context. The submitted faulted runs averaged 6.02 steps, 9,665 input tokens and 13.3 seconds. Clean runs averaged 4.76 steps, 7,492 input tokens and 11.0 seconds. These are group averages, and different cases could be harder or longer even without a fault.

The submitter also compared faulted and clean runs within the same model and case. It reports a median 21.5% extra token cost for a faulted run, with per-model increases from 14% for Gemini 3.6 Flash to 29% for Claude Haiku 4.5. That matched-case design is more useful than comparing every faulted run with every clean run, but the underlying pairings were not provided for independent review.

The injected 503s were not the only reported failures. Six GPT-4.1 mini runs and five GPT-5 mini runs included a tool call rejected by the tool layer for a separate reason, such as a missing argument or unknown identifier. The submission reports none of these rejected calls for the other four models. These observations depend on the exact tool schema and policy used here.

The most consequential reported failure was not visible in the token total. In one Claude Haiku 4.5 run, send_email returned a simulated 503; the agent treated it as final and reported the case complete. The submitted final state says the $1,079.10 refund was correct, but no customer email was sent. This example illustrates why completion must be checked against the tool log, not just the model’s closing sentence.

Steps run in sequence, so latency multiplies

The tested agent loop waited for each model response and tool result before issuing the next dependent call. Its task time was therefore the sum of serial steps. Some agent designs can parallelize independent work, but this refund workflow did not. The submission reports 4.7 to 5.9 calls per task on average, so per-call delays accumulated across a run.

Submitted latency averages

  • GPT-4.1 mini: 1.27-second median call, 1.59-second 90th percentile call, 7.5 seconds per task, 45 output tokens per call.
  • DeepSeek V4 Flash: 1.65-second median call, 2.16-second 90th percentile call, 9.2 seconds per task, 134 output tokens per call.
  • Claude Haiku 4.5: 1.69-second median call, 4.60-second 90th percentile call, 10.7 seconds per task, 153 output tokens per call.
  • Gemini 3.6 Flash: 1.87-second median call, 3.21-second 90th percentile call, 12.4 seconds per task, 220 output tokens per call.
  • Claude Sonnet 5.5: 2.08-second median call, 2.92-second 90th percentile call, 13.1 seconds per task, 121 output tokens per call.
  • GPT-5 mini: 3.05-second median call, 7.65-second 90th percentile call, 21.3 seconds per task, 189 output tokens per call.

GPT-5 mini took 21.3 seconds per task, almost three times as long as GPT-4.1 mini at 7.5 seconds. Its output per call was 189 tokens, of which 64 percent was reasoning. GPT-4.1 mini produced 45 tokens per call and no reasoning. For these two models, a longer wait came with more billed output tokens.

Some models also have a long tail of slow calls. Claude Haiku 4.5 had a median call time of 1.69 seconds and a 90th percentile of 4.60 seconds. GPT-5 mini had 3.05 and 7.65. A task makes about five calls, and one call at the 90th percentile adds 2.9 seconds on Claude Haiku 4.5 and 4.6 seconds on GPT-5 mini compared with a median call.

Failures add waiting time too. Runs with an injected failure took 2.3 seconds longer on average. These measurements come from one session on one day and will move with provider load.

Cost per correct task is the number to compare

In the submitted grading policy, a run counted as correct only if it issued the right refund amount, or escalated when required, and then emailed the customer. A cost-per-correct-task estimate divides total modeled token spend by the number of correct runs. This metric makes failed work visible, although it still does not price staff review, infrastructure, network overhead or the real-world cost of a bad refund.

Submitted correctness and cost per correct run

  • DeepSeek V4 Flash: 36 of 36 correct; $0.00184 per run; $0.00184 per correct task.
  • GPT-5 mini: 36 of 36 correct; $0.00403 per run; $0.00403 per correct task.
  • GPT-4.1 mini: 26 of 36 correct; $0.00295 per run; $0.00408 per correct task.
  • Claude Haiku 4.5: 26 of 36 correct; $0.01792 per run; $0.02482 per correct task.
  • Gemini 3.6 Flash: 35 of 36 correct; $0.02840 per run; $0.02921 per correct task.
  • Claude Sonnet 5.5: 36 of 36 correct; $0.04139 per run; $0.04139 per correct task.

GPT-4.1 mini is 27 percent cheaper per run than GPT-5 mini. Per correct task the two cost the same, $0.0041 and $0.0040. Claude Haiku 4.5 moves from $0.0179 to $0.0248, which is a 38 percent increase.

The submitted failure notes show why cheap tokens can be expensive operationally. For GPT-4.1 mini, they describe five out-of-window refunds totaling $751.59, three missing shipping refunds for gold customers and one unnecessary escalation. For Claude Haiku 4.5, they describe three missed restocking fees, four refunds based on list rather than discounted prices, and two out-of-window refunds. Those notes do not enumerate every wrong run in the 26-of-36 totals; the complete run-level grading sheet was not supplied. The reported overpayment total for the specified Haiku cases was $123.53.

The submission says each wrong run ended normally: the agent made its final tool call, summarized the case and stopped. One reported wrong refund of $116.10 is about 39,000 times a $0.00295 token bill. That ratio is arithmetically sound, but it is an illustration of outcome risk, not a prediction of expected loss. A final-state check of refund amount, escalation and email delivery would identify errors the token usage field cannot.

The sample is intentionally narrow: one support workflow, one refund policy and 36 runs per model. The apparent 36-of-36 success rows are not proof of perfect reliability, and a 26-of-36 result is not a general model ranking. Without run-level logs, seed values and the full harness, outside readers cannot independently reproduce the reported accuracy or latency. Use the study to design a buyer’s own test, not to declare a universal winner.

Four changes that lower the cost of an agent

Keep a stable beginning to the prompt and measure cache hits. OpenAI’s prompt-caching guide explains automatic reuse of matching prefixes, DeepSeek’s context-caching guide describes its default cache behavior, and Anthropic’s documentation explains its cache controls and minimum prompt lengths. The exact thresholds, rates and provider behavior can change. The submitted test’s savings of 62% on DeepSeek V4 Flash and 27% on GPT-4.1 mini should be treated as results for its particular prompts and historical prices.

Trim tool responses to what the decision needs. The submitted GPT-4.1 mini trace says the context grew by 1,378 tokens between its first and second calls, largely after an order lookup returned ten past orders the task did not use. In a loop that resends prior tool results, those fields can be charged again on later calls. The precise saving depends on caching and the data retained for audit or safety checks.

Handle transient tool errors in application code when it is safe to retry. A 503 resolved by the tool wrapper need not become another model turn; returning it to the model may add a call with the full context. The submitted matched-case comparison put the median extra token bill for an injected fault at 21.5%. Retry policy must still avoid duplicate refunds or emails: idempotency and final-state verification are part of the design.

Score completed outcomes, not just token use. A test harness should compare the amount refunded, whether escalation occurred and whether the customer email was recorded against an expected result for every case. AI/ML API reports that its checker caught each wrong run and that all 216 modeled runs cost $3.47 at its October 2 rate card. Those totals are vendor supplied; TECHi could check the table arithmetic but did not rerun the experiment.

How to measure your own agent

Start with a small set of cases whose correct outcomes you can define before running any model. Record the model ID, API endpoint, exact prompt and tool schema for each run. Log input, cached-input and output tokens from the provider response, the quoted rate card and the elapsed time for every call. Preserve errors and tool results so a retry is visible. Then inspect the final state: did the refund match policy, was the customer told, and did any duplicate action occur? Sum total spend and divide it by accepted runs. Repeat enough times to see whether a failure is stable or occasional. This procedure is more useful than a price-per-token comparison, but it does not replace a production pilot with real service-level and human-review costs.

How the test was run

According to the material supplied by AI/ML API, a Python agent loop called six models through its chat-completions endpoint with function calling. The mock support agent had five tools: get_order, get_customer, issue_refund, escalate_to_human and send_email. A six-rule refund policy included a 30-day window, a 10% restocking fee, a gold-tier exception and escalation for customers with three or more recent refunds. The company did not supply the full prompt, code or run logs to TECHi.

The submission describes 12 cases: eight single-item orders and four multi-item orders with a 10% coupon on line totals. Each case ran three times per model, giving 36 runs per model and 216 in all. The loop stopped when the model made no tool call or after 12 steps. A run was graded correct if it made exactly one refund for the right amount, or escalated exactly once when policy required it, and sent at least one customer email. These rules define the study’s reported accuracy; they do not test every way a real support team might judge a case.

The submitter says the first get_order, get_customer and send_email calls could each receive a simulated 503 with 25% probability. It says token counts and cache figures came from each API response’s usage field, while client-side call times included network delay. Costs used AI/ML API prices recorded by the submitter for October 2, 2026. The study does not provide the individual response records or a preserved price snapshot, so its results cannot be independently replicated from the article alone.

This is one vendor-supplied benchmark of one policy on one day. Its fault rate is higher than a healthy production tool would normally aim for. Cache setup differed by model, and latency reflects one session and its network conditions. Model behavior, prices and provider availability can change. The useful lesson is the measurement method: record each turn’s usage and time, then compare spending with a verified final outcome.