Quick Take
- The finding: Across 20 third-party LLM API relays advertising GPT-6 Astra and Claude Opus 5, first-round accuracy checks flagged 7 of 10 low-price services and 2 of 10 same- or higher-price ones for behavioral differences versus the official APIs.
- The billing risk: For identical requests, five relays reported 4,200 to about 60,000 more input tokens than OpenAI or Anthropic. Where balances were available, charges tracked those reported counts—including unexplained surplus at one service.
- What else broke: No prompt injection showed up in 3,332 screened code and setup responses, yet four relays still returned tool calls after clients set
tool_choice: none.
Third-party API relays promise cheaper or more convenient access to frontier models. You send a request for GPT or Claude; the relay forwards it and bills you. A new measurement study from Queen’s University, Kingston—”What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays”—asks whether that middle hop preserves answer quality, tool permissions and honest usage reporting. The authors anonymize every service as R01–R20 and compare them against batch-matched official OpenAI and Anthropic calls.
What the study shows
The team measured three client-visible properties on public APIs: answer accuracy on a fixed knowledge bank, returned content and tool permissions, and reported input tokens versus official references (plus balance changes where available).
- Accuracy flags cluster on the cheap tier—but not significantly at 0.05: In the primary first-round analysis, 7/10 low-price relays and 2/10 same- or higher-price relays had at least one flagged cell (two-sided Fisher p = 0.0698, reported as 0.07). A post-hoc “all four specifications” tally reaches 7/10 versus 1/10 (p = 0.02), but the authors treat that as non-primary.
- Some gaps are large: Relay R01’s Claude Opus 5 hit rate was 0.704 versus 0.992 officially (risk difference RD = 0.288). Pilot/extension cells for R20 Opus reached 0.409 versus 1.000 (RD = 0.591) and R11 Opus RD = 0.574. Several Astra cells were also Confirmed or Suspected.
- Flags are not forever: Retesting 35 pairs 11–18 days later, seven cells gained or lost a flag under the original rules (six under matched correction families). Six services stayed flagged in both rounds (R01, R08, R10, R11, R17, R20).
- Tool permissions leaked: Across 3,332 screened non-adversarial responses, no injection was confirmed. Still, in round 1 four relays returned tool calls despite
tool_choice: none(counts: R04 = 1, R11 = 24, R10 = 1, R17 = 4); R08 returned twoweb_searchcalls with no tools supplied. Round 2 showed none of those tool-call leaks. - Token reports can inflate the bill: Five relays reported 4,200–60,000 extra input tokens versus the official API on identical requests. On one Astra item, R01 and R20 reported 4,416 input tokens (4,224 cached) where the official API reported 36. R10 Opus showed a median surplus near 59,500 tokens. For the six services with readable balances, deductions matched reported usage at listed prices; R11 sits in both the balance group and the >4,000-token surplus group, so its surplus entered the bill.
| Check | Key result | Caveat |
|---|---|---|
| Accuracy vs official | 7/10 low-price and 2/10 same/higher-price relays flagged | Primary Fisher p = 0.07; not causal proof of substitution |
| Tool permissions | 4 relays returned tool calls despite tool_choice: none (round 1) | Calls not executed; cause (relay vs upstream) unknown |
| Billed input tokens | 5 relays: +4.2k to ~60k tokens vs official | Surplus ≠ proven hidden text; balances follow reported usage |
Controlled mixture tests also show limits of accuracy fingerprints: a related cheaper substitute (Astra→Luna) often passed at 10% replacement and was flagged in only 4.0% of simulations at 25%, while Haiku or GLM replacements were usually caught by 25%.
Why it matters: Three checks beat a price tag
Unlike fingerprint-only audits, this study ties answer behavior, permission compliance and billed usage together on the same anonymized services, with official API references and two measurement rounds. The authors publish explicit scoring rules, controlled mixtures and what their tests cannot see (message retention, key misuse, gateway compromise). That makes the work unusually useful for buyers who cannot inspect a relay’s internals.
Method in brief
- Services: 20 relays offering GPT-6 Astra and Claude Opus 5, selected from ranking sites after access and payment filters; 10 low-price, 10 same- or higher-price. Not a random sample; expanded after a low-price pilot.
- Accuracy: Knowledge/capability bank with ~125 scored relay responses per cell versus batch-matched official chat-completions calls; Benjamini–Hochberg q-values; labels Confirmed / Suspected / No evidence.
- Content: Fixed non-adversarial code and setup workload screened for harmful packages, commands and destinations (3,332 responses).
- Billing: Compare reported input tokens to official references; reconcile balance endpoints at six services.
- Anonymization: Services labeled R01–R20; identity mapping kept private.
What remains open
- No backend proof: An accuracy flag shows behavioral inconsistency, not which model ran. Wrappers, refusals, decoding or retrieval can also move scores.
- Price is not destiny: The primary price-tier test misses p = 0.05; three low-price relays had no flag.
- Unexplained tokens: A surplus can be an added system prompt or altered accounting; only R11 of the large-surplus group also had a reconcilable balance.
- Coverage gaps: No long multi-turn sessions, no credential-leak tests, no software-vulnerability scans; four Astra cells missing from the main table for interface reasons.
- Reproducibility: Accompanying anonymized data and plot inputs are mentioned; no public named-service scoreboard. OpenAI Codex polished the manuscript language.
Context: From shadow APIs to billable middlemen
Prior work fingerprints models, audits shadow APIs for fake model claims, and probes gateways for consistency and latency. This paper sits closer to buyer due diligence: it treats the relay as an untrusted delivery path whose contract covers quality, permissions and usage reporting at once. For i10X readers who already follow ChatGPT and Claude pricing, the practical question is whether a cheaper route still delivers the model they paid for.
What it means for teams buying LLM API access
- Verify tokens, not just $/1M: Diff reported input tokens against a small official reference batch; watch cache vs non-cache rates separately.
- Enforce tool permissions client-side: Never trust the relay alone—drop unexpected tool calls before execution.
- Retest: Flags moved between rounds 11–18 days apart; treat relays as living services.
- Price is a clue, not a certificate: Some cheap relays looked fine; some same-price ones did not.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of Wu, Zhao, Li, Cheong, Li and Ni (2026), arXiv:2611.07650v1 (cs.SE, submitted November 5, 2026). All numbers come from the paper; i10X did not retest the relays. The preprint is licensed CC BY 4.0. The chart was redrawn by i10X from the primary price-tier tallies in §V-B. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team. More coverage lives in our Benchmarks & Evals studies and Infrastructure section.
i10x Perspective
API relays are infrastructure, not magic discounts. If your product’s answer quality, tool policy or unit economics depend on GPT or Claude behaving as billed, treat the middleman as part of the threat model—and measure it the way this paper does: answers, permissions and invoices in the same audit.
