The data

Completion of our forced-output contract, by model and language

The share of pages where every analysis stage returned a valid result under our forced-output contract, reached through OpenRouter’s default routing, out of thirty pages per language. These are not clean verdicts on the models: a number here reflects our schema and this serving path on these days, and the lowest numbers carry attribution caveats set out below and on the methodology page. Read the columns together, not the rows in isolation.

Completion of our forced-output contract via OpenRouter default routing, thirty pages per leg, not a standalone model benchmark. GPT-5.2’s +23.3 gap is the only one meeting the threshold, and all seven of its European failures are dropped-connection errors, not contract failures (see the taxonomy below). DeepSeek and Kimi’s low numbers carry the attribution caveats below. The study was powered to detect a ten-point cliff, not a subtle slope, so “zero of five open models show the predicted drop” means no cliff appeared, not that the languages are identical.

The clean test: the same page in three languages

The table above compares different pages across languages, so a gap there is not purely a language gap. This one removes that: eight English pages, each machine-translated to French and German by a single translator, run through every model. Same content, only the language changes. Completion here means the same thing as in the table above: the analysis ultimately produced a full, valid result, with the pipeline’s normal retries allowed. The numbers are model-attributable completion out of eight, with transient serving failures treated as passes because a retry cleared them. It is a small test, eight pages, so it catches a completion cliff, not a subtle slope.

Completion on identical content, out of eight pages, after re-running every genuine failure three times to separate a model fault from serving noise (the same attribution discipline as the main study). Single-run numbers before that separation: GLM 7 in English, Mistral 7 in French and 7 in German, GPT-5.2 6 in French; every one of those misses except one cleared on retry. The single reproducible failure in the whole arm is Mistral on the longest page, in German, at the concept-extraction stage, and that same page is the one GPT-5.2 stumbled on transiently in French. That pattern tracks page length, not language. The translator was GPT-4.1-mini, one translator for all pages, disclosed on the methodology page along with the machine-translation limits.

Two supporting reads on the same pages. The reference model placed five of the eight pages in the same coverage band across all three languages; the three that moved shifted by a single adjacent band with no consistent direction, which is within the run-to-run score noise, so the pipeline scores identical content about the same whatever language it is written in. And the exploratory quality lean does not reproduce cleanly here: GLM still leans behind the reference on French and German, but Mistral does not, and GPT-5.2 leans ahead of the reference on French, where its same-family translator caveat applies (see the methodology page). On identical content the lean is not a consistent cross-model effect. That is consistency with our current model, not correctness, and it is why we keep it a loose end.

Failure taxonomy

How the failing analyses break down, by cause, per leg. The point of the English column is that the same failure classes appear there: these are contract failures, not language failures.

DeepSeek’s European failures split into two kinds: 17 where it returned no structured call and 11 where it returned our format with broken syntax. The broken-syntax kind is a genuine limitation; the no-call kind we could not fully separate from the serving path, and the control meant to settle it was disrupted by a serving-layer outage (see the methodology page). We report the counts as observed, not as a verdict on the model’s underlying ability. Kimi’s cap overrun, by contrast, reproduces on every retry, so we are confident it is the model.

Failure attribution

Failures re-run to sort the model from the infrastructure. “Persistent” means the model fails the same way on retry; “intermittent” means it recovered on a second attempt. Kimi is all model; the rest are mostly flaky serving.

Quality panel (exploratory)

Blind head-to-head against the reference model, on the pages each model completed, per leg. Reported as reference wins / challenger wins / ties. This is the panel we did not human-calibrate, so it is exploratory and does not carry the headline. It measures consistency with our current model, not correctness.

The capable models (GLM, Mistral, GPT-5.2) move from an even tie in English toward leaning behind the reference in European languages. Qwen leans behind in both, so for Qwen this is not a language effect. DeepSeek and Kimi produced too few complete pages to judge.

Score consistency, the mechanical check on the residual

The quality lean above comes from the judge panel we did not calibrate. Our mechanical measure, how far each model’s score banding drifts from the reference model on the same pages, does not corroborate a clean language story: the drift is about as large in English as in European, and larger in English for Qwen. This is why we treat the quality lean as a loose end, not a finding.

Maximum per-band distribution shift versus the reference model. GPT-5.2’s European figure sits on the reduced set left after its streaming failures, so read it with that in mind.

The Mistral recovery arc

Completion rate before and after re-phrasing the concept limit as a hard instruction. Compliance rose; the blind quality standing above did not, which is the whole point of the sub-experiment.

Cost, for completeness

The open models are cheaper to run than the reference model, by roughly three to nine times per analysis on the stages under test. It is a property of the models, not the finding: it is why the study was worth running, not what the study concluded. Reliability, not price, is what separated the models here.

Corpus: thirty European and thirty English production pages, six per language, URLs withheld as customer data. Design and disclosures on the methodology page.