We were sure competence would not survive the jump to other languages. Our own controls proved us wrong.

What looked like “these models break on non-English” kept turning out to be something else wearing its costume. Five times we caught an impostor: dropped connections, unequal page sets, retryable serving errors, a panel lean that does not reproduce, and once, our own measurement. What remained after all five is plainer and more useful: models that fail this task fail it in every language, and what breaks the rest is the weight of the page, not the language it arrives in.

First, the disclosure

Since early July, ContentGrapher’s English analyses have run on GLM 5.2, an open-weight model, in place of the model we used before. We did not swap it in on a hunch. It passed seven pre-registered gates and a seven-day production monitoring window with no regressions. We are writing that up here, in the same piece, because it is the honest setup for everything that follows: it is what made us confident the same model would hold in French and German too. Our reliability study covers how we test one model against another; this one asks whether that competence carries across languages. If you use ContentGrapher, this is the kind of scrutiny that sits behind the model reading your pages, and the reason we will change engines only when the evidence says so.

The hypothesis we held

Model cards advertise language support as a checkbox: “supports 11 languages.” That measures fluency, not whether a model can do a specific structured job in that language. Our job is strict: read a page, and return a fixed, machine-checkable analysis under a forced output format. We assumed the non-English version of that job would be meaningfully harder, and we were confident enough to name the effect and design a study to prove it.

One point of precision, before any number. On a French page our pipeline reads French but writes its analysis in English. So we are not testing “can the model speak French.” We are testing cross-lingual operation: read the other language, reason and answer in English. That is what real multilingual production work is.

The control, and the reversal

We ran six models on the same thirty European pages and thirty English pages, each model measured against its own English self. The chart is completion rate: the share of pages where every stage returned a valid result. Read each pair of bars together.

The transfer cliff we predicted is not there. GLM is near-perfect in both. Mistral is equally imperfect in both. Qwen is actually worse in English. DeepSeek and Kimi barely function in either language. Our pre-registered prediction was that at least two of the five open models would drop by ten points or more from English to European. Zero did.

The one exception that looked like our finding, and was not. GPT-5.2 shows a clean twenty-three point European gap. It is the single result that fits the story we set out to tell. It is also an artifact: all seven of its European failures are the same error, a dropped connection before the response finished, not a model struggling with French. The one clean transfer story in the whole study is a plumbing problem, and we only caught it because we re-ran the failures.

What actually governs reliability

If it is not language, what is it? Two things. The first is whether the model can hold the contract at all, and the same failures appear in English, so this is not a language story. Kimi is the clean case: it overruns a hard limit on how many concepts it may return, on almost every page, in both languages, and it does so on every retry. That is genuine model behaviour.

DeepSeek is the honest, messier case, and we will not overclaim it. About a third of its failures are malformed structured output, the model returning our format but with broken syntax, which is a real limitation. The rest are the model returning no structured call at all, and that one we could not fully pin on the model rather than the serving path. We tried to settle it by re-running those failures with the output forced a different way. The attempt was itself disrupted by an authentication outage on the serving layer, so it is inconclusive. We are leaving it inconclusive rather than dressing it up, and the irony is not lost on us: the study is about not mistaking the plumbing for the model, and the plumbing interfered with the very check meant to rule it out.

The second is serving noise. Open models are reached through a layer that spreads requests across many independent hosts at different quality settings. We re-ran every failure to sort real from flaky.

Kimi’s failures are genuinely the model. But most of DeepSeek’s and Qwen’s melt away on a second attempt, which means a single-shot failure rate overstates how bad the model is and understates how much the infrastructure contributes. Had we not controlled for this, we would have published router flakiness as a model verdict.

The cleaner test: the same page in three languages

The control above has a gap we named at the time: our English and European sets were different pages, and different pages can be differently hard, so a gap there is not purely a language gap. So we ran the version without that gap. We took eight English pages that the capable models complete, machine-translated each into French and German with one translator, and ran all three versions through every model. Same content, only the language changed. It is a small test, eight pages, built to catch a cliff, not a subtle slope.

On identical content, completion did not fall on non-English. The reference model is flat, eight of eight in all three languages. GLM matches it once its lone English miss is retried away. Mistral drops a single German page and nothing more. Even the frontier model GPT-5.2, whose French column briefly looked like the story we originally set out to tell (six of eight on a single run), cleared every French page on retry: both of its misses were one-shot serving failures, one a dropped connection, one an empty first response. Once again, the dramatic-looking language gap dissolved into plumbing.

The one failure that does reproduce is instructive for a different reason. Mistral fails the longest page of the eight, reproducibly, at the concept-extraction stage, in German, and that same page is the one GPT-5.2 stumbled on transiently in French. That pattern tracks page length and stage load, not language, which is the reversal’s point restated: what breaks these models is the weight of the job, not the language it arrives in.

Two smaller checks on the same pages point the same way. The reference model placed five of the eight pages in the same score band across all three languages, and the three that moved shifted by a single adjacent band with no consistent direction, which is within our run-to-run noise. And the panel’s quality lean does not reproduce cleanly here: GLM still leans behind the reference on French and German, but Mistral does not, so on identical content it is not a consistent cross-model effect. Eight pages is a small sample, so we say it precisely: this arm is consistent with the reversal and fails to refute it on the cleanest design we have. It does not prove language can never matter.

Five impostors, one lesson

It is worth lining them up, because the pattern is the finding. Five times, this study produced something that looked exactly like a language effect. Five times, a control took it apart.

1. 01The frontier model’s clean twenty-three point gap. The one result that matched our prediction: GPT-5.2 dropping from perfect English to three-quarters European. All seven failures were the same dropped connection before the response finished. Re-running the failures unmasked it.
2. 02The language contrast itself. Our English and European legs were different pages, so a harder language was tangled with harder pages. The paired arm removed the confound: the same eight pages in three languages, nothing different but the language. The cliff still did not appear.
3. 03The paired arm’s French dip. GPT-5.2 briefly read six of eight in French on a single pass. Both misses were one-shot serving failures that cleared on every retry.
4. 04The quality lean. A blind panel had the capable models slightly behind our reference model in European languages. The panel is uncalibrated, our mechanical consistency measure does not corroborate it, and on identical content it does not reproduce across models. A loose end worth another study, not a finding.
5. 05Our own yardstick. A draft of the paired arm counted an analysis as failed if any stage needed a retry, manufacturing a French weakness out of repaired serving errors. A review pass caught it before publication, and the methodology page discloses it in full. The fifth impostor was ours.

What survived every control is unglamorous. Kimi overruns the concept limit on almost every page, in both languages, on every retry. DeepSeek cannot hold the format in either language. And the longest page in the paired set breaks the heaviest stages for two different models in two different languages. Contract and load. Nothing about French or German.

A sub-story: you can prompt obedience, not judgment

Mistral gave us a clean side-experiment. It failed by overrunning the concept limit, so we re-phrased that limit as a blunt, hard instruction. Compliance improved: it stopped overrunning as often, and pages that always failed started passing. Its quality standing did not move. Instruction-following is a wording problem you can fix. Judgment is not.

What we cannot claim

1. 01This study was built to catch a cliff, not a slope. With thirty pages per language, we had the power to detect a large drop from English to European, not a subtle one. Read our result as "no cliff appeared," not as "the languages are proven identical." A smaller true effect could hide under this sample.
2. 02Our main English and European legs are different pages, not the same pages translated, so that contrast on its own is correlational. We ran the paired version to close that gap, the same eight pages in English, French, and German, and it agreed: no non-English completion cliff on identical content, and the only reproducible failure tracked page length, not language. But that arm is only eight pages, it uses machine translation rather than native prose, and its translator shares a maker with one of the models under test. It is consistent with the reversal; at that size it cannot prove language never matters.
3. 03Latin-script European languages only. Nothing here extends to Japanese, Korean, Chinese, or Arabic.
4. 04Six pages per language supports cohort-level statements, not per-language ones.
5. 05Every model was reached through OpenRouter’s default routing across many independent hosts, and we did not contact the model makers for comment before publishing. A model’s completion number here reflects our forced-output contract through that routing layer on these days, not a clean verdict on the model itself. We would welcome a maker showing a better result on their own endpoint.
6. 06For the two models that failed most (DeepSeek and Kimi), we cannot cleanly separate model incapacity from serving-path issues in every case. Kimi’s failure is deterministic and reproduces, so we are confident it is the model; a share of DeepSeek’s is not fully attributable, and our control to settle it was disrupted by a serving-layer outage. We report the completion numbers as observed, not as a verdict on the models’ underlying ability.
7. 07The quality-drift residual leans on a judge panel we did not human-calibrate, and our mechanical consistency measure does not corroborate it.
8. 08We sell this product and we ran the judging. We registered the checks before seeing results, we did move English off our previous model when the evidence supported it, and the harness is in our public repository. Named, not hidden.
9. 09Correction: an earlier internal read of ours held that one model passed on English and collapsed on non-English. The controlled English run shows it fails on English too. That earlier read came from a narrower, earlier task and was wrong. A study about controlling your assumptions should show its own.

How to test an LLM before putting it in production

1. 01Test the exact contract, not a benchmark. "Supports French" tells you nothing about whether a model returns your schema in French.
2. 02Control against your primary language. Most of what looks like a language weakness is a task weakness the model has everywhere.
3. 03Re-run failures before you believe them. Routing across many hosts is noisy; single-shot rates mislead in both directions.
4. 04Separate compliance from quality. A model that obeys your format is not the same as a model with good judgment.
5. 05Audit your own yardstick. Our draft metric counted repaired analyses as failures and invented a language gap. The artifacts you bring outweigh the effect you seek; measure the measurement before you trust it.

The practical decision on our side did not change: non-English analysis stays on our reference model, on the migration evidence. What changed is the question. We set out to ask which models lose competence in French, and the study kept answering a different and better one: what actually fails in production, and does it care about language. For this task, across six models and three controls, the answer was the contract and the load, and no, it does not.

Six models, thirty European pages and thirty English pages of real production traffic, six pages per language. Full design, gates, and disclosures on the methodology page; every number on the data page.