Methodology

The pre-registration, and the outcome

We locked one confirmatory hypothesis before running any model: that a challenger’s completion rate would drop by at least ten points from its English pages to its European pages, with a significant test, in at least two of the five open models. That hypothesis failed. Zero of five showed the drop. We are reporting the hypothesis as we wrote it and marking it failed, rather than quietly rewriting it to match what we found. The published finding is the reversal that the controls produced, not a narrowed version of the original claim.

A word on power, because it bounds what the null means. Thirty pages per language gives us the ability to detect a large drop, the ten-point cliff we pre-registered, not a subtle few-point slope. So “zero of five” is evidence that no cliff appeared across the range we tested, not proof the languages perform identically. A smaller true effect could sit under this sample. The pattern makes a large hidden effect unlikely (one model runs the wrong way, two are floored in both languages), but we do not claim more certainty than thirty pages can carry.

The corpus

Thirty native European pages (six each in French, German, Spanish, Italian, Dutch) and thirty English pages, all drawn from real analyses that ContentGrapher users submitted, one page per domain, spanning short to long. Because these are customer-submitted URLs, we withhold the list: publishing it would reveal who uses the product. The data page reports language, page category, and size class per anonymized page, never the address.

The three controls

1. AThe within-model English leg. Every model was also run on the English corpus, so each model is its own control. This is what isolates the language axis: a failure that also happens in English is not a language failure.
2. BFailure attribution. Every failed analysis was re-run three times to classify it as persistent (the model fails the same way) or intermittent (flaky), and once pinned to the model maker’s own serving endpoint. No failure-mode claim is published without this classification.
3. CThe translation-paired arm. The within-model leg compares different pages across languages, which isolates language only if the two page sets are equally hard. The paired arm removes even that assumption: the same eight pages in English, French, and German, so the only thing that changes across the three is the language. Described in full below.

A caveat on the pinning: for several models the maker’s own endpoint also failed, or rejected the forced output format outright, so pinning could not always isolate the model from its rehosts. We record that rather than pretend the attribution is cleaner than it is.

One control we attempted and could not complete: for the models whose dominant failure was returning no structured call, we tried re-running with the output forced a different way, to separate the model from the serving layer. That run was disrupted by an authentication outage on the serving path (the same key succeeded on direct calls but failed through the SDK for every model at the time), so its results were not trustworthy and we discarded them. The question it was meant to answer, whether a share of DeepSeek’s failures is the serving path rather than the model, therefore remains open, and we say so on the overview rather than fill the gap with a guess.

The translation-paired arm, in detail

We chose eight English pages that the capable models complete, spanning short to long and mixing product docs, commercial pages, and editorial. One translator, GPT-4.1-mini, translated each into French and German. That yields eight originals and sixteen translations, twenty-four page instances, run through the reference model and the capable challengers on the exact production stages. Because the bytes are the only thing that differs across a page’s three versions, a completion or quality gap across them is a language effect and nothing else, which is what the natural corpora could not give us.

What counts as completion: the analysis ultimately produced a full, valid result, with the pipeline’s normal stage-level retries allowed, the same definition as the main tables. That matters because a one-shot serving error that the pipeline repairs on the next attempt is not a model failure; it is the serving noise this study is about. An earlier draft of this arm used a stricter accounting that counted a repaired analysis as a failure; a review pass caught it before publication, and correcting it dissolved what had briefly looked like a French-specific weakness in the frontier comparator. We note that here because a study about measurement artifacts should disclose its own.

The limits are real and we state them. Eight pages is small: this arm can catch a completion cliff on identical content, not a subtle few-point slope, so we describe the result as consistent with the reversal, not as proof that language never matters. The text is machine translation, not native prose, and our French translations ran roughly a tenth to a quarter longer than their English source, which can nudge how many concepts a page appears to carry, so we read single-band score differences as noise rather than signal. The translator is an OpenAI model, which overlaps one member of the judge panel (GPT-4.1-mini) and one challenger (GPT-5.2); when GPT-5.2 is the challenger, the GPT-4.1-mini judge is already excluded by the same-family rule, but the translator itself could still produce French and German that GPT-5.2 reads more fluently than the reference does. The French quality lean toward GPT-5.2 in this arm runs in exactly that direction, so we do not read it as a clean win for that model. Every genuine failure in this arm was re-run three times on default routing to separate a model fault from serving noise, the same discipline as control B, though here we did not additionally pin the maker’s endpoint, so “persistent” in this arm means it did not recover on retries, a slightly weaker claim than in the main legs.

The result agreed with the main study and is reported on the data page: on identical content, completion did not fall on non-English once serving noise was removed, for any of the four models. The single reproducible failure in the arm is the longest page, at the concept-extraction stage (Mistral, in German), and the same page tripped GPT-5.2 transiently in French: a length and stage-load signal, not a language one.

What each condition holds fixed

Only the model behind the four analysis stages changes between conditions. The page text is scraped once and replayed to both, the earlier extraction stages are identical, the prompts are identical, temperature is zero, and the output format is forced. Two model-specific prompt lines exist and are disclosed: a style note for GLM, and the hard concept-limit instruction added for Mistral (see below). The base prompts every other model receives are identical to the byte.

The judge panel, and the calibration waiver

For the secondary quality comparison, each analysis pair was scored blind by a panel of models from different makers, with presentation order counterbalanced, and with the two contestants’ own families barred from judging. We chose not to run a human calibration pass on the panel. That is a deliberate limit, and it is why the quality result is exploratory and never carries the headline: our own reliability study found panel agreement (82%) is not the same as human agreement (57%). The headline of this study rests only on mechanical measurements: completion rates and error logs, which need no calibration.

The Mistral sub-experiment

Mistral’s dominant failure was overrunning a hard limit on concept count. We re-phrased that limit as a blunt system-prompt constraint, scoped to Mistral alone, and re-ran. Completion improved; the blind quality standing did not. We report both runs: the first as the failure, the second as the recovery attempt. This is the evidence behind “compliance is promptable, judgment is not,” and it is a single-model observation, not a general law.

Provider routing and quantization

Open models were reached through a routing layer that serves each model from many independent hosts at differing numerical precisions. The main runs used default routing, which is what a real deployment would hit. We did not capture which host served each call on the first runs; the attribution re-runs partly close this, and we disclose the gap rather than imply we controlled it. Because of this, a low completion number here is a statement about a model through this routing layer under our contract, not a clean benchmark of the model itself.

We did not contact the model makers for comment before publishing. We would rather correct the record than defend a number: if a maker can show a materially better result on their own endpoint, under this same contract, we will run it and update this page. The harness is public, so the test is reproducible.

Conflict of interest

ContentGrapher sells the product under study, and we ran the judging. Three things bound that: the confirmatory hypothesis and its pass criteria were registered before we saw results; we did move our English engine off our previous, more expensive model when the evidence supported it, which cuts against a motivated conclusion; and the study harness and its gate definitions are in our public repository. We state this plainly rather than leave it for a reader to infer.

Related: the language study (the pipeline reads non-English) and the concept validity study (the framework’s concepts hold across languages). This study asks the third question: do the models hold.