multilingual NLP · accepted
Translation Repairs Relations, Not Retrieval
Populating a knowledge base from low-resource text fails in two different ways, and standard evaluation scores them as one number. This paper separates them, then tests whether translating to English fixes either.
Problem
Populating Wikidata from low-resource language text fails in two distinct ways, and a monolithic triple score cannot tell them apart. A model can fail to understand the source sentence, or it can understand it perfectly and still fail to name the canonical identifier of an entity it has only ever seen written in another script.
Approach
Two open 8B models, eight Indic languages, two arms. Qwen3-8B and Llama-3.1-8B-Instruct run DIRECT and PIVOT (translate first via IndicTrans2, then extract), scored component by component so every error is attributable either to relation identification or to entity grounding. The hypothesis, the decision thresholds and the outcome-to-claim mapping were frozen in a version-controlled pre-registration before any model output was scored.
Findings
Translation repairs relations. It does not repair retrieval. Relation-F1 improves for Qwen3-8B in all eight languages and for Llama-3.1-8B-Instruct in seven of eight. Entity grounding yields no verdict-eligible gap, and for Qwen3-8B translation makes grounding measurably worse.
The question.
Knowledge bases like Wikidata get populated from text, and in low-resource languages that process fails often. The interesting part is that it fails for two unrelated reasons. A model can fail to understand the source sentence, or it can understand it perfectly and still fail to ground what it understood into a canonical identifier.
Standard evaluation cannot tell you which one happened. Extracted triples are scored monolithically, so a missed triple never says whether the model misread the sentence or simply could not name the Wikidata identifier of an entity it had only seen in another script. The conflation is expensive because the two failures call for opposite remedies: better multilingual pretraining addresses the first and does nothing for the second.
Approach.
Eight Indic languages (Hindi, Bengali, Telugu, Tamil, Kannada, Malayalam, Marathi, Gujarati) plus an English control. Between them they are spoken by over a billion people and remain low-resource by knowledge-base coverage. Two open 8B-class instruction-tuned models, Qwen3-8B and Llama-3.1-8B-Instruct, each run in a DIRECT arm and in a PIVOT arm that translates to English first.
The benchmark was built by reversing the manually annotated test split of XAlign, a fact-to-text alignment resource for Indic languages, and deterministically re-grounding its English fact labels into Wikidata identifiers. Then the benchmark itself was audited, because a measuring instrument of unknown accuracy is not evidence.
Decisions.
The verdict was frozen before the scoring
The hypothesis, its decision thresholds and the mapping from outcome to claim were committed to a version-controlled pre-registration before a single model output was scored. That commit ordering is the whole defence: nobody can have chosen the threshold that made the result look good, because the threshold existed first.
Score the components, not the triple
Every error is attributed to relation identification or to entity grounding by construction, rather than collapsed into a single number. This is the only reason the headline finding can exist at all, because it is a statement about which half of the problem translation actually touches.
Audit the instrument before trusting it
Predictions absent from the gold were cross-checked against live Wikidata, with a screened native-speaker check for one language on top. That puts a quantified floor under how incomplete the adapted gold is: at most 4.8% of them are in fact true in the live knowledge base, pooled at 1.3%. Roughly a fifth of predictions never enter that classification at all, because they lack a resolvable subject, property or object identifier, and the paper reports that rather than quietly excluding it.
Findings.
Relation-F1, Qwen3-8B
8/8
Relation-F1, Llama-3.1-8B
7/8
Entity-F1, Qwen3-8B
7/8
The verdict is mixed and the paper says so plainly. The confirmation criterion runs over the five Qwen3-8B languages that have a measurable baseline gap. All five move in the hypothesised direction, giving an exact one-sided sign test of p = .031, which does not survive the conservative family correction applied across the decision table.
Entity grounding never produces a verdict at all, for two different reasons worth separating. Qwen3-8B has no measurable gap left to recover, and Llama-3.1-8B-Instruct's four usable languages fall below the five-language bar the pre-registration set in advance.
The Qwen3-8B entity result is the one worth sitting with. Translation makes grounding worse, and decomposing the archived candidates attributes that to retrieval rather than selection: once the correct entity is retrieved at all, the model picks it in 92.9% to 99.5% of calls. Llama-3.1-8B-Instruct does not replicate the pattern, and its retrieval direction inverts.
A model can understand a translated sentence perfectly and still be unable to name the identifier of an entity it has only ever seen written in another script.
— on why the two failure modes need opposite fixes
Accepted at SPELLL 2026, Track 7, for publication by Springer in CCIS.
Next
The benchmark, the audit protocol and the evaluation harness are public, with the pre-registration commit intact as the first entry in the repository's history. The half of the problem translation does not touch, canonicalizing an entity across scripts, is the one worth attacking next.