Anjishnu Mukherjee

COLM 2026

Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMs

Anjishnu Mukherjee, Chutong Meng, Antonios Anastasopoulos

George Mason University

Which languages do frontier LLMs really support, and what goes wrong when agents built on the same model end up working in different languages?

The languages an LLM claims to support and the languages it handles in practice are far apart. Across five frontier models, the self-reported list changes with the wording of the question (Llama 4 Maverick names between 1 and 183 languages), and in a proof-of-concept attack, pushing seven agents into different languages lowers MMLU-Redux accuracy from 75.0 to 67.5.

Two-panel chart for GPT-5.4, Claude Opus 4.6, Gemini 3.1 Flash-Lite, Llama 4 Maverick and Qwen 3.5 397B. The top panel has paired violin plots of reported supported-language counts, raw and LangID verified, with unions such as 187 and 134. The bottom panel has stacked bars of correct and incorrect answers for language identification and yes or no self-report.
The number of languages each model claims to support varies widely across 20 paraphrases and shrinks after language-ID verification (top), and models identify their own verified languages less reliably than they claim to support them (bottom).

Key findings

  • Self-reported language lists are unstable. We asked the same question in 20 English paraphrases, and Llama 4 Maverick alone named between 1 and 183 languages. The union of claimed languages ranges from 18 for Gemini 3.1 Flash-Lite to 187 for GPT-5.4 and Claude Opus 4.6. (Figure 1, Section 3.2)
  • Verification shrinks every list. After GlotLID checks a sentence generated in each claimed language, the union falls for every model, for example from 187 to 134 for GPT-5.4. Only 25 languages for GPT-5.4, 33 for Claude Opus 4.6 and 1 (English) for Llama 4 Maverick are confirmed in all 20 runs. (Table 3 (appendix), Section 3.2)
  • Models give wrong answers instead of refusing. On story writing, teacher dialogue, translation and code generation across 203 languages, non-refusal rates typically stay above 0.94 while correctness is much lower. On PolyWrite long-form writing, the rate of staying in the requested language in the lowest-resource class ranges from 0.893 for Gemini 3.1 Flash-Lite to 0.696 for Llama 4 Maverick. (Figure 2, Table 4 (appendix), Section 3.2; Figure 3, Table 4 (appendix))
  • A language switch can break an agent team. As an illustrative proof of concept, we ran a DyLAN-style system of seven role-based agents on MMLU-Redux. Accuracy drops from 75.0 when all agents use English to 67.5 when each agent is pushed into a different language. (Section 4 (results table labeled Figure 4 in the paper))
Four stacked panels for five models: target-language generation match rates for short stories and teacher conversations, translation target-language accuracy, translation quality as chrF divided by 100 (between about 0.25 and 0.45), and code generation with parseable versus functional fractions. Bars are split into correct, incorrect and refusal.
Across 203 languages the models keep answering with very few refusals, while target-language generation, translation quality and functional code all fall short.
Slope chart of average language-ID match on PolyWrite across language resource classes 5 down to 0 for GPT-5.4, Claude Opus 4.6, Gemini 3.1 Flash-Lite, Llama 4 Maverick and Qwen 3.5 397B. All lines start near 1.0 at class 5 and fan out to between about 0.7 and 0.9 at class 0.
The models stay in the requested language for high-resource classes and spread apart for the lowest-resource classes 1 and 0.

What we did

This is a position paper with a focused empirical study of five frontier models (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Flash-Lite, Llama 4 Maverick and Qwen 3.5 397B). We elicit declared language support with 20 English prompt variants and verify it with GlotLID. We test behavior on short stories, teacher conversations, translation (chrF++) and code generation (mHumanEval) over a 203-language FLORES/NLLB set, and on long-form writing in 141 PolyWrite languages. The agent experiment is an illustrative proof of concept with seven role-based agents in a DyLAN-style setup on MMLU-Redux.

Abstract

This paper argues that contemporary multilingual NLP has converged on a fragile and misleading paradigm of incidental multilingualism. Today's LLMs appear multilingual largely because they are trained on massive, uneven web corpora, not because multilingual or multicultural competence has been treated as a core design objective. We contend that this paradigm systematically produces unequal, brittle, and opaque behavior across languages, with severe consequences in real-world and agentic deployments where models must reason, plan, and act across multiple linguistic contexts. We report a focused empirical study of two practical questions: which languages models self-report as supported and which languages they actually respond in across multilingual prompts. We additionally demonstrate how even a simple language-change attack can surface these failures and expose hidden assumptions about language in LLM-based systems. To address this, we call for a shift toward multilingualism by design: a research agenda that treats equitable multilingual performance, cultural grounding, and cross-lingual behavioral understanding as first-class goals in all aspects of the model pipeline.

Citation

@inproceedings{mukherjee2026lost,
  title = {Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMs},
  author = {Mukherjee, Anjishnu and Meng, Chutong and Anastasopoulos, Antonios},
  year = {2026},
  booktitle = {Third Conference on Language Modeling},
  url = {https://openreview.net/forum?id=Drhe92Fj3J}
}