EMNLP 2026 (Main)Outstanding Paper Award
Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations
George Mason University
If a language model learns a second language alongside English, does its representation of English concepts change?
Adding a second language to pretraining changes how a model represents English concepts, even after we align the English word embeddings of the two models. In all 32 controlled comparisons, bilingual and English-only models disagreed on held-out English concepts more than English-only models trained with different random seeds.
Key findings
- The effect survives four training controls. We matched either English exposure or total training steps, with shared or separate English documents. The median contextual difference stayed above English-only seed variation in all four comparisons, between 0.16 and 0.53, so less English, more training or reused English text cannot explain the difference. (Table 1 and Figure 3)
- Aligned embeddings hide contextual differences. Across sixteen shared-document comparisons, the seed-corrected disagreement along the 50 semantic contrasts is 0.014 for token embeddings and 0.386 for contextual states under an orthogonal alignment. A more flexible affine alignment still leaves the contextual value at 0.580. (Table 2, Section 4.4)
- The difference peaks in the middle layers. When we fit a separate alignment at each of the 12 transformer layers, the average difference is small at the input, rises through the middle layers and falls in the final layers. The contextual difference is already present at the first saved checkpoint, 500 steps into training, and remains through 3,000 steps. (Figure 6, Section 4.3; Figure 7, Section 4.4)
- The direction depends on the second language. The same English concept can move toward opposite ends of a contrast depending on the language learned with English. The paper shows twelve such reversals across the eight added languages, including 'wedding dress' on the white versus red contrast. (Figure 2)
What we did
We trained 40 decoder-only models with about 310M parameters each, all with the same 12-layer architecture, tokenizer, training recipe and data source (BabyBabelLM). The bilingual models pair English with Chinese, French, Persian, Dutch, Ukrainian, Bulgarian, Indonesian or German, and English-only runs with different seeds give the reference for ordinary training variation. We align model pairs with orthogonal Procrustes on 3,000 common English words and compare them on 1,000 held-out English concepts along 50 fixed semantic contrasts in nine themes, with 100 physical terms as negative controls.
Abstract
A concept can carry different associations across languages, while modern language models learn English alongside many other languages during pretraining. Yet comparisons among existing models cannot easily isolate how any one language changes the way these models represent English concepts because their training corpora, compute, architectures, and random seeds all differ. We study this question through a controlled experiment with 40 matched 310M-parameter decoder-only models that share an architecture, tokenizer, training recipe, and English data source. Each bilingual condition adds one of eight languages, while four experimental comparisons separately account for English exposure, total training, and English-document overlap. We align each model pair using 3,000 common English words, then measure where 1,000 held-out English concepts fall along 50 fixed semantic contrasts, such as red versus white. Across 32 experimental comparisons, English concept positions differ more between bilingual and English-only conditions than between English-only runs with different random seeds. These differences are larger in contextual states than in token embeddings and peak in middle layers. The language learned alongside English can therefore change how a model represents English concepts even when its English input representations are explicitly aligned.
Citation
@inproceedings{mukherjee-etal-2026-double,
title = "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations",
author = "Mukherjee, Anjishnu and
Zhu, Ziwei and
Anastasopoulos, Antonios",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Junyi Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.2062/",
doi = "10.18653/2026.emnlp-main.2062",
pages = "46953--46977",
ISBN = "979-8-89176-459-0"
}