EMNLP 2023 (Main)
Global Voices, Local Biases: Socio-cultural Prejudices across Languages
George Mason University
* Equal contribution
Do multilingual language models show the same social biases in every language, and can WEAT measure bias in languages other than English?
Social biases in language models change with the language. WEATHub extends the Word Embedding Association Test (WEAT) to 24 languages with word lists checked by native speakers, and the measured effect sizes differ in strength and even in direction from one language to the next.

Key findings
- Bias differs across languages. With DistilmBERT, the six WEAT tests give significant positive effects in some languages and significant reversed effects in others, and many languages show nothing significant. Chinese and Telugu are often positive and significant, while Urdu, Tagalog and Russian mostly show small effects. (Figure 3, Section 4.1)
- Human translations find more bias. Word lists translated by native speakers gave the larger effect size in 52% of cases with BERT embeddings averaged over all layers, against 33% for machine translation. For Korean, human translation gave the larger absolute effect size in all six WEAT categories. (Figure 2, Section 3.3)
- Monolingual models can reverse the result. For Thai on WEAT 1 (flowers and insects), multilingual DistilmBERT gives d = 1.062 (p = 0.000) but a monolingual Thai BERT gives d = -0.629 (p = 0.996), a significant reversed association that fits the Thai annotator's view of insects as food. For Turkish, WEAT 1 is significant only in the monolingual model. (Section 4.2; Tables 5 and 7 (Appendix F))
- WEATHub adds five new bias dimensions. The new tests cover toxicity, ableism, sexuality, education and immigration. Immigration terms are significantly tied to disrespectful words in Spanish, Italian, Kurmanji Kurdish, Turkish, Korean, Chinese, Bengali, Marathi, Punjabi and Urdu, while English and Hindi show the reverse. (Figure 5, Section 4.3)



What we did
WEATHub covers 24 languages (Arabic, Bengali, Sorani Kurdish, Danish, German, Greek, Spanish, Persian, French, Hindi, Italian, Japanese, Korean, Kurmanji Kurdish, Marathi, Punjabi, Russian, Telugu, Thai, Tagalog, Turkish, Urdu, Vietnamese, Mandarin Chinese). It has six original WEAT tests (1, 2, 6, 7, 8, 9), five new dimensions and India-specific tests for caste, religion, occupation and gender. Native speakers corrected the translations and added five language-specific words per category. We compare 10 ways of extracting word embeddings from DistilmBERT, XLM-RoBERTa, monolingual BERT models, IndicBERT and FastText.
Talk
Watch the talk on YouTube
Abstract
Human biases are ubiquitous but not uniform: disparities exist across linguistic, cultural, and societal borders. As large amounts of recent literature suggest, language models (LMs) trained on human data can reflect and often amplify the effects of these social biases. However, the vast majority of existing studies on bias are heavily skewed towards Western and European languages. In this work, we scale the Word Embedding Association Test (WEAT) to 24 languages, enabling broader studies and yielding interesting findings about LM bias. We additionally enhance this data with culturally relevant information for each language, capturing local contexts on a global scale. Further, to encompass more widely prevalent societal biases, we examine new bias dimensions across toxicity, ableism, and more. Moreover, we delve deeper into the Indian linguistic landscape, conducting a comprehensive regional bias analysis across six prevalent Indian languages. Finally, we highlight the significance of these social biases and the new dimensions through an extensive comparison of embedding methods, reinforcing the need to address them in pursuit of more equitable language models.
Citation
@inproceedings{mukherjee-etal-2023-global,
title = "{G}lobal {V}oices, Local Biases: Socio-Cultural Prejudices across Languages",
author = "Mukherjee, Anjishnu and
Raj, Chahat and
Zhu, Ziwei and
Anastasopoulos, Antonios",
editor = "Bouamor, Houda and
Pino, Juan and
Bali, Kalika",
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2023",
address = "Singapore",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.emnlp-main.981/",
doi = "10.18653/v1/2023.emnlp-main.981",
pages = "15828--15845"
}