EMNLP 2023 (Main)

Global Voices, Local Biases: Socio-cultural Prejudices across Languages

Anjishnu Mukherjee*, Chahat Raj*, Ziwei Zhu, Antonios Anastasopoulos

George Mason University

* Equal contribution

Do multilingual language models show the same social biases in every language, and can WEAT measure bias in languages other than English?

Social biases in language models change with the language. WEATHub extends the Word Embedding Association Test (WEAT) to 24 languages with word lists checked by native speakers, and the measured effect sizes differ in strength and even in direction from one language to the next.

Six bar charts, one per WEAT test (flowers vs insects, instruments vs weapons, names with career vs family, math vs art, science vs art, mental vs physical disease), each with one bar per language from Arabic to Chinese; only some bars are colored as significant and several point below zero.
WEAT effect sizes for 24 languages in DistilmBERT, where colored bars are significant at the 95% level and bars below zero are reversed associations.

Key findings

  • Bias differs across languages. With DistilmBERT, the six WEAT tests give significant positive effects in some languages and significant reversed effects in others, and many languages show nothing significant. Chinese and Telugu are often positive and significant, while Urdu, Tagalog and Russian mostly show small effects. (Figure 3, Section 4.1)
  • Human translations find more bias. Word lists translated by native speakers gave the larger effect size in 52% of cases with BERT embeddings averaged over all layers, against 33% for machine translation. For Korean, human translation gave the larger absolute effect size in all six WEAT categories. (Figure 2, Section 3.3)
  • Monolingual models can reverse the result. For Thai on WEAT 1 (flowers and insects), multilingual DistilmBERT gives d = 1.062 (p = 0.000) but a monolingual Thai BERT gives d = -0.629 (p = 0.996), a significant reversed association that fits the Thai annotator's view of insects as food. For Turkish, WEAT 1 is significant only in the monolingual model. (Section 4.2; Tables 5 and 7 (Appendix F))
  • WEATHub adds five new bias dimensions. The new tests cover toxicity, ableism, sexuality, education and immigration. Immigration terms are significantly tied to disrespectful words in Spanish, Italian, Kurmanji Kurdish, Turkish, Korean, Chinese, Bengali, Marathi, Punjabi and Urdu, while English and Hindi show the reverse. (Figure 5, Section 4.3)
Heatmap with seven rows (sexuality-valence, sexuality-perception, ableism-valence, ableism-gender, immigration, education, toxicity) and 25 language columns including English, with values between about -1.4 and 1.6 and significant values in bold.
Effect sizes for the new bias dimensions (toxicity, education, immigration, ableism, sexuality) in each language, with significant cells in bold.
Six radar charts for Bengali, Hindi, Marathi, Punjabi, Telugu and Urdu, each with axes for WEAT 1, 2, 6, 7, 8 and 9 and three outlines for IndicBERT, distilBERT-multilingual and monolingual BERT.
For six Indian languages, monolingual BERT models (blue) often give larger effect sizes than IndicBERT (red) or multilingual DistilBERT (green).
Left: paired bars for machine and human translation over six WEAT categories in Korean. Right: two pie charts; FastText shows human translation better 43%, machine translation better 36%, similar 21%; all hidden layers shows 52%, 33% and 14%.
Human translations give larger effect sizes than machine translations, shown per category for Korean on the left and as a share of cases across languages on the right.

What we did

WEATHub covers 24 languages (Arabic, Bengali, Sorani Kurdish, Danish, German, Greek, Spanish, Persian, French, Hindi, Italian, Japanese, Korean, Kurmanji Kurdish, Marathi, Punjabi, Russian, Telugu, Thai, Tagalog, Turkish, Urdu, Vietnamese, Mandarin Chinese). It has six original WEAT tests (1, 2, 6, 7, 8, 9), five new dimensions and India-specific tests for caste, religion, occupation and gender. Native speakers corrected the translations and added five language-specific words per category. We compare 10 ways of extracting word embeddings from DistilmBERT, XLM-RoBERTa, monolingual BERT models, IndicBERT and FastText.

Talk

Video thumbnail for the WEATHub talkWatch the talk on YouTube

Abstract

Human biases are ubiquitous but not uniform: disparities exist across linguistic, cultural, and societal borders. As large amounts of recent literature suggest, language models (LMs) trained on human data can reflect and often amplify the effects of these social biases. However, the vast majority of existing studies on bias are heavily skewed towards Western and European languages. In this work, we scale the Word Embedding Association Test (WEAT) to 24 languages, enabling broader studies and yielding interesting findings about LM bias. We additionally enhance this data with culturally relevant information for each language, capturing local contexts on a global scale. Further, to encompass more widely prevalent societal biases, we examine new bias dimensions across toxicity, ableism, and more. Moreover, we delve deeper into the Indian linguistic landscape, conducting a comprehensive regional bias analysis across six prevalent Indian languages. Finally, we highlight the significance of these social biases and the new dimensions through an extensive comparison of embedding methods, reinforcing the need to address them in pursuit of more equitable language models.

Citation

@inproceedings{mukherjee-etal-2023-global,
    title = "{G}lobal {V}oices, Local Biases: Socio-Cultural Prejudices across Languages",
    author = "Mukherjee, Anjishnu  and
      Raj, Chahat  and
      Zhu, Ziwei  and
      Anastasopoulos, Antonios",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.981/",
    doi = "10.18653/v1/2023.emnlp-main.981",
    pages = "15828--15845"
}