NAACL 2024 (Main)
Global Gallery: The Fine Art of Painting Culture Portraits through Multilingual Instruction Tuning
George Mason University · University of Washington
Does instruction tuning an LLM in a language improve its knowledge of the culture where that language is spoken?
Instruction tuning LLaMA 2 in the main language of a culture does not consistently improve what the model knows about that culture. At 70B, English tuning with English prompts scores higher on GeoMLaMA than native-language tuning with native prompts for China (0.46 against 0.39), India and Iran, while bilingual pretraining and curated tuning data help more.

Key findings
- Native-language tuning gives no consistent gain. On GeoMLaMA with LLaMA 2 70B, English Alpaca tuning with English prompts beats native-language tuning with native prompts for China (0.46 vs 0.39), India (0.45 vs 0.33), Iran (0.28 vs 0.14) and Greece (0.38 vs 0.34). Kenya is the exception, with 0.28 for English against 0.34 for Swahili. (Table 3, Section 4.1)
- Bilingual pretraining and curated data help. The Chinese-English Yi 34B model with Chinese tuning and Chinese prompts scores 0.54 on China questions, against 0.39 for LLaMA 2 70B in the same setting. For Kenya, the Swahili model Uliza with a curated bilingual LoRA and Swahili prompts scores 0.41, against 0.31 with machine-translated Swahili Alpaca. (Table 4, Sections 4.2 and 4.3)
- The model defaults to China and Iran. When no answer option comes from the correct culture, LLaMA 2 70B picks options from China and Iran most often. With three correct options and one from another culture, the model still picks the wrong one nearly 70% of the time for male names in the US. (Figure 3, Section 4.4)
- The model knows cultures only coarsely. Given four options from the right culture where only one fits the category in the question, LLaMA 2 70B picks a wrong-category option 60% (USA) to 76% (India) of the time overall. The 13B model rated 'Goji Berry tea' a more likely name for a girl than 'Chen'. (Table 5, Section 4.4)


What we did
We machine-translate the 52k Alpaca instructions with NLLB into Hindi, Mandarin Chinese, Persian, Swahili and Greek, and train LoRA adapters with 4-bit QLoRA on LLaMA 2 at 7B, 13B and 70B, plus the bilingual base models Yi (6B, 34B) and Uliza (7B). We evaluate on GeoMLaMA in question-answer form, 900 questions with 150 per language across the USA, China, India, Iran, Kenya and Greece, and on an adapted English CAMeL set with nine categories such as food, names, clothing and literature. Answers are scored by option log-probability, without free-text generation.
Talk
Watch the talk on YouTube
Abstract
Exploring the intersection of language and culture in Large Language Models (LLMs), this study critically examines their capability to encapsulate cultural nuances across diverse linguistic landscapes. Central to our investigation are three research questions: the efficacy of language-specific instruction tuning, the impact of pretraining on dominant language data, and the identification of optimal approaches to elicit accurate cultural knowledge from LLMs. Utilizing the GeoMLaMA benchmark for multilingual commonsense knowledge and an adapted CAMeL dataset (English-only) for evaluation of nuanced cultural aspects, our experiments span six different languages and cultural contexts, revealing the extent of LLMs’ cultural awareness. Our findings highlight a nuanced landscape: while language-specific tuning and bilingual pretraining enhance cultural understanding in certain contexts, they also uncover inconsistencies and biases, particularly in non-Western cultures. This work expands our understanding of LLMs’ cultural competence and emphasizes the importance of integrating diverse cultural perspectives in their development, aiming for a more globally representative and equitable approach in language modeling.
Citation
@inproceedings{mukherjee-etal-2024-global,
title = "Global Gallery: The Fine Art of Painting Culture Portraits through Multilingual Instruction Tuning",
author = "Mukherjee, Anjishnu and
Caliskan, Aylin and
Zhu, Ziwei and
Anastasopoulos, Antonios",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Proceedings of the 2024 Conference of the {N}orth {A}merican Chapter of the {A}ssociation for {C}omputational {L}inguistics: Human Language Technologies (Volume 1: Long Papers)",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.naacl-long.355/",
doi = "10.18653/v1/2024.naacl-long.355",
pages = "6398--6415"
}