WACV 2025Oral
Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models
George Mason University
How well do large multimodal models recognize the culture shown in an image, and can the visual cues they associate with each country be used to adapt an image to another culture?
LLaVA and GPT-4V identify the region of culturally themed DALL-E 3 images more accurately than human annotators (78.05% and 56.31% against 47.63%), but the same models score between 19% and 42% on the Dollar Street and MaRVL photographs. We release DalleStreet, 9,935 generated images from 67 countries, and CultureAdapt, a pipeline that edits an image toward a target culture.

Key findings
- Models beat humans on generated images. On DalleStreet, LLaVA predicts the geographic subregion of an image with 78.05% accuracy and GPT-4V with 56.31%. Fourteen human annotators reach 47.63% at the subregion level and 22.16% at the country level on a subset. (Table 1 and Table 2)
- The advantage shrinks on real photographs. On Dollar Street photographs, GPT-4V scores 36.28% and LLaVA 36.83%. On MaRVL, GPT-4V scores 41.59% and LLaVA 19.14%, which is about as good as random guessing. (Table 2, Section 3.2)
- GPT-4V extracts over 18,000 cultural artifacts. Running GPT-4V extraction over DalleStreet gives more than 18,000 unique cultural artifacts, and the generated images also tie Greece to blue and the Netherlands to red. In a human check of 100 images, over 90% of the artifacts are present in the image and 60 to 70% are judged culturally relevant. (Section 4, Figures 5 to 7)
- CultureAdapt moves images toward the target country. Across 20 country pairs, 54% of CultureAdapt edits move the image toward the target country and away from the source country by CLIPScore, against 50% for the cap-edit baseline (McNemar p = 7.43e-5). Structural similarity to the source image stays close, at 0.94 against 0.97. (Section 5.2, Figure 10)



What we did
We study about 20,000 images from three datasets: DalleStreet (9,935 DALL-E 3 images, 67 countries, 19 geographic regions, 10 concept classes such as home, kitchen, plate of food and wardrobe), Dollar Street (4,137 photographs from 63 countries) and MaRVL (4,914 images). LLaVA-NeXT and GPT-4V predict the United Nations geoscheme subregion of each image in a zero-shot setting, and 14 human annotators give a baseline. CultureAdapt combines GPT-4V object detection, Grounding DINO and Stable Diffusion 2 inpainting.
Talk
Watch the talk on YouTube
Abstract
We present a comprehensive three-phase study to examine (1) the cultural understanding of Large Multimodal Models (LMMs) by introducing DalleStreet, a large-scale dataset generated by DALL-E 3 and validated by humans, containing 9,935 images of 67 countries and 10 concept classes; (2) the underlying implicit and potentially stereotypical cultural associations with a cultural artifact extraction task; and (3) an approach to adapt cultural representation in an image based on extracted associations using a modular pipeline, CultureAdapt. We find disparities in cultural understanding at geographic sub-region levels with both open-source (LLaVA) and closed-source (GPT-4V) models on DalleStreet and other existing benchmarks, which we try to understand using over 18,000 artifacts that we identify in association to different countries. Our findings reveal a nuanced picture of the cultural competence of LMMs, highlighting the need to develop culture-aware systems. Dataset and code are available at https://github.com/iamshnoo/crossroads
Citation
@InProceedings{Mukherjee_2025_WACV,
author = {Mukherjee, Anjishnu and Zhu, Ziwei and Anastasopoulos, Antonios},
title = {Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models},
booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
month = {February},
year = {2025},
pages = {1755-1764}
}