WACV 2025Oral

Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models

Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos

George Mason University

How well do large multimodal models recognize the culture shown in an image, and can the visual cues they associate with each country be used to adapt an image to another culture?

LLaVA and GPT-4V identify the region of culturally themed DALL-E 3 images more accurately than human annotators (78.05% and 56.31% against 47.63%), but the same models score between 19% and 42% on the Dollar Street and MaRVL photographs. We release DalleStreet, 9,935 generated images from 67 countries, and CultureAdapt, a pipeline that edits an image toward a target culture.

Three-row diagram. Row one shows ten generated images of Greece with the question 'Which geographical region is this?'. Row two shows a wardrobe image and a model answer listing brown wardrobe, white tunic, wicker basket, brown sandals and Ancient Greek pottery. Row three shows the wardrobe image edited into an Indian version with colorful tunics.
The three tasks in the paper are guessing the region of an image, extracting the cultural artifacts in the image, and editing those artifacts to adapt the image to another country.

Key findings

  • Models beat humans on generated images. On DalleStreet, LLaVA predicts the geographic subregion of an image with 78.05% accuracy and GPT-4V with 56.31%. Fourteen human annotators reach 47.63% at the subregion level and 22.16% at the country level on a subset. (Table 1 and Table 2)
  • The advantage shrinks on real photographs. On Dollar Street photographs, GPT-4V scores 36.28% and LLaVA 36.83%. On MaRVL, GPT-4V scores 41.59% and LLaVA 19.14%, which is about as good as random guessing. (Table 2, Section 3.2)
  • GPT-4V extracts over 18,000 cultural artifacts. Running GPT-4V extraction over DalleStreet gives more than 18,000 unique cultural artifacts, and the generated images also tie Greece to blue and the Netherlands to red. In a human check of 100 images, over 90% of the artifacts are present in the image and 60 to 70% are judged culturally relevant. (Section 4, Figures 5 to 7)
  • CultureAdapt moves images toward the target country. Across 20 country pairs, 54% of CultureAdapt edits move the image toward the target country and away from the source country by CLIPScore, against 50% for the cap-edit baseline (McNemar p = 7.43e-5). Structural similarity to the source image stays close, at 0.94 against 0.97. (Section 5.2, Figure 10)
Left: pipeline diagram where a Vietnamese food image passes through GPT-4V object detection, Grounding DINO bounding boxes and a Stable Diffusion inpainting prompt to become a Japanese version, scored with CLIPScore. Right: a grid comparing CultureAdapt, DALL-E 3, e2e-instruct and cap-edit on a Greek doorway adapted to China, India and the USA.
CultureAdapt detects objects with GPT-4V, grounds them with Grounding DINO and inpaints them with Stable Diffusion for the target country, while keeping the layout of the source image.
Five generated images labeled Kazakhstan, South Korea, India, Ukraine and Mexico with yellow outlines around detected objects. Captions list the artifacts: yurt, horse, mountains, wooden carts; sake, floral teapot; saree, tea, dupatta, cow, spices; sunflowers, slavic thatched roof village house and attire; jalapeno, beans, avocado, cauldron.
The artifacts most strongly associated with five countries in DalleStreet, such as yurt and horse for Kazakhstan and sunflowers and thatched roofs for Ukraine.
Confusion matrix of true versus predicted United Nations subregions for GPT-4V on DalleStreet, with 19 subregions plus Invalid and ResponsibleAI labels. The diagonal cells are darkest, Western Asia has 600 correct and no errors, and the last column holds policy-violation responses.
GPT-4V subregion predictions on DalleStreet show a strong diagonal, with 100% accuracy for Western Asia and many refusals in the ResponsibleAI column.

What we did

We study about 20,000 images from three datasets: DalleStreet (9,935 DALL-E 3 images, 67 countries, 19 geographic regions, 10 concept classes such as home, kitchen, plate of food and wardrobe), Dollar Street (4,137 photographs from 63 countries) and MaRVL (4,914 images). LLaVA-NeXT and GPT-4V predict the United Nations geoscheme subregion of each image in a zero-shot setting, and 14 human annotators give a baseline. CultureAdapt combines GPT-4V object detection, Grounding DINO and Stable Diffusion 2 inpainting.

Talk

Video thumbnail for the Crossroads of Continents talkWatch the talk on YouTube

Abstract

We present a comprehensive three-phase study to examine (1) the cultural understanding of Large Multimodal Models (LMMs) by introducing DalleStreet, a large-scale dataset generated by DALL-E 3 and validated by humans, containing 9,935 images of 67 countries and 10 concept classes; (2) the underlying implicit and potentially stereotypical cultural associations with a cultural artifact extraction task; and (3) an approach to adapt cultural representation in an image based on extracted associations using a modular pipeline, CultureAdapt. We find disparities in cultural understanding at geographic sub-region levels with both open-source (LLaVA) and closed-source (GPT-4V) models on DalleStreet and other existing benchmarks, which we try to understand using over 18,000 artifacts that we identify in association to different countries. Our findings reveal a nuanced picture of the cultural competence of LMMs, highlighting the need to develop culture-aware systems. Dataset and code are available at https://github.com/iamshnoo/crossroads

Citation

@InProceedings{Mukherjee_2025_WACV,
    author    = {Mukherjee, Anjishnu and Zhu, Ziwei and Anastasopoulos, Antonios},
    title     = {Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models},
    booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
    month     = {February},
    year      = {2025},
    pages     = {1755-1764}
}