AIES 2024Oral
Breaking Bias, Building Bridges: Evaluation and Mitigation of Social Biases in LLMs via Contact Hypothesis
George Mason University · University of Washington
Does the Contact Hypothesis from social psychology hold for LLMs, and can simulated contact between groups reduce social bias in LLMs?
LLM answers follow the Contact Hypothesis from social psychology. A sentence of positive contact with a group lowers biased answers and negative contact raises them, and one epoch of instruction tuning LLaMA 2 on such prompts (Social Contact Debiasing) reduces biased responses by up to 40%.

Key findings
- LLMs follow the Contact Hypothesis. On yes/no (certainty) prompts, LLaMA 2 gives a biased answer 27.47% of the time with no contact, 18.79% after positive contact and 37.95% after negative contact. Tulu and NousHermes move in the same two directions. (Table 3)
- One epoch of tuning removes most of the bias. We instruction-tuned LLaMA 2 on about 10k prompts with unbiased answers and tested on held-out prompts (Setting 1). Biased responses fall from 41.64% to 2.40% with no contact, from 37.99% to 2.42% with positive contact and from 45.69% to 2.05% with negative contact. (Figure 4)
- The gains transfer to other question types and to BBQ. Tuning only on likelihood prompts cuts bias on certainty prompts from 27.51% to 1.74%, and tuning only on certainty prompts cuts bias on likelihood prompts from 50% to 5.41%. On the BBQ benchmark, which uses a different prompt format, all eight tuned models score above the untuned model overall. (Table 4; Table 6 (Table 5 in the arXiv version))
- Answer quality is preserved. On 100 WikiMovies questions, BERTScore F1 is 0.7965 before debiasing and 0.7963 after. Two annotators (Cohen's kappa 0.76) judged 35 of 100 answers fluent after debiasing against 39 before, and 50 relevant against 31 before. (Section 'RQ3: Bias Mitigation Results'; Table 5 (Table 6 in the arXiv version))



What we did
The dataset has 108,000 prompt sets built from 6 contact principles, 5 scenarios (education, workplace, community, sports, healthcare), about 600 HolisticBias descriptors over 13 demographic dimensions, 2 action types and 3 answer scales (certainty, likelihood, frequency). Each set has a no-contact, a positive-contact and a negative-contact prompt. We measure bias in LLaMA 2 Chat 13B, Tulu 13B (DPO) and NousHermes 13B, and instruction-tune LLaMA 2 in eight settings that hold out prompt scales, scenarios, principles, bias dimensions or a whole second dataset of 106,560 items. All prompts are in English.
Abstract
Large Language Models (LLMs) perpetuate social biases, reflecting prejudices in their training data and reinforcing societal stereotypes and inequalities. Our work explores the potential of the Contact Hypothesis, a concept from social psychology for debiasing LLMs. We simulate various forms of social contact through LLM prompting to measure their influence on the model's biases, mirroring how intergroup interactions can reduce prejudices in social contexts. We create a dataset of 108,000 prompts following a principled approach replicating social contact to measure biases in three LLMs (LLaMA 2, Tulu, and NousHermes) across 13 social bias dimensions. We propose a unique debiasing technique, Social Contact Debiasing (SCD), that instruction-tunes these models with unbiased responses to prompts. Our research demonstrates that LLM responses exhibit social biases when subject to contact probing, but more importantly, these biases can be significantly reduced by up to 40% in 1 epoch of instruction tuning LLaMA 2 following our SCD strategy. Our code and data are available at https://github.com/chahatraj/breakingbias.
Citation
@article{Raj_Mukherjee_Caliskan_Anastasopoulos_Zhu_2024,
title = {Breaking Bias, Building Bridges: Evaluation and Mitigation of Social Biases in LLMs via Contact Hypothesis},
author = {Raj, Chahat and Mukherjee, Anjishnu and Caliskan, Aylin and Anastasopoulos, Antonios and Zhu, Ziwei},
journal = {Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society},
volume = {7},
number = {1},
pages = {1180--1189},
year = {2024},
month = {Oct.},
DOI = {10.1609/aies.v7i1.31715},
url = {https://ojs.aaai.org/index.php/AIES/article/view/31715}
}