Diffusing DeBias:
Synthetic Bias Amplification for Model Debiasing

Massimiliano Ciranni^1*, Vito Paolo Pastore^1,2*, Roberto Di Via^1*, Enzo Tartaglione³, Francesca Odone¹, Vittorio Murino^2,4

* equal contributions

¹MaLGa-DIBRIS, University of Genoa, Italy ²AI for Good (AIGO), Istituto Italiano di Tecnologia, Genova, Italy ³Telecom-Paris / École Polytechnique, France ⁴Dept. of Computer Science, University of Verona, Italy

Paper (PDF) Poster (PDF) Code NeurIPS 2025 — Workshop / Conference

We introduce Diffusing DeBias (DDB), a plug-in debiasing framework that leverages conditional diffusion models to generate synthetic bias-aligned images and train a robust Bias Amplifier for unsupervised model debiasing.

Figure 1: Comparison of bias amplifier training approaches. (a) Traditional methods that use real biased datasets lead to the memorization of bias-conflicting samples, resulting in suboptimal auxiliary models. (b) Our proposed approach leverages conditional diffusion models to learn class-specific biases and amplify them into synthetic images. Such generations can replace the original training set for learning an effective Bias Amplifier, eliminating memorization effects by avoiding exposure to real data.

Abstract

The effectiveness of deep learning models is often limited by spurious correlations present in training data. These biases lead models to rely on shortcut features rather than learning robust, generalizable representations. Existing debiasing methods typically require either explicit bias labels or sophisticated techniques to identify bias-conflicting samples, which can be unreliable or expensive.

We propose Diffusing DeBias (DDB), a novel framework that exploits conditional diffusion probabilistic models (CDPM) to generate synthetic bias-aligned images. These synthetic samples are used to train a robust Bias Amplifier (BA) that deliberately learns to predict bias attributes without memorizing the scarce bias-conflicting examples present in real data. The trained BA can then be seamlessly integrated into both two-step and end-to-end unsupervised debiasing recipes.

DDB achieves state-of-the-art results on multiple popular biased benchmarks (Waterbirds, BFFHQ, BAR, ImageNet-9/A, UrbanCars) while maintaining competitive performance on unbiased test data, demonstrating its effectiveness and practical applicability.

Method Overview

Figure 2: Schematic representation of our DDB framework. The debiasing process consists of two key steps: (A) Diffusing the Bias uses a conditional diffusion model with classifier-free guidance to generate synthetic images that preserve training dataset biases, and (B) employs a Bias Amplifier firstly trained on such synthetic data, and subsequently used during inference to extract supervisory bias signals from real images. These signals are used to guide the training process of a target debiased model by designing two debiasing recipes (i.e., 2-step and end-to-end methods).

Key Innovation: Synthetic Bias Amplification

The core idea of DDB is to avoid training the Bias Amplifier on real data. Traditional approaches train bias models on the real training set, which leads to memorization of bias-conflicting samples—the very samples we want to upweight during debiasing. Instead, DDB:

Trains a class-conditional diffusion model on the biased training dataset
Generates synthetic bias-aligned images by leveraging classifier-free guidance (CFG) with high guidance scale, which amplifies the dominant spurious correlations
Trains the Bias Amplifier exclusively on synthetic data, ensuring it learns to recognize bias patterns without memorizing real bias-conflicting examples
Integrates the BA into debiasing recipes (Recipe I: two-step with GroupDRO; Recipe II: end-to-end with LfF-style reweighting)

Why This Works

By training on purely synthetic bias-aligned data, the Bias Amplifier learns a clean decision boundary that separates bias-aligned from bias-conflicting samples based on bias attributes alone. This avoids the common pitfall where bias models trained on real data learn to identify individual bias-conflicting samples, leading to poor generalization when used for debiasing.

Synthetic Bias-Aligned Samples

Figure 3: Examples of synthetic bias-aligned images generated by our conditional diffusion model with high CFG scale. These images strongly exhibit the spurious correlations present in the training data (e.g., waterbirds on water backgrounds, landbirds on land backgrounds).

The synthetic images capture and amplify the dataset's bias patterns. For instance, in Waterbirds, the diffusion model learns that "waterbirds" are strongly associated with "water backgrounds" and "landbirds" with "land backgrounds." By using high classifier-free guidance scale during sampling, we generate images that exclusively follow these spurious correlations.

Integration with Debiasing Recipes

Recipe I: Two-Step Debiasing

Recipe I: GroupDRO-based two-step debiasing

Figure 4: Recipe I uses the BA to create pseudo-groups, then trains a target model with GroupDRO for robust worst-group performance.

Recipe II: End-to-End Debiasing

Recipe II: End-to-end LfF-style debiasing

Figure 5: Recipe II jointly trains the BA and target model end-to-end with sample reweighting based on BA predictions.

Recipe I (Two-Step): First train the BA on synthetic data, use it to predict bias attributes for real samples and create pseudo-groups (class × bias), then train a target classifier using GroupDRO to optimize worst-group accuracy.

Recipe II (End-to-End): Jointly train the BA and target model, where the BA provides per-sample weights that upweight bias-conflicting samples during target model training, following an LfF-style approach.

Experimental Results

We evaluate DDB on five popular biased benchmarks with different types of spurious correlations:

Waterbirds: Birds on correlated backgrounds (water vs. land)
BFFHQ: Face gender classification with age bias
BAR: Actions correlated with specific backgrounds
ImageNet-9/A: Natural image classification with background bias
UrbanCars: Vehicle co-occurrences in urban scenes

Main Results: Recipe I (Two-Step)

Table 1: Comparison with state-of-the-art unsupervised debiasing methods using Recipe I (GroupDRO-based two-step approach). DDB achieves the best worst-group accuracy across all benchmarks.

Main Results: Recipe II (End-to-End)

Table 2: Results using Recipe II (end-to-end LfF-style training). DDB consistently outperforms baseline methods on bias-conflicting samples while maintaining strong average performance.

Key Findings

State-of-the-art performance: DDB achieves the best worst-group accuracy compared to other unsupervised debiasing methods across all benchmarks
Synthetic data is crucial: Training the BA on synthetic data significantly outperforms training on real data, confirming our hypothesis about avoiding memorization
Efficient synthesis: Only 1,000-2,000 synthetic images per class are needed for effective debiasing
CFG scale matters: Higher CFG guidance scales (7-15) produce more strongly bias-aligned samples, leading to better BA performance
No degradation on unbiased data: DDB maintains competitive performance on unbiased test sets, showing it doesn't sacrifice general accuracy

Key Contributions

Novel synthetic bias amplification approach: First method to use conditional diffusion models to generate bias-aligned synthetic data for training bias models
Memorization-free Bias Amplifier: By training exclusively on synthetic data, we avoid memorizing bias-conflicting real samples, leading to better generalization
Plug-and-play framework: DDB can be integrated into various debiasing recipes (both two-step and end-to-end approaches)
State-of-the-art results: Superior performance on five challenging benchmarks while maintaining competitiveness on unbiased data
Comprehensive analysis: Extensive ablations studying CFG scale, synthetic data quantity, and comparison with real-data training

Citation

BibTeX

@inproceedings{ciranni2025diffusing,
  title={Diffusing DeBias: Synthetic Bias Amplification for Model Debiasing},
  author={Ciranni, Massimiliano and Pastore, Vito Paolo and Di Via, Roberto and 
          Tartaglione, Enzo and Odone, Francesca and Murino, Vittorio},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}

APA Citation

Ciranni, M., Pastore, V. P., Di Via, R., Tartaglione, E., Odone, F., & Murino, V. (2025). Diffusing DeBias: Synthetic Bias Amplification for Model Debiasing. In Advances in Neural Information Processing Systems (NeurIPS 2025).