Seedlabs
Source research

Towards Improving Speech Emotion Recognition Using Synthetic Data Augmentation from Emotion Conversion

Karim M. Ibrahim, Antony Perzo, Simon Leglaive · 2024 · 15 citationsRead the paper

One of the main challenges in speech emotion recognition is the lack of large labelled datasets. The progress in speech synthesis allows us to generate reliable and realistic expressive speech. In this work, we propose using a state-of-the-art end-to-end speech emotion conversion model to generate new synthetic data for training speech emotion recognition models. We first evaluate the quality of the converted speech on new unseen datasets, which proves to be on par with the training data. Then, we study the effect of using the synthesized speech as data augmentation. We show that this approach improves the overall performance of emotion recognition models on two different datasets, IEMOCAP and RAVDESS, both in the cases of speaker dependent and independent emotion recognition using a fine-tuned wav2vec 2.0.

What the full text shows

Working prototype
  • Augmenting training data with synthetic speech improved SER accuracy over using original data alone for IEMOCAP (SD) and RAVDESS (SD). — IEMOCAP SD: 76.19% ± 1.95% vs 74.16% ± 2.00%; RAVDESS SD: 93.05% ± 2.12% vs 91.08% ± 2.93% (Fine-tuned wav2vec 2.0, Speaker Dependent (SD) setup)
  • Synthetic data augmentation provided a performance gain over original data for IEMOCAP (SI), though it performed slightly lower than traditional audio augmentation. — 66.06% ± 2.21% (Synthetic+Original) vs 63.97% ± 2.93% (Original) vs 65.32% ± 3.44% (Baseline) (Fine-tuned wav2vec 2.0, Speaker Independent (SI) setup)
  • Synthetic data augmentation performed similarly to traditional audio augmentation for RAVDESS (SI). — 81.29% ± 2.77% (Synthetic+Original) vs 82.29% ± 2.96% (Baseline) (Fine-tuned wav2vec 2.0, Speaker Independent (SI) setup)
  • The emotion conversion model generalized to unseen datasets with perceived quality on par with training data. — Comparable Mean-Opinion-Score (MOS) across 9 raters (Evaluated on IEMOCAP and RAVDESS datasets)
Setup
5-fold cross-validation on IEMOCAP and RAVDESS datasets; synthetic augmentation ratios tested at 25%, 50%, 75%, and 100% of original dataset size.
How
The authors used a textless end-to-end speech emotion conversion model (HuBERT for phonetic representation, Seq2Seq for unit translation, CNNs for duration/F0 prediction, and HiFi-GAN vocoder) to generate synthetic audio. This synthetic data was used to augment the training of a fine-tuned wav2vec 2.0 emotion recognition model.
Limitations
The emotion conversion model was trained only on the EmoV dataset. · Performance gains in speaker-independent setups were marginal or slightly lower than traditional augmentation for some datasets.

Summarized by Gemma 4 from the open-access full text · 8 Oct 2026. Read it