ISCA Archive Interspeech 2022
ISCA Archive Interspeech 2022

Complex Paralinguistic Analysis of Speech: Predicting Gender, Emotions and Deception in a Hierarchical Framework

Alena Velichko, Maxim Markitantov, Heysem Kaya, Alexey Karpov

In this paper, we present a hierarchical framework for complex paralinguistic analysis of speech including gender, emotions and deception recognition. The main idea of the framework is built upon the research on interrelation between various paralinguistic phenomena. It uses gender information to predict emotional states, and the outcome of the emotion recognition to predict the truthfulness of the speech. We use multiple datasets (aGender, Ruslana, EmoDB and DSD) to perform within-corpus and cross-corpus experiments using various performance measures. The experimental results reveal that gender-specific models improve the effectiveness of automatic speech emotion recognition in terms of Unweighted Average Recall up to an absolute 5.7%, and the integration of emotion predictions improves the F-score of automatic deception detection compared to our baseline by an absolute 4.7%. The obtained cross-validation results of 88.4 +/- 1.5% for deception detection beat the existing state-of-the-art by an absolute 2.8%.


doi: 10.21437/Interspeech.2022-11294

Cite as: Velichko, A., Markitantov, M., Kaya, H., Karpov, A. (2022) Complex Paralinguistic Analysis of Speech: Predicting Gender, Emotions and Deception in a Hierarchical Framework. Proc. Interspeech 2022, 4735-4739, doi: 10.21437/Interspeech.2022-11294

@inproceedings{velichko22_interspeech,
  author={Alena Velichko and Maxim Markitantov and Heysem Kaya and Alexey Karpov},
  title={{Complex Paralinguistic Analysis of Speech: Predicting Gender, Emotions and Deception in a Hierarchical Framework}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={4735--4739},
  doi={10.21437/Interspeech.2022-11294}
}