Cross-Language Perception of Audio-visual Attitudinal Expressions

Hansjörg Mixdorff, Angelika Hönemann, Albert Rilliard, Tan Lee, Matthew Ma


This paper presents results from a cross-language free labeling experiment employing short audio-visual utterances of Cantonese produced with varying attitudinal expressions. German perceivers were asked to specify a single word that best described these stimuli, some of which were presented in audio-only and video-only modality. The resulting terms were classified with respect to the emotional dimensions of valence, activation and dominance, as well as the linguistic dimension of assertion/interrogation. We compared the results with the outcomes from a similar experiment employing German stimuli. Most types of attitudes presented in Cantonese were rated less positive and portrayed with less activation than those presented by Germans. Video-supported stimuli yielded significantly higher activation and dominance levels than audio-only ones. The main dimensions separating expressions are assertive vs. interrogation, valence, and dominance. Illocutionary strength is associated with the perceived activation, and primarily linked to the visual channel, while linguistic information is primarily conveyed by acoustic cues, but of course only for the German stimuli.


 DOI: 10.21437/AVSP.2017-23

Cite as: Mixdorff, H., Hönemann, A., Rilliard, A., Lee, T., Ma, M. (2017) Cross-Language Perception of Audio-visual Attitudinal Expressions. Proc. The 14th International Conference on Auditory-Visual Speech Processing, 119-124, DOI: 10.21437/AVSP.2017-23.


@inproceedings{Mixdorff2017,
  author={Hansjörg Mixdorff and Angelika Hönemann and Albert Rilliard and Tan Lee and Matthew Ma},
  title={ Cross-Language Perception of Audio-visual Attitudinal Expressions},
  year=2017,
  booktitle={Proc. The 14th International Conference on Auditory-Visual Speech Processing},
  pages={119--124},
  doi={10.21437/AVSP.2017-23},
  url={http://dx.doi.org/10.21437/AVSP.2017-23}
}