Referential Gaze Makes a Difference in Spoken Language Comprehension: Human Speaker vs. Virtual Agent Listener Gaze

Eva Maria Nunnemann, Kirsten Bergmann, Helene Kreysa, Pia Knoeferle


An interlocutor's referential gaze is of great importance in face-to-face communication as it facilitates spoken language comprehension. People are also able to exploit virtual agent gaze in interactions. Our study addressed effects of human speaker gaze vs. virtual agent listener gaze on reaction times, accuracy and eye movements. Participants saw videos in which a static scene depicting three characters was visible on a screen. We manipulated: (1) whether the human speaker - uttering the sentence - was visible, (2) whether the agent listener was present and (3) whether the template following each video matched the scene. Eye movements were recorded as participants listened to German SVO sentences describing an inter- action between two of these three characters. After each trial a template schematically depicting three characters and the interaction appeared on screen. Participants verified a match be- tween sentence and template. Participants solved the matching task very well across all conditions. They responded faster to matches than mismatches between sentence and template. Participants were slower when the agent was present. Eye movement results suggest that while hearing the second noun phrase participants tended to look at its referent to a greater extent when the speaker was present compared to the other conditions.


 DOI: 10.21437/AVSP.2017-4

Cite as: Nunnemann, E.M., Bergmann, K., Kreysa, H., Knoeferle, P. (2017) Referential Gaze Makes a Difference in Spoken Language Comprehension: Human Speaker vs. Virtual Agent Listener Gaze. Proc. The 14th International Conference on Auditory-Visual Speech Processing, 16-20, DOI: 10.21437/AVSP.2017-4.


@inproceedings{Nunnemann2017,
  author={Eva Maria Nunnemann and Kirsten Bergmann and Helene Kreysa and Pia Knoeferle},
  title={ Referential Gaze Makes a Difference in Spoken Language Comprehension: Human Speaker vs. Virtual Agent Listener Gaze},
  year=2017,
  booktitle={Proc. The 14th International Conference on Auditory-Visual Speech Processing},
  pages={16--20},
  doi={10.21437/AVSP.2017-4},
  url={http://dx.doi.org/10.21437/AVSP.2017-4}
}