Speech and Language Technology in Education (SLaTE 2013)

Grenoble, France
August 30-September 1, 2013

Fusing Eye-gaze and Speech Recognition for Tracking in an Automatic Reading Tutor – A Step in the Right Direction?

Morten Højfeldt Rasmussen (1), Zheng-Hua Tan (2)

(1) SpeechOp ApS, Aalborg, Denmark
(2) Department of Electronic Systems, Aalborg University, Aalborg, Denmark

In this paper we present a novel approach for automatically tracking the reading progress using a combination of eye-gaze tracking and speech recognition. The two are fused by first generating word probabilities based on eye-gaze information and then using these probabilities to augment the language model probabilities during speech recognition. Experimental results on a small dataset show that the tracking error rate of the system using only speech recognition is 37.9% whereas the tracking error rate for the system that incorporates eye-gaze tracking into the speech recognizer is 35.8%.

Index Terms: automatic reading tutor, eye-gaze tracking, speech recognition

Full Paper

Bibliographic reference.  Rasmussen, Morten Højfeldt / Tan, Zheng-Hua (2013): "Fusing eye-gaze and speech recognition for tracking in an automatic reading tutor – a step in the right direction?", In SLaTE-2013, 112-115.