Second Workshop on Child, Computer and Interaction (WOCCI 2009)

Cambridge, MA, USA
November 5, 2009

Conversations with a Virtual Science Tutor in Multimedia Learning Environments

Wayne H. Ward, Ronald A. Cole

Boulder Language Technologies, Boulder, CO, USA

During the past two years, a team of researchers, developers and teachers at Boulder Language Technologies has been collaborating with colleagues at the University of Colorado and the University of Pittsburgh to develop My Science Tutor (MyST), an intelligent tutoring system that is designed to improve science achievement of third, fourth and fifth grade students through conversational spoken dialogs with a virtual science tutor. The virtual tutor is a lifelike computer character that produces accurate visual speech (synchronized with recorded or synthesized speech) and emotions. We are developing spoken dialogs that use open ended questions such as “So, what did you learn about today?” “Ah, you built a circuit, tell me more about that.” The dialogs are guided by principles of Questioning the Author, an approach designed to manage classroom conversations to improve comprehension of stories, which we have modified for one on one tutoring in collaboration with Margaret Mckeown, one of the co-developers of the program. In our project, children leave the classroom to interact with My Science Tutor following classroom science investigations that are part of the FOSS science program, used in over 100,000 classrooms by over 2 million children in the United States. The conversational dialogs in MyST are designed to help children arrive at accurate explanations of the science observations, data and concepts encountered in their recent hands-on classroom science investigations. In addition to open ended questions, MyST presents illustrations and flash animations at appropriate times to focus the dialog on specific phenomena and concepts.
   In our talk, we will describe both the system architecture (the spoken dialog system and component modules) and the process of developing and integrating the speech recognition, natural language understanding, dialog modeling and character animation technologies used in the system. The project also involves the development of a corpus of annotated dialogs collected during the tutoring sessions. A Wizard of Oz procedure is used for data collection. In this procedure, students are interacting with the virtual tutor, while a human tutor at a second computer is mediating the interaction to ensure a fluent dialog. All interaction data for the sessions are logged by the system. The student speech is transcribed and the logs are then annotated for semantic content. To date we have recorded and transcribed data for over 1000 individual tutoring sessions; over 100 hours of children’s speech. We will demonstrate the system in both Wizard of Oz and standalone modes, and present initial results comparing students’ knowledge of science concepts on standardized tests for students in the same classroom who either did or did not use the system.

Full Paper

Bibliographic reference.  Ward, Wayne H. / Cole, Ronald A. (2009): "Conversations with a virtual science tutor in multimedia learning environments", In WOCCI-2009, 121-122.