Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Abhinav Kumar, Jillian Aurisano, Barbara Di Eugenio, Andrew Johnson, Abeer Alsaiari, Nigel Flowers, Alberto Gonzalez, Jason Leigh


The goals of our work are twofold: gain insight into how humans interact with complex data and visualizations thereof in order to make discoveries; and use our findings to develop a dialogue system for exploring data visualizations. In our work to date, we have collected a multimodal dialogue corpus and developed a pipeline that creates visualizations from spoken utterances. In this paper, we focus on the multimodal and contextual information that can give the system additional insight into the user’s intention, both through gestures and utterances nearby the actual request. To this end, we have annotated our corpus for context and both speech and gesture information; we developed a classifier for identifying rich themes found in the annotated context, in order to refine the interpretation of the request; and we have developed gesture detection using Kinect.


 DOI: 10.21437/SemDial.2017-5

Cite as: Kumar, A., Aurisano, J., Di Eugenio, B., Johnson, A., Alsaiari, A., Flowers, N., Gonzalez, A., Leigh, J. (2017) Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification. Proc. SEMDIAL 2017 (SaarDial) Workshop on the Semantics and Pragmatics of Dialogue, 41-51, DOI: 10.21437/SemDial.2017-5.


@inproceedings{Kumar2017,
  author={Abhinav Kumar and Jillian Aurisano and Barbara {Di Eugenio} and Andrew Johnson and Abeer Alsaiari and Nigel Flowers and Alberto Gonzalez and Jason Leigh},
  title={Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification},
  year=2017,
  booktitle={Proc. SEMDIAL 2017 (SaarDial) Workshop on the Semantics and Pragmatics of Dialogue},
  pages={41--51},
  doi={10.21437/SemDial.2017-5},
  url={http://dx.doi.org/10.21437/SemDial.2017-5}
}