Off-Topic Spoken Response Detection Using Siamese Convolutional Neural Networks

Chong Min Lee, Su-Youn Yoon, Xihao Wang, Matthew Mulholland, Ikkyu Choi, Keelan Evanini


In this study, we developed an off-topic response detection system to be used in the context of the automated scoring of non-native English speakers’ spontaneous speech. Based on transcriptions generated from an ASR system trained on non-native speakers’ speech and various semantic similarity features, the system classified each test response as an on-topic or off-topic response. The recent success of deep neural networks (DNN) in text similarity detection led us to explore DNN-based document similarity features. Specifically, we used a siamese adaptation of the convolutional network, due to its efficiency in learning similarity patterns simultaneously from both responses and questions used to elicit responses. In addition, a baseline system was developed using a standard vector space model (VSM) trained on sample responses for each question. The accuracy of the siamese CNN-based system was 0.97 and there was a 50% relative error reduction compared to the standard VSM-based system. Furthermore, the accuracy of the siamese CNN-based system was consistent across different questions.


 DOI: 10.21437/Interspeech.2017-1174

Cite as: Lee, C.M., Yoon, S., Wang, X., Mulholland, M., Choi, I., Evanini, K. (2017) Off-Topic Spoken Response Detection Using Siamese Convolutional Neural Networks. Proc. Interspeech 2017, 1427-1431, DOI: 10.21437/Interspeech.2017-1174.


@inproceedings{Lee2017,
  author={Chong Min Lee and Su-Youn Yoon and Xihao Wang and Matthew Mulholland and Ikkyu Choi and Keelan Evanini},
  title={Off-Topic Spoken Response Detection Using Siamese Convolutional Neural Networks},
  year=2017,
  booktitle={Proc. Interspeech 2017},
  pages={1427--1431},
  doi={10.21437/Interspeech.2017-1174},
  url={http://dx.doi.org/10.21437/Interspeech.2017-1174}
}