Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model

Da-Rong Liu, Chunxi Liu, Frank Zhang, Gabriel Synnaeve, Yatharth Saraf, Geoffrey Zweig


Videos uploaded on social media are often accompanied with textual descriptions. In building automatic speech recognition (ASR) systems for videos, we can exploit the contextual information provided by such video metadata. In this paper, we explore ASR lattice rescoring by selectively attending to the video descriptions. We first use an attention based method to extract contextual vector representations of video metadata, and use these representations as part of the inputs to a neural language model during lattice rescoring. Secondly, we propose a hybrid pointer network approach to explicitly interpolate the word probabilities of the word occurrences in metadata. We perform experimental evaluations on both language modeling and ASR tasks, and demonstrate that both proposed methods provide performance improvements by selectively leveraging the video metadata.


 DOI: 10.21437/Interspeech.2020-1344

Cite as: Liu, D., Liu, C., Zhang, F., Synnaeve, G., Saraf, Y., Zweig, G. (2020) Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model. Proc. Interspeech 2020, 3650-3654, DOI: 10.21437/Interspeech.2020-1344.


@inproceedings{Liu2020,
  author={Da-Rong Liu and Chunxi Liu and Frank Zhang and Gabriel Synnaeve and Yatharth Saraf and Geoffrey Zweig},
  title={{Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model}},
  year=2020,
  booktitle={Proc. Interspeech 2020},
  pages={3650--3654},
  doi={10.21437/Interspeech.2020-1344},
  url={http://dx.doi.org/10.21437/Interspeech.2020-1344}
}