Audio-Visual Speech Recognition using Sequence to Sequence Models
Audio-Visual Speech Recognition (AVSR) research system using sequence-to-sequence neural networks based on TensorFlow 1.13
AVSR-tf1 is an open-source research system for Speech Recognition.
Written entirely in Python, AVSR-tf1 aims to provide a simple and reproducible way of training and evaluating
speech recognition models based on sequence to sequence neural networks. AVSR-tf1 can exploit both auditory
and visual speech modalities, considered either independently (ASR, VSR) or jointly (AVSR).
Rather than providing a dense documentation to the users and contributors, the AVSR-tf1 code is designed
(or strives) to be intuitive and self-explanatory, encouraging researchers and developers to understand the entire
codebase and propose improvements at its lowest levels. Hence we want it to be more of a flexible research system than
a black box for production.
A typical workflow is as follows:
Please refer to the attached examples for running audio-only, visual-only, or audio-visual speech recognition experiments.
To prepare the data, you can use the two scripts extract_faces.py and write_records_tcd.py
For visual/audio-visual experiments, please compile from source install OpenFace
The other dependencies are popular and easy to install Python packages, so feel free to use your preferred sources.
The supported TensorFlow version for this repository is 1.13.1, and the recommended install source is: pip install tensorflow_gpu==1.13.1.
Please get in touch in case you face any issues.
We are grateful to Eugene Brevdo of Google for his remarkable help and advice during the early stages of development. In addition, we would like to thank
Derek Murray, Andreas Steiner, Khe Chai Sim for the assistance and interesting conversations, and also every
TensorFlow contributor on GitHub and StackOverflow.
Our work is supported by NVIDIA, which granted us a Titan Xp GPU through its academic program.
If you use this work, please cite it as:
George Sterpu, Christian Saam. Naomi Harte.
How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition.
IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020. https://doi.org/10.1109/TASLP.2020.2980436
@ARTICLE{Sterpu2020,
author={G. {Sterpu} and C. {Saam} and N. {Harte}},
journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
title={How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition},
year={2020},
volume={},
number={},
pages={1-1},
}
or
George Sterpu, Christian Saam, and Naomi Harte. 2018.
Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition.
In 2018 International Conference on Multimodal Interaction (ICMI ’18), October 16–20, 2018, Boulder, CO, USA.
ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3242969.3243014
@inproceedings{sterpu_icmi18,
author = {George Sterpu and Christian Saam and Naomi Harte},
title = {Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition},
year = {2018},
publisher = {{ACM, New York, NY, USA}},
booktitle = {2018 International Conference on Multimodal Interaction (ICMI '18), October 16--20, 2018, Boulder, CO, USA},
url = {http://doi.acm.org/10.1145/3242969.3243014},
doi = {10.1145/3242969.3243014},
}
We are delighted to receive your feedback and help on improving AVSR-tf1. On the technical side, this could be
an advice or a pull request for code refactoring (we are not Python/TensorFlow experts), adding implementations of popular
features, bug reports, performance improvements, language models, support for computation in 16 bit precision or on
Google TPU devices.
[1] Sequence to Sequence Learning with Neural Networks
https://arxiv.org/abs/1409.3215
[2] Neural Machine Translation by Jointly Learning to Align and Translate
https://arxiv.org/abs/1409.0473
[3] Listen, Attend and Spell
https://arxiv.org/abs/1508.01211
[4] Lip Reading Sentences in the Wild
https://arxiv.org/abs/1611.05358
[5] Can DNNs Learn to Lipread Full Sentences?
https://arxiv.org/abs/1805.11685
[6] Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition
https://arxiv.org/abs/1809.01728
[7] How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
https://ieeexplore.ieee.org/document/9035650