Geoffrey Hinton
Nitish Srivastava,
Kevin Swersky
Tijmen Tieleman
Abdel-rahman Mohamed
Neural Networks for Machine Learning
Lecture 7a
Modeling sequences: A brief overview
Getting targets when modeling sequences
• When applying machine learning to sequences, we often want to turn an input
sequence into an output sequence that lives in a different domain.
– E. g. turn a sequence of sound pressures into a sequence of word identities.
• When there is no separate target sequence, we can get a teaching signal by trying
to predict the next term in the input sequence.
– The target output sequence is the input sequence with an advance of 1 step.
– This seems much more natural than trying to predict one pixel in an image
from the other pixels, or one patch of an image from the rest of the image.
– For temporal sequences there is a natural order for the predictions.
• Predicting the next term in a sequence blurs the distinction between supervise
sequence/input/target/image/sequences/predict/output/– /natural/turn/
sequence/input/target/image/sequences/predict/output/– /natural/turn/
-->