Keep pulling the thread on Chris Olah.
Long Short Term Memory networks (LSTMs) perform significantly better than standard Recurrent Neural Network (RNN) versions for many tasks.
Almost all significant results based on recurrent neural networks are achieved with Long Short Term Memory networks (LSTMs).
Standard Recurrent Neural Networks (RNNs) are unable to learn to connect information as the time gap between the relevant information and its point of use grows.
Long Short Term Memory networks (LSTMs) are explicitly designed to avoid the long-term dependency problem, making remembering information for long periods their default behavior.
The next major step in Recurrent Neural Network (RNN) research after LSTMs is believed by many researchers to be attention mechanisms.
Long Short Term Memory networks (LSTMs) were introduced by Hochreiter & Schmidhuber in 1997.
A popular Long Short Term Memory (LSTM) variant with 'peephole connections,' which allow gate layers to view the cell state, was introduced by Gers & Schmidhuber in 2000.
The Gated Recurrent Unit (GRU), introduced by Cho, et al. in 2014, is a simpler variant of LSTM that combines the forget and input gates into a single 'update gate' and merges the cell and hidden states.
A 2015 comparison of popular Long Short Term Memory (LSTM) variants by Greff, et al. found that their performance is roughly equivalent.
A 2015 study by Jozefowicz, et al. that tested over ten thousand RNN architectures found some that outperformed LSTMs on certain tasks.
Grid LSTMs, introduced by Kalchbrenner, et al. in 2015, are considered an extremely promising area of Recurrent Neural Network (RNN) research.
Recurrent neural networks (RNNs) use loops to allow information to persist, addressing a shortcoming of traditional neural networks which cannot use reasoning about previous events to inform later ones.