4 / 4 Architectures (CNN / RNN)

Forget Me Not

A vanilla RNN and an LSTM are both trained on a sequence task requiring information from 200 time steps earlier to be preserved. The vanilla RNN fails to learn this dependency while the LSTM succeeds. What LSTM mechanism most directly explains this difference?

A

The LSTM processes the sequence in both directions simultaneously, so it never needs to carry information forward

B

The LSTM applies dropout to the recurrent weights, which prevents the vanishing gradient problem by design

C

Gated cell state updates (forget, input, and output gates) that let the LSTM additively preserve or selectively overwrite information in its cell state instead of repeatedly squashing it through a nonlinearity at every step

D

The LSTM uses a larger hidden state dimension, so it simply has more raw capacity to memorize the sequence

E

The LSTM replaces backpropagation through time with a convolution over the whole sequence, avoiding recurrence entirely