Recurrent Neural Networks and Sequence Data

The one idea: A feedforward network (and even a CNN) processes a fixed-size input and has no built-in notion of "before" and "after" — every input dimension is just another number in a vector. A recurrent neural network instead carries a hidden state forward from one time step to the next, feeding each new input in alongside a summary of everything it's seen so far. The same small set of weights gets reused at every position in the sequence, the same way a convolution kernel gets reused at every position in an image — except now the sharing is across time, not space.

⚠️ The trap: Deeper usually means more powerful, and for a plain RNN reading a long sequence, "deeper" happens automatically — a 50-word sentence is a 50-layer-deep network by the time you unroll the recurrence. **The same vanishing-gradient mechanism that plagues deep sigmoid networks (from the ANN lesson) shows up here in an even harsher form, because it's the same weight matrix multiplied against itself at every step — and it isn't a training inconvenience to patch around, it's mathematically unavoidable for a plain RNN that's actually storing long-term memory. That's the specific, structural reason LSTM and GRU exist, not a generic "add more tricks" upgrade.**

Why feedforward and convolutional nets can't naturally handle sequences

The Artificial Neural Networks lesson built networks that take a fixed-size input vector and produce an output — nothing about that architecture assumes any ordering or relationship between one input and the next. The Convolutional Neural Networks lesson improved on that for images by sharing a small kernel across every position in…