Artificial Neural Networks

The one idea: A neural network is a stack of simple weighted-sum-then-squash steps (layers), and it learns by running that stack forward to get a prediction, measuring how wrong the prediction was, then pushing the blame for that error backward through the stack — one layer at a time, via the chain rule — to figure out how much each individual weight contributed and nudge it in the direction that would have helped.

⚠️ The trap: Stacking more layers should make a network smarter. With the wrong activation function it makes it untrainable instead. **Sigmoid activations squash their output into a narrow range, and the chain rule multiplies those small squashing derivatives together at every layer — by the time the error signal reaches an early layer of a deep sigmoid network, it has been multiplied by so many numbers under 0.25 that it's effectively zero, and that layer stops learning. This is the vanishing gradient problem, and it's the specific, mechanical reason ReLU replaced sigmoid as the default hidden-layer activation.**

A neural network is not a mysterious black box — it's a composition of ordinary arithmetic. Every "deep learning" system, from a spam filter to a large language model, is built from the same core unit repeated and connected: take a weighted sum of your inputs, add a bias, pass the result through a nonlinear function. Stack enough of those units in enough layers and the network can approximate almost any function you can define — that's not a metaphor, it's a provable result (the universal approximation theorem). The catch is that "can approximate" doesn't mean "can learn to." Getting there requires a…