NumPy Neural Network from Scratch
98.07% on MNIST · SGD, Momentum & Adam compared- Python
- NumPy
- Deep Learning
What I implemented by hand
Forward pass, backpropagation, Xavier initialisation, and SGD / Momentum / Adam optimizers written by hand in NumPy. No autograd, no ML framework. Same 784 → 128 → 10 architecture as my earlier C++ network.
How I evaluated it
Same 784 → 128 → 10 network, same weight seeds, 100 epochs, batch size 32, MNIST's standard 60,000 / 10,000 train/test split. SGD and Momentum (β = 0.9) at learning rate 0.01, Adam at 0.001. One run per optimizer; reproducible with python run_comparison.py.
Results
At 100 epochs all three finish between 97.90% and 98.07% — too close for a single run to rank them. The real difference is speed: Adam passes 97% by epoch 2 and classical momentum by epoch 4, while plain SGD takes until epoch 26.
- SGD
- Momentum
- Adam
Test accuracy vs epoch (floor 90%)
Hover or focus the chart and use the arrow keys to inspect an epoch.
Training loss vs epoch (log scale)
Hover or focus the chart and use the arrow keys to inspect an epoch.
View data
| Epoch | SGD acc | Momentum acc | Adam acc | SGD loss | Momentum loss | Adam loss |
|---|---|---|---|---|---|---|
| 1 | 90.49% | 94.87% | 95.93% | 0.6438 | 0.3048 | 0.2601 |
| 10 | 95.33% | 97.81% | 97.79% | 0.1654 | 0.0310 | 0.0157 |
| 20 | 96.68% | 97.88% | 97.85% | 0.1057 | 0.0082 | 0.0057 |
| 30 | 97.22% | 98.07% | 98.07% | 0.0770 | 0.0032 | 0.0040 |
| 40 | 97.39% | 98.04% | 97.72% | 0.0597 | 0.0019 | 0.0022 |
| 50 | 97.66% | 98.07% | 98.03% | 0.0477 | 0.0013 | 0.0016 |
| 60 | 97.77% | 98.08% | 97.95% | 0.0392 | 9.82e-4 | 0.0030 |
| 70 | 97.85% | 98.07% | 97.78% | 0.0328 | 7.89e-4 | 0.0031 |
| 80 | 97.85% | 98.04% | 97.79% | 0.0277 | 6.55e-4 | 0.0017 |
| 90 | 97.88% | 98.08% | 97.79% | 0.0235 | 5.56e-4 | 0.0019 |
| 100 | 97.90% | 98.07% | 97.91% | 0.0202 | 4.83e-4 | 0.0011 |
| SGD — final test accuracy (epoch 100) | 97.90% |
| Momentum — final test accuracy (epoch 100) | 98.07% |
| Adam — final test accuracy (epoch 100) | 97.91% |
| SGD — first epoch ≥ 97% test accuracy | 26 |
| Momentum — first epoch ≥ 97% test accuracy | 4 |
| Adam — first epoch ≥ 97% test accuracy | 2 |
| SGD — test accuracy at epoch 5 | 93.83% |
| Momentum — test accuracy at epoch 5 | 97.44% |
| Adam — test accuracy at epoch 5 | 97.42% |
What broke
- My first 'momentum' was the EMA form, v = βv + (1−β)g. At the same learning rate that's just SGD with extra smoothing, and its curve sat exactly on top of SGD's. Switching to classical momentum (v = βv + g) took it from epoch 26 to epoch 4 to reach 97%. A unit test now checks the velocity sequence (g, 1.9g, 2.71g), and it fails on the old code.
- The logged training loss was the last mini-batch's, not the epoch average, so the curve was mostly noise. It's now the batch-size-weighted mean over each epoch.