A Recipe for Training Neural Networks

The first step in a neural network project should be a thorough inspection of the data before writing any model code.

A key step to verify a model and training setup is to confirm it can overfit a single small batch of data and achieve the lowest possible loss.

When starting a new project, one should copy the simplest high-performing architecture from the most related research paper instead of designing a novel architecture.

The most effective way to regularize a model and improve its performance is to add more real training data.

Adding more data is the only method guaranteed to monotonically improve the performance of a well-configured neural network almost indefinitely.

Misconfigured neural networks often continue to train without throwing errors, but will perform worse than correctly configured models.

Data augmentation should be disabled during the initial setup of a training and evaluation skeleton to reduce potential sources of bugs.

To verify correct initialization, a softmax classifier's loss should start at -log(1/number_of_classes).

For image classification tasks, a ResNet-50 is a good default architecture to start with.

The Adam optimizer with a learning rate of 3e-4 is a good default choice for early-stage model baselining.

The Adam optimizer is more forgiving to hyperparameter choices, including learning rate, than SGD.

For convolutional networks, a well-tuned SGD optimizer will almost always slightly outperform the Adam optimizer.