Draft:Layer normalization

• Comment: One section is completely unsourced. Hitro talk 08:22, 21 September 2023 (UTC)

---

Layer normalization (also known as layer norm) helps make training of artificial neural networks faster and more stable. It is presented as an alternative to batch normalization. In layer norm, neural networks compute a mean and variance using all the summed inputs to the neuron in a single training step. This information is then used to normalize the summed input to that neuron for each and every training step. Layer normalization is commonly found in recurrent neural networks and in the Transformer (machine learning model).

Introduction and Motivation

Complex artificial neural networks trained with stochastic gradient descent often require multiple days of training and large amounts of computing power. One approach to reducing training time is to standardize the neuron inputs performed during feedforward to make the learning easier. Batch normalization achieves this by standardizing each neuron's input using the data's mean and standard deviation. However, batch normalization requires storing the mean and standard deviation for each layer. This is not possible with an RNN because RNNs typically vary in sequence length. Layer normalization mitigates this by directly producing normalization statistics from the layer inputs so that the statistics do not need to be stored between training cases.

Procedure

The outputs of hidden layers in a deep neural network tend to be highly correlated with one another. This means the mean and variance of the summed layer inputs can be fixed as constants. The computation for layer norm statistics for all neurons in the same hidden layer is:

Assume that $H$ is the number of neurons in the $l^{\text{th}}$ hidden layer.

The mean is:

$$\mu ^{l}={\frac {1}{H}}\sum _{i=1}^{H}a_{i}^{l}$$

The variance is:

$$\sigma ^{l}={\sqrt {{\frac {1}{H}}\sum _{i=1}^{H}(a_{i}^{l}-\mu ^{l})^{2}}}$$