View source on GitHub |
Batch normalization.
tf.nn.batch_normalization(
x, mean, variance, offset, scale, variance_epsilon, name=None
)
Normalizes a tensor by mean
and variance
, and applies (optionally) a
scale
\(\gamma\) to it, as well as an offset
\(\beta\):
\(\frac{\gamma(x-\mu)}{\sigma}+\beta\)
mean
, variance
, offset
and scale
are all expected to be of one of two
shapes:
- In all generality, they can have the same number of dimensions as the
input
x
, with identical sizes asx
for the dimensions that are not normalized over (the 'depth' dimension(s)), and dimension 1 for the others which are being normalized over.mean
andvariance
in this case would typically be the outputs oftf.nn.moments(..., keepdims=True)
during training, or running averages thereof during inference. - In the common case where the 'depth' dimension is the last dimension in
the input tensor
x
, they may be one dimensional tensors of the same size as the 'depth' dimension. This is the case for example for the common[batch, depth]
layout of fully-connected layers, and[batch, height, width, depth]
for convolutions.mean
andvariance
in this case would typically be the outputs oftf.nn.moments(..., keepdims=False)
during training, or running averages thereof during inference.
See equation 11 in Algorithm 2 of source: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; S. Ioffe, C. Szegedy.
Returns | |
---|---|
the normalized, scaled, offset tensor. |
References | |
---|---|
Batch Normalization - Accelerating Deep Network Training by Reducing Internal Covariate Shift: Ioffe et al., 2015 (pdf) |