tf.nn.batch_normalization
Stay organized with collections
Save and categorize content based on your preferences.
Batch normalization.
tf.nn.batch_normalization(
x, mean, variance, offset, scale, variance_epsilon, name=None
)
Normalizes a tensor by mean
and variance
, and applies (optionally) a
scale
\(\gamma\) to it, as well as an offset
\(\beta\):
\(\frac{\gamma(x-\mu)}{\sigma}+\beta\)
mean
, variance
, offset
and scale
are all expected to be of one of two
shapes:
- In all generality, they can have the same number of dimensions as the
input
x
, with identical sizes as x
for the dimensions that are not
normalized over (the 'depth' dimension(s)), and dimension 1 for the
others which are being normalized over.
mean
and variance
in this case would typically be the outputs of
tf.nn.moments(..., keepdims=True)
during training, or running averages
thereof during inference.
- In the common case where the 'depth' dimension is the last dimension in
the input tensor
x
, they may be one dimensional tensors of the same
size as the 'depth' dimension.
This is the case for example for the common [batch, depth]
layout of
fully-connected layers, and [batch, height, width, depth]
for
convolutions.
mean
and variance
in this case would typically be the outputs of
tf.nn.moments(..., keepdims=False)
during training, or running averages
thereof during inference.
See equation 11 in Algorithm 2 of source:
Batch Normalization: Accelerating Deep Network Training by
Reducing Internal Covariate Shift; S. Ioffe, C. Szegedy.
Args |
x
|
Input Tensor of arbitrary dimensionality.
|
mean
|
A mean Tensor .
|
variance
|
A variance Tensor .
|
offset
|
An offset Tensor , often denoted \(\beta\) in equations, or
None. If present, will be added to the normalized tensor.
|
scale
|
A scale Tensor , often denoted \(\gamma\) in equations, or
None . If present, the scale is applied to the normalized tensor.
|
variance_epsilon
|
A small float number to avoid dividing by 0.
|
name
|
A name for this operation (optional).
|
Returns |
the normalized, scaled, offset tensor.
|
References |
Batch Normalization - Accelerating Deep Network Training by Reducing
Internal Covariate Shift:
Ioffe et al., 2015
(pdf)
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Some content is licensed under the numpy license.
Last updated 2023-03-17 UTC.
[{
"type": "thumb-down",
"id": "missingTheInformationINeed",
"label":"Missing the information I need"
},{
"type": "thumb-down",
"id": "tooComplicatedTooManySteps",
"label":"Too complicated / too many steps"
},{
"type": "thumb-down",
"id": "outOfDate",
"label":"Out of date"
},{
"type": "thumb-down",
"id": "samplesCodeIssue",
"label":"Samples / code issue"
},{
"type": "thumb-down",
"id": "otherDown",
"label":"Other"
}]
[{
"type": "thumb-up",
"id": "easyToUnderstand",
"label":"Easy to understand"
},{
"type": "thumb-up",
"id": "solvedMyProblem",
"label":"Solved my problem"
},{
"type": "thumb-up",
"id": "otherUp",
"label":"Other"
}]
{"lastModified": "Last updated 2023-03-17 UTC."}
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2023-03-17 UTC."],[],[]]