TensorFlow 1 version
|
View source on GitHub
|
Indicates how a distributed variable will be aggregated.
tf.distribute.Strategy distributes a model by making multiple copies
(called "replicas") acting data-parallel on different elements of the input
batch. When performing some variable-update operation, say
var.assign_add(x), in a model, we need to resolve how to combine the
different values for x computed in the different replicas.
NONE: This is the default, giving an error if you use a variable-update operation with multiple replicas.SUM: Add the updates across replicas.MEAN: Take the arithmetic mean ("average") of the updates across replicas.ONLY_FIRST_REPLICA: This is for when every replica is performing the same update, but we only want to perform the update once. Used, e.g., for the global step counter.
Class Variables | |
|---|---|
| MEAN |
tf.VariableAggregation
|
| NONE |
tf.VariableAggregation
|
| ONLY_FIRST_REPLICA |
tf.VariableAggregation
|
| SUM |
tf.VariableAggregation
|