tf.distribute.CrossDeviceOps

TensorFlow 1 version View source on GitHub

Base class for cross-device reduction and broadcasting algorithms.

tf.distribute.CrossDeviceOps()

Methods

batch_reduce

View source

batch_reduce(
    reduce_op, value_destination_pairs, experimental_hints=None
)

Reduce PerReplica objects in a batch.

Reduce each first element in value_destination_pairs to each second element which indicates the destinations.

This can be faster than multiple individual reduces because we can fuse several tensors into one or multiple packs before reduction.

Args
reduce_op An instance of tf.distribute.ReduceOp that indicates how the per_replica_value will be reduced.
value_destination_pairs A list or a tuple of PerReplica objects (or tensors with device set if there is one device) and destinations.
experimental_hints A tf.distrbute.experimental.CollectiveHints. Hints to perform collective operations.

Returns
a list of Mirrored objects.

Raises
ValueError if value_destination_pairs is not an iterable of tuples of PerReplica objects and destinations.

batch_reduce_implementation

View source

batch_reduce_implementation(
    reduce_op, value_destination_pairs, experimental_hints
)

Implementation of reduce PerReplica objects in a batch.

Overriding this method is useful for subclass implementers.

Reduce each first element in value_destination_pairs to each second element which indicates the destinations.

Args
reduce_op An instance of tf.distribute.ReduceOp that indicates how per_replica_value will be reduced.
value_destination_pairs An iterable of tuples of PerReplica objects (or tensors with device set if there is one device) and destinations.
experimental_hints A tf.distrbute.experimental.CollectiveHints. Hints to perform collective operations.

Returns
a list of Mirrored objects.

Raises
ValueError if value_destination_pairs is not an iterable of tuples of PerReplica objects and destinations

broadcast

View source

broadcast(
    tensor, destinations
)

Broadcast the tensor to destinations.

Args
tensor the tensor to broadcast.
destinations the broadcast destinations.

Returns
a Mirrored object.

broadcast_implementation

View source

broadcast_implementation(
    tensor, destinations
)

Implementation of broadcast the tensor to destinations.

Args
tensor the tensor to broadcast.
destinations the broadcast destinations.

Returns
a Mirrored object.

reduce

View source

reduce(
    reduce_op, per_replica_value, destinations, experimental_hints=None
)

Reduce per_replica_value to destinations.

It runs the reduction operation defined by reduce_op and put the result on destinations.

Args
reduce_op An instance of tf.distribute.ReduceOp that indicates how per_replica_value will be reduced.
per_replica_value A PerReplica object or a tensor with device set.
destinations the reduction destinations.
experimental_hints A tf.distrbute.experimental.CollectiveHints. Hints to perform collective operations.

Returns
a Mirrored object.

Raises
ValueError if per_replica_value can't be converted to a PerReplica object or if destinations aren't strings, Variables or DistributedValues

reduce_implementation

View source

reduce_implementation(
    reduce_op, per_replica_value, destinations, experimental_hints
)

The implementation of reduce of per_replica_value to destinations.

Overriding this method is useful for subclass implementers.

It runs the reduction operation defined by reduce_op and put the result on destinations.

Args
reduce_op An instance tf.distribute.ReduceOp that indicates of how per_replica_value will be reduced.
per_replica_value A PerReplica object or a tensor with device set.
destinations the reduction destinations.
experimental_hints A tf.distrbute.experimental.CollectiveHints. Hints to perform collective operations.

Returns
a Mirrored object.

Raises
ValueError if per_replica_value can't be converted to a PerReplica object.