tf.data.experimental.DistributeOptions

TensorFlow 1 version View source on GitHub

Represents options for distributed data processing.

tf.data.experimental.DistributeOptions()

You can set the distribution options of a dataset through the experimental_distribute property of tf.data.Options; the property is an instance of tf.data.experimental.DistributeOptions.

options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = AutoShardPolicy.OFF
dataset = dataset.with_options(options)

auto_shard_policy The type of sharding that auto-shard should attempt. If this is set to FILE, then we will attempt to shard by files (each worker will get a set of files to process). If we cannot find a set of files to shard for at least one file per worker, we will error out. When this option is selected, make sure that you have enough files so that each worker gets at least one file. There will be a runtime error thrown if there are insufficient files. If this is set to DATA, then we will shard by elements produced by the dataset, and each worker will process the whole dataset and discard the portion that is not for itself. If this is set to OFF, then we will not autoshard, and each worker will receive a copy of the full dataset. This option is set to AUTO by default, AUTO will attempt to first shard by FILE, and fall back to sharding by DATA if we cannot find a set of files to shard.
num_devices The number of devices attached to this input pipeline. This will be automatically set by MultiDeviceIterator.

Methods

__eq__

View source

__eq__(
    other
)

Return self==value.

__ne__

View source

__ne__(
    other
)

Return self!=value.