tf.keras.layers.experimental.preprocessing.IntegerLookup

Reindex integer inputs to be in a contiguous range, via a dict lookup.

Inherits From: PreprocessingLayer, Layer, Module

tf.keras.layers.experimental.preprocessing.IntegerLookup(
    max_tokens=None, num_oov_indices=1, mask_token=0, oov_token=-1, vocabulary=None,
    invert=False, output_mode=index_lookup.INT, sparse=False,
    pad_to_max_tokens=False, **kwargs
)

This layer maps a set of arbitrary integer input tokens into indexed integer output via a table-based vocabulary lookup. The layer's output indices will be contiguously arranged up to the maximum vocab size, even if the input tokens are non-continguous or unbounded. The layer supports multiple options for encoding the output via output_mode, and has optional support for out-of-vocabulary (OOV) tokens and masking.

The vocabulary for the layer can be supplied on construction or learned via adapt(). During adapt(), the layer will analyze a data set, determine the frequency of individual integer tokens, and create a vocabulary from them. If the vocabulary is capped in size, the most frequent tokens will be used to create the vocabulary and all others will be treated as OOV.

There are two possible output modes for the layer. When output_mode is "int", input integers are converted to their index in the vocabulary (an integer). When output_mode is "binary", "count", or "tf-idf", input integers are encoded into an array where each dimension corresponds to an element in the vocabulary.

The vocabulary can optionally contain a mask token as well as an OOV token (which can optionally occupy multiple indices in the vocabulary, as set by num_oov_indices). The position of these tokens in the vocabulary is fixed. When output_mode is "int", the vocabulary will begin with the mask token at index 0, followed by OOV indices, followed by the rest of the vocabulary. When output_mode is "binary", "count", or "tf-idf" the vocabulary will begin with OOV indices and instances of the mask token will be dropped.

max_tokens The maximum size of the vocabulary for this layer. If None, there is no cap on the size of the vocabulary. Note that this size includes the OOV and mask tokens. Default to None.
num_oov_indices The number of out-of-vocabulary tokens to use. If this value is more than 1, OOV inputs are modulated to determine their OOV value. If this value is 0, OOV inputs will map to -1 when output_mode is "int" and are dropped otherwise. Defaults to 1.
mask_token An integer token that represents masked inputs. When output_mode is "int", the token is included in vocabulary and mapped to index 0. In other output modes, the token will not appear in the vocabulary and instances of the mask token in the input will be dropped. If set to None, no mask term will be added. Defaults to 0.
oov_token Only used when invert is True. The token to return for OOV indices. Defaults to -1.
vocabulary An optional list of integer tokens, or a path to a text file containing a vocabulary to load into this layer. The file should contain one integer token per line. If the list or file contains the same token multiple times, an error will be thrown.
invert Only valid when output_mode is "int". If True, this layer will map indices to vocabulary items instead of mapping vocabulary items to indices. Default to False.
output_mode Specification for the output of the layer. Defaults to "int". Values can be "int", "binary", "count", or "tf-idf" configuring the layer as follows: "int": Return the vocabulary indices of the input tokens. "binary": Outputs a single int array per sample, of either vocabulary size or max_tokens size, containing 1s in all elements where the token mapped to that index exists at least once in the sample. "count": Like "binary", but the int array contains a count of the number of times the token at that index appeared in the sample. "tf-idf": As "binary", but the TF-IDF algorithm is applied to find the value in each token slot.
pad_to_max_tokens Only applicable when output_mode is "binary", "count", or "tf-idf". If True, the output will have its feature axis padded to max_tokens even if the number of unique tokens in the vocabulary is less than max_tokens, resulting in a tensor of shape [batch_size, max_tokens] regardless of vocabulary size. Defaults to False.
sparse Boolean. Only applicable when output_mode is "binary", "count", or "tf-idf". If True, returns a SparseTensor instead of a dense Tensor. Defaults to False.

Examples:

Creating a lookup layer with a known vocabulary

This example creates a lookup layer with a pre-existing vocabulary.

vocab = [12, 36, 1138, 42]
data = tf.constant([[12, 1138, 42], [42, 1000, 36]])  # Note OOV tokens
layer = IntegerLookup(vocabulary=vocab)
layer(data)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[2, 4, 5],
       [5, 1, 3]])>

Creating a lookup layer with an adapted vocabulary

This example creates a lookup layer and generates the vocabulary by analyzing the dataset.

data = tf.constant([[12, 1138, 42], [42, 1000, 36]])
layer = IntegerLookup()
layer.adapt(data)
layer.get_vocabulary()
[0, -1, 42, 1138, 1000, 36, 12]

Note how the mask token 0 and the OOV token -1 have been added to the vocabulary. The remaining tokens are sorted by frequency (1138, which has 2 occurrences, is first) then by inverse sort order.

data = tf.constant([[12, 1138, 42], [42, 1000, 36]])
layer = IntegerLookup()
layer.adapt(data)
layer(data)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[6, 3, 2],
       [2, 4, 5]])>

Lookups with multiple OOV indices

This example demonstrates how to use a lookup layer with multiple OOV indices. When a layer is created with more than one OOV index, any OOV tokens are hashed into the number of OOV buckets, distributing OOV tokens in a deterministic fashion across the set.

vocab = [12, 36, 1138, 42]
data = tf.constant([[12, 1138, 42], [37, 1000, 36]])
layer = IntegerLookup(vocabulary=vocab, num_oov_indices=2)
layer(data)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[3, 5, 6],
       [2, 1, 4]])>

Note that the output for OOV token 37 is 2, while the output for OOV token 1000 is 1. The in-vocab terms have their output index increased by 1 from earlier examples (12 maps to 3, etc) in order to make space for the extra OOV token.

Multi-hot output

Configure the layer with output_mode='binary'. Note that the first num_oov_indices dimensions in the binary encoding represent OOV tokens

vocab = [12, 36, 1138, 42]
data = tf.constant([[12, 1138, 42, 42], [42, 7, 36, 7]]) # Note OOV tokens
layer = IntegerLookup(vocabulary=vocab, output_mode='binary')
layer(data)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
  array([[0., 1., 0., 1., 1.],
         [1., 0., 1., 0., 1.]], dtype=float32)>

Token count output

Configure the layer with output_mode='count'. As with binary output, the first num_oov_indices dimensions in the output represent OOV tokens.

vocab = [12, 36, 1138, 42]
data = tf.constant([[12, 1138, 42, 42], [42, 7, 36, 7]]) # Note OOV tokens
layer = IntegerLookup(vocabulary=vocab, output_mode='count')
layer(data)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
  array([[0., 1., 0., 1., 2.],
         [2., 0., 1., 0., 1.]], dtype=float32)>

TF-IDF output

Configure the layer with output_mode='tf-idf'. As with binary output, the first num_oov_indices dimensions in the output represent OOV tokens.

Each token bin will output token_count * idf_weight, where the idf weights are the inverse document frequency weights per token. These should be provided along with the vocabulary. Note that the idf_weight for OOV tokens will default to the average of all idf weights passed in.

vocab = [12, 36, 1138, 42]
idf_weights = [0.25, 0.75, 0.6, 0.4]
data = tf.constant([[12, 1138, 42, 42], [42, 7, 36, 7]]) # Note OOV tokens
layer = IntegerLookup(output_mode='tf-idf')
layer.set_vocabulary(vocab, idf_weights=idf_weights)
layer(data)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
  array([[0.  , 0.25, 0.  , 0.6 , 0.8 ],
         [1.0 , 0.  , 0.75, 0.  , 0.4 ]], dtype=float32)>

To specify the idf weights for oov tokens, you will need to pass the entire vocabularly including the leading oov token.

vocab = [-1, 12, 36, 1138, 42]
idf_weights = [0.9, 0.25, 0.75, 0.6, 0.4]
data = tf.constant([[12, 1138, 42, 42], [42, 7, 36, 7]]) # Note OOV tokens
layer = IntegerLookup(output_mode='tf-idf')
layer.set_vocabulary(vocab, idf_weights=idf_weights)
layer(data)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
  array([[0.  , 0.25, 0.  , 0.6 , 0.8 ],
         [1.8 , 0.  , 0.75, 0.  , 0.4 ]], dtype=float32)>

When adapting the layer in tf-idf mode, each input sample will be considered a document, and idf weight per token will be calculated as log(1 + num_documents / (1 + token_document_count)).

Inverse lookup

This example demonstrates how to map indices to tokens using this layer. (You can also use adapt() with inverse=True, but for simplicity we'll pass the vocab in this example.)

vocab = [12, 36, 1138, 42]
data = tf.constant([[2, 4, 5], [5, 1, 3]])
layer = IntegerLookup(vocabulary=vocab, invert=True)
layer(data)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[  12, 1138,   42],
       [  42,   -1,   36]])>

Note that the first two indices correspond to the mask and oov token by default. This behavior can be disabled by setting mask_token=None and num_oov_indices=0.

Forward and inverse lookup pairs

This example demonstrates how to use the vocabulary of a standard lookup layer to create an inverse lookup layer.

vocab = [12, 36, 1138, 42]
data = tf.constant([[12, 1138, 42], [42, 1000, 36]])
layer = IntegerLookup(vocabulary=vocab)
i_layer = IntegerLookup(vocabulary=layer.get_vocabulary(), invert=True)
int_data = layer(data)
i_layer(int_data)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[  12, 1138,   42],
       [  42,   -1,   36]])>

In this example, the input token 1000 resulted in an output of -1, since 1000 was not in the vocabulary - it got represented as an OOV, and all OOV tokens are returned as -1 in the inverse layer. Also, note that for the inverse to work, you must have already set the forward layer vocabulary either directly or via fit() before calling get_vocabulary().

is_adapted Whether the layer has been fit to data already.
streaming Whether adapt can be called twice without resetting the state.

Methods

adapt

View source

adapt(
    data, reset_state=True
)

Fits the state of the preprocessing layer to the dataset.

Overrides the default adapt method to apply relevant preprocessing to the inputs before passing to the combiner.

Args
data The data to train on. It can be passed either as a tf.data Dataset, or as a numpy array.
reset_state Optional argument specifying whether to clear the state of the layer at the start of the call to adapt. This must be True for this layer, which does not support repeated calls to adapt.

compile

View source

compile(
    run_eagerly=None, steps_per_execution=None
)

Configures the layer for adapt.

Arguments
run_eagerly Bool. Defaults to False. If True, this Model's logic will not be wrapped in a tf.function. Recommended to leave this as None unless your Model cannot be run inside a tf.function. steps_per_execution: Int. Defaults to 1. The number of batches to run during each tf.function call. Running multiple batches inside a single tf.function call can greatly improve performance on TPUs or small models with a large Python overhead.

finalize_state

View source

finalize_state()

Finalize the statistics for the preprocessing layer.

This method is called at the end of adapt. This method handles any one-time operations that should occur after all data has been seen.

get_vocabulary

View source

get_vocabulary()

make_adapt_function

View source

make_adapt_function()

Creates a function to execute one step of adapt.

This method can be overridden to support custom adapt logic. This method is called by PreprocessingLayer.adapt.

Typically, this method directly controls tf.function settings, and delegates the actual state update logic to PreprocessingLayer.update_state.

This function is cached the first time PreprocessingLayer.adapt is called. The cache is cleared whenever PreprocessingLayer.compile is called.

Returns
Function. The function created by this method should accept a tf.data.Iterator, retrieve a batch, and update the state of the layer.

merge_state

View source

merge_state(
    layers
)

Merge the statistics of multiple preprocessing layers.

This layer will contain the merged state.

Arguments
layers Layers whose statistics should be merge with the statistics of this layer.

reset_state

View source

reset_state()

Resets the statistics of the preprocessing layer.

set_vocabulary

View source

set_vocabulary(
    vocabulary, idf_weights=None
)

Sets vocabulary (and optionally document frequency) data for this layer.

This method sets the vocabulary and idf weights for this layer directly, instead of analyzing a dataset through 'adapt'. It should be used whenever the vocab (and optionally document frequency) information is already known. If vocabulary data is already present in the layer, this method will replace it.

Args
vocabulary An array of hashable tokens.
idf_weights An array of inverse document frequency weights with equal length to vocab. Only necessary if the layer output_mode is TFIDF.

Raises
ValueError If there are too many inputs, the inputs do not match, or input data is missing.
RuntimeError If the vocabulary cannot be set when this function is called. This happens when "binary", "count", and "tfidf" modes, if "pad_to_max_tokens" is False and the layer itself has already been called.

update_state

View source

update_state(
    data
)

vocab_size

View source

vocab_size()

vocabulary_size

View source

vocabulary_size()

Gets the current size of the layer's vocabulary.

Returns
The integer size of the voculary, including optional mask and oov indices.