tf.lite.Interpreter

TensorFlow 1 version

View source on GitHub

Interpreter interface for TensorFlow Lite Models.

tf.lite.Interpreter(
    model_path=None, model_content=None, experimental_delegates=None,
    num_threads=None,
    experimental_op_resolver_type=tf.lite.experimental.OpResolverType.AUTO,
    experimental_preserve_all_tensors=False
)

This makes the TensorFlow Lite interpreter accessible in Python. It is possible to use this interpreter in a multithreaded Python environment, but you must be sure to call functions of a particular instance from only one thread at a time. So if you want to have 4 threads running different inferences simultaneously, create an interpreter for each one as thread-local data. Similarly, if you are calling invoke() in one thread on a single interpreter but you want to use tensor() on another thread once it is done, you must use a synchronization primitive between the threads to ensure invoke has returned before calling tensor().

Methods

allocate_tensors

View source

allocate_tensors()

get_input_details

View source

get_input_details()

Gets model input details.

get_output_details

View source

get_output_details()

Gets model output details.

get_signature_list

View source

get_signature_list()

Gets list of SignatureDefs in the model.

Example,

signatures = interpreter.get_signature_list()
print(signatures)

# {
#   'add': {'inputs': ['x', 'y'], 'outputs': ['output_0']}
# }

Then using the names in the signature list you can get a callable from
get_signature_runner().

get_signature_runner

View source

get_signature_runner(
    method_name=None
)

Gets callable for inference of specific SignatureDef.

Example usage,

interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()
fn = interpreter.get_signature_runner('div_with_remainder')
output = fn(x=np.array([3]), y=np.array([2]))
print(output)
# {
#   'quotient': array([1.], dtype=float32)
#   'remainder': array([1.], dtype=float32)
# }

None can be passed for method_name if the model has a single Signature only.

All names used are this specific SignatureDef names.

get_tensor

View source

get_tensor(
    tensor_index
)

Gets the value of the output tensor (get a copy).

If you wish to avoid the copy, use tensor(). This function cannot be used to read intermediate results.

get_tensor_details

View source

get_tensor_details()

Gets tensor details for every tensor with valid tensor details.

Tensors where required information about the tensor is not found are not added to the list. This includes temporary tensors without a name.

invoke

View source

invoke()

Invoke the interpreter.

Be sure to set the input sizes, allocate tensors and fill values before calling this. Also, note that this function releases the GIL so heavy computation can be done in the background while the Python interpreter continues. No other function on this object should be called while the invoke() call has not finished.

reset_all_variables

View source

reset_all_variables()

resize_tensor_input

View source

resize_tensor_input(
    input_index, tensor_size, strict=False
)

Resizes an input tensor.

Usage:

interpreter = Interpreter(model_content=tflite_model)
interpreter.resize_tensor_input(0, [num_test_images, 224, 224, 3])
interpreter.allocate_tensors()
interpreter.set_tensor(0, test_images)
interpreter.invoke()

set_tensor

View source

set_tensor(
    tensor_index, value
)

Sets the value of the input tensor.

Note this copies data in value.

If you want to avoid copying, you can use the tensor() function to get a numpy buffer pointing to the input buffer in the tflite interpreter.

tensor

View source

tensor(
    tensor_index
)

Returns function that gives a numpy view of the current tensor buffer.

This allows reading and writing to this tensors w/o copies. This more closely mirrors the C++ Interpreter class interface's tensor() member, hence the name. Be careful to not hold these output references through calls to allocate_tensors() and invoke(). This function cannot be used to read intermediate results.

Usage:

interpreter.allocate_tensors()
input = interpreter.tensor(interpreter.get_input_details()[0]["index"])
output = interpreter.tensor(interpreter.get_output_details()[0]["index"])
for i in range(10):
  input().fill(3.)
  interpreter.invoke()
  print("inference %s" % output())

Notice how this function avoids making a numpy array directly. This is because it is important to not hold actual numpy views to the data longer than necessary. If you do, then the interpreter can no longer be invoked, because it is possible the interpreter would resize and invalidate the referenced tensors. The NumPy API doesn't allow any mutability of the the underlying buffers.

WRONG:

input = interpreter.tensor(interpreter.get_input_details()[0]["index"])()
output = interpreter.tensor(interpreter.get_output_details()[0]["index"])()
interpreter.allocate_tensors()  # This will throw RuntimeError
for i in range(10):
  input.fill(3.)
  interpreter.invoke()  # this will throw RuntimeError since input,output