Migrar evaluación

Ver en TensorFlow.org Ejecutar en Google Colab Ver fuente en GitHub Descargar libreta

La evaluación es una parte fundamental de los modelos de medición y evaluación comparativa.

Esta guía demuestra cómo migrar las tareas del evaluador de TensorFlow 1 a TensorFlow 2. En Tensorflow 1, esta funcionalidad se implementa mediante tf.estimator.train_and_evaluate , cuando la API se ejecuta de forma distribuida. En Tensorflow 2, puedes usar el tf.keras.utils.SidecarEvaluator o un bucle de evaluación personalizado en la tarea del evaluador.

Hay opciones simples de evaluación en serie tanto en TensorFlow 1 ( tf.estimator.Estimator.evaluate ) como en TensorFlow 2 ( Model.fit(..., validation_data=(...)) o Model.evaluate ). La tarea del evaluador es preferible cuando desea que sus trabajadores no cambien entre capacitación y evaluación, y la evaluación integrada en Model.fit es preferible cuando desea que se distribuya su evaluación.

Configuración

import tensorflow.compat.v1 as tf1
import tensorflow as tf
import numpy as np
import tempfile
import time
import os

mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
11501568/11490434 [==============================] - 0s 0us/step

TensorFlow 1: evaluación mediante tf.estimator.train_and_evaluate

En TensorFlow 1, puedes configurar un tf.estimator para evaluar el estimador usando tf.estimator.train_and_evaluate .

En este ejemplo, comience definiendo el tf.estimator.Estimator y especificando las especificaciones de capacitación y evaluación:

feature_columns = [tf1.feature_column.numeric_column("x", shape=[28, 28])]

classifier = tf1.estimator.DNNClassifier(
    feature_columns=feature_columns,
    hidden_units=[256, 32],
    optimizer=tf1.train.AdamOptimizer(0.001),
    n_classes=10,
    dropout=0.2
)

train_input_fn = tf1.estimator.inputs.numpy_input_fn(
    x={"x": x_train},
    y=y_train.astype(np.int32),
    num_epochs=10,
    batch_size=50,
    shuffle=True,
)

test_input_fn = tf1.estimator.inputs.numpy_input_fn(
    x={"x": x_test},
    y=y_test.astype(np.int32),
    num_epochs=10,
    shuffle=False
)

train_spec = tf1.estimator.TrainSpec(input_fn=train_input_fn, max_steps=10)
eval_spec = tf1.estimator.EvalSpec(input_fn=test_input_fn,
                                   steps=10,
                                   throttle_secs=0)

INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpv82biaa9
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpv82biaa9', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /tmp/ipykernel_20878/122738158.py:11: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.

WARNING:tensorflow:From /tmp/ipykernel_20878/122738158.py:11: The name tf.estimator.inputs.numpy_input_fn is deprecated. Please use tf.compat.v1.estimator.inputs.numpy_input_fn instead.

Luego, entrene y evalúe el modelo. La evaluación se ejecuta sincrónicamente entre el entrenamiento porque está limitada como ejecución local en este cuaderno y alterna entre entrenamiento y evaluación. Sin embargo, si el estimador se usa de forma distribuida, el evaluador se ejecutará como una tarea de evaluador dedicada. Para más información, consulta la guía de migración sobre formación distribuida .

tf1.estimator.train_and_evaluate(estimator=classifier,
                                train_spec=train_spec,
                                eval_spec=eval_spec)

INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_queue_runner.py:65: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_functions.py:491: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py:914: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpv82biaa9/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 118.02926, step = 0
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 10...
INFO:tensorflow:Saving checkpoints for 10 into /tmp/tmpv82biaa9/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 10...
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2022-01-19T02:31:38
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpv82biaa9/model.ckpt-10
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [1/10]
INFO:tensorflow:Evaluation [2/10]
INFO:tensorflow:Evaluation [3/10]
INFO:tensorflow:Evaluation [4/10]
INFO:tensorflow:Evaluation [5/10]
INFO:tensorflow:Evaluation [6/10]
INFO:tensorflow:Evaluation [7/10]
INFO:tensorflow:Evaluation [8/10]
INFO:tensorflow:Evaluation [9/10]
INFO:tensorflow:Evaluation [10/10]
INFO:tensorflow:Inference Time : 0.29827s
INFO:tensorflow:Finished evaluation at 2022-01-19-02:31:38
INFO:tensorflow:Saving dict for global step 10: accuracy = 0.4953125, average_loss = 1.8270489, global_step = 10, loss = 233.86226
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10: /tmp/tmpv82biaa9/model.ckpt-10
INFO:tensorflow:Loss for final step: 92.23195.
({'accuracy': 0.4953125,
  'average_loss': 1.8270489,
  'loss': 233.86226,
  'global_step': 10},
 [])

TensorFlow 2: evaluación de un modelo de Keras

En TensorFlow 2, si usa la API Keras Model.fit para el entrenamiento, puede evaluar el modelo con tf.keras.utils.SidecarEvaluator . También puede visualizar las métricas de evaluación en Tensorboard que no se muestran en esta guía.

Para ayudar a demostrar esto, primero comencemos definiendo y entrenando el modelo:

def create_model():
  return tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10)
  ])

loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

model = create_model()
model.compile(optimizer='adam',
              loss=loss,
              metrics=['accuracy'],
              steps_per_execution=10,
              run_eagerly=True)

log_dir = tempfile.mkdtemp()
model_checkpoint = tf.keras.callbacks.ModelCheckpoint(
    filepath=os.path.join(log_dir, 'ckpt-{epoch}'),
    save_weights_only=True)

model.fit(x=x_train,
          y=y_train,
          epochs=1,
          callbacks=[model_checkpoint])

1875/1875 [==============================] - 27s 14ms/step - loss: 0.2202 - accuracy: 0.9350
<keras.callbacks.History at 0x7f534c8dbed0>

Luego, evalúe el modelo usando tf.keras.utils.SidecarEvaluator . En la capacitación real, se recomienda usar un trabajo separado para realizar la evaluación para liberar recursos de los trabajadores para la capacitación.

data = tf.data.Dataset.from_tensor_slices((x_test, y_test))
data = data.batch(64)

tf.keras.utils.SidecarEvaluator(
    model=model,
    data=data,
    checkpoint_dir=log_dir,
    max_evaluations=1
).start()

INFO:tensorflow:Waiting for new checkpoint at /tmp/tmpl6y5s71p
INFO:tensorflow:Found new checkpoint at /tmp/tmpl6y5s71p/ckpt-1
INFO:tensorflow:Evaluation starts: Model weights loaded from latest checkpoint file /tmp/tmpl6y5s71p/ckpt-1
157/157 - 2s - loss: 0.1006 - accuracy: 0.9697 - 2s/epoch - 10ms/step
INFO:tensorflow:End of evaluation. Metrics: loss=0.10060054063796997 accuracy=0.9696999788284302
INFO:tensorflow:Last checkpoint evaluated. SidecarEvaluator stops.

Próximos pasos