 TensorFlow.org'da görüntüleyin |  Google Colab'da çalıştırın |  GitHub'da görüntüle |  Not defterini indir |  TF Hub modeline bakın |
genel bakış
Adil Göstergeler üzerine inşa araçları paketidir TensorFlow Modeli Analizi (TFMA) ürün boru hatları adalet ölçütlerinin düzenli değerlendirilmesini sağlar. TFMA, hem TensorFlow hem de TensorFlow olmayan makine öğrenimi modellerini değerlendirmek için bir kitaplıktır. Modellerinizi dağıtılmış bir şekilde büyük miktarda veri üzerinde değerlendirmenize, farklı veri dilimleri üzerinde grafik içi ve diğer metrikleri hesaplamanıza ve bunları not defterlerinde görselleştirmenize olanak tanır.
Adil Göstergeler ile paketlenir TensorFlow Veri Doğrulama (TFDV) ve Ne-Eğer Aracı . Adalet Göstergelerini kullanmak şunları yapmanızı sağlar:
- Tanımlanmış kullanıcı grupları arasında dilimlenmiş model performansını değerlendirin
- Güven aralıkları ve çoklu eşiklerde değerlendirmeler ile sonuçlar hakkında güven kazanın
- Veri kümelerinin dağılımını değerlendirin
- Temel nedenleri ve iyileştirme fırsatlarını keşfetmek için bireysel dilimlere derinlemesine dalın
Bu defterin içinde, kullanmakta tren bir modelde adalet sorunları gidermek için Adil Göstergelerini kullanacak Sivil veri kümesi Yorumlar . Bunu izle videoyu gerçek dünya senaryosunda bu konuda daha ayrıntılı bilgi ve bağlam için hangi dayanmaktadır ayrıca Adil Göstergeleri oluşturmak için birincil motivasyon biridir.
veri kümesi
Bu defter, sen çalışacak Sivil veri kümesi Yorumlar yaklaşık 2 milyon kamu yorumlar tarafından kamuoyuna açıkladı Civil platformu Yorumlar süregelen araştırma için 2017 yılında. Bu çaba sponsor oldu Jigsaw sıra istenmeyen modeli önyargı minimize olarak toksik yorumlarla sınıflandırmak yardımına Kaggle üzerinde yarışmalar ağırladık.
Veri kümesindeki her bir metin yorumu, bir toksiklik etiketine sahiptir; etiket, yorum toksik ise 1 ve yorum toksik değilse 0 olur. Veriler içinde, bir yorum alt kümesi, cinsiyet, cinsel yönelim, din ve ırk veya etnik köken kategorileri dahil olmak üzere çeşitli kimlik özellikleriyle etiketlenmiştir.
Kurmak
Yükleme fairness-indicators ve witwidget .
pip install -q -U pip==20.2pip install -q fairness-indicatorspip install -q witwidget
Yüklemeden sonra Colab çalışma zamanını yeniden başlatmanız gerekir. Seç Runtime> CoLab menüsünden Yeniden çalışma zamanı.
İlk önce çalışma zamanını yeniden başlatmadan bu öğreticinin geri kalanına devam etmeyin.
Diğer tüm gerekli kitaplıkları içe aktarın.
import os
import tempfile
import apache_beam as beam
import numpy as np
import pandas as pd
from datetime import datetime
import pprint
from google.protobuf import text_format
import tensorflow_hub as hub
import tensorflow as tf
import tensorflow_model_analysis as tfma
import tensorflow_data_validation as tfdv
from tfx_bsl.tfxio import tensor_adapter
from tfx_bsl.tfxio import tf_example_record
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from tensorflow_model_analysis.addons.fairness.view import widget_view
from fairness_indicators.tutorial_utils import util
from witwidget.notebook.visualization import WitConfigBuilder
from witwidget.notebook.visualization import WitWidget
from tensorflow_metadata.proto.v0 import schema_pb2
Verileri indirin ve analiz edin
Varsayılan olarak, bu not defteri bu veri kümesinin önceden işlenmiş bir sürümünü indirir, ancak orijinal veri kümesini kullanabilir ve isterseniz işleme adımlarını yeniden çalıştırabilirsiniz. Orijinal veri setinde, her yorum, bir yorumun belirli bir kimliğe karşılık geldiğine inanan değerlendiricilerin yüzdesiyle etiketlenir. Örneğin, bir yorum şu şekilde etiketlenebilir: { erkek: 0.3, kadın: 1.0, transgender: 0.0, heteroseksüel: 0.8, homoseksüel_gay_or_lesbian: 1.0 } İşlem adımı, kimliği kategoriye göre gruplandırır (cinsiyet, cinsel_oryantasyon, vb.) ve kaldırır 0,5'ten düşük bir puana sahip kimlikler. Dolayısıyla yukarıdaki örnek şuna dönüştürülebilir: bir yorumun belirli bir kimliğe karşılık geldiğine inanan değerlendiriciler. Örneğin, yorum şu şekilde etiketlenir: { cinsiyet: [kadın], cinsel_oryantasyon: [heteroseksüel, homoseksüel_gay_or_lesbian] }
download_original_data = False
if download_original_data:
train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')
# The identity terms list will be grouped together by their categories
# (see 'IDENTITY_COLUMNS') on threshould 0.5. Only the identity term column,
# text column and label column will be kept after processing.
train_tf_file = util.convert_comments_data(train_tf_file)
validate_tf_file = util.convert_comments_data(validate_tf_file)
else:
train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')
Verileri analiz etmek ve eksik değerler ve veri dengesizlikleri gibi adalet farklılıklarına yol açabilecek olası sorunları bulmak için TFDV'yi kullanın.
stats = tfdv.generate_statistics_from_tfrecord(data_location=train_tf_file)
tfdv.visualize_statistics(stats)
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)` WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)`
TFDV, verilerde önyargılı model sonuçlarına yol açabilecek bazı önemli dengesizlikler olduğunu göstermektedir.
Toksisite etiketi (model tarafından tahmin edilen değer) dengesiz. Eğitim setindeki örneklerin sadece %8'i toksiktir, bu da bir sınıflandırıcının tüm yorumların toksik olmadığını tahmin ederek %92 doğruluk elde edebileceği anlamına gelir.
Kimlik terimleriyle ilgili alanlarda, 1,08 milyon (%0,61) eğitim örneğinden yalnızca 6,6 bin'i eşcinsellikle ilgilidir ve biseksüellikle ilgili olanlar daha da nadirdir. Bu, eğitim verilerinin eksikliği nedeniyle bu dilimlerdeki performansın düşebileceğini gösterir.
Verileri hazırlayın
Verileri ayrıştırmak için bir özellik haritası tanımlayın. Her örnek bir etiket, yorum metni olacak ve kimlik özellikleri sexual orientation , gender , religion , race ve disability metin ile ilişkilidir.
BASE_DIR = tempfile.gettempdir()
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'
FEATURE_MAP = {
# Label:
LABEL: tf.io.FixedLenFeature([], tf.float32),
# Text:
TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),
# Identities:
'sexual_orientation':tf.io.VarLenFeature(tf.string),
'gender':tf.io.VarLenFeature(tf.string),
'religion':tf.io.VarLenFeature(tf.string),
'race':tf.io.VarLenFeature(tf.string),
'disability':tf.io.VarLenFeature(tf.string),
}
Ardından, modele veri beslemek için bir giriş işlevi ayarlayın. Her örneğe bir ağırlık sütunu ekleyin ve TFDV tarafından tanımlanan sınıf dengesizliğini hesaba katmak için toksik örnekleri artırın. Eğitim sırasında modele yalnızca yorumlar beslendiğinden, değerlendirme aşamasında yalnızca kimlik özelliklerini kullanın.
def train_input_fn():
def parse_function(serialized):
parsed_example = tf.io.parse_single_example(
serialized=serialized, features=FEATURE_MAP)
# Adds a weight column to deal with unbalanced classes.
parsed_example['weight'] = tf.add(parsed_example[LABEL], 0.1)
return (parsed_example,
parsed_example[LABEL])
train_dataset = tf.data.TFRecordDataset(
filenames=[train_tf_file]).map(parse_function).batch(512)
return train_dataset
Modeli eğit
Veriler üzerinde bir derin öğrenme modeli oluşturun ve eğitin.
model_dir = os.path.join(BASE_DIR, 'train', datetime.now().strftime(
"%Y%m%d-%H%M%S"))
embedded_text_feature_column = hub.text_embedding_column(
key=TEXT_FEATURE,
module_spec='https://tfhub.dev/google/nnlm-en-dim128/1')
classifier = tf.estimator.DNNClassifier(
hidden_units=[500, 100],
weight_column='weight',
feature_columns=[embedded_text_feature_column],
optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.003),
loss_reduction=tf.losses.Reduction.SUM,
n_classes=2,
model_dir=model_dir)
classifier.train(input_fn=train_input_fn, steps=1000)
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210923-205025', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210923-205025', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2021-09-23 20:50:26.540914: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 808. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/head/base_head.py:512: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/head/base_head.py:512: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/keras/optimizer_v2/adagrad.py:84: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/keras/optimizer_v2/adagrad.py:84: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210923-205025/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210923-205025/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 59.34932, step = 0
INFO:tensorflow:loss = 59.34932, step = 0
INFO:tensorflow:global_step/sec: 108.435
INFO:tensorflow:global_step/sec: 108.435
INFO:tensorflow:loss = 56.416668, step = 100 (0.924 sec)
INFO:tensorflow:loss = 56.416668, step = 100 (0.924 sec)
INFO:tensorflow:global_step/sec: 116.367
INFO:tensorflow:global_step/sec: 116.367
INFO:tensorflow:loss = 47.250374, step = 200 (0.859 sec)
INFO:tensorflow:loss = 47.250374, step = 200 (0.859 sec)
INFO:tensorflow:global_step/sec: 116.333
INFO:tensorflow:global_step/sec: 116.333
INFO:tensorflow:loss = 55.81682, step = 300 (0.860 sec)
INFO:tensorflow:loss = 55.81682, step = 300 (0.860 sec)
INFO:tensorflow:global_step/sec: 116.844
INFO:tensorflow:global_step/sec: 116.844
INFO:tensorflow:loss = 55.814293, step = 400 (0.856 sec)
INFO:tensorflow:loss = 55.814293, step = 400 (0.856 sec)
INFO:tensorflow:global_step/sec: 114.434
INFO:tensorflow:global_step/sec: 114.434
INFO:tensorflow:loss = 41.805046, step = 500 (0.874 sec)
INFO:tensorflow:loss = 41.805046, step = 500 (0.874 sec)
INFO:tensorflow:global_step/sec: 115.693
INFO:tensorflow:global_step/sec: 115.693
INFO:tensorflow:loss = 45.53726, step = 600 (0.864 sec)
INFO:tensorflow:loss = 45.53726, step = 600 (0.864 sec)
INFO:tensorflow:global_step/sec: 115.772
INFO:tensorflow:global_step/sec: 115.772
INFO:tensorflow:loss = 51.17028, step = 700 (0.864 sec)
INFO:tensorflow:loss = 51.17028, step = 700 (0.864 sec)
INFO:tensorflow:global_step/sec: 116.131
INFO:tensorflow:global_step/sec: 116.131
INFO:tensorflow:loss = 47.696205, step = 800 (0.861 sec)
INFO:tensorflow:loss = 47.696205, step = 800 (0.861 sec)
INFO:tensorflow:global_step/sec: 115.609
INFO:tensorflow:global_step/sec: 115.609
INFO:tensorflow:loss = 47.800926, step = 900 (0.865 sec)
INFO:tensorflow:loss = 47.800926, step = 900 (0.865 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210923-205025/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210923-205025/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 50.67367.
INFO:tensorflow:Loss for final step: 50.67367.
<tensorflow_estimator.python.estimator.canned.dnn.DNNClassifierV2 at 0x7f113351ebd0>
Modeli analiz edin
Eğitilmiş modeli elde ettikten sonra, TFMA ve Adalet Göstergelerini kullanarak adalet metriklerini hesaplamak için onu analiz edin. Bir açıdan modeli ihraç ederek başlayın SavedModel .
Kaydedilen Modeli Dışa Aktar
def eval_input_receiver_fn():
serialized_tf_example = tf.compat.v1.placeholder(
dtype=tf.string, shape=[None], name='input_example_placeholder')
# This *must* be a dictionary containing a single key 'examples', which
# points to the input placeholder.
receiver_tensors = {'examples': serialized_tf_example}
features = tf.io.parse_example(serialized_tf_example, FEATURE_MAP)
features['weight'] = tf.ones_like(features[LABEL])
return tfma.export.EvalInputReceiver(
features=features,
receiver_tensors=receiver_tensors,
labels=features[LABEL])
tfma_export_dir = tfma.export.export_eval_savedmodel(
estimator=classifier,
export_dir_base=os.path.join(BASE_DIR, 'tfma_eval_model'),
eval_input_receiver_fn=eval_input_receiver_fn)
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2021-09-23 20:50:39.359797: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 808. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20210923-205025/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20210923-205025/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1632430239/assets INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1632430239/assets INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1632430239/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1632430239/saved_model.pb
Hesaplama Adaleti Metrikleri
Sağdaki paneldeki açılır menüyü kullanarak metriklerin hesaplanacağı kimliği ve güven aralıklarıyla çalıştırılıp çalıştırılmayacağını seçin.
Adalet Göstergeleri Hesaplama Seçenekleri
tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')
slice_selection = 'sexual_orientation'
print(f'Slice selection: {slice_selection}')
compute_confidence_intervals = False
print(f'Compute confidence intervals: {compute_confidence_intervals}')
# Define slices that you want the evaluation to run on.
eval_config_pbtxt = """
model_specs {
label_key: "%s"
}
metrics_specs {
metrics {
class_name: "FairnessIndicators"
config: '{ "thresholds": [0.1, 0.3, 0.5, 0.7, 0.9] }'
}
}
slicing_specs {} # overall slice
slicing_specs {
feature_keys: ["%s"]
}
options {
compute_confidence_intervals { value: %s }
disabled_outputs { values: "analysis" }
}
""" % (LABEL, slice_selection, compute_confidence_intervals)
eval_config = text_format.Parse(eval_config_pbtxt, tfma.EvalConfig())
eval_shared_model = tfma.default_eval_shared_model(
eval_saved_model_path=tfma_export_dir)
schema = text_format.Parse(
"""
tensor_representation_group {
key: ""
value {
tensor_representation {
key: "comment_text"
value {
dense_tensor {
column_name: "comment_text"
shape {}
}
}
}
}
}
feature {
name: "comment_text"
type: BYTES
}
feature {
name: "toxicity"
type: FLOAT
}
feature {
name: "sexual_orientation"
type: BYTES
}
feature {
name: "gender"
type: BYTES
}
feature {
name: "religion"
type: BYTES
}
feature {
name: "race"
type: BYTES
}
feature {
name: "disability"
type: BYTES
}
""", schema_pb2.Schema())
tfxio = tf_example_record.TFExampleRecord(
file_pattern=validate_tf_file,
schema=schema,
raw_record_column_name=tfma.ARROW_INPUT_COLUMN)
tensor_adapter_config = tensor_adapter.TensorAdapterConfig(
arrow_schema=tfxio.ArrowSchema(),
tensor_representations=tfxio.TensorRepresentations())
with beam.Pipeline() as pipeline:
(pipeline
| 'ReadFromTFRecordToArrow' >> tfxio.BeamSource()
| 'ExtractEvaluateAndWriteResults' >> tfma.ExtractEvaluateAndWriteResults(
eval_config=eval_config,
eval_shared_model=eval_shared_model,
output_path=tfma_eval_result_path,
tensor_adapter_config=tensor_adapter_config))
eval_result = tfma.load_eval_result(output_path=tfma_eval_result_path)
Slice selection: sexual_orientation Compute confidence intervals: False WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1632430239/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1632430239/variables/variables WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching:
What-if Aracını kullanarak verileri görselleştirin
Bu bölümde, verileri mikro düzeyde keşfetmek ve işlemek için What-If Aracı'nın etkileşimli görsel arayüzünü kullanacaksınız.
Sağ paneldeki dağılım grafiğindeki her nokta, araca yüklenen alt kümedeki örneklerden birini temsil eder. Sol panelde bu özel örnekle ilgili ayrıntıları görmek için noktalardan birine tıklayın. Yorum metni, temel doğruluk toksisitesi ve uygulanabilir kimlikler gösterilir. Bu sol panelin alt kısmında, az önce eğittiğiniz modelden çıkarım sonuçlarını görürsünüz.
Örneğin metnini değiştirin ve değişiklikler değişime algılanan toksisite tahminini neden nasıl ardından görünümüne Çalıştır çıkarsama butonuna tıklayın.
DEFAULT_MAX_EXAMPLES = 1000
# Load 100000 examples in memory. When first rendered,
# What-If Tool should only display 1000 of these due to browser constraints.
def wit_dataset(file, num_examples=100000):
dataset = tf.data.TFRecordDataset(
filenames=[file]).take(num_examples)
return [tf.train.Example.FromString(d.numpy()) for d in dataset]
wit_data = wit_dataset(train_tf_file)
config_builder = WitConfigBuilder(wit_data[:DEFAULT_MAX_EXAMPLES]).set_estimator_and_feature_spec(
classifier, FEATURE_MAP).set_label_vocab(['non-toxicity', LABEL]).set_target_feature(LABEL)
wit = WitWidget(config_builder)
Hakkaniyet Göstergeleri Oluşturun
Dışa aktarılan değerlendirme sonuçlarıyla Adalet Göstergeleri pencere öğesini oluşturun.
Aşağıda, seçilen metriklerdeki verilerin her bir diliminin performansını gösteren çubuk grafikler göreceksiniz. Görselleştirmenin üst kısmındaki açılır menüleri kullanarak temel karşılaştırma dilimini ve görüntülenen eşik(ler)i ayarlayabilirsiniz.
Adalet Göstergesi widget'ı, yukarıda oluşturulan What-If Aracı ile entegre edilmiştir. Çubuk grafikte verilerin bir dilimini seçerseniz, What-If Aracı size seçilen dilimden örnekler gösterecek şekilde güncellenecektir. Yukarıda Ne-Eğer Aracı veri Çoğul, toksisitesi için Renk By değiştirerek çalıştığınızda. Bu, örneklerin toksisite dengesini dilimlere göre görsel olarak anlamanızı sağlayabilir.
event_handlers={'slice-selected':
wit.create_selection_callback(wit_data, DEFAULT_MAX_EXAMPLES)}
widget_view.render_fairness_indicator(eval_result=eval_result,
slicing_column=slice_selection,
event_handlers=event_handlers
)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'prediction/…
Bu özel veri seti ve görevle, belirli kimlikler için sistematik olarak daha yüksek yanlış pozitif ve yanlış negatif oranları olumsuz sonuçlara yol açabilir. Örneğin, bir içerik denetleme sisteminde, belirli bir grup için genelden daha yüksek bir yanlış pozitif oranı, bu seslerin susturulmasına neden olabilir. Bu nedenle, modelleri geliştirirken ve iyileştirirken bu tür kriterleri düzenli olarak değerlendirmek ve olası sorunları aydınlatmaya yardımcı olmak için Adillik Göstergeleri, TFDV ve WIT gibi araçları kullanmak önemlidir. Adaletle ilgili sorunları belirledikten sonra, düşük performans gösteren gruplarda performansı artırmak için yeni veri kaynakları, veri dengeleme veya diğer teknikleri deneyebilirsiniz.
Bkz burada Adil Göstergelerini nasıl kullanılacağına dair daha fazla bilgi ve rehberlik için.
Adalet değerlendirme sonuçlarını kullanın
eval_result nesne, yukarıda render render_fairness_indicator() , kendi programlarına TFMA sonuçlarını okumaya kaldıraç ki kendi API vardır.
Değerlendirilen dilimleri ve metrikleri alın
Kullanım get_slice_names() ve get_metric_names() sırasıyla değerlendirildi dilimleri ve ölçümleri almak için.
pp = pprint.PrettyPrinter()
print("Slices:")
pp.pprint(eval_result.get_slice_names())
print("\nMetrics:")
pp.pprint(eval_result.get_metric_names())
Slices:
[(),
(('sexual_orientation', 'homosexual_gay_or_lesbian'),),
(('sexual_orientation', 'heterosexual'),),
(('sexual_orientation', 'bisexual'),),
(('sexual_orientation', 'other_sexual_orientation'),)]
Metrics:
['fairness_indicators_metrics/negative_rate@0.1',
'fairness_indicators_metrics/positive_rate@0.7',
'fairness_indicators_metrics/false_discovery_rate@0.9',
'fairness_indicators_metrics/false_negative_rate@0.3',
'fairness_indicators_metrics/false_omission_rate@0.1',
'accuracy',
'fairness_indicators_metrics/false_discovery_rate@0.7',
'fairness_indicators_metrics/false_negative_rate@0.7',
'label/mean',
'fairness_indicators_metrics/true_positive_rate@0.5',
'fairness_indicators_metrics/false_positive_rate@0.1',
'recall',
'fairness_indicators_metrics/false_omission_rate@0.7',
'fairness_indicators_metrics/false_positive_rate@0.7',
'auc_precision_recall',
'fairness_indicators_metrics/negative_rate@0.7',
'fairness_indicators_metrics/negative_rate@0.3',
'fairness_indicators_metrics/false_discovery_rate@0.3',
'fairness_indicators_metrics/true_negative_rate@0.9',
'fairness_indicators_metrics/false_omission_rate@0.3',
'fairness_indicators_metrics/false_negative_rate@0.1',
'fairness_indicators_metrics/true_negative_rate@0.3',
'fairness_indicators_metrics/true_positive_rate@0.7',
'fairness_indicators_metrics/false_positive_rate@0.3',
'fairness_indicators_metrics/true_positive_rate@0.1',
'fairness_indicators_metrics/true_positive_rate@0.9',
'fairness_indicators_metrics/false_negative_rate@0.9',
'fairness_indicators_metrics/positive_rate@0.5',
'fairness_indicators_metrics/positive_rate@0.9',
'fairness_indicators_metrics/negative_rate@0.9',
'fairness_indicators_metrics/true_negative_rate@0.1',
'fairness_indicators_metrics/false_omission_rate@0.5',
'post_export_metrics/example_count',
'fairness_indicators_metrics/false_omission_rate@0.9',
'fairness_indicators_metrics/negative_rate@0.5',
'fairness_indicators_metrics/false_positive_rate@0.5',
'fairness_indicators_metrics/positive_rate@0.3',
'prediction/mean',
'accuracy_baseline',
'fairness_indicators_metrics/true_negative_rate@0.5',
'fairness_indicators_metrics/false_discovery_rate@0.5',
'fairness_indicators_metrics/false_discovery_rate@0.1',
'precision',
'fairness_indicators_metrics/false_positive_rate@0.9',
'fairness_indicators_metrics/true_positive_rate@0.3',
'auc',
'average_loss',
'fairness_indicators_metrics/positive_rate@0.1',
'fairness_indicators_metrics/false_negative_rate@0.5',
'fairness_indicators_metrics/true_negative_rate@0.7']
Kullanım get_metrics_for_slice() için bir sözlük haritalama metrik adları gibi belli bir bölümü için ölçümleri almak için metrik değerler .
baseline_slice = ()
heterosexual_slice = (('sexual_orientation', 'heterosexual'),)
print("Baseline metric values:")
pp.pprint(eval_result.get_metrics_for_slice(baseline_slice))
print("\nHeterosexual metric values:")
pp.pprint(eval_result.get_metrics_for_slice(heterosexual_slice))
Baseline metric values:
{'accuracy': {'doubleValue': 0.7174859642982483},
'accuracy_baseline': {'doubleValue': 0.9198060631752014},
'auc': {'doubleValue': 0.796409547328949},
'auc_precision_recall': {'doubleValue': 0.3000231087207794},
'average_loss': {'doubleValue': 0.5615971088409424},
'fairness_indicators_metrics/false_discovery_rate@0.1': {'doubleValue': 0.9139404145348933},
'fairness_indicators_metrics/false_discovery_rate@0.3': {'doubleValue': 0.8796606156634021},
'fairness_indicators_metrics/false_discovery_rate@0.5': {'doubleValue': 0.816806708107944},
'fairness_indicators_metrics/false_discovery_rate@0.7': {'doubleValue': 0.7090802784427505},
'fairness_indicators_metrics/false_discovery_rate@0.9': {'doubleValue': 0.4814937210839392},
'fairness_indicators_metrics/false_negative_rate@0.1': {'doubleValue': 0.006079867348348763},
'fairness_indicators_metrics/false_negative_rate@0.3': {'doubleValue': 0.08696628437197734},
'fairness_indicators_metrics/false_negative_rate@0.5': {'doubleValue': 0.2705713693519414},
'fairness_indicators_metrics/false_negative_rate@0.7': {'doubleValue': 0.5445108470360647},
'fairness_indicators_metrics/false_negative_rate@0.9': {'doubleValue': 0.891598728755009},
'fairness_indicators_metrics/false_omission_rate@0.1': {'doubleValue': 0.006604499315158452},
'fairness_indicators_metrics/false_omission_rate@0.3': {'doubleValue': 0.017811407791031682},
'fairness_indicators_metrics/false_omission_rate@0.5': {'doubleValue': 0.03187681488249431},
'fairness_indicators_metrics/false_omission_rate@0.7': {'doubleValue': 0.04993640137936933},
'fairness_indicators_metrics/false_omission_rate@0.9': {'doubleValue': 0.07271999842219298},
'fairness_indicators_metrics/false_positive_rate@0.1': {'doubleValue': 0.9202700382800194},
'fairness_indicators_metrics/false_positive_rate@0.3': {'doubleValue': 0.5818879187535954},
'fairness_indicators_metrics/false_positive_rate@0.5': {'doubleValue': 0.28355525303665063},
'fairness_indicators_metrics/false_positive_rate@0.7': {'doubleValue': 0.09679333307231039},
'fairness_indicators_metrics/false_positive_rate@0.9': {'doubleValue': 0.00877639469079322},
'fairness_indicators_metrics/negative_rate@0.1': {'doubleValue': 0.07382367199944595},
'fairness_indicators_metrics/negative_rate@0.3': {'doubleValue': 0.39155620195304386},
'fairness_indicators_metrics/negative_rate@0.5': {'doubleValue': 0.6806884133250225},
'fairness_indicators_metrics/negative_rate@0.7': {'doubleValue': 0.8744414433132488},
'fairness_indicators_metrics/negative_rate@0.9': {'doubleValue': 0.9832342960038783},
'fairness_indicators_metrics/positive_rate@0.1': {'doubleValue': 0.926176328000554},
'fairness_indicators_metrics/positive_rate@0.3': {'doubleValue': 0.6084437980469561},
'fairness_indicators_metrics/positive_rate@0.5': {'doubleValue': 0.3193115866749775},
'fairness_indicators_metrics/positive_rate@0.7': {'doubleValue': 0.12555855668675117},
'fairness_indicators_metrics/positive_rate@0.9': {'doubleValue': 0.016765703996121616},
'fairness_indicators_metrics/true_negative_rate@0.1': {'doubleValue': 0.0797299617199806},
'fairness_indicators_metrics/true_negative_rate@0.3': {'doubleValue': 0.41811208124640464},
'fairness_indicators_metrics/true_negative_rate@0.5': {'doubleValue': 0.7164447469633494},
'fairness_indicators_metrics/true_negative_rate@0.7': {'doubleValue': 0.9032066669276896},
'fairness_indicators_metrics/true_negative_rate@0.9': {'doubleValue': 0.9912236053092068},
'fairness_indicators_metrics/true_positive_rate@0.1': {'doubleValue': 0.9939201326516512},
'fairness_indicators_metrics/true_positive_rate@0.3': {'doubleValue': 0.9130337156280227},
'fairness_indicators_metrics/true_positive_rate@0.5': {'doubleValue': 0.7294286306480586},
'fairness_indicators_metrics/true_positive_rate@0.7': {'doubleValue': 0.45548915296393533},
'fairness_indicators_metrics/true_positive_rate@0.9': {'doubleValue': 0.10840127124499102},
'label/mean': {'doubleValue': 0.08019392192363739},
'post_export_metrics/example_count': {'doubleValue': 721950.0},
'precision': {'doubleValue': 0.18319329619407654},
'prediction/mean': {'doubleValue': 0.3998037576675415},
'recall': {'doubleValue': 0.7294286489486694} }
Heterosexual metric values:
{'accuracy': {'doubleValue': 0.5203251838684082},
'accuracy_baseline': {'doubleValue': 0.7601625919342041},
'auc': {'doubleValue': 0.6672822833061218},
'auc_precision_recall': {'doubleValue': 0.4065391719341278},
'average_loss': {'doubleValue': 0.8273133039474487},
'fairness_indicators_metrics/false_discovery_rate@0.1': {'doubleValue': 0.7541666666666667},
'fairness_indicators_metrics/false_discovery_rate@0.3': {'doubleValue': 0.7272727272727273},
'fairness_indicators_metrics/false_discovery_rate@0.5': {'doubleValue': 0.7062937062937062},
'fairness_indicators_metrics/false_discovery_rate@0.7': {'doubleValue': 0.655367231638418},
'fairness_indicators_metrics/false_discovery_rate@0.9': {'doubleValue': 0.4473684210526316},
'fairness_indicators_metrics/false_negative_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.3': {'doubleValue': 0.0847457627118644},
'fairness_indicators_metrics/false_negative_rate@0.5': {'doubleValue': 0.288135593220339},
'fairness_indicators_metrics/false_negative_rate@0.7': {'doubleValue': 0.4830508474576271},
'fairness_indicators_metrics/false_negative_rate@0.9': {'doubleValue': 0.8220338983050848},
'fairness_indicators_metrics/false_omission_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_omission_rate@0.3': {'doubleValue': 0.10416666666666667},
'fairness_indicators_metrics/false_omission_rate@0.5': {'doubleValue': 0.1650485436893204},
'fairness_indicators_metrics/false_omission_rate@0.7': {'doubleValue': 0.18095238095238095},
'fairness_indicators_metrics/false_omission_rate@0.9': {'doubleValue': 0.21365638766519823},
'fairness_indicators_metrics/false_positive_rate@0.1': {'doubleValue': 0.9679144385026738},
'fairness_indicators_metrics/false_positive_rate@0.3': {'doubleValue': 0.7700534759358288},
'fairness_indicators_metrics/false_positive_rate@0.5': {'doubleValue': 0.5401069518716578},
'fairness_indicators_metrics/false_positive_rate@0.7': {'doubleValue': 0.31016042780748665},
'fairness_indicators_metrics/false_positive_rate@0.9': {'doubleValue': 0.045454545454545456},
'fairness_indicators_metrics/negative_rate@0.1': {'doubleValue': 0.024390243902439025},
'fairness_indicators_metrics/negative_rate@0.3': {'doubleValue': 0.1951219512195122},
'fairness_indicators_metrics/negative_rate@0.5': {'doubleValue': 0.4186991869918699},
'fairness_indicators_metrics/negative_rate@0.7': {'doubleValue': 0.6402439024390244},
'fairness_indicators_metrics/negative_rate@0.9': {'doubleValue': 0.9227642276422764},
'fairness_indicators_metrics/positive_rate@0.1': {'doubleValue': 0.975609756097561},
'fairness_indicators_metrics/positive_rate@0.3': {'doubleValue': 0.8048780487804879},
'fairness_indicators_metrics/positive_rate@0.5': {'doubleValue': 0.5813008130081301},
'fairness_indicators_metrics/positive_rate@0.7': {'doubleValue': 0.3597560975609756},
'fairness_indicators_metrics/positive_rate@0.9': {'doubleValue': 0.07723577235772358},
'fairness_indicators_metrics/true_negative_rate@0.1': {'doubleValue': 0.03208556149732621},
'fairness_indicators_metrics/true_negative_rate@0.3': {'doubleValue': 0.22994652406417113},
'fairness_indicators_metrics/true_negative_rate@0.5': {'doubleValue': 0.45989304812834225},
'fairness_indicators_metrics/true_negative_rate@0.7': {'doubleValue': 0.6898395721925134},
'fairness_indicators_metrics/true_negative_rate@0.9': {'doubleValue': 0.9545454545454546},
'fairness_indicators_metrics/true_positive_rate@0.1': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.3': {'doubleValue': 0.9152542372881356},
'fairness_indicators_metrics/true_positive_rate@0.5': {'doubleValue': 0.711864406779661},
'fairness_indicators_metrics/true_positive_rate@0.7': {'doubleValue': 0.5169491525423728},
'fairness_indicators_metrics/true_positive_rate@0.9': {'doubleValue': 0.17796610169491525},
'label/mean': {'doubleValue': 0.2398373931646347},
'post_export_metrics/example_count': {'doubleValue': 492.0},
'precision': {'doubleValue': 0.2937062978744507},
'prediction/mean': {'doubleValue': 0.5578703880310059},
'recall': {'doubleValue': 0.7118644118309021} }
Kullanım get_metrics_for_all_slices() çalıştırmakta elde ettiğimiz karşılık gelen ölçümlere her dilim sözlüğe bir sözlük haritalama gibi tüm dilimleri için ölçümleri almak için get_metrics_for_slice() üzerinde.
pp.pprint(eval_result.get_metrics_for_all_slices())
{(): {'accuracy': {'doubleValue': 0.7174859642982483},
'accuracy_baseline': {'doubleValue': 0.9198060631752014},
'auc': {'doubleValue': 0.796409547328949},
'auc_precision_recall': {'doubleValue': 0.3000231087207794},
'average_loss': {'doubleValue': 0.5615971088409424},
'fairness_indicators_metrics/false_discovery_rate@0.1': {'doubleValue': 0.9139404145348933},
'fairness_indicators_metrics/false_discovery_rate@0.3': {'doubleValue': 0.8796606156634021},
'fairness_indicators_metrics/false_discovery_rate@0.5': {'doubleValue': 0.816806708107944},
'fairness_indicators_metrics/false_discovery_rate@0.7': {'doubleValue': 0.7090802784427505},
'fairness_indicators_metrics/false_discovery_rate@0.9': {'doubleValue': 0.4814937210839392},
'fairness_indicators_metrics/false_negative_rate@0.1': {'doubleValue': 0.006079867348348763},
'fairness_indicators_metrics/false_negative_rate@0.3': {'doubleValue': 0.08696628437197734},
'fairness_indicators_metrics/false_negative_rate@0.5': {'doubleValue': 0.2705713693519414},
'fairness_indicators_metrics/false_negative_rate@0.7': {'doubleValue': 0.5445108470360647},
'fairness_indicators_metrics/false_negative_rate@0.9': {'doubleValue': 0.891598728755009},
'fairness_indicators_metrics/false_omission_rate@0.1': {'doubleValue': 0.006604499315158452},
'fairness_indicators_metrics/false_omission_rate@0.3': {'doubleValue': 0.017811407791031682},
'fairness_indicators_metrics/false_omission_rate@0.5': {'doubleValue': 0.03187681488249431},
'fairness_indicators_metrics/false_omission_rate@0.7': {'doubleValue': 0.04993640137936933},
'fairness_indicators_metrics/false_omission_rate@0.9': {'doubleValue': 0.07271999842219298},
'fairness_indicators_metrics/false_positive_rate@0.1': {'doubleValue': 0.9202700382800194},
'fairness_indicators_metrics/false_positive_rate@0.3': {'doubleValue': 0.5818879187535954},
'fairness_indicators_metrics/false_positive_rate@0.5': {'doubleValue': 0.28355525303665063},
'fairness_indicators_metrics/false_positive_rate@0.7': {'doubleValue': 0.09679333307231039},
'fairness_indicators_metrics/false_positive_rate@0.9': {'doubleValue': 0.00877639469079322},
'fairness_indicators_metrics/negative_rate@0.1': {'doubleValue': 0.07382367199944595},
'fairness_indicators_metrics/negative_rate@0.3': {'doubleValue': 0.39155620195304386},
'fairness_indicators_metrics/negative_rate@0.5': {'doubleValue': 0.6806884133250225},
'fairness_indicators_metrics/negative_rate@0.7': {'doubleValue': 0.8744414433132488},
'fairness_indicators_metrics/negative_rate@0.9': {'doubleValue': 0.9832342960038783},
'fairness_indicators_metrics/positive_rate@0.1': {'doubleValue': 0.926176328000554},
'fairness_indicators_metrics/positive_rate@0.3': {'doubleValue': 0.6084437980469561},
'fairness_indicators_metrics/positive_rate@0.5': {'doubleValue': 0.3193115866749775},
'fairness_indicators_metrics/positive_rate@0.7': {'doubleValue': 0.12555855668675117},
'fairness_indicators_metrics/positive_rate@0.9': {'doubleValue': 0.016765703996121616},
'fairness_indicators_metrics/true_negative_rate@0.1': {'doubleValue': 0.0797299617199806},
'fairness_indicators_metrics/true_negative_rate@0.3': {'doubleValue': 0.41811208124640464},
'fairness_indicators_metrics/true_negative_rate@0.5': {'doubleValue': 0.7164447469633494},
'fairness_indicators_metrics/true_negative_rate@0.7': {'doubleValue': 0.9032066669276896},
'fairness_indicators_metrics/true_negative_rate@0.9': {'doubleValue': 0.9912236053092068},
'fairness_indicators_metrics/true_positive_rate@0.1': {'doubleValue': 0.9939201326516512},
'fairness_indicators_metrics/true_positive_rate@0.3': {'doubleValue': 0.9130337156280227},
'fairness_indicators_metrics/true_positive_rate@0.5': {'doubleValue': 0.7294286306480586},
'fairness_indicators_metrics/true_positive_rate@0.7': {'doubleValue': 0.45548915296393533},
'fairness_indicators_metrics/true_positive_rate@0.9': {'doubleValue': 0.10840127124499102},
'label/mean': {'doubleValue': 0.08019392192363739},
'post_export_metrics/example_count': {'doubleValue': 721950.0},
'precision': {'doubleValue': 0.18319329619407654},
'prediction/mean': {'doubleValue': 0.3998037576675415},
'recall': {'doubleValue': 0.7294286489486694} },
(('sexual_orientation', 'bisexual'),): {'accuracy': {'doubleValue': 0.5258620977401733},
'accuracy_baseline': {'doubleValue': 0.8017241358757019},
'auc': {'doubleValue': 0.6252922415733337},
'auc_precision_recall': {'doubleValue': 0.3546649217605591},
'average_loss': {'doubleValue': 0.7461641430854797},
'fairness_indicators_metrics/false_discovery_rate@0.1': {'doubleValue': 0.7870370370370371},
'fairness_indicators_metrics/false_discovery_rate@0.3': {'doubleValue': 0.7816091954022989},
'fairness_indicators_metrics/false_discovery_rate@0.5': {'doubleValue': 0.7666666666666667},
'fairness_indicators_metrics/false_discovery_rate@0.7': {'doubleValue': 0.7037037037037037},
'fairness_indicators_metrics/false_discovery_rate@0.9': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.3': {'doubleValue': 0.17391304347826086},
'fairness_indicators_metrics/false_negative_rate@0.5': {'doubleValue': 0.391304347826087},
'fairness_indicators_metrics/false_negative_rate@0.7': {'doubleValue': 0.6521739130434783},
'fairness_indicators_metrics/false_negative_rate@0.9': {'doubleValue': 0.9130434782608695},
'fairness_indicators_metrics/false_omission_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_omission_rate@0.3': {'doubleValue': 0.13793103448275862},
'fairness_indicators_metrics/false_omission_rate@0.5': {'doubleValue': 0.16071428571428573},
'fairness_indicators_metrics/false_omission_rate@0.7': {'doubleValue': 0.16853932584269662},
'fairness_indicators_metrics/false_omission_rate@0.9': {'doubleValue': 0.18421052631578946},
'fairness_indicators_metrics/false_positive_rate@0.1': {'doubleValue': 0.9139784946236559},
'fairness_indicators_metrics/false_positive_rate@0.3': {'doubleValue': 0.7311827956989247},
'fairness_indicators_metrics/false_positive_rate@0.5': {'doubleValue': 0.4946236559139785},
'fairness_indicators_metrics/false_positive_rate@0.7': {'doubleValue': 0.20430107526881722},
'fairness_indicators_metrics/false_positive_rate@0.9': {'doubleValue': 0.0},
'fairness_indicators_metrics/negative_rate@0.1': {'doubleValue': 0.06896551724137931},
'fairness_indicators_metrics/negative_rate@0.3': {'doubleValue': 0.25},
'fairness_indicators_metrics/negative_rate@0.5': {'doubleValue': 0.4827586206896552},
'fairness_indicators_metrics/negative_rate@0.7': {'doubleValue': 0.7672413793103449},
'fairness_indicators_metrics/negative_rate@0.9': {'doubleValue': 0.9827586206896551},
'fairness_indicators_metrics/positive_rate@0.1': {'doubleValue': 0.9310344827586207},
'fairness_indicators_metrics/positive_rate@0.3': {'doubleValue': 0.75},
'fairness_indicators_metrics/positive_rate@0.5': {'doubleValue': 0.5172413793103449},
'fairness_indicators_metrics/positive_rate@0.7': {'doubleValue': 0.23275862068965517},
'fairness_indicators_metrics/positive_rate@0.9': {'doubleValue': 0.017241379310344827},
'fairness_indicators_metrics/true_negative_rate@0.1': {'doubleValue': 0.08602150537634409},
'fairness_indicators_metrics/true_negative_rate@0.3': {'doubleValue': 0.26881720430107525},
'fairness_indicators_metrics/true_negative_rate@0.5': {'doubleValue': 0.5053763440860215},
'fairness_indicators_metrics/true_negative_rate@0.7': {'doubleValue': 0.7956989247311828},
'fairness_indicators_metrics/true_negative_rate@0.9': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.1': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.3': {'doubleValue': 0.8260869565217391},
'fairness_indicators_metrics/true_positive_rate@0.5': {'doubleValue': 0.6086956521739131},
'fairness_indicators_metrics/true_positive_rate@0.7': {'doubleValue': 0.34782608695652173},
'fairness_indicators_metrics/true_positive_rate@0.9': {'doubleValue': 0.08695652173913043},
'label/mean': {'doubleValue': 0.1982758641242981},
'post_export_metrics/example_count': {'doubleValue': 116.0},
'precision': {'doubleValue': 0.23333333432674408},
'prediction/mean': {'doubleValue': 0.4908219575881958},
'recall': {'doubleValue': 0.6086956262588501} },
(('sexual_orientation', 'heterosexual'),): {'accuracy': {'doubleValue': 0.5203251838684082},
'accuracy_baseline': {'doubleValue': 0.7601625919342041},
'auc': {'doubleValue': 0.6672822833061218},
'auc_precision_recall': {'doubleValue': 0.4065391719341278},
'average_loss': {'doubleValue': 0.8273133039474487},
'fairness_indicators_metrics/false_discovery_rate@0.1': {'doubleValue': 0.7541666666666667},
'fairness_indicators_metrics/false_discovery_rate@0.3': {'doubleValue': 0.7272727272727273},
'fairness_indicators_metrics/false_discovery_rate@0.5': {'doubleValue': 0.7062937062937062},
'fairness_indicators_metrics/false_discovery_rate@0.7': {'doubleValue': 0.655367231638418},
'fairness_indicators_metrics/false_discovery_rate@0.9': {'doubleValue': 0.4473684210526316},
'fairness_indicators_metrics/false_negative_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.3': {'doubleValue': 0.0847457627118644},
'fairness_indicators_metrics/false_negative_rate@0.5': {'doubleValue': 0.288135593220339},
'fairness_indicators_metrics/false_negative_rate@0.7': {'doubleValue': 0.4830508474576271},
'fairness_indicators_metrics/false_negative_rate@0.9': {'doubleValue': 0.8220338983050848},
'fairness_indicators_metrics/false_omission_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_omission_rate@0.3': {'doubleValue': 0.10416666666666667},
'fairness_indicators_metrics/false_omission_rate@0.5': {'doubleValue': 0.1650485436893204},
'fairness_indicators_metrics/false_omission_rate@0.7': {'doubleValue': 0.18095238095238095},
'fairness_indicators_metrics/false_omission_rate@0.9': {'doubleValue': 0.21365638766519823},
'fairness_indicators_metrics/false_positive_rate@0.1': {'doubleValue': 0.9679144385026738},
'fairness_indicators_metrics/false_positive_rate@0.3': {'doubleValue': 0.7700534759358288},
'fairness_indicators_metrics/false_positive_rate@0.5': {'doubleValue': 0.5401069518716578},
'fairness_indicators_metrics/false_positive_rate@0.7': {'doubleValue': 0.31016042780748665},
'fairness_indicators_metrics/false_positive_rate@0.9': {'doubleValue': 0.045454545454545456},
'fairness_indicators_metrics/negative_rate@0.1': {'doubleValue': 0.024390243902439025},
'fairness_indicators_metrics/negative_rate@0.3': {'doubleValue': 0.1951219512195122},
'fairness_indicators_metrics/negative_rate@0.5': {'doubleValue': 0.4186991869918699},
'fairness_indicators_metrics/negative_rate@0.7': {'doubleValue': 0.6402439024390244},
'fairness_indicators_metrics/negative_rate@0.9': {'doubleValue': 0.9227642276422764},
'fairness_indicators_metrics/positive_rate@0.1': {'doubleValue': 0.975609756097561},
'fairness_indicators_metrics/positive_rate@0.3': {'doubleValue': 0.8048780487804879},
'fairness_indicators_metrics/positive_rate@0.5': {'doubleValue': 0.5813008130081301},
'fairness_indicators_metrics/positive_rate@0.7': {'doubleValue': 0.3597560975609756},
'fairness_indicators_metrics/positive_rate@0.9': {'doubleValue': 0.07723577235772358},
'fairness_indicators_metrics/true_negative_rate@0.1': {'doubleValue': 0.03208556149732621},
'fairness_indicators_metrics/true_negative_rate@0.3': {'doubleValue': 0.22994652406417113},
'fairness_indicators_metrics/true_negative_rate@0.5': {'doubleValue': 0.45989304812834225},
'fairness_indicators_metrics/true_negative_rate@0.7': {'doubleValue': 0.6898395721925134},
'fairness_indicators_metrics/true_negative_rate@0.9': {'doubleValue': 0.9545454545454546},
'fairness_indicators_metrics/true_positive_rate@0.1': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.3': {'doubleValue': 0.9152542372881356},
'fairness_indicators_metrics/true_positive_rate@0.5': {'doubleValue': 0.711864406779661},
'fairness_indicators_metrics/true_positive_rate@0.7': {'doubleValue': 0.5169491525423728},
'fairness_indicators_metrics/true_positive_rate@0.9': {'doubleValue': 0.17796610169491525},
'label/mean': {'doubleValue': 0.2398373931646347},
'post_export_metrics/example_count': {'doubleValue': 492.0},
'precision': {'doubleValue': 0.2937062978744507},
'prediction/mean': {'doubleValue': 0.5578703880310059},
'recall': {'doubleValue': 0.7118644118309021} },
(('sexual_orientation', 'homosexual_gay_or_lesbian'),): {'accuracy': {'doubleValue': 0.5851936340332031},
'accuracy_baseline': {'doubleValue': 0.7182232141494751},
'auc': {'doubleValue': 0.7057511806488037},
'auc_precision_recall': {'doubleValue': 0.469566285610199},
'average_loss': {'doubleValue': 0.7369641661643982},
'fairness_indicators_metrics/false_discovery_rate@0.1': {'doubleValue': 0.7107050831576481},
'fairness_indicators_metrics/false_discovery_rate@0.3': {'doubleValue': 0.6717557251908397},
'fairness_indicators_metrics/false_discovery_rate@0.5': {'doubleValue': 0.6172690763052209},
'fairness_indicators_metrics/false_discovery_rate@0.7': {'doubleValue': 0.5427319211102994},
'fairness_indicators_metrics/false_discovery_rate@0.9': {'doubleValue': 0.4092664092664093},
'fairness_indicators_metrics/false_negative_rate@0.1': {'doubleValue': 0.0016168148746968471},
'fairness_indicators_metrics/false_negative_rate@0.3': {'doubleValue': 0.06143896523848019},
'fairness_indicators_metrics/false_negative_rate@0.5': {'doubleValue': 0.22958771220695232},
'fairness_indicators_metrics/false_negative_rate@0.7': {'doubleValue': 0.4939369442198868},
'fairness_indicators_metrics/false_negative_rate@0.9': {'doubleValue': 0.8763136620856912},
'fairness_indicators_metrics/false_omission_rate@0.1': {'doubleValue': 0.01652892561983471},
'fairness_indicators_metrics/false_omission_rate@0.3': {'doubleValue': 0.08909730363423213},
'fairness_indicators_metrics/false_omission_rate@0.5': {'doubleValue': 0.14947368421052631},
'fairness_indicators_metrics/false_omission_rate@0.7': {'doubleValue': 0.20225091029460443},
'fairness_indicators_metrics/false_omission_rate@0.9': {'doubleValue': 0.2624061970467199},
'fairness_indicators_metrics/false_positive_rate@0.1': {'doubleValue': 0.9622581668252458},
'fairness_indicators_metrics/false_positive_rate@0.3': {'doubleValue': 0.7535680304471931},
'fairness_indicators_metrics/false_positive_rate@0.5': {'doubleValue': 0.4874722486520774},
'fairness_indicators_metrics/false_positive_rate@0.7': {'doubleValue': 0.2356485886457342},
'fairness_indicators_metrics/false_positive_rate@0.9': {'doubleValue': 0.03361877576910879},
'fairness_indicators_metrics/negative_rate@0.1': {'doubleValue': 0.0275626423690205},
'fairness_indicators_metrics/negative_rate@0.3': {'doubleValue': 0.19430523917995443},
'fairness_indicators_metrics/negative_rate@0.5': {'doubleValue': 0.4328018223234624},
'fairness_indicators_metrics/negative_rate@0.7': {'doubleValue': 0.6881548974943053},
'fairness_indicators_metrics/negative_rate@0.9': {'doubleValue': 0.941002277904328},
'fairness_indicators_metrics/positive_rate@0.1': {'doubleValue': 0.9724373576309795},
'fairness_indicators_metrics/positive_rate@0.3': {'doubleValue': 0.8056947608200455},
'fairness_indicators_metrics/positive_rate@0.5': {'doubleValue': 0.5671981776765376},
'fairness_indicators_metrics/positive_rate@0.7': {'doubleValue': 0.31184510250569475},
'fairness_indicators_metrics/positive_rate@0.9': {'doubleValue': 0.05899772209567198},
'fairness_indicators_metrics/true_negative_rate@0.1': {'doubleValue': 0.0377418331747542},
'fairness_indicators_metrics/true_negative_rate@0.3': {'doubleValue': 0.24643196955280686},
'fairness_indicators_metrics/true_negative_rate@0.5': {'doubleValue': 0.5125277513479226},
'fairness_indicators_metrics/true_negative_rate@0.7': {'doubleValue': 0.7643514113542658},
'fairness_indicators_metrics/true_negative_rate@0.9': {'doubleValue': 0.9663812242308912},
'fairness_indicators_metrics/true_positive_rate@0.1': {'doubleValue': 0.9983831851253031},
'fairness_indicators_metrics/true_positive_rate@0.3': {'doubleValue': 0.9385610347615198},
'fairness_indicators_metrics/true_positive_rate@0.5': {'doubleValue': 0.7704122877930477},
'fairness_indicators_metrics/true_positive_rate@0.7': {'doubleValue': 0.5060630557801131},
'fairness_indicators_metrics/true_positive_rate@0.9': {'doubleValue': 0.12368633791430882},
'label/mean': {'doubleValue': 0.2817767560482025},
'post_export_metrics/example_count': {'doubleValue': 4390.0},
'precision': {'doubleValue': 0.3827309310436249},
'prediction/mean': {'doubleValue': 0.5428739786148071},
'recall': {'doubleValue': 0.770412266254425} },
(('sexual_orientation', 'other_sexual_orientation'),): {'accuracy': {'doubleValue': 0.6000000238418579},
'accuracy_baseline': {'doubleValue': 0.800000011920929},
'auc': {'doubleValue': 1.0},
'auc_precision_recall': {'doubleValue': 1.0},
'average_loss': {'doubleValue': 0.7521011829376221},
'fairness_indicators_metrics/false_discovery_rate@0.1': {'doubleValue': 0.8},
'fairness_indicators_metrics/false_discovery_rate@0.3': {'doubleValue': 0.75},
'fairness_indicators_metrics/false_discovery_rate@0.5': {'doubleValue': 0.6666666666666666},
'fairness_indicators_metrics/false_discovery_rate@0.7': {'doubleValue': 0.5},
'fairness_indicators_metrics/false_discovery_rate@0.9': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.3': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.5': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.7': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_negative_rate@0.9': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_omission_rate@0.1': {'doubleValue': 'NaN'},
'fairness_indicators_metrics/false_omission_rate@0.3': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_omission_rate@0.5': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_omission_rate@0.7': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_omission_rate@0.9': {'doubleValue': 0.0},
'fairness_indicators_metrics/false_positive_rate@0.1': {'doubleValue': 1.0},
'fairness_indicators_metrics/false_positive_rate@0.3': {'doubleValue': 0.75},
'fairness_indicators_metrics/false_positive_rate@0.5': {'doubleValue': 0.5},
'fairness_indicators_metrics/false_positive_rate@0.7': {'doubleValue': 0.25},
'fairness_indicators_metrics/false_positive_rate@0.9': {'doubleValue': 0.0},
'fairness_indicators_metrics/negative_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/negative_rate@0.3': {'doubleValue': 0.2},
'fairness_indicators_metrics/negative_rate@0.5': {'doubleValue': 0.4},
'fairness_indicators_metrics/negative_rate@0.7': {'doubleValue': 0.6},
'fairness_indicators_metrics/negative_rate@0.9': {'doubleValue': 0.8},
'fairness_indicators_metrics/positive_rate@0.1': {'doubleValue': 1.0},
'fairness_indicators_metrics/positive_rate@0.3': {'doubleValue': 0.8},
'fairness_indicators_metrics/positive_rate@0.5': {'doubleValue': 0.6},
'fairness_indicators_metrics/positive_rate@0.7': {'doubleValue': 0.4},
'fairness_indicators_metrics/positive_rate@0.9': {'doubleValue': 0.2},
'fairness_indicators_metrics/true_negative_rate@0.1': {'doubleValue': 0.0},
'fairness_indicators_metrics/true_negative_rate@0.3': {'doubleValue': 0.25},
'fairness_indicators_metrics/true_negative_rate@0.5': {'doubleValue': 0.5},
'fairness_indicators_metrics/true_negative_rate@0.7': {'doubleValue': 0.75},
'fairness_indicators_metrics/true_negative_rate@0.9': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.1': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.3': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.5': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.7': {'doubleValue': 1.0},
'fairness_indicators_metrics/true_positive_rate@0.9': {'doubleValue': 1.0},
'label/mean': {'doubleValue': 0.20000000298023224},
'post_export_metrics/example_count': {'doubleValue': 5.0},
'precision': {'doubleValue': 0.3333333432674408},
'prediction/mean': {'doubleValue': 0.6101843118667603},
'recall': {'doubleValue': 1.0} } }