{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "headers" }, "source": [ "Project: /overview/_project.yaml\n", "Book: /overview/_book.yaml\n", "\n", "\n", "\n", "\n", "\n", "\n", "{% comment %}\n", "The source of truth file can be found [here]: http://google3/zz\n", "{% endcomment %}" ] }, { "cell_type": "markdown", "metadata": { "id": "metadata" }, "source": [ "
TensorFlow.org で表示 | \n", "Google Colab で実行 | \n", "GitHub でソースを表示 | \n", "ノートブックをダウンロード | \n", "
tf.function
内で初めに実行される可能性が高いため、 `__call__()` で作成される変数は tf.init_scope
でラップする必要があります。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0b7a45f57610"
},
"source": [
"## Layers are recursively composable\n",
"\n",
"レイヤーインスタンスを別のレイヤーの属性として割り当てると、外部レイヤーは内部レイヤーの重みを追跡し始めます。\n",
"\n",
"`__init__()` メソッドでこのようなサブレイヤーを作成し、それを最初の `__call__()` で、重みの作成をトリガーすることをお勧めします。"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"execution": {
"iopub.execute_input": "2022-12-14T21:46:06.788079Z",
"iopub.status.busy": "2022-12-14T21:46:06.787512Z",
"iopub.status.idle": "2022-12-14T21:46:06.818587Z",
"shell.execute_reply": "2022-12-14T21:46:06.817970Z"
},
"id": "1aaaf82ab8ce"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"weights: 6\n",
"trainable weights: 6\n"
]
}
],
"source": [
"class MLPBlock(keras.layers.Layer):\n",
" def __init__(self):\n",
" super(MLPBlock, self).__init__()\n",
" self.linear_1 = Linear(32)\n",
" self.linear_2 = Linear(32)\n",
" self.linear_3 = Linear(1)\n",
"\n",
" def call(self, inputs):\n",
" x = self.linear_1(inputs)\n",
" x = tf.nn.relu(x)\n",
" x = self.linear_2(x)\n",
" x = tf.nn.relu(x)\n",
" return self.linear_3(x)\n",
"\n",
"\n",
"mlp = MLPBlock()\n",
"y = mlp(tf.ones(shape=(3, 64))) # The first call to the `mlp` will create the weights\n",
"print(\"weights:\", len(mlp.weights))\n",
"print(\"trainable weights:\", len(mlp.trainable_weights))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2bf11b296bd2"
},
"source": [
"## The `add_loss()` method\n",
"\n",
"When writing the `call()` method of a layer, you can create loss tensors that you will want to use later, when writing your training loop. This is doable by calling `self.add_loss(value)`:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"execution": {
"iopub.execute_input": "2022-12-14T21:46:06.822206Z",
"iopub.status.busy": "2022-12-14T21:46:06.821612Z",
"iopub.status.idle": "2022-12-14T21:46:06.825652Z",
"shell.execute_reply": "2022-12-14T21:46:06.825060Z"
},
"id": "ba2782dc0879"
},
"outputs": [],
"source": [
"# A layer that creates an activity regularization loss\n",
"class ActivityRegularizationLayer(keras.layers.Layer):\n",
" def __init__(self, rate=1e-2):\n",
" super(ActivityRegularizationLayer, self).__init__()\n",
" self.rate = rate\n",
"\n",
" def call(self, inputs):\n",
" self.add_loss(self.rate * tf.reduce_sum(inputs))\n",
" return inputs\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a883b230a9e9"
},
"source": [
"These losses (including those created by any inner layer) can be retrieved via `layer.losses`. This property is reset at the start of every `__call__()` to the top-level layer, so that `layer.losses` always contains the loss values created during the last forward pass."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"execution": {
"iopub.execute_input": "2022-12-14T21:46:06.828552Z",
"iopub.status.busy": "2022-12-14T21:46:06.828181Z",
"iopub.status.idle": "2022-12-14T21:46:06.838070Z",
"shell.execute_reply": "2022-12-14T21:46:06.837488Z"
},
"id": "b56d223a30cd"
},
"outputs": [],
"source": [
"class OuterLayer(keras.layers.Layer):\n",
" def __init__(self):\n",
" super(OuterLayer, self).__init__()\n",
" self.activity_reg = ActivityRegularizationLayer(1e-2)\n",
"\n",
" def call(self, inputs):\n",
" return self.activity_reg(inputs)\n",
"\n",
"\n",
"layer = OuterLayer()\n",
"assert len(layer.losses) == 0 # No losses yet since the layer has never been called\n",
"\n",
"_ = layer(tf.zeros(1, 1))\n",
"assert len(layer.losses) == 1 # We created one loss value\n",
"\n",
"# `layer.losses` gets reset at the start of each __call__\n",
"_ = layer(tf.zeros(1, 1))\n",
"assert len(layer.losses) == 1 # This is the loss created during the call above"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0809dec680ff"
},
"source": [
"In addition, the `loss` property also contains regularization losses created for the weights of any inner layer:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2022-12-14T21:46:06.841180Z",
"iopub.status.busy": "2022-12-14T21:46:06.840669Z",
"iopub.status.idle": "2022-12-14T21:46:06.861003Z",
"shell.execute_reply": "2022-12-14T21:46:06.860415Z"
},
"id": "41016153e983"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[