View source on GitHub |
Layer that projects its inputs into a random feature space.
tf.keras.layers.experimental.RandomFourierFeatures(
output_dim,
kernel_initializer='gaussian',
scale=None,
trainable=False,
name=None,
**kwargs
)
This layer implements a mapping from input space to a space with
output_dim
dimensions, which approximates shift-invariant kernels. A
kernel function K(x, y)
is shift-invariant if K(x, y) == k(x - y)
for
some function k
. Many popular Radial Basis Functions (RBF), including
Gaussian and Laplacian kernels, are shift-invariant.
The implementation of this layer is based on the following paper: "Random Features for Large-Scale Kernel Machines" by Ali Rahimi and Ben Recht.
The distribution from which the parameters of the random features map (layer) are sampled determines which shift-invariant kernel the layer approximates (see paper for more details). You can use the distribution of your choice. The layer supports out-of-the-box approximations of the following two RBF kernels:
- Gaussian:
K(x, y) == exp(- square(x - y) / (2 * square(scale)))
- Laplacian:
K(x, y) = exp(-abs(x - y) / scale))
Usage: Typically, this layer is used to "kernelize" linear models by applying a non-linear transformation (this layer) to the input features and then training a linear model on top of the transformed features. Depending on the loss function of the linear model, the composition of this layer and the linear model results to models that are equivalent (up to approximation) to kernel SVMs (for hinge loss), kernel logistic regression (for logistic loss), kernel linear regression (for squared loss), etc.
Examples:
A kernel multinomial logistic regression model with Gaussian kernel for MNIST:
model = keras.Sequential([
keras.Input(shape=(784,)),
RandomFourierFeatures(
output_dim=4096,
scale=10.,
kernel_initializer='gaussian'),
layers.Dense(units=10, activation='softmax'),
])
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['categorical_accuracy']
)
A quasi-SVM classifier for MNIST:
model = keras.Sequential([
keras.Input(shape=(784,)),
RandomFourierFeatures(
output_dim=4096,
scale=10.,
kernel_initializer='gaussian'),
layers.Dense(units=10),
])
model.compile(
optimizer='adam',
loss='hinge',
metrics=['categorical_accuracy']
)
To use another kernel, just replace the layer creation line with:
random_features_layer = RandomFourierFeatures(
output_dim=500,
kernel_initializer=<my_initializer>,
scale=...,
...)