tfma.utils.CombineFnWithModels

Abstract class for CombineFns that need the shared models.

Methods

add_input

Return result of folding element into accumulator.

CombineFn implementors must override add_input.

Args
mutable_accumulator the current accumulator, may be modified and returned for efficiency
element the element to add, should not be mutated
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

add_inputs

Returns the result of folding each element in elements into accumulator.

This is provided in case the implementation affords more efficient bulk addition of elements. The default implementation simply loops over the inputs invoking add_input for each one.

Args
mutable_accumulator the current accumulator, may be modified and returned for efficiency
elements the elements to add, should not be mutated
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

apply

Returns result of applying this CombineFn to the input values.

Args
elements the set of values to combine.
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

compact

Optionally returns a more compact representation of the accumulator.

This is called before an accumulator is sent across the wire, and can be useful in cases where values are buffered or otherwise lazily kept unprocessed when added to the accumulator. Should return an equivalent, though possibly modified, accumulator.

By default returns the accumulator unmodified.

Args
accumulator the current accumulator
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

create_accumulator

Return a fresh, empty accumulator for the combine operation.

Args
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

default_label

default_type_hints

display_data

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns
Dict[str, Any]: A dictionary containing key:value pairs. The value might be an integer, float or string value; a :class:DisplayDataItem for values that have more data (e.g. short value, label, url); or a :class:HasDisplayData instance that has more display data that should be picked up. For example::

{ 'key1': 'string_value', 'key2': 1234, 'key3': 3.14159265, 'key4': DisplayDataItem('apache.org', url='http://apache.org'), 'key5': subComponent }

extract_output

Return result of converting accumulator into the output value.

Args
accumulator the final accumulator value computed by this CombineFn for the entire input key or PCollection. Can be modified for efficiency.
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

for_input_type

Returns a specialized implementation of self, if it exists.

Otherwise, returns self.

Args
input_type the type of input elements.

from_callable

from_runner_api

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_accumulator_coder

get_type_hints

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order:

  • Using self.default_type_hints().
  • Using self.class type hints.

maybe_from_callable

merge_accumulators

Returns the result of merging several accumulators to a single accumulator value.

Args
accumulators the accumulators to merge. Only the first accumulator may be modified and returned for efficiency; the other accumulators should not be mutated, because they may be shared with other code and mutating them could lead to incorrect results or data corruption.
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

register_pickle_urn

Registers and implements the given urn via pickling.

register_urn

Registers a urn with a constructor.

For example, if 'beam:fn:foo' had parameter type FooPayload, one could write RunnerApiFn.register_urn('bean:fn:foo', FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple ('beam:fn:foo', FooPayload)

setup

View source

Called to prepare an instance for combining.

This method can be useful if there is some state that needs to be loaded before executing any of the other methods. The resources can then be disposed of in CombineFn.teardown.

If you are using Dataflow, you need to enable Dataflow Runner V2 before using this feature.

Args
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

teardown

Called to clean up an instance before it is discarded.

If you are using Dataflow, you need to enable Dataflow Runner V2 before using this feature.

Args
*args Additional arguments and side inputs.
**kwargs Additional arguments and side inputs.

to_runner_api

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter

with_input_types

with_output_types