View source on GitHub |
Abstract class for CombineFns that need the shared models.
tfma.utils.CombineFnWithModels(
model_loaders: Dict[str, tfma.types.ModelLoader
]
)
Methods
add_input
add_input(
mutable_accumulator, element, *args, **kwargs
)
Return result of folding element into accumulator.
CombineFn implementors must override add_input.
Args | |
---|---|
mutable_accumulator
|
the current accumulator, may be modified and returned for efficiency |
element
|
the element to add, should not be mutated |
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
add_inputs
add_inputs(
mutable_accumulator, elements, *args, **kwargs
)
Returns the result of folding each element in elements into accumulator.
This is provided in case the implementation affords more efficient bulk addition of elements. The default implementation simply loops over the inputs invoking add_input for each one.
Args | |
---|---|
mutable_accumulator
|
the current accumulator, may be modified and returned for efficiency |
elements
|
the elements to add, should not be mutated |
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
apply
apply(
elements, *args, **kwargs
)
Returns result of applying this CombineFn to the input values.
Args | |
---|---|
elements
|
the set of values to combine. |
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
compact
compact(
accumulator, *args, **kwargs
)
Optionally returns a more compact representation of the accumulator.
This is called before an accumulator is sent across the wire, and can be useful in cases where values are buffered or otherwise lazily kept unprocessed when added to the accumulator. Should return an equivalent, though possibly modified, accumulator.
By default returns the accumulator unmodified.
Args | |
---|---|
accumulator
|
the current accumulator |
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
create_accumulator
create_accumulator(
*args, **kwargs
)
Return a fresh, empty accumulator for the combine operation.
Args | |
---|---|
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
default_label
default_label()
default_type_hints
default_type_hints()
display_data
display_data()
Returns the display data associated to a pipeline component.
It should be reimplemented in pipeline components that wish to have static display data.
Returns | |
---|---|
Dict[str, Any]: A dictionary containing key:value pairs.
The value might be an integer, float or string value; a
:class:DisplayDataItem for values that have more data
(e.g. short value, label, url); or a :class:HasDisplayData instance
that has more display data that should be picked up. For example::
{ 'key1': 'string_value', 'key2': 1234, 'key3': 3.14159265, 'key4': DisplayDataItem('apache.org', url='http://apache.org'), 'key5': subComponent } |
extract_output
extract_output(
accumulator, *args, **kwargs
)
Return result of converting accumulator into the output value.
Args | |
---|---|
accumulator
|
the final accumulator value computed by this CombineFn for the entire input key or PCollection. Can be modified for efficiency. |
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
for_input_type
for_input_type(
input_type
)
Returns a specialized implementation of self, if it exists.
Otherwise, returns self.
Args | |
---|---|
input_type
|
the type of input elements. |
from_callable
@staticmethod
from_callable( fn )
from_runner_api
@classmethod
from_runner_api( fn_proto: beam_runner_api_pb2.FunctionSpec, context: 'PipelineContext' ) -> RunnerApiFnT
Converts from an FunctionSpec to a Fn object.
Prefer registering a urn with its parameter type and constructor.
get_accumulator_coder
get_accumulator_coder()
get_type_hints
get_type_hints()
Gets and/or initializes type hints for this object.
If type hints have not been set, attempts to initialize type hints in this order:
- Using self.default_type_hints().
- Using self.class type hints.
maybe_from_callable
@staticmethod
maybe_from_callable( fn, has_side_inputs=True )
merge_accumulators
merge_accumulators(
accumulators, *args, **kwargs
)
Returns the result of merging several accumulators to a single accumulator value.
Args | |
---|---|
accumulators
|
the accumulators to merge. Only the first accumulator may be modified and returned for efficiency; the other accumulators should not be mutated, because they may be shared with other code and mutating them could lead to incorrect results or data corruption. |
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
register_pickle_urn
@classmethod
register_pickle_urn( pickle_urn )
Registers and implements the given urn via pickling.
register_urn
@classmethod
register_urn( urn, parameter_type, fn=None )
Registers a urn with a constructor.
For example, if 'beam:fn:foo' had parameter type FooPayload, one could
write RunnerApiFn.register_urn('bean:fn:foo', FooPayload, foo_from_proto)
where foo_from_proto took as arguments a FooPayload and a PipelineContext.
This function can also be used as a decorator rather than passing the
callable in as the final parameter.
A corresponding to_runner_api_parameter method would be expected that returns the tuple ('beam:fn:foo', FooPayload)
setup
setup()
Called to prepare an instance for combining.
This method can be useful if there is some state that needs to be loaded
before executing any of the other methods. The resources can then be
disposed of in CombineFn.teardown
.
If you are using Dataflow, you need to enable Dataflow Runner V2 before using this feature.
Args | |
---|---|
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
teardown
teardown(
*args, **kwargs
)
Called to clean up an instance before it is discarded.
If you are using Dataflow, you need to enable Dataflow Runner V2 before using this feature.
Args | |
---|---|
*args
|
Additional arguments and side inputs. |
**kwargs
|
Additional arguments and side inputs. |
to_runner_api
to_runner_api(
context: 'PipelineContext'
) -> beam_runner_api_pb2.FunctionSpec
Returns an FunctionSpec encoding this Fn.
Prefer overriding self.to_runner_api_parameter.
to_runner_api_parameter
to_runner_api_parameter(
context
)
with_input_types
with_input_types(
*arg_hints, **kwarg_hints
) -> WithTypeHintsT
with_output_types
with_output_types(
*arg_hints, **kwarg_hints
) -> WithTypeHintsT