Module core.components.data_gen.utils

Utils for data_gen

Functions

append_tf_example(data: Dict[str, Any], schema: Dict[str, Any]) ‑> tensorflow.core.example.example_pb2.Example : Add tf example to row

Classes

DataType(value, names=None, *, module=None, qualname=None, type=None, start=1) : An enumeration.

### Ancestors (in MRO)

* enum.IntEnum
* builtins.int
* enum.Enum

### Class variables

`BYTES`
:

`FLOAT`
:

`INT`
:

`UNKNOWN`
:

DtypeInferrer(*unused_args, **unused_kwargs) : A beam.CombineFn to infer data types

### Ancestors (in MRO)

* apache_beam.transforms.core.CombineFn
* apache_beam.typehints.decorators.WithTypeHints
* apache_beam.transforms.display.HasDisplayData
* apache_beam.utils.urns.RunnerApiFn
* abc.ABC

### Methods

`add_input(self, accumulator: Dict[str, core.components.data_gen.utils.DataType], element: Dict[str, bytes], **kwargs) ‑> Dict[str, core.components.data_gen.utils.DataType]`
:   Return result of folding element into accumulator.
    
    CombineFn implementors must override add_input.
    
    Args:
      mutable_accumulator: the current accumulator,
        may be modified and returned for efficiency
      element: the element to add, should not be mutated
      *args: Additional arguments and side inputs.
      **kwargs: Additional arguments and side inputs.

`create_accumulator(self, **kwargs) ‑> Dict[str, core.components.data_gen.utils.DataType]`
:   Return a fresh, empty accumulator for the combine operation.
    
    Args:
      *args: Additional arguments and side inputs.
      **kwargs: Additional arguments and side inputs.

`extract_output(self, accumulator: Dict[str, Any], **kwargs) ‑> Dict[str, str]`
:   Return result of converting accumulator into the output value.
    
    Args:
      accumulator: the final accumulator value computed by this CombineFn
        for the entire input key or PCollection. Can be modified for
        efficiency.
      *args: Additional arguments and side inputs.
      **kwargs: Additional arguments and side inputs.

`merge_accumulators(self, accumulators: List[Dict[str, core.components.data_gen.utils.DataType]], **kwargs) ‑> Dict[str, core.components.data_gen.utils.DataType]`
:   Returns the result of merging several accumulators
    to a single accumulator value.
    
    Args:
      accumulators: the accumulators to merge.
        Only the first accumulator may be modified and returned for efficiency;
        the other accumulators should not be mutated, because they may be
        shared with other code and mutating them could lead to incorrect
        results or data corruption.
      *args: Additional arguments and side inputs.
      **kwargs: Additional arguments and side inputs.