Module core.components.sequencer.utils

Functions

df_to_example(instance) ‑> tensorflow.core.example.example_pb2.Example : Auxiliary function to create a tf.train.Example from a pandas dataframe

:param instance: pd.DataFrame, the datapoint
:return: tf.train.Example, the result

serialize(instance) : Helper function to serialize a tf.train.Example in the beam pipeline

:param instance: the input, tf.train.Example
:return: serialized version of the input

Classes

ConvertToDataframe(*unused_args, **unused_kwargs) : Beam PTransform responsible for converting the incoming Arrow table into a pandas dataframe

### Ancestors (in MRO)

* apache_beam.transforms.core.DoFn
* apache_beam.typehints.decorators.WithTypeHints
* apache_beam.transforms.display.HasDisplayData
* apache_beam.utils.urns.RunnerApiFn

### Methods

`process(self, element, *args, **kwargs)`
:   Method to use for processing elements.
    
    This is invoked by ``DoFnRunner`` for each element of a input
    ``PCollection``.
    
    The following parameters can be used as default values on ``process``
    arguments to indicate that a DoFn accepts the corresponding parameters. For
    example, a DoFn might accept the element and its timestamp with the
    following signature::
    
      def process(element=DoFn.ElementParam, timestamp=DoFn.TimestampParam):
        ...
    
    The full set of parameters is:
    
    - ``DoFn.ElementParam``: element to be processed, should not be mutated.
    - ``DoFn.SideInputParam``: a side input that may be used when processing.
    - ``DoFn.TimestampParam``: timestamp of the input element.
    - ``DoFn.WindowParam``: ``Window`` the input element belongs to.
    - ``DoFn.TimerParam``: a ``userstate.RuntimeTimer`` object defined by the
      spec of the parameter.
    - ``DoFn.StateParam``: a ``userstate.RuntimeState`` object defined by the
      spec of the parameter.
    - ``DoFn.KeyParam``: key associated with the element.
    - ``DoFn.RestrictionParam``: an ``iobase.RestrictionTracker`` will be
      provided here to allow treatment as a Splittable ``DoFn``. The restriction
      tracker will be derived from the restriction provider in the parameter.
    - ``DoFn.WatermarkEstimatorParam``: a function that can be used to track
      output watermark of Splittable ``DoFn`` implementations.
    
    Args:
      element: The element to be processed
      *args: side inputs
      **kwargs: other keyword arguments.
    
    Returns:
      An Iterable of output elements or None.