Module core.components.sequencer.utils¶
Functions¶
df_to_example(instance) ‑> tensorflow.core.example.example_pb2.Example
: Auxiliary function to create a tf.train.Example from a pandas dataframe
:param instance: pd.DataFrame, the datapoint
:return: tf.train.Example, the result
serialize(instance)
: Helper function to serialize a tf.train.Example in the beam pipeline
:param instance: the input, tf.train.Example
:return: serialized version of the input
Classes¶
ConvertToDataframe(*unused_args, **unused_kwargs)
: Beam PTransform responsible for converting the incoming Arrow table into
a pandas dataframe
### Ancestors (in MRO)
* apache_beam.transforms.core.DoFn
* apache_beam.typehints.decorators.WithTypeHints
* apache_beam.transforms.display.HasDisplayData
* apache_beam.utils.urns.RunnerApiFn
### Methods
`process(self, element, *args, **kwargs)`
: Method to use for processing elements.
This is invoked by ``DoFnRunner`` for each element of a input
``PCollection``.
The following parameters can be used as default values on ``process``
arguments to indicate that a DoFn accepts the corresponding parameters. For
example, a DoFn might accept the element and its timestamp with the
following signature::
def process(element=DoFn.ElementParam, timestamp=DoFn.TimestampParam):
...
The full set of parameters is:
- ``DoFn.ElementParam``: element to be processed, should not be mutated.
- ``DoFn.SideInputParam``: a side input that may be used when processing.
- ``DoFn.TimestampParam``: timestamp of the input element.
- ``DoFn.WindowParam``: ``Window`` the input element belongs to.
- ``DoFn.TimerParam``: a ``userstate.RuntimeTimer`` object defined by the
spec of the parameter.
- ``DoFn.StateParam``: a ``userstate.RuntimeState`` object defined by the
spec of the parameter.
- ``DoFn.KeyParam``: key associated with the element.
- ``DoFn.RestrictionParam``: an ``iobase.RestrictionTracker`` will be
provided here to allow treatment as a Splittable ``DoFn``. The restriction
tracker will be derived from the restriction provider in the parameter.
- ``DoFn.WatermarkEstimatorParam``: a function that can be used to track
output watermark of Splittable ``DoFn`` implementations.
Args:
element: The element to be processed
*args: side inputs
**kwargs: other keyword arguments.
Returns:
An Iterable of output elements or None.