Module core.steps.data.postgres_data_step

Base interface for Postgres Data Step

Functions

ReadFromPostgres(p: apache_beam.pipeline.Pipeline, username: str, password: str, database: str, table: str, host: str = 'localhost', port: int = 5432, query_limit: int = None, schema: Dict = None) ‑> apache_beam.pvalue.PCollection : The Beam PTransform used to read data from a specific BQ table.

Args:
    p: Input beam.Pipeline object coming from a TFX Executor.
    host: Host of database.
    username: Username of database user.
    password: Password to connect to database.
    port: Port to connect to with database (default 5432)
    database: Name of the target database.
    table: Name of the target table.
    query_limit: Max number of rows to fetch.
    schema: Dict specifying schema.

Returns:
    A beam.PCollection of data points. Each row in the BigQuery table
     represents a single data point.

Classes

PostgresDataStep(username: str, password: str, database: str, table: str, host: str = 'localhost', port: int = 5432, query_limit: int = None, schema: dict = None) : A step that reads in data from a Google BigQuery table supplied on construction.

Postgres data step constructor. Targets a single Postgres table.

Args:
    host: Host of database.
    username: Username of database user.
    password: Password to connect to database.
    port: Port to connect to with database (default 5432)
    database: Name of the target database.
    table: Name of the target table.
    query_limit: Max number of rows to fetch.
    schema: Dict specifying schema.

### Ancestors (in MRO)

* zenml.core.steps.data.base_data_step.BaseDataStep
* zenml.core.steps.base_step.BaseStep

### Methods

`read_from_source(self)`
: