Module core.steps.data.postgres_data_step¶
Base interface for Postgres Data Step
Functions¶
ReadFromPostgres(p: apache_beam.pipeline.Pipeline, username: str, password: str, database: str, table: str, host: str = 'localhost', port: int = 5432, query_limit: int = None, schema: Dict = None) ‑> apache_beam.pvalue.PCollection
: The Beam PTransform used to read data from a specific BQ table.
Args:
p: Input beam.Pipeline object coming from a TFX Executor.
host: Host of database.
username: Username of database user.
password: Password to connect to database.
port: Port to connect to with database (default 5432)
database: Name of the target database.
table: Name of the target table.
query_limit: Max number of rows to fetch.
schema: Dict specifying schema.
Returns:
A beam.PCollection of data points. Each row in the BigQuery table
represents a single data point.
Classes¶
PostgresDataStep(username: str, password: str, database: str, table: str, host: str = 'localhost', port: int = 5432, query_limit: int = None, schema: dict = None)
: A step that reads in data from a Google BigQuery table supplied on
construction.
Postgres data step constructor. Targets a single Postgres table.
Args:
host: Host of database.
username: Username of database user.
password: Password to connect to database.
port: Port to connect to with database (default 5432)
database: Name of the target database.
table: Name of the target table.
query_limit: Max number of rows to fetch.
schema: Dict specifying schema.
### Ancestors (in MRO)
* zenml.core.steps.data.base_data_step.BaseDataStep
* zenml.core.steps.base_step.BaseStep
### Methods
`read_from_source(self)`
: