Fan in, fan out
Running steps in parallel.
The fan-out/fan-in pattern is a common pipeline architecture where a single step splits into multiple parallel operations (fan-out) and then consolidates the results back into a single step (fan-in). This pattern is particularly useful for parallel processing, distributed workloads, or when you need to process data through different transformations and then aggregate the results. For example, you might want to process different chunks of data in parallel and then aggregate the results:
The fan-out pattern allows for parallel processing and better resource utilization, while the fan-in pattern enables aggregation and consolidation of results. This is particularly useful for:
Parallel data processing
Distributed model training
Ensemble methods
Batch processing
Data validation across multiple sources
Note that when implementing the fan-in step, you'll need to use the ZenML Client to query the results from previous parallel steps, as shown in the example above, and you can't pass in the result directly.
The fan-in, fan-out method has the following limitations:
Steps run sequentially rather than in parallel if the underlying orchestrator does not support parallel step runs (e.g. with the local orchestrator)
The number of steps need to be known ahead-of-time, and ZenML does not yet support the ability to dynamically create steps on the fly.

Last updated
Was this helpful?