ZenML
  • Getting Started
    • Introduction
    • Installation
    • Core concepts
    • System Architecture
    • Deploying ZenML
      • Deploy with Docker
      • Deploy with Helm
      • Deploy using HuggingFace Spaces
      • Deploy with custom images
      • Secret management
      • Custom secret stores
    • ZenML Pro
      • Core Concepts
        • Organizations
        • Tenants
        • Teams
        • Roles & Permissions
      • Self-hosted deployment
      • Accessing the ZenML Pro API
  • User Guide
    • Starter guide
      • Create an ML pipeline
      • Cache previous executions
      • Manage artifacts
      • Track ML models
      • A starter project
    • Production guide
      • Deploying ZenML
      • Understanding stacks
      • Connecting remote storage
      • Orchestrate on the cloud
      • Configure your pipeline to add compute
      • Configure a code repository
      • Set up CI/CD
      • An end-to-end project
    • LLMOps guide
      • RAG with ZenML
        • RAG in 85 lines of code
        • Understanding Retrieval-Augmented Generation (RAG)
        • Data ingestion and preprocessing
        • Embeddings generation
        • Storing embeddings in a vector database
        • Basic RAG inference pipeline
      • Evaluation and metrics
        • Evaluation in 65 lines of code
        • Retrieval evaluation
        • Generation evaluation
        • Evaluation in practice
      • Reranking for better retrieval
        • Understanding reranking
        • Implementing reranking in ZenML
        • Evaluating reranking performance
      • Improve retrieval by finetuning embeddings
        • Synthetic data generation
        • Finetuning embeddings with Sentence Transformers
        • Evaluating finetuned embeddings
      • Finetuning LLMs with ZenML
        • Finetuning in 100 lines of code
        • Why and when to finetune LLMs
        • Starter choices with finetuning
        • Finetuning with 🤗 Accelerate
        • Evaluation for finetuning
        • Deploying finetuned models
        • Next steps
  • How-To
    • Manage your ZenML server
      • Connect to a server
        • Connect in with your User (interactive)
        • Connect with an API Token
        • Connect with a Service Account
      • Upgrade your ZenML server
      • Best practices for upgrading ZenML
      • Using ZenML server in production
      • Troubleshoot your ZenML server
      • Migration guide
        • Migration guide 0.13.2 → 0.20.0
        • Migration guide 0.23.0 → 0.30.0
        • Migration guide 0.39.1 → 0.41.0
        • Migration guide 0.58.2 → 0.60.0
    • Project Setup and Management
      • Set up a ZenML project
        • Set up a repository
        • Connect your git repository
      • Collaborate with your team
        • Project templates
          • Create your own template
        • Shared components for teams
        • Setting up Stacks, pipelines and models
        • Access management
      • Interact with secrets
    • Pipeline Development
      • Build a pipeline
        • Use pipeline/step parameters
        • Configuring a pipeline at runtime
        • Reference environment variables in configurations
        • Step output typing and annotation
        • Control caching behavior
        • Schedule a pipeline
        • Deleting a pipeline
        • Compose pipelines
        • Automatically retry steps
        • Run pipelines asynchronously
        • Control execution order of steps
        • Using a custom step invocation ID
        • Name your pipeline runs
        • Tag your pipeline runs
        • Use failure/success hooks
        • Fan in, fan out
        • Hyperparameter tuning
        • Access secrets in a step
        • Run an individual step
        • Fetching pipelines
        • Get past pipeline/step runs
      • Develop locally
        • Use config files to develop locally
        • Keep your pipelines and dashboard clean
      • Use configuration files
        • How to configure a pipeline with a YAML
        • What can be configured
        • Runtime settings for Docker, resources, and stack components
        • Configuration hierarchy
        • Find out which configuration was used for a run
        • Autogenerate a template yaml file
      • Train with GPUs
        • Distributed Training with 🤗 Accelerate
      • Run remote pipelines from notebooks
        • Limitations of defining steps in notebook cells
        • Run a single step from a notebook
      • Configure Python environments
        • Handling dependencies
        • Configure the server environment
    • Trigger a pipeline
      • Use templates: Python SDK
      • Use templates: CLI
      • Use templates: Dashboard
      • Use templates: Rest API
    • Customize Docker builds
      • Docker settings on a pipeline
      • Docker settings on a step
      • Use a prebuilt image for pipeline execution
      • Specify pip dependencies and apt packages
      • How to use a private PyPI repository
      • Use your own Dockerfiles
      • Which files are built into the image
      • How to reuse builds
      • Define where an image is built
    • Data and Artifact Management
      • Understand ZenML artifacts
        • How ZenML stores data
        • Return multiple outputs from a step
        • Delete an artifact
        • Artifacts naming
        • Organize data with tags
        • Get arbitrary artifacts in a step
        • Handle custom data types
        • Load artifacts into memory
      • Complex use-cases
        • Datasets in ZenML
        • Manage big data
        • Skipping materialization
        • Passing artifacts between pipelines
        • Register Existing Data as a ZenML Artifact
      • Visualizing artifacts
        • Default visualizations
        • Creating custom visualizations
        • Displaying visualizations in the dashboard
        • Disabling visualizations
    • Model Management and Metrics
      • Use the Model Control Plane
        • Registering a Model
        • Deleting a Model
        • Associate a pipeline with a Model
        • Connecting artifacts via a Model
        • Controlling Model versions
        • Load a Model in code
        • Promote a Model
        • Linking model binaries/data to a Model
        • Load artifacts from Model
      • Track metrics and metadata
        • Attach metadata to a step
        • Attach metadata to a run
        • Attach metadata to an artifact
        • Attach metadata to a model
        • Group metadata
        • Special Metadata Types
        • Fetch metadata within steps
        • Fetch metadata during pipeline composition
    • Stack infrastructure and deployment
      • Manage stacks & components
        • Deploy a cloud stack with ZenML
        • Deploy a cloud stack with Terraform
        • Register a cloud stack
        • Export and install stack requirements
        • Reference secrets in stack configuration
        • Implement a custom stack component
      • Infrastructure as code
        • Manage your stacks with Terraform
        • ZenML & Terraform Best Practices
      • Connect services via connectors
        • Service Connectors guide
        • Security best practices
        • Docker Service Connector
        • Kubernetes Service Connector
        • AWS Service Connector
        • GCP Service Connector
        • Azure Service Connector
        • HyperAI Service Connector
    • Control logging
      • View logs on the dashboard
      • Enable or disable logs storage
      • Set logging verbosity
      • Set logging format
      • Disable rich traceback output
      • Disable colorful logging
    • Popular integrations
      • Run on AWS
      • Run on GCP
      • Run on Azure
      • Kubeflow
      • Kubernetes
      • MLflow
      • Skypilot
    • Contribute to/Extend ZenML
      • Implement a custom integration
    • Debug and solve issues
  • Stack Components
    • Overview
    • Orchestrators
      • Local Orchestrator
      • Local Docker Orchestrator
      • Kubeflow Orchestrator
      • Kubernetes Orchestrator
      • Google Cloud VertexAI Orchestrator
      • AWS Sagemaker Orchestrator
      • AzureML Orchestrator
      • Databricks Orchestrator
      • Tekton Orchestrator
      • Airflow Orchestrator
      • Skypilot VM Orchestrator
      • HyperAI Orchestrator
      • Lightning AI Orchestrator
      • Develop a custom orchestrator
    • Artifact Stores
      • Local Artifact Store
      • Amazon Simple Cloud Storage (S3)
      • Google Cloud Storage (GCS)
      • Azure Blob Storage
      • Develop a custom artifact store
    • Container Registries
      • Default Container Registry
      • DockerHub
      • Amazon Elastic Container Registry (ECR)
      • Google Cloud Container Registry
      • Azure Container Registry
      • GitHub Container Registry
      • Develop a custom container registry
    • Data Validators
      • Great Expectations
      • Deepchecks
      • Evidently
      • Whylogs
      • Develop a custom data validator
    • Experiment Trackers
      • Comet
      • MLflow
      • Neptune
      • Weights & Biases
      • Google Cloud VertexAI Experiment Tracker
      • Develop a custom experiment tracker
    • Model Deployers
      • MLflow
      • Seldon
      • BentoML
      • Hugging Face
      • Databricks
      • vLLM
      • Develop a Custom Model Deployer
    • Step Operators
      • Amazon SageMaker
      • AzureML
      • Google Cloud VertexAI
      • Kubernetes
      • Modal
      • Spark
      • Develop a Custom Step Operator
    • Alerters
      • Discord Alerter
      • Slack Alerter
      • Develop a Custom Alerter
    • Image Builders
      • Local Image Builder
      • Kaniko Image Builder
      • AWS Image Builder
      • Google Cloud Image Builder
      • Develop a Custom Image Builder
    • Annotators
      • Argilla
      • Label Studio
      • Pigeon
      • Prodigy
      • Develop a Custom Annotator
    • Model Registries
      • MLflow Model Registry
      • Develop a Custom Model Registry
    • Feature Stores
      • Feast
      • Develop a Custom Feature Store
  • Examples
    • Quickstart
    • End-to-End Batch Inference
    • Basic NLP with BERT
    • Computer Vision with YoloV8
    • LLM Finetuning
    • More Projects...
  • Reference
    • llms.txt
    • Python Client
    • Global settings
    • Environment Variables
    • API reference
    • SDK & CLI reference
    • How do I...?
    • Community & content
    • FAQ
Powered by GitBook