> For the complete documentation index, see [llms.txt](https://docs.zenml.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zenml.io/stacks/stack-components/step-operators/ssh.md). # SSH The SSH step operator allows you to execute individual ZenML steps on a remote Linux host via SSH. Steps run inside detached Docker containers on the remote machine, with optional GPU selection. This is ideal for teams that have dedicated GPU machines they want to use for training without needing a full cluster orchestration platform. {% hint style="info" %} If you want to run *entire* pipelines on a remote host (not just individual steps), see the sibling [SSH orchestrator](/stacks/stack-components/orchestrators/ssh.md). The two components share the same SSH connection layer and can target the same machine. {% endhint %} {% hint style="warning" %} When a step operator step runs under the [SSH orchestrator](/stacks/stack-components/orchestrators/ssh.md), it executes inside a container on the remote host rather than on your client. In that case configure the key via `ssh_private_key` instead of `ssh_key_path`, and set `verify_host_key=False` (or point `known_hosts_path` at a file baked into the image), since the container has no local key file or `known_hosts`. {% endhint %} ### When to use it You should use the SSH step operator if: * You have a **dedicated remote machine** (e.g., a GPU workstation or server) reachable via SSH. * You want to run compute-heavy steps (training, inference) on that machine while keeping your orchestrator lightweight. * You want to target **specific GPUs** on the host via `--gpus`. * You prefer a simple setup without Kubernetes or cloud-managed services. ### How to deploy it The SSH step operator connects to an existing remote host; it does not provision infrastructure. Your remote host must meet these requirements: * **Linux** operating system. * **Docker** installed and accessible to the SSH user (the user should be in the `docker` group). * **Network reachability** from the machine running your orchestrator to `hostname:port` over SSH. * If using GPUs: the **NVIDIA Container Toolkit** must be installed so Docker supports `--gpus`. ### How to use it To use the SSH step operator, you need: * The ZenML `ssh` integration installed. If you haven't done so, run: ```shell zenml integration install ssh ``` This installs [paramiko](https://www.paramiko.org/), the Python SSH library used under the hood. * A [remote artifact store](https://docs.zenml.io/stacks/artifact-stores/) as part of your stack (the remote container cannot access local files). * A [remote container registry](https://docs.zenml.io/stacks/container-registries/) to store the Docker images that will be pulled on the remote host. * An [image builder](https://docs.zenml.io/stacks/image-builders/) to build the Docker images for your steps. We can then register the step operator. SSH credentials are configured directly on the step operator: a `username` and **at least one of** `ssh_key_path` **or** `ssh_private_key` **must be provided** for authentication. **Using a key file:** ```shell zenml step-operator register \ --flavor=ssh \ --hostname= \ --username= \ --ssh_key_path=~/.ssh/id_ed25519 ``` **Using a key stored in a ZenML secret:** ```shell zenml secret create ssh_secret \ --private_key='' \ --passphrase='' zenml step-operator register \ --flavor=ssh \ --hostname= \ --username= \ --ssh_private_key={{ssh_secret.private_key}} \ --ssh_key_passphrase={{ssh_secret.passphrase}} ``` We can then add the step operator to our active stack: ```shell zenml stack update -s ``` Once you have added the step operator to your active stack, you can use it to execute individual steps of your pipeline by specifying it in the `@step` decorator: ```python from zenml import step @step(step_operator="") def trainer(...) -> ...: """Train a model on the remote host.""" # This step will be executed via SSH + Docker ``` {% hint style="info" %} ZenML will build a Docker image which includes your code and use it to run your steps on the remote host. Check out [this page](https://docs.zenml.io/how-to/customize-docker-builds/) if you want to learn more about how ZenML builds these images and how you can customize them. {% endhint %} #### What happens at runtime When a step runs via the SSH step operator: 1. ZenML builds and pushes the step image using your stack's image builder and container registry. 2. The step operator opens an SSH connection to the remote host. 3. A **preflight check** verifies that `docker` is available. 4. An **env-file** (with the step's environment variables) is uploaded to the remote working directory. 5. The Docker image is **pulled** on the remote host. 6. The container is started **detached** (`docker run -d`), and the env-file is deleted immediately afterwards so the step's secrets do not linger on the host. 7. ZenML polls the step's status with `docker inspect` until the container exits — the Docker daemon is the single source of truth — and reports success or failure. Step logs are captured by ZenML like any other step. ### GPU execution The SSH step operator supports targeting specific GPUs on the host. #### Selecting GPUs Use the `gpu_indices` setting to specify which GPU devices to expose to the container: ```python from zenml import step from zenml.integrations.ssh.flavors import SSHStepOperatorSettings ssh_settings = SSHStepOperatorSettings( gpu_indices=[0, 1], docker_run_args=["--shm-size=2g"], ) @step( step_operator="", settings={"step_operator": ssh_settings}, ) def train(): """Train with GPUs 0 and 1.""" ... ``` GPU indices are passed to Docker as `--gpus "device=0,1"`. They must be non-negative integers matching the device indices reported by `nvidia-smi` on the remote host. {% hint style="info" %} The step operator does not coordinate GPU access between concurrent steps. If you run multiple GPU steps on the same host at once, give them distinct `gpu_indices` so they don't oversubscribe the same device. {% endhint %} #### CPU-only execution If `gpu_indices` is not set (the default), the step runs without GPU access: ```python ssh_settings = SSHStepOperatorSettings() # CPU-only @step(step_operator="", settings={"step_operator": ssh_settings}) def preprocess(): ... ``` ### Configuration options #### Step Operator Configuration (set during registration) | Option | Type | Required | Default | Description | | -------------------- | ----- | -------- | ---------------- | ---------------------------------------------------------------- | | `hostname` | str | Yes | - | Hostname or IP of the remote SSH server | | `username` | str | No\* | - | SSH username (must have Docker permissions) | | `port` | int | No | `22` | SSH port on the remote host | | `ssh_key_path` | str | No\* | - | Path to SSH private key file (RSA, Ed25519, ECDSA) | | `ssh_private_key` | str | No\* | - | SSH private key content (supports `{{secret.key}}` references) | | `ssh_key_passphrase` | str | No | - | Passphrase for encrypted private key (supports `{{secret.key}}`) | | `verify_host_key` | bool | No | `True` | Verify remote host key against known\_hosts | | `known_hosts_path` | str | No | - | Path to known\_hosts file (defaults to system known\_hosts) | | `connection_timeout` | float | No | `10.0` | SSH connection timeout in seconds | | `keepalive_interval` | int | No | `30` | Seconds between SSH keepalive packets (0 to disable) | | `remote_workdir` | str | No | `/tmp/zenml-ssh` | Directory for temporary files on the remote host | | `docker_binary` | str | No | `docker` | Path to Docker binary on the remote host | \* `username` and at least one of `ssh_key_path` or `ssh_private_key` must be provided. #### Step Settings (per-step configuration) | Option | Type | Default | Description | | ----------------- | ---------- | ------- | ------------------------------------------------------------- | | `gpu_indices` | list\[int] | `None` | GPU device indices to expose via `--gpus` flag | | `docker_run_args` | list\[str] | `None` | Additional `docker run` arguments (e.g., `["--shm-size=2g"]`) | {% hint style="warning" %} Do not include secrets in `docker_run_args` as they may appear in process listings on the remote host. {% endhint %} ### Security notes #### Host key verification By default, `verify_host_key` is `True`, which uses paramiko's **RejectPolicy** to reject connections to unknown hosts. This protects against man-in-the-middle attacks. If you haven't connected to the remote host before, add its host key first: ```shell ssh-keyscan -H >> ~/.ssh/known_hosts ``` For ephemeral or test hosts, you can set `verify_host_key=False` to auto-accept unknown host keys (less secure). #### Secrets handling * The env-file uploaded to the remote host is created with **`0600` permissions** (owner read/write only) and is **automatically deleted** after the step completes. * Use ZenML secrets with `{{secret.key}}` syntax for `ssh_private_key` and `ssh_key_passphrase` to avoid storing credentials in plaintext. Check out the [SDK docs](https://sdkdocs.zenml.io/latest/) for the full API reference.
ZenML Scarf
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.zenml.io/stacks/stack-components/step-operators/ssh.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.