SSH

Executing individual steps on a remote host via SSH and Docker.

The SSH step operator allows you to execute individual ZenML steps on a remote Linux host via SSH. Steps run inside detached Docker containers on the remote machine, with optional GPU selection. This is ideal for teams that have dedicated GPU machines they want to use for training without needing a full cluster orchestration platform.

If you want to run entire pipelines on a remote host (not just individual steps), see the sibling SSH orchestrator. The two components share the same SSH connection layer and can target the same machine.

When a step operator step runs under the SSH orchestrator, it executes inside a container on the remote host rather than on your client. In that case configure the key via ssh_private_key instead of ssh_key_path, and set verify_host_key=False (or point known_hosts_path at a file baked into the image), since the container has no local key file or known_hosts.

When to use it

You should use the SSH step operator if:

• You have a dedicated remote machine (e.g., a GPU workstation or server) reachable via SSH.
• You want to run compute-heavy steps (training, inference) on that machine while keeping your orchestrator lightweight.
• You want to target specific GPUs on the host via --gpus.
• You prefer a simple setup without Kubernetes or cloud-managed services.

How to deploy it

The SSH step operator connects to an existing remote host; it does not provision infrastructure. Your remote host must meet these requirements:

• Linux operating system.
• Docker installed and accessible to the SSH user (the user should be in the docker group).
• Network reachability from the machine running your orchestrator to hostname:port over SSH.
• If using GPUs: the NVIDIA Container Toolkit must be installed so Docker supports --gpus.

How to use it

To use the SSH step operator, you need:

• The ZenML ssh integration installed. If you haven't done so, run:

  zenml integration install ssh

  This installs paramiko, the Python SSH library used under the hood.
• A remote artifact store as part of your stack (the remote container cannot access local files).
• A remote container registry to store the Docker images that will be pulled on the remote host.
• An image builder to build the Docker images for your steps.

We can then register the step operator. SSH credentials are configured directly on the step operator: a username and at least one of ssh_key_path or ssh_private_key must be provided for authentication.

Using a key file:

zenml step-operator register <NAME> \
    --flavor=ssh \
    --hostname=<HOST> \
    --username=<USER> \
    --ssh_key_path=~/.ssh/id_ed25519

Using a key stored in a ZenML secret:

zenml secret create ssh_secret \
    --private_key='<KEY_CONTENT>' \
    --passphrase='<PASSPHRASE>'

zenml step-operator register <NAME> \
    --flavor=ssh \
    --hostname=<HOST> \
    --username=<USER> \
    --ssh_private_key={{ssh_secret.private_key}} \
    --ssh_key_passphrase={{ssh_secret.passphrase}}

We can then add the step operator to our active stack:

zenml stack update -s <NAME>

Once you have added the step operator to your active stack, you can use it to execute individual steps of your pipeline by specifying it in the @step decorator:

from zenml import step


@step(step_operator="<NAME>")
def trainer(...) -> ...:
    """Train a model on the remote host."""
    # This step will be executed via SSH + Docker

ZenML will build a Docker image which includes your code and use it to run your steps on the remote host. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.

What happens at runtime

When a step runs via the SSH step operator:

1. ZenML builds and pushes the step image using your stack's image builder and container registry.
2. The step operator opens an SSH connection to the remote host.
3. A preflight check verifies that docker is available.
4. An env-file (with the step's environment variables) is uploaded to the remote working directory.
5. The Docker image is pulled on the remote host.
6. The container is started detached (docker run -d), and the env-file is deleted immediately afterwards so the step's secrets do not linger on the host.
7. ZenML polls the step's status with docker inspect until the container exits — the Docker daemon is the single source of truth — and reports success or failure. Step logs are captured by ZenML like any other step.

GPU execution

The SSH step operator supports targeting specific GPUs on the host.

Selecting GPUs

Use the gpu_indices setting to specify which GPU devices to expose to the container:

from zenml import step
from zenml.integrations.ssh.flavors import SSHStepOperatorSettings

ssh_settings = SSHStepOperatorSettings(
    gpu_indices=[0, 1],
    docker_run_args=["--shm-size=2g"],
)


@step(
    step_operator="<NAME>",
    settings={"step_operator": ssh_settings},
)
def train():
    """Train with GPUs 0 and 1."""
    ...

GPU indices are passed to Docker as --gpus "device=0,1". They must be non-negative integers matching the device indices reported by nvidia-smi on the remote host.

The step operator does not coordinate GPU access between concurrent steps. If you run multiple GPU steps on the same host at once, give them distinct gpu_indices so they don't oversubscribe the same device.

CPU-only execution

If gpu_indices is not set (the default), the step runs without GPU access:

ssh_settings = SSHStepOperatorSettings()  # CPU-only

@step(step_operator="<NAME>", settings={"step_operator": ssh_settings})
def preprocess():
    ...

Configuration options

Step Operator Configuration (set during registration)

Option	Type	Required	Default	Description
hostname	str	Yes	-	Hostname or IP of the remote SSH server
username	str	No*	-	SSH username (must have Docker permissions)
port	int	No	22	SSH port on the remote host
ssh_key_path	str	No*	-	Path to SSH private key file (RSA, Ed25519, ECDSA)
ssh_private_key	str	No*	-	SSH private key content (supports {{secret.key}} references)
ssh_key_passphrase	str	No	-	Passphrase for encrypted private key (supports {{secret.key}})
verify_host_key	bool	No	True	Verify remote host key against known_hosts
known_hosts_path	str	No	-	Path to known_hosts file (defaults to system known_hosts)
connection_timeout	float	No	10.0	SSH connection timeout in seconds
keepalive_interval	int	No	30	Seconds between SSH keepalive packets (0 to disable)
remote_workdir	str	No	/tmp/zenml-ssh	Directory for temporary files on the remote host
docker_binary	str	No	docker	Path to Docker binary on the remote host

* username and at least one of ssh_key_path or ssh_private_key must be provided.

Step Settings (per-step configuration)

Option	Type	Default	Description
gpu_indices	list[int]	None	GPU device indices to expose via --gpus flag
docker_run_args	list[str]	None	Additional docker run arguments (e.g., ["--shm-size=2g"])

Do not include secrets in docker_run_args as they may appear in process listings on the remote host.

Security notes

Host key verification

By default, verify_host_key is True, which uses paramiko's RejectPolicy to reject connections to unknown hosts. This protects against man-in-the-middle attacks.

If you haven't connected to the remote host before, add its host key first:

ssh-keyscan -H <HOST> >> ~/.ssh/known_hosts

For ephemeral or test hosts, you can set verify_host_key=False to auto-accept unknown host keys (less secure).

Secrets handling

• The env-file uploaded to the remote host is created with 0600 permissions (owner read/write only) and is automatically deleted after the step completes.
• Use ZenML secrets with {{secret.key}} syntax for ssh_private_key and ssh_key_passphrase to avoid storing credentials in plaintext.

Check out the SDK docs for the full API reference.