For the complete documentation index, see llms.txt. This page is also available as Markdown.

SSH

Executing individual steps on a remote host via SSH and Docker.

The SSH step operator allows you to execute individual ZenML steps on a remote Linux host via SSH. Steps run inside detached Docker containers on the remote machine, with optional GPU selection. This is ideal for teams that have dedicated GPU machines they want to use for training without needing a full cluster orchestration platform.

If you want to run entire pipelines on a remote host (not just individual steps), see the sibling SSH orchestrator. The two components share the same SSH connection layer and can target the same machine.

When to use it

You should use the SSH step operator if:

  • You have a dedicated remote machine (e.g., a GPU workstation or server) reachable via SSH.

  • You want to run compute-heavy steps (training, inference) on that machine while keeping your orchestrator lightweight.

  • You want to target specific GPUs on the host via --gpus.

  • You prefer a simple setup without Kubernetes or cloud-managed services.

How to deploy it

The SSH step operator connects to an existing remote host; it does not provision infrastructure. Your remote host must meet these requirements:

  • Linux operating system.

  • Docker installed and accessible to the SSH user (the user should be in the docker group).

  • Network reachability from the machine running your orchestrator to hostname:port over SSH.

  • If using GPUs: the NVIDIA Container Toolkit must be installed so Docker supports --gpus.

How to use it

To use the SSH step operator, you need:

  • The ZenML ssh integration installed. If you haven't done so, run:

    This installs paramiko, the Python SSH library used under the hood.

  • A remote artifact store as part of your stack (the remote container cannot access local files).

  • A remote container registry to store the Docker images that will be pulled on the remote host.

  • An image builder to build the Docker images for your steps.

We can then register the step operator. SSH credentials are configured directly on the step operator: a username and at least one of ssh_key_path or ssh_private_key must be provided for authentication.

Using a key file:

Using a key stored in a ZenML secret:

We can then add the step operator to our active stack:

Once you have added the step operator to your active stack, you can use it to execute individual steps of your pipeline by specifying it in the @step decorator:

ZenML will build a Docker image which includes your code and use it to run your steps on the remote host. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.

What happens at runtime

When a step runs via the SSH step operator:

  1. ZenML builds and pushes the step image using your stack's image builder and container registry.

  2. The step operator opens an SSH connection to the remote host.

  3. A preflight check verifies that docker is available.

  4. An env-file (with the step's environment variables) is uploaded to the remote working directory.

  5. The Docker image is pulled on the remote host.

  6. The container is started detached (docker run -d), and the env-file is deleted immediately afterwards so the step's secrets do not linger on the host.

  7. ZenML polls the step's status with docker inspect until the container exits — the Docker daemon is the single source of truth — and reports success or failure. Step logs are captured by ZenML like any other step.

GPU execution

The SSH step operator supports targeting specific GPUs on the host.

Selecting GPUs

Use the gpu_indices setting to specify which GPU devices to expose to the container:

GPU indices are passed to Docker as --gpus "device=0,1". They must be non-negative integers matching the device indices reported by nvidia-smi on the remote host.

The step operator does not coordinate GPU access between concurrent steps. If you run multiple GPU steps on the same host at once, give them distinct gpu_indices so they don't oversubscribe the same device.

CPU-only execution

If gpu_indices is not set (the default), the step runs without GPU access:

Configuration options

Step Operator Configuration (set during registration)

Option
Type
Required
Default
Description

hostname

str

Yes

-

Hostname or IP of the remote SSH server

username

str

No*

-

SSH username (must have Docker permissions)

port

int

No

22

SSH port on the remote host

ssh_key_path

str

No*

-

Path to SSH private key file (RSA, Ed25519, ECDSA)

ssh_private_key

str

No*

-

SSH private key content (supports {{secret.key}} references)

ssh_key_passphrase

str

No

-

Passphrase for encrypted private key (supports {{secret.key}})

verify_host_key

bool

No

True

Verify remote host key against known_hosts

known_hosts_path

str

No

-

Path to known_hosts file (defaults to system known_hosts)

connection_timeout

float

No

10.0

SSH connection timeout in seconds

keepalive_interval

int

No

30

Seconds between SSH keepalive packets (0 to disable)

remote_workdir

str

No

/tmp/zenml-ssh

Directory for temporary files on the remote host

docker_binary

str

No

docker

Path to Docker binary on the remote host

* username and at least one of ssh_key_path or ssh_private_key must be provided.

Step Settings (per-step configuration)

Option
Type
Default
Description

gpu_indices

list[int]

None

GPU device indices to expose via --gpus flag

docker_run_args

list[str]

None

Additional docker run arguments (e.g., ["--shm-size=2g"])

Security notes

Host key verification

By default, verify_host_key is True, which uses paramiko's RejectPolicy to reject connections to unknown hosts. This protects against man-in-the-middle attacks.

If you haven't connected to the remote host before, add its host key first:

For ephemeral or test hosts, you can set verify_host_key=False to auto-accept unknown host keys (less secure).

Secrets handling

  • The env-file uploaded to the remote host is created with 0600 permissions (owner read/write only) and is automatically deleted after the step completes.

  • Use ZenML secrets with {{secret.key}} syntax for ssh_private_key and ssh_key_passphrase to avoid storing credentials in plaintext.

Check out the SDK docs for the full API reference.

ZenML Scarf

Last updated

Was this helpful?