remote

how winch reaches a host and runs there

winch works on a host through one SSH connection, the session, with a helper program at the other end.

What a host needs

• Debian 13, Ubuntu 24.04, or Ubuntu 26.04, on amd64 or arm64.
• Login as root, or as a user with passwordless sudo.
• A POSIX login shell that prints nothing on stdout before winch starts a command.
• /var/lib/winch/ on a file system that allows running programs.

Nothing else needs to be installed. The machine that runs winch is Linux or macOS with the OpenSSH client.

SSH settings

winch runs your ssh, so ~/.ssh/config applies: Port, IdentityFile, ProxyJump, and every other setting. Its Host blocks match the host name of the inventory, also when the host has an address. The address and the user of the inventory override the SSH config. --user overrides both.

# ~/.ssh/config
Host *.acme.com
  Port 2222
  IdentityFile ~/.ssh/acme_ed25519

# inventories/production.yaml
hosts:
  web1.acme.com:
    address: 192.0.2.10
    user: admin

# winch logs in to 192.0.2.10 on port 2222 as admin,
# with the key ~/.ssh/acme_ed25519.

ssh_config in winch.yaml names another SSH config file, relative to the project directory. winch passes it with -F to every ssh it runs: the sessions, the access checks, and each hop of a ProxyJump. ssh then reads this file instead of ~/.ssh/config. Use it for the SSH config that a tool such as Lima or Vagrant writes. To keep ~/.ssh/config, include it:

# winch.yaml
name: acme
ssh_config: ssh_config

# ssh_config
Include ~/.ssh/config
Include ~/.lima/default/ssh.config

winch also passes these options. They win over the SSH config:

Host keys and passphrases

When stdin and stderr are terminals, ssh can ask for a new host key or a key passphrase. Sessions then start one at a time, in inventory order, so each question belongs to one host. After the start, hosts run in parallel.

Without a terminal, or with --batch, winch passes BatchMode=yes and sessions start in parallel. A host that needs a question is then unreachable. Accept the host key and load the key into ssh-agent first:

ssh -o HostName=192.0.2.10 -l admin web1.acme.com
ssh-add ~/.ssh/acme_ed25519
winch apply site --batch

# ssh asks no questions; the apply question stays.

The access checks and the connections after a reboot always use BatchMode=yes, so they cannot ask for anything. Run winch doc workflow for the access checks. At a terminal, a session therefore logs in with BatchMode=yes first. When the host refuses that login, or its host key is new, winch logs in again, and ssh can ask. A host that refused the first login plans as usual, but apply stops before its first change and names ssh-add. After a new host key, apply runs one access check before the first change.

When a connection fails

A host is unreachable when ssh fails before the session starts: the name does not resolve, the connection is refused or takes more than 15 seconds, or the host key or the login fails. winch then exits 4 when no step failed. For an unknown host key, the error names the ssh command to run once. For a refused login, it names ssh-add.

Only one winch run works on a host at a time. When another run holds the host, the error names that run and the process ID of its helper. When that run is gone, end the process.

The session

A session is one ssh process with the helper at the other end. All work on a host goes through it. It opens at the first work of the host and stays open across stages and through the question. The host lock stays held while the question waits; Ctrl-C releases it. A session closes when its host has no more work. The access checks open their own connections.

winch keeps the helper in /var/lib/winch/bin/ and sends it when it is missing. The helper exits when the session ends and leaves no process, except a confirm_access timer until it ends.

The helpers for amd64 and arm64 are inside the winch binary. A winch built with go install has none: plan, apply, run, and exec fail before they contact a host. Install a release.

How commands run

Every command on the host, also one that a module runs, such as apt-get or systemctl, runs as root in /, with only this environment:

Nothing comes from the login session, such as the TMPDIR that libpam-tmpdir sets.

The commands of cmd, assert, unless, validate, then, health, and the certbot hooks run through sh -e, with stdin from /dev/null. A command of several lines stops at its first failed line.

A cmd with dir runs in that directory. A cmd with user runs as that user, with the home of that user as HOME, and its name as USER and LOGNAME. It starts no login session: PAM does not run, so the login umask and limits of the user do not apply.

The time limit

A command stops after 10 minutes, after the timeout of a cmd step in seconds, or after the --timeout of winch exec. The helper sends SIGTERM to the command and its child processes, and SIGKILL 10 seconds later to those that still run. The step fails, and the error shows the last 10 lines of output. apt-get in pkg, repo, and upgrade has no time limit.

When the session ends, the helper lets the running command finish within its limit, and then exits. A package install is not cut off in the middle. winch exec is different: on Ctrl-C, or when its session breaks, its command gets SIGINT, and SIGKILL after 10 seconds.

What winch leaves on the host

winch keeps its files in /var/lib/winch/:

To remove winch from a host, delete /var/lib/winch/. Pending handlers, then commands, health gates, asserts, and certbot hooks are then lost. The next run of winch starts from the state of the host.