Troubleshooting connectivity & logs

The dagster-slurm presets hide most SSH and Slurm plumbing, but a few deployment-specific quirks are worth calling out. The tips below keep log streaming responsive, explain how to retain run artefacts for inspection, and silence noisy shells in the Docker sandbox.

SSH connection multiplexing

Supervising a Slurm job is chatty: status polling, log tailing and Pipes message reading all talk to the login node. Without multiplexing every one of those would be a separate DNS lookup, TCP handshake and authentication. HPC centres rate-limit — and increasingly block — accounts that do this, so dagster-slurm funnels every SSH and SCP invocation through a single shared OpenSSH ControlMaster connection per user@host:port.

The socket lives at ~/.ssh/dagster-slurm/cm-<user>@<host>:<port> and is deliberately not per-process: a master started by one Dagster step, sensor tick or CLI helper is reused by the next one. It is left running after a run finishes and expires on its own via ControlPersist.

Variable	Default	Purpose
DAGSTER_SLURM_SSH_CONTROL_DIR	~/.ssh/dagster-slurm	Where control sockets are stored. Point it at a writable directory if your site restricts socket paths.
DAGSTER_SLURM_SSH_CONTROL_PERSIST	10m	How long an idle master stays alive after the last command.
DAGSTER_SLURM_SSH_CLOSE_MASTER_ON_EXIT	unset	Set to 1 to tear the master down when a run ends instead of leaving it for reuse.
DAGSTER_SLURM_SSH_FAILURE_BACKOFF_BASE	60	Seconds to wait after a failed master start before the next login. 0 disables the backoff.
DAGSTER_SLURM_SSH_FAILURE_BACKOFF_MAX	1800	Upper limit in seconds for that wait. The wait doubles for each consecutive failure.

Note that these command-line options take precedence over ~/.ssh/config, so a ControlPath entry there will not change what dagster-slurm does — it only affects SSH commands you run yourself.

When multiplexing is unavailable

The transport stops when the master cannot start or recover. It never falls back to one connection per command, transfer, sensor tick or log stream:

SSH MULTIPLEXING FAILED for <host>. Refusing unpooled SSH commands and transfers.

Commands attach with ControlMaster=no and ProxyCommand=false. The latter blocks OpenSSH's own direct-connection fallback if the socket disappears between a health check and command execution. A live master bypasses the proxy command. Master creation is serialized across processes for each target.

Breaking change: configurations with an explicit target or jump-host password are rejected before connecting. Use a key or inherited ssh-agent authentication. This implementation does not authenticate a background master through interactive prompts. A cluster that forbids multiplexing cannot use this transport safely.

Backoff after a refused login

Some login nodes ban accounts that log in too often, and each refused login can extend the ban. When a master start fails, dagster-slurm writes the time, the count of consecutive failures and the reason to <control socket>.failed. Until the backoff window ends, every new master start for that user@host:port stops at once with SSHMasterBackoffError (a RuntimeError) and does not run ssh. The error names the time of the next attempt. All processes that use the same control directory share the marker, so sensors, run workers and helpers make one login attempt per window together.

The window is 1 minute after the first failure and doubles for each consecutive failure, up to 30 minutes. A successful master start removes the marker. A master that is already alive is used, whatever the marker says. To try again at once after you fix the cause, delete the .failed file.

The orphan-reconcile sensor (build_slurm_orphan_reconcile_sensor) opens the SSH connection only when a started run has a Slurm job ID and a run directory. A tick without such runs does not log in.

Raw commands returned by get_ssh_base_command() and get_scp_base_command() require an existing master. Use SSHConnectionPool to establish it first; the legacy ssh_run helper does this automatically. Jump-host connections have their own shared master and are established before the target master.

Slurm status polling cadence

While a job runs, its state is polled over the shared connection. The interval starts at status_poll_interval_seconds and backs off by status_poll_backoff_factor up to status_poll_max_interval_seconds for as long as the state stays unchanged; any state transition resets it, so completion is still noticed promptly.

Variable	Field	Default
SLURM_STATUS_POLL_INTERVAL	status_poll_interval_seconds	2.0
SLURM_STATUS_POLL_MAX_INTERVAL	status_poll_max_interval_seconds	5.0
SLURM_STATUS_POLL_BACKOFF	status_poll_backoff_factor	1.5

Raise these if your site asks for fewer squeue/sacct queries; lower them if you need tighter completion latency and the site is happy with the load. At the defaults a four-hour job costs about 2,880 status queries instead of 14,400, and completion is never noticed more than status_poll_max_interval_seconds late.

The sleep is clamped to the remaining poll_timeout budget, so backing off can never step over the deadline and report a job that finished in time as a timeout.

How SSH settings interact with ~/.ssh/config

OpenSSH always prefers a command-line -o over anything in a config file, so every option SSHConnectionResource emits overrides your ~/.ssh/config for that connection. This is deliberate: a Dagster daemon should behave the same on your laptop, a colleague's machine, and a CI runner, rather than depending on whatever happens to be in the operator's home directory.

Where that is not what you want, each setting can be handed back to OpenSSH:

Setting	Explicit (default)	Deferred
Host-key policy	host_key_checking="strict" emits StrictHostKeyChecking and UserKnownHostsFile	host_key_checking="inherit" emits neither
Authentication	key_path=...	leave both unset — ssh-agent, IdentityFile and friends apply
Prompting	batch_mode=True emits BatchMode=yes	batch_mode=None emits nothing
Jump host	a jump_host uses an attach-only ProxyCommand	unspecified jump-host options remain inherited

Each explicit jump host has its own shared master. The target master attaches to it through ProxyCommand; commands and streams attach to the target master.

extra_opts overrides authentication and host-key defaults. Transport options (ControlPath, ControlMaster, ControlPersist, ProxyCommand, ProxyJump and their command-line flags) are reserved for the pool and rejected there. The create argument is removed from the multiplexing option builders: master creation belongs to the pool's serialized startup.

:::caution BatchMode and key passphrases The default batch_mode=True refuses every interactive prompt, which is what a non-interactive daemon needs — otherwise a hidden passphrase prompt would hang the run forever. A passphrase-protected key therefore fails with Permission denied unless it is loaded into ssh-agent. Set batch_mode=None if you need OpenSSH to ask. :::

Keep or delete remote run directories

Every successful run now schedules an asynchronous cleanup on the edge node:

nohup rm -rf /path/to/dagster_runs/runs/<run_id> >/dev/null 2>&1 &

That keeps storage quotas tidy without delaying Dagster shutdown. You can still opt out temporarily:

• Set debug_mode=True on the relevant ComputeResource to keep run folders around for inspection.
• Manually remove old folders with a periodic cron job if you prefer a deterministic retention policy:

find /home/$USER/dagster_runs/runs -mindepth 1 -maxdepth 1 -mtime +3 -exec rm -rf {} +

The cleanup_on_failure flag continues to apply—failed runs respect debug_mode and skip removal.

Laptop and one-off researcher deployments

Not every HPC workflow needs a persistent Dagster server. A common pattern is running Dagster directly on a researcher's laptop or workstation: submit long-running Slurm jobs, close the lid, and check results hours or days later. dagster-slurm is designed to support this workflow.

Why this works without persistent infrastructure:

• Slurm is the durable layer. Once sbatch submits a job, the HPC scheduler owns it. The job runs to completion regardless of what happens to the Dagster process that submitted it. Logs, outputs, and exit codes are stored on the cluster.
• Dagster is the orchestration layer. It packages the environment, submits the job, and streams logs while it is running. But it does not need to be running for the Slurm job to finish.
• Reattach on restart. When you reopen your laptop and start Dagster again, dagster-slurm detects the still-running (or already-completed) Slurm job and picks up where it left off. It never resubmits — the retry only monitors or collects results from the existing job.

Recommended setup for laptop deployments:

from dagster import Definitions, define_asset_job, in_process_executor

# In-process executor keeps runs green through Dagster restarts.
# For laptop use this is almost always the right choice — there is
# no benefit to multiprocess when you are submitting to Slurm anyway.
slurm_job = define_asset_job(
    "slurm_job",
    executor_def=in_process_executor,
)

With this setup, you can:

1. Start Dagster and materialize assets — jobs are submitted to the cluster.
2. Stop Dagster (or close your laptop) — Slurm jobs continue running on the cluster.
3. Restart Dagster — if the process was still monitoring a job when it was stopped, the in-process executor catches the signal and the run completes on the next startup. If the process was already gone, a retry reattaches.

There is no need for a persistent Dagster daemon, database server, or always-on VM. The HPC cluster provides the durability; Dagster provides the developer experience (log streaming, metadata, lineage, retries).

:::tip When to deploy persistent Dagster infrastructure

Consider a dedicated Dagster server (e.g. on a VM or Kubernetes) when you need:

• Scheduled runs (cron-like triggers via ScheduleDefinition)
• Sensors that react to external events
• Team-wide visibility across multiple users' runs
• Automated backfill orchestration across many partitions

For a single researcher iterating on HPC workloads, a laptop deployment is simpler and sufficient.

:::

Dagster restarts and Slurm job survival

When Dagster restarts (e.g. a deploy, code location reload, or process crash), it sends SIGTERM to running processes. How this affects your Slurm jobs depends on the executor you are using.

In-process executor (recommended for Slurm workloads)

With the in-process executor, dagster-slurm catches the SIGTERM, lets the Slurm job continue running, and keeps monitoring it. The run completes successfully — the Slurm job is never cancelled.

This is the recommended configuration when you expect frequent Dagster restarts and want green run results:

from dagster import Definitions, define_asset_job, in_process_executor

slurm_job = define_asset_job(
    "slurm_job",
    executor_def=in_process_executor,
)

Multiprocess executor

:::warning Multiprocess executor always fails runs on SIGTERM

With the multiprocess executor, each asset step runs in a separate child process. On SIGTERM, the parent executor process unconditionally raises DagsterExecutionInterruptedError and marks the run as FAILURE — this is Dagster core behaviour that dagster-slurm cannot override.

The Slurm job itself is not cancelled and continues running on the cluster. However, the Dagster run will show as failed.

:::

To recover automatically, configure run-level retries in your dagster.yaml. This tells the dagster-daemon to re-launch failed runs — and dagster-slurm never resubmits the Slurm job. It queries the cluster to find the existing job and either reattaches (if still running) or collects its results (if already completed):

# dagster.yaml
run_retries:
  max_retries: 3
  retry_on_asset_or_op_failure: false # only retry on crashes/interrupts

On re-launch, dagster-slurm looks up the Slurm job submitted by the previous run using run tags. Three outcomes are possible:

1. Job still running — reattach and monitor to completion. No new submission.
2. Job already completed — collect metrics from sacct and mark the run as successful. No new submission.
3. Job failed or not found — only then does dagster-slurm submit a new job (the previous one genuinely failed).

This means run retries are always safe for HPC workloads: a lengthy training run will never be duplicated, even if Dagster restarts multiple times while the job is in progress.

:::note run_retries requires the dagster-daemon

The dagster-daemon process monitors failed runs and re-launches them. For laptop deployments without a daemon, use the in-process executor instead (no retry needed — the run survives SIGTERM).

:::

Summary:

Executor	On SIGTERM	Run result	Slurm job	Recommendation
In-process	Monitoring continues	Success	Keeps running	Use this for laptop / no-daemon deployments
Multiprocess	Run fails (unavoidable)	Failure, then Success on retry	Keeps running	Configure run_retries in dagster.yaml + daemon

For partitioned backfills using the multiprocess executor (parallel partitions), each partition independently reattaches to its own Slurm job on retry.