Troubleshooting connectivity & logs

The dagster-slurm presets hide most SSH and Slurm plumbing, but a few deployment-specific quirks are worth calling out. The tips below keep log streaming responsive, explain how to retain run artefacts for inspection, and silence noisy shells in the Docker sandbox.

SSH connection multiplexing

Supervising a Slurm job is chatty: status polling, log tailing and Pipes message reading all talk to the login node. Without multiplexing every one of those would be a separate DNS lookup, TCP handshake and authentication. HPC centres rate-limit — and increasingly block — accounts that do this, so dagster-slurm funnels every SSH and SCP invocation through a single shared OpenSSH ControlMaster connection per user@host:port.

The socket lives at ~/.ssh/dagster-slurm/cm-<user>@<host>:<port> and is deliberately not per-process: a master started by one Dagster step, sensor tick or CLI helper is reused by the next one. It is left running after a run finishes and expires on its own via ControlPersist.

Variable	Default	Purpose
DAGSTER_SLURM_SSH_CONTROL_DIR	~/.ssh/dagster-slurm	Where control sockets are stored. Point it at a writable directory if your site restricts socket paths.
DAGSTER_SLURM_SSH_CONTROL_PERSIST	10m	How long an idle master stays alive after the last command.
DAGSTER_SLURM_SSH_CLOSE_MASTER_ON_EXIT	unset	Set to 1 to tear the master down when a run ends instead of leaving it for reuse.

Note that these command-line options take precedence over ~/.ssh/config, so a ControlPath entry there will not change what dagster-slurm does — it only affects SSH commands you run yourself.

When multiplexing is unavailable

Password-based authentication (on the target or on a jump host) cannot be multiplexed, because a shared connection has nowhere to answer a prompt. In that case Dagster falls back to one-off connections and polls the remote log files instead of tailing them, printing a debug line such as:

Streaming ... via polling fallback (ControlMaster unavailable)

Use key-based authentication wherever possible. If the master fails to start, or dies mid-run and cannot be restarted, Dagster logs a run-level error:

SSH MULTIPLEXING FAILED for <host>. Dagster is falling back to one-off SSH connections. ...

Treat that as actionable: stop the run and fix the SSH configuration before the connection rate becomes a problem for the site.

Other tips:

• Leave SLURM_EDGE_NODE_FORCE_TTY=false unless your cluster explicitly requires a TTY. Password-based sessions request one automatically.
• If you rely on jump hosts or OTP prompts, prefer a local ~/.ssh/config entry. The same configuration is reused by the Pipes log tailer.

Slurm status polling cadence

While a job runs, its state is polled over the shared connection. The interval starts at status_poll_interval_seconds and backs off by status_poll_backoff_factor up to status_poll_max_interval_seconds for as long as the state stays unchanged; any state transition resets it, so completion is still noticed promptly.

Variable	Field	Default
SLURM_STATUS_POLL_INTERVAL	status_poll_interval_seconds	2.0
SLURM_STATUS_POLL_MAX_INTERVAL	status_poll_max_interval_seconds	5.0
SLURM_STATUS_POLL_BACKOFF	status_poll_backoff_factor	1.5

Raise these if your site asks for fewer squeue/sacct queries; lower them if you need tighter completion latency and the site is happy with the load. At the defaults a four-hour job costs about 2,880 status queries instead of 14,400, and completion is never noticed more than status_poll_max_interval_seconds late.

The sleep is clamped to the remaining poll_timeout budget, so backing off can never step over the deadline and report a job that finished in time as a timeout.

How SSH settings interact with ~/.ssh/config

OpenSSH always prefers a command-line -o over anything in a config file, so every option SSHConnectionResource emits overrides your ~/.ssh/config for that connection. This is deliberate: a Dagster daemon should behave the same on your laptop, a colleague's machine, and a CI runner, rather than depending on whatever happens to be in the operator's home directory.

Where that is not what you want, each setting can be handed back to OpenSSH:

Setting	Explicit (default)	Deferred
Host-key policy	host_key_checking="strict" emits StrictHostKeyChecking and UserKnownHostsFile	host_key_checking="inherit" emits neither
Authentication	key_path=... or password=...	leave both unset — ssh-agent, IdentityFile and friends apply
Prompting	batch_mode=True emits BatchMode=yes	batch_mode=None emits nothing
Jump host	a jump_host with any of the above set becomes an explicit ProxyCommand	a jump_host with none of them set becomes plain ssh -J

A jump host is only rebuilt as an explicit ProxyCommand when the resource actually configures something for it — that is what makes the bastion's own key and host-key policy apply. If you configure nothing beyond host, user and port, it falls back to -J and your own bastion settings keep working.

extra_opts is applied before the generated options, and OpenSSH takes the first value it sees, so it overrides anything above.

:::caution BatchMode and key passphrases The default batch_mode=True refuses every interactive prompt, which is what a non-interactive daemon needs — otherwise a hidden passphrase prompt would hang the run forever. A passphrase-protected key therefore fails with Permission denied unless it is loaded into ssh-agent. Set batch_mode=None if you need OpenSSH to ask. :::

Keep or delete remote run directories

Every successful run now schedules an asynchronous cleanup on the edge node:

nohup rm -rf /path/to/dagster_runs/runs/<run_id> >/dev/null 2>&1 &

That keeps storage quotas tidy without delaying Dagster shutdown. You can still opt out temporarily:

• Set debug_mode=True on the relevant ComputeResource to keep run folders around for inspection.
• Manually remove old folders with a periodic cron job if you prefer a deterministic retention policy:

find /home/$USER/dagster_runs/runs -mindepth 1 -maxdepth 1 -mtime +3 -exec rm -rf {} +

The cleanup_on_failure flag continues to apply—failed runs respect debug_mode and skip removal.

Laptop and one-off researcher deployments

Not every HPC workflow needs a persistent Dagster server. A common pattern is running Dagster directly on a researcher's laptop or workstation: submit long-running Slurm jobs, close the lid, and check results hours or days later. dagster-slurm is designed to support this workflow.

Why this works without persistent infrastructure:

• Slurm is the durable layer. Once sbatch submits a job, the HPC scheduler owns it. The job runs to completion regardless of what happens to the Dagster process that submitted it. Logs, outputs, and exit codes are stored on the cluster.
• Dagster is the orchestration layer. It packages the environment, submits the job, and streams logs while it is running. But it does not need to be running for the Slurm job to finish.
• Reattach on restart. When you reopen your laptop and start Dagster again, dagster-slurm detects the still-running (or already-completed) Slurm job and picks up where it left off. It never resubmits — the retry only monitors or collects results from the existing job.

Recommended setup for laptop deployments:

from dagster import Definitions, define_asset_job, in_process_executor

# In-process executor keeps runs green through Dagster restarts.
# For laptop use this is almost always the right choice — there is
# no benefit to multiprocess when you are submitting to Slurm anyway.
slurm_job = define_asset_job(
    "slurm_job",
    executor_def=in_process_executor,
)

With this setup, you can:

1. Start Dagster and materialize assets — jobs are submitted to the cluster.
2. Stop Dagster (or close your laptop) — Slurm jobs continue running on the cluster.
3. Restart Dagster — if the process was still monitoring a job when it was stopped, the in-process executor catches the signal and the run completes on the next startup. If the process was already gone, a retry reattaches.

There is no need for a persistent Dagster daemon, database server, or always-on VM. The HPC cluster provides the durability; Dagster provides the developer experience (log streaming, metadata, lineage, retries).

:::tip When to deploy persistent Dagster infrastructure

Consider a dedicated Dagster server (e.g. on a VM or Kubernetes) when you need:

• Scheduled runs (cron-like triggers via ScheduleDefinition)
• Sensors that react to external events
• Team-wide visibility across multiple users' runs
• Automated backfill orchestration across many partitions

For a single researcher iterating on HPC workloads, a laptop deployment is simpler and sufficient.

:::

Dagster restarts and Slurm job survival

When Dagster restarts (e.g. a deploy, code location reload, or process crash), it sends SIGTERM to running processes. How this affects your Slurm jobs depends on the executor you are using.

In-process executor (recommended for Slurm workloads)

With the in-process executor, dagster-slurm catches the SIGTERM, lets the Slurm job continue running, and keeps monitoring it. The run completes successfully — the Slurm job is never cancelled.

This is the recommended configuration when you expect frequent Dagster restarts and want green run results:

from dagster import Definitions, define_asset_job, in_process_executor

slurm_job = define_asset_job(
    "slurm_job",
    executor_def=in_process_executor,
)

Multiprocess executor

:::warning Multiprocess executor always fails runs on SIGTERM

With the multiprocess executor, each asset step runs in a separate child process. On SIGTERM, the parent executor process unconditionally raises DagsterExecutionInterruptedError and marks the run as FAILURE — this is Dagster core behaviour that dagster-slurm cannot override.

The Slurm job itself is not cancelled and continues running on the cluster. However, the Dagster run will show as failed.

:::

To recover automatically, configure run-level retries in your dagster.yaml. This tells the dagster-daemon to re-launch failed runs — and dagster-slurm never resubmits the Slurm job. It queries the cluster to find the existing job and either reattaches (if still running) or collects its results (if already completed):

# dagster.yaml
run_retries:
  max_retries: 3
  retry_on_asset_or_op_failure: false # only retry on crashes/interrupts

On re-launch, dagster-slurm looks up the Slurm job submitted by the previous run using run tags. Three outcomes are possible:

1. Job still running — reattach and monitor to completion. No new submission.
2. Job already completed — collect metrics from sacct and mark the run as successful. No new submission.
3. Job failed or not found — only then does dagster-slurm submit a new job (the previous one genuinely failed).

This means run retries are always safe for HPC workloads: a lengthy training run will never be duplicated, even if Dagster restarts multiple times while the job is in progress.

:::note run_retries requires the dagster-daemon

The dagster-daemon process monitors failed runs and re-launches them. For laptop deployments without a daemon, use the in-process executor instead (no retry needed — the run survives SIGTERM).

:::

Summary:

Executor	On SIGTERM	Run result	Slurm job	Recommendation
In-process	Monitoring continues	Success	Keeps running	Use this for laptop / no-daemon deployments
Multiprocess	Run fails (unavoidable)	Failure, then Success on retry	Keeps running	Configure run_retries in dagster.yaml + daemon

For partitioned backfills using the multiprocess executor (parallel partitions), each partition independently reattaches to its own Slurm job on retry.