Skip to main content

dagster-slurm

Develop locally. Run on Slurm. Orchestrate the whole pipeline.

Configure a cluster once, then materialize Dagster assets with the payload from your current checkout and a Pixi-locked environment. Keep local feedback, Slurm telemetry, and data dependencies in one graph.

Get startedView on GitHub
server_cluster

Configure once, then click Materialize

dagster-slurm transfers the payload from your current checkout, provides its Pixi-locked environment, submits it with sbatch, and returns the job status and logs to Dagster.

Fast feedback before HPC

Run the same asset locally without SSH or Slurm. Developers and coding agents can iterate quickly, then switch the compute configuration when the workload is ready for the cluster.

monitor

Advanced HPC controls

Dagster reports Slurm state, logs, CPU efficiency, memory, elapsed time, and node-hours. Choose environments per asset and define a separate ComputeResource per cluster. Run-scoped Ray can reuse one allocation; shared-allocation sessions and HET jobs are experimental.

Before, during, and after HPC

Before: ingest and validate data. During: send the heavy asset to Slurm. After: aggregate, publish, or serve the result. Dagster dependencies and lineage connect every stage, even when they run in different places.