What Did My Job Actually Use? Slurm Accounting with sacct and seff on Your HPC Auto Scale Cluster

Learn how to check what your Slurm jobs really used on a RONIN HPC Auto Scale Cluster with sacct and seff, then right-size your next job so it asks for what it needs.

RONIN feature image with the title What Did My Job Actually Use? and a seff report card showing CPU Efficiency 99.34% and Memory Efficiency 73.38% for a completed job

Your job finished. Great! But did it actually use the 2 CPUs and 6 GB of memory you asked for? Or did it sit on one CPU and a few hundred MB, holding a whole node while other jobs waited in the queue?

Every HPC Auto Scale Cluster in RONIN runs native Slurm with job accounting switched on for you, so you do not have to guess. In this tutorial, we will run a handful of small jobs, then use seff to get a one-page report card for each job, sacct to dig into the details, and both to find out why a job failed. Then we will use the report card to right-size a job so it asks for what it needs.

This guide is for researchers who are comfortable submitting Slurm jobs with sbatch. If you have not met the new clusters yet, start with Meet the New HPC Auto Scale Clusters.

Why accounting matters

Each job gets the CPUs and memory it asks for, and several jobs share a node until its CPUs or memory are used up. That makes your requests important:

  • Ask for too much and your job holds CPUs and memory it never uses. Other jobs (including your own) cannot use them, so they wait in the queue or launch extra compute nodes.
  • Ask for too little and your job can run slowly, use more memory than was set aside for it.
  • Ask for the right amount and more jobs fit on each node, so your work starts sooner and finishes sooner.

Slurm records what every job asked for and what it actually used. Checking that record after a test run is the easiest way to get your next request right.

How it works

On your cluster:

  • While a job runs, Slurm checks its memory use every 30 seconds and records its total CPU time when each step ends.
  • slurmctld, the Slurm scheduler on your head node, keeps track of every job and job step.
  • slurmdbd, also on your head node, saves each job's record to the accounting database.
  • sacct and seff read those records back for you, from the head node.
Diagram. On the compute node, Slurm checks your job's memory use every 30 seconds and records its total CPU time when each step ends. On the head node, slurmctld tracks every job and step, slurmdbd saves job records to the accounting database, and sacct and seff show what your jobs used. A banner reads: Job accounting is switched on for you. Run sacct and seff on the head node, no setup needed.
How job accounting data gets from your job to sacct and seff.

Before you begin

You will need:

  • A running HPC Auto Scale Cluster. Any size will do. A cluster with Min nodes set to 0 is perfect for this tutorial.
  • Your cluster's SSH key, so you can log in to the head node.
  • A few finished jobs to look at. Any jobs you have run today will work, or follow along with the examples below.
  • A basic idea of how Slurm job scripts work. If #SBATCH directives are new to you, start with our simple Slurm guide for beginners.

The examples in this post come from a cluster with t3.large compute nodes (2 vCPUs and 8 GiB of memory each), Min nodes set to 0 and Max nodes set to 4. Slurm reports 7,372 MB of memory on each of these nodes.

Step 1: Connect to your head node

Open a terminal on your head node. The terminal button on your cluster's card in RONIN LINK is the quickest way. You can also SSH to your cluster's address from your own terminal, using your cluster's SSH key:

ssh -i ~/.ssh/your-cluster-key.pem ubuntu@slurm-accounting.hpc.ronin.cloud
A running cluster card in RONIN showing the OPEN IN RONIN LINK button, the cluster address, 22 SSH, the ubuntu user, a T3.LARGE head node and T3.LARGE compute nodes with Min nodes 0 and Max nodes 4
Your cluster's address and the OPEN IN RONIN LINK button are on your cluster's card.

Step 2: Run some test jobs

We ran a small set of example jobs, each designed to tell a different story. Every job ran a short Python script that keeps CPUs busy and holds a set amount of memory, so the numbers are easy to read:

Job What it asked for What it did
well-sized 2 CPUs, 1 GB Kept both CPUs busy and used about 750 MB
over-asked 2 CPUs, 6 GB Used 1 CPU and about 315 MB
right-sized 1 CPU, 512 MB The same work as over-asked, with a smaller request
oom-demo 1 CPU, 256 MB Used more memory than it asked for, about 1 GB
fail-demo 1 CPU, 256 MB Tried to read a file that does not exist
timeout-demo 1 CPU, 256 MB and a 1 minute time limit Tried to run for 5 minutes
cancel-demo 1 CPU, 256 MB Cancelled with scancel
array-demo 4 tasks, 1 CPU and 512 MB each One task had a missing input file

We will refer to these jobs as examples throughout this blog post.

💡
Tip: Your own jobs are the best test. Run one small, representative piece of your real work, then check it with the steps below before you submit the full run.

Step 3: Get a report card with seff

Once your jobs finish, start with seff. It prints a short report card for one finished job, showing how it ended and how much of its CPUs and memory it actually used, with the maths done for you. For most jobs, this is all you need. seff is installed on the head node, so just give it a job ID e.g. seff 1 to get a report card for job 1.

💡
Tip: Your Slurm job ID is printed when you submit your job. You can also find it with squeue while the job is in the queue, or with sacct after it has finished, which we cover in the next section. For a job array, give seff the task ID, for example seff 7_3.

Here is the job script for the over-asked example job, which asks for 2 CPUs and 6 GB of memory:

#!/bin/bash
#SBATCH --job-name=over-asked
#SBATCH --cpus-per-task=2
#SBATCH --mem=6G
#SBATCH --output=%x-%j.out
python3 busy.py 1 150 300

And here is the seff report card for the over-asked job:

seff 2
Job ID: 2
Cluster: slurm-accounting-head
User/Group: ubuntu/ubuntu
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2
CPU Utilized: 00:02:30
CPU Efficiency: 50.00% of 00:05:00 core-walltime
Job Wall-clock time: 00:02:30
Memory Utilized: 314.72 MB
Memory Efficiency: 5.12% of 6.00 GB

The two lines that matter most:

  • CPU Efficiency is the CPU time the job used, divided by its CPUs × its wall-clock time (the core-walltime). 100% would mean every CPU it asked for was busy for the whole run.
  • Memory Efficiency is the most memory the job used (its MaxRSS), divided by the memory it asked for. Somewhere around 60 to 80% leaves sensible headroom. Over 100% means the job used more memory than it asked for (see the memory note in Step 4).

50.00% CPU efficiency on 2 CPUs means one CPU did all the work while the other sat idle. 5.12% memory efficiency means the job used 314.72 MB of the 6 GB it held. That tells us exactly what this job should ask for instead:

  • It used 1 CPU, so ask for --cpus-per-task=1.
  • It used 314.72 MB at most, so ask for --mem=512M. That leaves headroom without holding gigabytes it will never touch.

So let's fix the script. Here is the right-sized version, doing exactly the same work:

#!/bin/bash
#SBATCH --job-name=right-sized
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --output=%x-%j.out
python3 busy.py 1 150 300

And here is its new seff report card:

seff 11
Job ID: 11
Cluster: slurm-accounting-head
User/Group: ubuntu/ubuntu
State: COMPLETED (exit code 0)
Cores: 1
CPU Utilized: 00:02:30
CPU Efficiency: 100.00% of 00:02:30 core-walltime
Job Wall-clock time: 00:02:30
Memory Utilized: 315.02 MB
Memory Efficiency: 61.53% of 512.00 MB

Both numbers look much better now. CPU efficiency is 100.00%, because the one CPU the job asked for was busy the whole time. Memory efficiency is 61.53% of 512 MB, which leaves sensible headroom without holding gigabytes the job never touches.

Bar chart. CPUs: over-asked requested 2 CPUs and used 1 CPU on average, CPU efficiency 50.00%. Right-sized requested 1 CPU and used 1 CPU on average, CPU efficiency 100.00%. Memory: over-asked requested 6 GB and used 314.72 MB, memory efficiency 5.12%. Right-sized requested 512 MB and used 315.02 MB, memory efficiency 61.53%. Both jobs ran for 00:02:30.
The same work, asked for two ways. Requested and used CPUs and memory for job 2 (over-asked) and job 11 (right-sized), from seff.

Both jobs took 2 minutes 30 seconds, so the smaller request cost the job nothing. The difference is what it left for everyone else:

  • The over-asked job held both CPUs on ip-10-255-2-6 and 6 GB of the node's 7,372 MB, so no other job could use that node's CPUs while it ran.
  • The right-sized job held 1 CPU and 512 MB on ip-10-255-2-8. While it ran, another task could run alongside it on the same node.

Note that right-sizing works both ways. If seff shows CPU efficiency close to 100% on every CPU you asked for, your job may finish sooner if you give it more CPUs, as long as your software can use them. Check your software's documentation for how to set its thread count. Our well-sized job shows this. It asks for 2 CPUs and tells the Python script to run 2 workers, so it does twice the work of the right-sized job in the same time. We also gave it more memory, because it holds more:

#!/bin/bash
#SBATCH --job-name=well-sized
#SBATCH --cpus-per-task=2
#SBATCH --mem=1G
#SBATCH --output=%x-%j.out
srun python3 busy.py 2 150 700

The srun here is optional. We added it so this job shows a numbered step in sacct, which you'll see in Step 4.

Now let's see its seff report card:

seff 12
Job ID: 12
Cluster: slurm-accounting-head
User/Group: ubuntu/ubuntu
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2
CPU Utilized: 00:05:00
CPU Efficiency: 99.34% of 00:05:02 core-walltime
Job Wall-clock time: 00:02:31
Memory Utilized: 751.39 MB
Memory Efficiency: 73.38% of 1.00 GB

This time the job kept both CPUs busy for almost the whole run (99.34%) and used about 750 MB of its 1 GB, which still leaves sensible headroom.

Step 4: See your jobs with sacct

seff covers one job at a time. sacct is for when you want more detail: all your jobs at once, each step inside a job, and the exact columns you choose. Run it with no options to start:

sacct
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
2            over-asked       main    default          2  COMPLETED      0:0 
2.batch           batch               default          2  COMPLETED      0:0 
3              oom-demo       main    default          1  COMPLETED      0:0 
3.batch           batch               default          1  COMPLETED      0:0 
4             fail-demo       main    default          1     FAILED      1:0 
4.batch           batch               default          1     FAILED      1:0 
5            timeout-d+       main    default          1    TIMEOUT      0:0 
5.batch           batch               default          1  CANCELLED     0:15 
6            cancel-de+       main    default          1 CANCELLED+      0:0 
6.batch           batch               default          1  CANCELLED     0:15 
7_1          array-demo       main    default          1  COMPLETED      0:0 
7_1.batch         batch               default          1  COMPLETED      0:0 
7_2          array-demo       main    default          1  COMPLETED      0:0 
7_2.batch         batch               default          1  COMPLETED      0:0 
7_3          array-demo       main    default          1     FAILED      2:0 
7_3.batch         batch               default          1     FAILED      2:0 
7_4          array-demo       main    default          1  COMPLETED      0:0 
7_4.batch         batch               default          1  COMPLETED      0:0 
11           right-siz+       main    default          1  COMPLETED      0:0 
11.batch          batch               default          1  COMPLETED      0:0 
12           well-sized       main    default          2  COMPLETED      0:0 
12.batch          batch               default          2  COMPLETED      0:0 
12.0            python3               default          2  COMPLETED      0:0 

With no options, sacct shows your jobs since midnight, in the cluster's time zone, which is usually UTC. Each job can take up more than one line, because Slurm tracks the job as a whole and also the separate parts that ran inside it, which it calls job steps:

  • The line with just the job ID (for example 12) is the whole job. Its state, elapsed time and resources cover everything the job did from start to finish.
  • 12.batch is your job script. When you submit a script with sbatch, Slurm runs it as the batch step. If your script just runs commands directly, like python3 analysis.py, all of that work happens inside .batch, and you'll only see the job line and the .batch line. That's normal, and it's how most simple jobs look.
  • 12.0, 12.1 and so on only appear if your script uses srun. Each srun command inside the script starts a new numbered step, so you can see how long each part took and how much memory it used. Job 12 in this example ran its Python program with srun, so it has a 12.0 step named python3. If you don't use srun, you won't see these lines.

In short, if you just run sbatch with a plain script, look at the job line and the .batch line. The numbered steps only matter if you use srun to split your job into parts. We cover these Slurm options in more detail in our simple Slurm guide for beginners.

Job arrays get one job line (and one .batch line) per task, with IDs like 7_1 to 7_4. Use sacct -j 7 to see every task in array 7 at once, and compare their MaxRSS values to pick one --mem that suits them all.

A + at the end of a value means it was cut off to fit the column. You can widen any column, as shown below.

What the states mean

The State column tells you how each job ended. The ones you will meet most:

Four cards. COMPLETED: finished with exit code 0, check seff to see if you asked for the right amount. FAILED: finished with a non-zero exit code such as 1:0, check the job's output file. CANCELLED: cancelled with scancel by you or an admin. TIMEOUT: reached its --time limit and was stopped, ask for more time or split the work. A note below the cards reads: You won't see OUT_OF_MEMORY on HPC Auto Scale Clusters. Memory limits aren't enforced, because the cluster is yours alone.
Common job states and what to check next.

The ExitCode column shows two numbers separated by a colon: the exit code your script returned, then the signal (if any) that stopped it (e.g. a Slurm cancellation signal).

💡
A note on TIMEOUT on RONIN clusters: RONIN clusters don't put a time limit on jobs because the cluster is yours alone, so you don't need to add #SBATCH --time to your scripts, and your jobs won't end with a TIMEOUT state just because they ran long. You'll only see TIMEOUT if you choose to set a time limit yourself, for example with --time to stop a runaway job, and the job runs past it.

Pick your columns and filter your jobs

Add these flags to sacct to see just what you need. You can combine them too.

  • --format picks the columns, for example --format=JobID,JobName%15,ReqMem,MaxRSS,State. A width like %15 stops values being cut off. Run sacct --helpformat to list every column.
  • --units=M shows memory in MB instead of K.
  • -j looks up particular jobs, for example -j 12 or -j 12,2,11.
  • -S and -E set a start and end time, for example -S 2026-10-29T05:30:00 -E 2026-10-29T05:35:00, or words like today, midnight and now. sacct shows every job that was running at any point in that window. Times are in the cluster's time zone.
  • --state shows only jobs in the states you list, for example --state=FAILED,TIMEOUT,CANCELLED. Always add an end time such as -E now, or the list usually comes back empty.
  • -X shows one line per job and hides the step lines. It's the tidiest way to scan a day's work, but MaxRSS is blank and TotalCPU shows 00:00:00, because those numbers come from the steps.

Here is a good everyday set of columns, for jobs 12 (well-sized), 2 (over-asked) and 11 (right-sized, from Step 3):

sacct -j 12,2,11 --units=M --format=JobID,JobName%15,AllocCPUS,ReqMem,MaxRSS,Elapsed,TotalCPU,State,NodeList
JobID                JobName  AllocCPUS     ReqMem     MaxRSS    Elapsed   TotalCPU      State        NodeList 
------------ --------------- ---------- ---------- ---------- ---------- ---------- ---------- --------------- 
2                 over-asked          2      6144M              00:02:30  02:30.252  COMPLETED   ip-10-255-2-6 
2.batch                batch          2               314.72M   00:02:30  02:30.252  COMPLETED   ip-10-255-2-6 
11               right-sized          1       512M              00:02:30  02:30.104  COMPLETED   ip-10-255-2-8 
11.batch               batch          1               315.02M   00:02:30  02:30.104  COMPLETED   ip-10-255-2-8 
12                well-sized          2      1024M              00:02:31  04:59.563  COMPLETED   ip-10-255-2-5 
12.batch               batch          2                12.12M   00:02:31  00:00.016  COMPLETED   ip-10-255-2-5 
12.0                 python3          2               751.39M   00:02:30  04:59.547  COMPLETED   ip-10-255-2-5 

The columns that matter most:

Column What it shows
JobID The job, a step inside it (like 12.batch) or an array task (like 7_3)
AllocCPUS CPUs Slurm gave the job
ReqMem Memory the job asked for
MaxRSS The most memory the job used (shown on the step lines)
Elapsed Wall-clock time the job ran for
TotalCPU CPU time the job actually used, across all its CPUs
State and ExitCode How the job ended
NodeList The compute node(s) the job ran on

These are the numbers seff uses: CPU efficiency is TotalCPU divided by Elapsed × AllocCPUS, and memory efficiency is MaxRSS divided by ReqMem.

⚠️
A note on jobs that use more memory than they asked for: On HPC Auto Scale Clusters, memory limits aren't enforced, because the cluster is yours alone. --mem tells Slurm how much memory to set aside when it places jobs on nodes, so a job that goes over keeps running and still shows COMPLETED. But because Slurm places jobs by what they ask for, it can squeeze the other jobs on its node. oom-demo asked for 256M and reached a MaxRSS of 1037.52M (405.28% memory efficiency in seff). To spot jobs like this, compare MaxRSS with ReqMem, then raise --mem next time. For oom-demo, --mem=1500M would give it room.

Step 5: Find out why a job failed

When a job doesn't end the way you expected, two things tell you what happened. Its state and exit code, which you met in Step 4, tell you that it failed. Its output file usually tells you why.

fail-demo did 30 seconds of work, then tried to read a file that was not there. seff 4 shows State: FAILED (exit code 1) at the top of its report card. The job's output file shows the reason:

cat fail-demo-4.out
holding 50 MB, running 1 busy worker(s) for 30s
done
reading input file...
cat: results/input.csv: No such file or directory

Your output file's name comes from the --output line in your script. These jobs used --output=%x-%j.out, which is the job name and job ID, so job 4 wrote to fail-demo-4.out.

Array tasks work the same way. Task 7_3 failed straight away with exit code 2, and its output file says why:

cat array-demo-7_3.out
task 3 processing inputs/sample_3.txt
missing inputs/sample_3.txt

Two other states you may see:

TIMEOUT means the job ran past a time limit you set with --time. On RONIN clusters you only see this if you set a limit yourself (see the TIMEOUT note in Step 4). The output file ends with a line like this one from timeout-demo:

slurmstepd-ip-10-255-2-8: error: *** JOB 5 ON ip-10-255-2-8 CANCELLED AT 2026-10-29T05:27:52 DUE TO TIME LIMIT ***

To fix it, give the job a longer limit, leave --time out, or split the work into smaller jobs.

CANCELLED means someone ran scancel. With a wide enough State column (for example State%20), sacct shows who: "CANCELLED by 1000" means the user with ID 1000, which is the default ubuntu user.

💡
Tip: To list every job that didn't go to plan today, use the --state flag from Step 4, for example sacct -X -S today -E now --state=FAILED,TIMEOUT,CANCELLED,OUT_OF_MEMORY.

Troubleshooting

sacct shows nothing. With no options, sacct only shows jobs since midnight in the cluster's time zone. Add -S with an earlier date, for example -S 2026-10-01, or look up a job directly with -j and its job ID.

--state returns an empty list. Add an end time, for example -S today -E now.

MaxRSS is blank. MaxRSS appears on the step lines (.batch, .0), not the job line, and not at all with -X. A job that ends within a few seconds may also finish before its memory is checked.

seff shows 0.00% for everything. The job probably ended almost straight away, like array task 7_3 in Step 5. Check State, ExitCode and the job's output file.

Some values end in +. The column is too narrow. Add a width, for example JobName%15 or State%20.

See the whole picture with Grafana

sacct and seff give you stats for each job: what it asked for, what it used and how it ended. Grafana shows you activity for each node in your cluster. Together, they build the whole picture: Grafana shows what each node is doing, and sacct and seff show which jobs ran there and whether their requests were the right size.

To find out how to use them, read How to Use Grafana Dashboards on Your HPC Auto Scale Cluster.

Wrapping up

You now know how to get a report card for any job with seff, dig into the details with sacct, and find out why a job failed. Most importantly, you can turn that report card into a better request for your next job.

If your jobs are consistently too big (or too small) for your nodes, read Changing Compute Nodes in an HPC Auto Scale Cluster. For the full tour of what is new, see Meet the New HPC Auto Scale Clusters.

Ask for what you need, check what you used, and your next job will start sooner and run more cost-efficiently.