How To Determine At Which Point In Python Script Step Memory Exceeded In Slurm

I have a python script that I am running on a SLURM cluster for multiple input files: #!/bin/bash #SBATCH -p standard #SBATCH -A overall #SBATCH --time=12:00:00 #SBATCH --output=

Solution 1:

You can get refined information by using srun to start the python scripts:

srun python $NORM_SCRIPT -data $file -path $OUTPUT_DIR

Slurm will then create one 'step' per instance of your python script, and report information (errors, return codes, memory used, etc.) for each step independently in the accounting, which you can interrogate with the sacct command.

If configured by the administrators, use the --profile option to get a timeline of the memory usage of each step.

In your python script you can use the memory_profile module to get a feedback on the memory usage of your scripts.