Pauli administration cheat-sheet

Job-related commands

  • Changing job timelimit
scontrol update jobid <id of the job> timelimit=<hh:mm:ss>
  • Change timelimit for all jobs of specific user
for i in $(squeue -u <username> -h -t PD -o %i) ; do scontrol update jobid $i qos=special timelimit=<hh:mm:ss>; done

Other job properties can also be changed same way. Ususally it is useful to change the following properties:

  • qos - if the default qos is not sufficient
  • partition - if the job has been submitted to a wrong partition (this could be useful if other partition became available)
  • timelimit - if the initial time was not enough (useful for already running jobs, if job is pending, highly likely it will never start if the new timelimit exceeds the qos timelimit)

Node-related commands

  • Restart node using slurm (Slurm will wait for jobs that are currently running on selected node to be finished)
scontrol reboot ASAP <node name>
  • If, after some hanging job, the node’s state is down, new jobs will not use it, to return it back to normal state use
scontrol update nodename=<nodename> state=resume