This page provides an example of submitting a simple
MPI job
using Python, and in particular the mpi4py
Python package.
This job basically runs a simple MPI enabled Hello World! type script which basically just prints a single line from each task identifying its rank and the node it is running on.
We provide linebyline descriptions of both the submission script as well of the python script.
The submission script is similar to other MPI jobs , and can be downloaded here. We also present it here:
Line#  Code 













































































































































































































The python script is somewhat simple (as far as MPI scripts go) because it really does not do much. The python script can be downloaded here. We also present it here:
This is a python script, so our shebang
runs python. In particular,
the sequence #!/bin/env python
is a common Unix idiom for shebangs to
instruct the system to use the python command in your PATH.
hello_mpi
functionhello_mpi
function for this script.
This function actually does all of the real work for this script, and
we examine it in detail below.
This particular block of comments just provides some basic identification of the script, what is does, where it came from, etc. Other comments throughout the script give a brief description of what is being done.
sys
module is generally useful, the platform
module is used to get
the name of the compute node the task is running on, and the mpi4py
module is what interfaces with the
MPI libraries
.
node
. Each
MPI task
will start it's own copy
of the python interpreter running this script. In general, an
MPI job
can run on multiple nodes,
the value saved in the variable node
can be different for different
tasks.
MPI_COMM_WORLD
communicator for use by our code. Communicator objects are what
MPI
uses to connect the various
MPI tasks
processes associated
with an MPI session, and the MPI_COMM_WORLD
communicator
includes all of the tasks that are part of the job.
Get_rank
and Get_size
methods to obtain the
rank
and size
. The size
is the number of processes
in the communicator, and so should be the same for all tasks. The rank
is an
unique integer identifier, ranging from 0 to size  1
, identifying the task.
So this will be distinct for all of the tasks in the job.
hello_mpi
function. This is a standard
Python idiom for allowing the same code to be used either as a library or a
script; it invokes the hello_mpi
function when the file is executed
as a Python script but not if imported.
The easiest way to run this example is with the
Job Composer
of the OnDemand portal,
using the Basic_Python_MPI
template.
To submit from the command line, just
sbatch python_mpi.sh
. This will submit
the job to the scheduler, and should return a message like
Submitted batch job 23767
 the number will vary (and is the
job number for this job). The job number can be used to reference
the job in Slurm, etc. (Please always give the job number(s) when requesting
help about a job you submitted).
Whichever method you used for submission, the job will be queued for the
debug partition and should run within 15 minutes or so. When it finishes
running, the slurmJOBNUMBER.out
should contain
the output from our diagnostic commands (time the job started, finished,
module list, etc). The output of the helloumd
will be in
the file hello.out
in the job specific work directory
created in your lustre directory. For the convenience of users of the
OnDemand portal, a symlink to this directory is created in the submission
directory. So if you used OnDemand, a symlink to the work directory will
appear in the Folder contents
section on the right.
The slurmJOBNUMBER.out
file will resemble
(from an Intel MPI example):
Hello, world! from MPI task 3 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 4 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 1 of 60 on node compute100.juggernaut.umd.edu
HELLO_MPI: Version 1.5
Hello, world! from MPI task 2 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 0 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 5 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 8 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 6 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 9 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 10 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 11 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 7 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 12 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 13 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 14 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 15 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 16 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 17 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 18 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 19 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 20 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 21 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 22 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 23 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 24 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 25 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 26 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 27 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 28 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 29 of 60 on node compute100.juggernaut.umd.edu
Hello, world! from MPI task 30 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 31 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 32 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 33 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 34 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 35 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 36 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 37 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 38 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 39 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 40 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 41 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 42 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 43 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 44 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 45 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 46 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 47 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 48 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 49 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 50 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 51 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 52 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 53 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 54 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 55 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 56 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 57 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 58 of 60 on node compute101.juggernaut.umd.edu
Hello, world! from MPI task 59 of 60 on node compute101.juggernaut.umd.edu
Basically, you should see a message from each task 0 to 59, in some random order. The identifying comments (with the version number) will appear somewhere in the mix. Because everything is running in parallel, the order will not be constant. Note that the tasks will be divided across multiple nodes (in this case compute100 and compute101). On Juggernaut, the 60 cores will require two nodes, and on Deepthought2 it would require three nodes.