Using the HPC clusters: The Basics

This page covers some preliminary information about using the environment on the HPC clusters. I.e., the stuff you need to know how to do before we can even start talking about submitting jobs to the system.

  1. Logging into the system
  2. Graphics
  3. Basic unix commands
  4. Setting up your environment
  5. Files and storage basics
  6. Transferring files to/from the HPCCs and other systems
  7. Compiling codes

Logging into the system

Each cluster has at least two nodes available for users to log into. From from these login nodes, you can submit and monitor your jobs, compile codes, look at the results of jobs, etc. On the two Deepthought clusters, these nodes can also be used for transferring files and data to/from the clusters and other systems.. For the MARCC/Bluecrab cluster, there are data transfer nodes (listed below which should be used instead of the login nodes for data transfers).

DO NOT RUN computationally intensive processes on the login nodes!!!. These are in violation of policy, interfere with other users of the clusters, and will be killed without warning. Repeated offenses can lead to suspension of your privilege to use the clusters.

For most tasks you will wish to accomplish, you will start by logging into one of the login nodes for the appropriate cluster. To log into the cluster, you need to use the Secure Shell protocol (SSH). This is usually standardly installed as ssh on Unix systems, and clients are available for Windows, Mac, and even Android, however on non-Unix systems you typically must install an SSH client .

Once you have an ssh client is installed on your system, you just tell it you wish to connect to the one of the login nodes for the cluster desired. Assuming your official UMD email address is,

Cluster Login Node Username
Deepthought johndoe
Deepthought2 johndoe
Login nodes
Data transfer nodes

Note that for MARCC/Bluecrab, you must append to your username (the is actually part of your username on that cluster).

Note that the MARCC/Bluecrab cluster has both login nodes and data transfer nodes. The data transfer nodes should only be used for transferring large amounts of data to or from the cluster.

NOTE: In or about Jan 2016, the MARCC staff will start requiring multi-factor authentication to access the Bluecrab cluster. You will need to download an app to your smart-phone (the phone you registered with MARCC staff when requesting an account on their web page) to generate an one-time passcode to be used in addition to your normal password. For more information, visit the MARCC web site for more information.

On an TerpConnect/Glue system, you would just issue the command ssh to connect to Deepthought, or similarly ssh to connect to Deepthought2. The unix ssh command by default assumes your login name on the remote system is the same as on the local system, which is true for the Deepthought clusters and for TerpConnect/Glue systems. From other Unix systems, you might need to specify your Deepthought/Deepthought2 username, e.g. ssh or ssh -l USERNAME , where USERNAME is your Deepthought/Deepthought2 username.

For MARCC/Bluecrab, because of the in your username, you will always need to specify the user name and should either use the -l option to ssh or quote your username, e.g. ssh -l johndoe or ssh ""

  • Do NOT try to log into or These are the master nodes for the clusters, and you do NOT have access to it.
  • Your username and password are your standard campus directory id (the part of your or email address to the left of the @)

The first time you connect to the system via ssh, you might receive a warning message like (color added to assist in discussion below):

The authenticity of host ' (' can't be established.
# RSA key fingerprint is b6:14:8f:b3:e2:65:c4:2c:4a:c1:75:71:3e:fa:75:64
# Are you sure you want to continue connecting (yes/no)?

The message sounds scary, but it is normal. SSH tries very hard to protect you from all sorts of hacking, and one such mechanism is to try to ensure you are talking to the system you think you are talking to. For every server you connect to, it remembers a secret (RSA fingerprint) to prove the identity of the server, and verifies that (for brevity, this is a gross oversimplification; for more information). But it cannot verify it the very first time you connect (unless, as is the case on some campus systems, systems staff have pre-populated SSH with information about the system you are connecting to). This message is just to inform you about that.

The IP address and the hostname (in green above) may vary, although the hostname should match the name of the system you want to connect to. The parts in red (the key type and the fingerprint) will depend on the system you are trying to communicate with. To be secure, you should verify that it matches one of the fingerprints listed below:

  • For
    • RSA key MD5 fingerprint b6:14:8f:b3:e2:65:c4:2c:4a:c1:75:71:3e:fa:75:64
    • RSA key SHA256 fingerprint h3B8hc5f+bmY6Di3K6AYVo4yFRYvDISIPcI+Hr1zTHs
    • DSA key MD5 fingerprint 04:56:4f:7c:23:2c:6e:df:7b:20:23:a2:54:1f:94:9a
    • DSA key SHA256 fingerprint Fs/mVsZ4ctslJ9m+zFLpHbqs7vpRQG9Yb1DkMNJ1cvA
  • For
    • RSA key MD5 fingerprint f1:ca:a9:c2:af:44:51:de:8b:34:e7:88:0a:a8:76:c1
    • RSA key SHA256 fingerprint YmasH7fEWPSXPpdJUy6Do2FHkhXOjGFCTkglBv9m0M0
    • DSA key MD5 fingerprint 90:02:49:d9:e1:76:8c:4c:59:20:8e:ca:3f:ad:31:dd
    • DSA key SHA256 fingerprint f0pBFbc4B+VwfCbhRBC5jKp0h+kbcJUdND+9/FyOaVE
  • For
    • RSA key MD5 fingerprint e4:b5:4d:3b:d3:bd:e4:39:d0:8e:f1:c6:96:b8:79:ce
    • RSA key SHA256 fingerprint fUjFRAH0uWwDYXfUF6PcbGVKru6sjsgqN0xiEW1XMoA
    • DSA key MD5 fingerprint 49:bc:ce:2d:e6:63:43:88:d8:87:db:82:ae:f0:98:88
    • DSA key SHA256 fingerprint LBGxzQG0f/4nvTvW7TFab00exMKvbnCOxRVYS3FNPU8
The MARCC/Bluecrab cluster is maintained by MARCC staff, not the Division of Information Technology at the University of Maryland. Therefore, the MARCC website should always be considered the definitive source of information about MARCC and the Bluecrab cluster. Any information provided on this UMD website is provided as a convenience to our users, but we cannot guarantee its accuracy or currency. In case of discrepency, you should assume the MARCC website is the correct one.

If the fingerprint does NOT match, you should NOT enter your password and contact system staff ASAP. If you enter your password in such a situation, as it is possible that someone is performing a man-in-the-middle attack and can obtain your password when you enter it.

If you see a message like

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.

you should always refrain from logging in and contact system staff as soon as possible. This means that the server you are connecting to did not know the secret remembered by SSH for that system, as described above, which means either system staff changed the keys, or someone is hacking you. As system staff do not change the keys often, and will send email to everyone well in advance of changing the keys warning you of this if we were to, this likely means someone is attacking you unless you received an email from systems staff. Do NOT enter your password and contact system staff.


The login nodes are capable of running graphical applications with the graphics appearing on your desktop using the network features of the X11 Windowing System.

In order for this to work, however, you need to be running an X11 server on your desktop. The exact procedure depends on what your desktop, etc. is, and you might wish to contact the UMD helpdesk for assistance.

  • For Linux or Unix/Unix-like systems, X11 is usually available by default. You should check your DISPLAY environmental variable (e.g. echo $DISPLAY should return something, usually your hostname followed by a colon and a number). You might need to add a -Y to your ssh command to allow ssh to forward X11 connections.
  • For Macs running OS X, see this or this knowledge base article .
  • Microsoft Windows does not ship with an X11 server, but there are third party servers available. Cygwin/X is an open-source/free implementation, and Xming is another popular free choice. There are also some proprietary implementations like Xwin32, MKS X/Server, and Exceed.
  • There is also at least one java implementation, WeirdX, that is able to run as an applet in a web browser.

Some software packages have their own protocols for remote visualization, including:

  • Matlab: The Matlab Distributed Computing Server (MDCS) allows one to run Matlab on one's desktop but have computation run on the HPC cluster. Among other benefits, you can do plotting on local machine using data generated on the cluster.
  • ParaView: Paraview supports a remote visualization mode wherein you can do calculations and even rendering on compute nodes and have graphics sent back to your desktop. More information forthcoming.
  • ViSit: ViSit supports a client/server remote visualization mode enabling you to visualize data on the HPC cluster filesystems and/or do rendering on the HPC compute nodes.

Basic unix commands

This is a very short list of some very basic Unix commands to help you get started in the HPC environment.

  • To list files and directories, use the ls command. With no arguments, it lists all files in the directory. You can give it a -l option to show more detail, and/or a file or directory name to show that file or directory.
  • For help on a command, use the man command. E.g., you can use man ls to find out all about the ls command and its options.
  • To make a directory, use the mkdir command. Give it the name of the new directory to make, e.g. mkdir foo.
  • To change directories, use the cd command. Give it the name of the directory to change to, e.g. cd foo. You can also use the pushd and popd to temporarily go to another directory and return. If I am in the directory foo and I issue pushd /export/lustre/newfoo, do some stuff, then popd, I will be back in the foo directory I started in.
  • To copy a file, use the cp command. Usage is cp FILE1 FILE2, which will copy an already existing file FILE1 to FILE2.
  • To move a file to a new directory, or to rename a file, use the mv command, e.g. mv FILE1 FILE2 to rename FILE1 to FILE2.
  • To move files back and forth between one of the HPC clusters and other systems (including the other HPC clusters), the best options are probably sftp and/or scp. If you are on one of the Deepthought/Deepthought2 login nodes or the Bluecrab data transfer node, and wish to transfer files to/from a system which can act as a server for the scp or sftp protocols, you do one of
    • scp FILE1 USERNAME@RHOST:FILE2 to copy FILE1 on a Deepthought/Deepthought2 login node, or a Bluecrab data transfer node, to the system RHOST, putting it with the name FILE2 on the remote system. You can leave out FILE2 to keep the same name (but be sure to keep the colon (:)). USERNAME is your username on the remote system; you can omit the USERNAME@ if it is the same as on the HPC system.
    • scp USERNAME@RHOST:FILE2 FILE1 does the reverse; it copies FILE2 from RHOST to the login node.
    • sftp RHOST will start an interactive SFTP session from one of the Deepthought clusters' login nodes or the Bluecrab data transfer node to the SFTP server on system RHOST. You can use the get command to copy files from RHOST to the local system, or put to copy files from the local system to RHOST. For more information, see the following tutorial on using sftp.

    MARCC/Bluecrab users should use the data transfer nodes for sftping or scping large amounts of data. The MARCC/Bluecrab cluster also provides a Globus end point, see the MARCC/Bluecrab FAQ for more information about that.

    If the system you wish to get the files from/copy the files to does NOT provide an sftp or scp server, you can still use sftp or scp to transfer files if you can run an sftp or scp client on the system. You might need to install such a client, e.g. if it is a Windows desktop. This helpdesk article willshow you how to install and use sftp/scp clients on various systems .

  • To view a file, use the less command. You can use something like less FILE1 to view the contents of the text file FILE1. You can use the space key to page forward, the "b" key to page back, up and down arrows to scroll a line up or down, and "q" to exit. You can also use it to read long output from another command, e.g. squeue | less to allow you to page through all the jobs running on the system.
  • There are several text editors available on the system. If you are not familiar with either vi or emacs, we recommend that you stick with pico.
  • Logging out: Just type logout

Hopefully, the above list will get you through most of the basic tasks you need to complete. A full tutorial on using Unix is beyond the scope of this document. However, there are many tutorials for beginning to use Unix on the web. A few are tutorials we recommend are:

Changing your default shell

The shell is the command line prompt that you interact with. It also is what processes shell scripts. Although there are a number of shells available on TerpConnect/Glue/HPCC systems, the most common are tcsh and bash. Your default shell is the shell you get when you log into a system; by default on the Deepthought HPC clusters it is set to /bin/tcsh as this is (debatably) considered to be the easiest shell for interactive use. The MARCC administrators are of the other opinion, and set the default shell to /bin/bash, presumably for the same reason.

You can change your default shell on the two Deepthought clusters using the chsh command. Your default shell is common across all Glue/TerpConnect/Deepthought* systems at the University of Maryland.

  • You must have valid Kerberos and AFS credentials to use the chsh command. To ensure that, you should run renew first, and enter your password if requested. If you do NOT have valid credentials, the chsh command will fail with an error message about being unable to create a file and ask if you are authenticated. Run renew and try again.
  • The chsh command also allows you to change your room and phone numbers as listed in the Unix password file (i.e. the output of the finger command). You can just hit return to keep the values unchanged.
  • If you use the chsh command, it can take up to an hour for the changes you request to take effect. And for changing the default shell, this will only impact future logins.
  • If changing your default shell, you MUST give full paths to the shell executable. I.e., give either /bin/tcsh or /bin/bash.
  • Your default shell does NOT affect what shell your job scripts run under. This is controlled by the specifying shell for batch jobs.

The default shell on the MARCC/Bluecrab cluster is bash. The chsh command is currently unavailable on the Bluecrab cluster; to change your default shell please contact the MARCC/Bluecrab systems staff.

Setting Up Your Environment

In order to provide an environment that can support a wide variety of users, many software packages are available for your use. However, in order to simplify each individual user's environment, a large number of those packages are not included in your default PATH.

Your account as provided gives you access to the basic tools needed to submit and monitor jobs, access basic Gnu compilers, etc. It is HIGHLY suggested that you DO NOT remove or modify the dot files (.cshrc, .profile, etc) that are provided for you on the Deepthought HPC clusters. Instead, add any customizations you need to the alternate set of files described here , e.g. .cshrc.mine and/or .bashrc.mine. If you choose to modify the system default files, you run the risk of losing any systemwide changes that are necessary to keep your account running smoothly.

DO NOT modify your .cshrc or .bashrc files on the Deepthought clusters. Doing so could cause you to either lose your changes or lose systemwide updates that are needed to keep things running smoothly. Instead, we provide some alternative files, e.g. .cshrc.mine and .bashrc.mine which will be sourced from the main .cshrc and .bashrc, respectively. So modify the dot files ending in .mine. See here formore information .
Bash users on the Deepthought clusters should NOT create a .bash_profile file in their home directory. This will prevent the execution of the system initialization scripts beginning with .profile. Again, .bashrc.mine is available for user customization.
Tcsh users on the Deepthought clusters should NOT create a .tcshrc file in their home directory. This will prevent the execution of the system initialization scripts beginning with .cshrc. Again, .cshrc.mine is available for user customization.

The MARCC/Bluecrab cluster does not use the .cshrc.mine or .bashrc.mine files. On the MARCC/Bluecrab cluster, you should simply edit you .cshrc (if you changed your default shell to tcsh or csh) or .bashrc file directly. It is HIGHLY advised that you DO NOT remove the part near the beginning that sources the global definitions, e.g. the part like

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
as doing so will lose any site wide set up.

For packages that are not included in your default environment, the module command is provided. When run, this command will modify your current session by adding the appropriate entries to your PATH and whatever other variables are necessary to ensure the proper functioning of the package in question. Note that these changes are temporary and only exist until you log out. If you want to have code run for you automatically, add the command module load PACKAGE. If you run the module avail command, it will provide a list of all available packages. module avail STRING will return a list of all available packages starting with the string STRING. The commands module whatis PACKAGE and module help PACKAGE will return a short one-line description and a longer, multiline help text, respectively, for the named package.

Note that many of the packages shown with module avail are NOT accessible on the compute nodes of cluster by default. This is due to the fact that the compute nodes have their own copy of the globally shared Glue software directory, and to conserve space only those packages considered useful for HPC environments have been copied. If you see a package listed under module avail which is not available on the compute nodes, let us know and if possible, we'll make it available. The software page contains listing of the various packages available. If you click on the package name, you will get detailed information as to which versions are available, and whether they are available on the compute nodes or not of a particular cluster.

NOTE: The tap, which performed a similar role to the module command but was unique to UMD and lacks some of the features of the latter, is no longer available on either Deepthought cluster. It is deprecated on the rest of Glue already, and the migration to the module command is slowly occuring there as well, but due to the need to have separate builds depending on compiler and MPI library for many HPC packages, the process has been expedited on the HPC clusters.
If you use the tap in your .cshrc.mine or .bashrc.mine scripts, be sure to give the -q argument before the package name, e.g. tap -q PACKAGE. This suppresses the normal help output, which can confuse some shells and break file transfer protocols.

For example, if you want to run Matlab, you'll want to do the following. Notice that Matlab is not available until after the module load command has been run.

f20-l1:~: matlab
matlab: Command not found.
f20-l1:~: module whatis matlab
matlab               : Matlab 2014b
f20-l1:~: module load matlab
f20-l1:~: matlab

                              < M A T L A B >
		  Copyright 1984-2014 The MathWorks, Inc.
                   R2014b ( 64-bit (glnxa64)
                             September 15, 2014

  To get started, type one of these: helpwin, helpdesk, or demo.
  For product information, visit
If you are running a bourne or bash script under sbatch but your default shell is csh or tcsh, remember that you must include a . ~/.profile in your script to enable the module load and/or tap commands.

For more information, see the section on the module command.

Preventing output from your dot files

It is strongly recommended that your dot files (e.g. .cshrc.mine and/or .bashrc.mine) do not produce any terminal output, or at least, only do so when run in an interactive shell. Output in non-interactive shells can cause problems, most notably with some file transfer protocols like scp and sftp --- in most cases, the stray output will confuse the file transfer protocol usually causing it to abort.

If commands in your .cshrc.mine, .bashrc.mine, or other dot files might produce output to the terminal, you should take measures to ensure such output is only produced in interactive shells. Otherwise, you might break file transfer protocols like sftp or scp.

If you have commands in your dot files which might produce output, you should consider running them only in interactive shells so as not to confuse file transfer protocols. The method varies by the type of shell; for csh and/or tcsh style shells, something like:

if ( $?prompt ) then
	#Only execute the following in interactive shells
	echo "Hello, today is "

#Here we redirect output (if any) to /dev/null
some_command >& /dev/null

For Bourne-like shells (e.g. sh and/or bash) something like:

if [ ! "x$PS1" = "x" ]; then
	#Only execute the following in interactive shells
	echo "Hello, today is "

#Here we redirect output (if any) to /dev/null
some_command > /dev/null 2> /dev/null

You can also redirect output to /dev/null, as is done in the case of some_command in the above examples. Be sure to remember to redirect both stdout and stderr. In some cases, the command has options to silent it, which can also be used (e.g. the old tap has a -q option), but this generally will still get output from errors.

Files and Storage Basics

On each cluster, you have several options available to you regarding where files are stored. A whole section is devoted to this on another page, but it is important and basic enough that a short discussion is merited here.

The file structure is somewhat different between the two Deepthought clusters and the MARCC/Bluecrab cluster.

Files and Storage Basics: the Deepthought clusters

With access to the cluster, you are granted a home directory, which is the directory you see when you first log in. It is distinct from your standard TerpConnect/Glue home directory. Also, if you have access to more than one HPC cluster, your home directory on each is distinct from the other(s). On the login nodes of the Deepthought clusters, your standard TerpConnect/Glue home directory is located as the subdirectory glue_home underneath your Deepthought/Deepthought2 home directory; this directory is NOT accessible from the compute nodes.

Your home directory is, by default, private to you, and should be used as little as possible for data storage. In particular, you should not run jobs out of your home directory --- run your jobs out of a /data volume or from the lustre filesystem; these (especially lustre) are optimized to provide better read and write performance to improve the speed of your job. After the job is finished, you might wish to copy the more critical results files back to your home directory, which gets backed up nightly. (The /data and lustre filesystems are NOT backed up.)

You should run jobs out of a /data volume or out of lustre. So one of your first actions on the cluster should be to create a directory for your jobs in lustre. To do this on Deepthought, mkdir /export/lustre_1/USERNAME. On Deepthought2, the directory /lustre/USERNAME has already been created for you. USERNAME is your Deepthought/Deepthought2 username. Then cd to that directory to run your jobs.

Files and Storage Basics: the Bluecrab cluster

With access to the cluster, you are granted a home directory, which is the directory you see when you first log in. It is distinct from your standard TerpConnect/Glue home directory. Also, if you have access to more than one HPC cluster, your home directory on each is distinct from the other(s).

Your home directory is, by default, private to you, and has a 20 GB quota. This is the only directory of yours which is backed up automatically. In your home directory are the symlinks scratch and work. Both of these are links into the lustre filesystem, and are NOT BACKED UP. The scratch directory is specific to you, and by default is private to you, and has a 512 GB quota. It is for files needed for active jobs on the cluster. The work directory is shared by all users in your project/allocation; it has a 5 TB limit, and is readable and writable by all members of the allocation.

Your home directory should be used as little as possible for data storage. In particular, you should not run jobs out of your home directory --- run your jobs out of the work or scratch directories.

Transferring files to/from the HPC clusters and other systems

Before long, you will need to transfer files to the HPC clusters from your computer, or from one of the HPC clusters to your computer. On the Deepthought clusters, this is best accomplished with the either the secure copy (SCP) or secure file transfer (SFTP) protocols. These are typically preinstalled on Unix systems. Windows or Mac users might need to install a scp/sftp client on their machine.

Once the client is installed, you just point it to the Deepthought/Deepthought2 login nodes as described in the section on logging into the clusters.. You can also use sftp or scp to transfer data to/from the MARCC/Bluecrab cluster, but in this case you should direct traffic at the data transfer node rather than the login node. Also, because your username on Bluecrab includes the @ symbol, which is what scp and sftp normally use to specify the username in file string, you will need to use the -o User=USERNAME option. I. e.,

unix-prompt> scp -o myfile 
myfile                               100%  867     0.9KB/s   00:00
unix-prompt> sftp -o
Connecting to
sftp> put morefiles* 
Uploading morefiles1 to /home-3/
morefiles1                              100%  132     0.1KB/s   00:00    
Uploading morefiles2 to /home-3/
morefiles2                              100%  308     0.3KB/s   00:00    
sftp> quit
The above example shows how to transfer files using scp and sftp for the user; you will obviously need to replace that username with your own.

This will by default allow you to move files to and from your home directory. For larger data sizes (more than a few GB) you almost certainly wish to place them in lustre or in one of the data directories..

Some more detailed information regarding the use of the scp command can be found in the section on basic Unix commands

Compiling codes

Compiling codes in an HPC environment can be tricky, as they often involve MPI and other external libraries. Furthermore, the Deepthought and Bluecrab clusters offer multiple compilers and versions, and multiple versions of many libraries, which can make this task even more complicated.

First off, you must choose your compiler family. The Deepthought clusters offer the "standard" GNU compiler suite, the Intel compiler suite, and the PGI compiler suite; Bluecrab offers GNU and Intel compiler suites. The GNU compiler suite is probably the most widely used for open source packages, most of which will most easily compile on this suite. Intel makes frequent updates to their compilers, although we usually only update annually. The Intel compilers have "bleeding edge" optimization options, but have also been known to have bugs. The PGI compiler suite has some enhancements which might make it preferable for CUDA programming.

In general, you should use the most recent version of the compiler available on the system unless you have a specific need for an older compiler. Bear in mind that the MPI libraries and a number of other external libraries (NetCDF, HDF, FFTW, etc) have dependencies on the compiler version --- you are strongly advised to ensure all dependent libraries and all parts of your code were built with the same compiler version and the same versions of any common dependent libraries. For system provided external libraries made accessible with the module command, the module avail subcommand will show you the different builds of the package available --- if module avail is showing tags with compilers, MPI library, etc. versions, the library is sensitive about such versions and you are advised to ensure that things match. The module load subcommand will complain if you try to load a version which does not match a previously loaded compiler, etc., but that is not perfect. If module avail does not show build tags with compiler, MPI library versions, etc., then the package should be usable with any compiler. See the section on the module command for more information.

Once you decide which compiler you want to use, you should module load it. The only exception to this is if you want to use the default compiler, which is a somewhat old version of the GNU compiler suite (gcc version 4.6.1 at the time of this writing). It is recommended that one first loads the desired compiler, then the desired MPI library (if any), and then any other library packages needed. For the MPI and other libraries, it is recommended that one just gives the package name possibly with a version number (the module command will look for an appropriate build based on the previously loaded compiler and/or MPI libraries).

The Deepthought clusters support various versions of the GNU Compiler Suite, the Intel Compiler Suite, and the PGI Compiler Suite; follow the links to see the different versions supported. With the exception of the default gcc compiler (version 4.6.1 at /usr/local/bin/gcc) you will need to issue a module load command to make the desired compilers available, e.g. module load gcc/4.9.3 or module load intel/2015 or module load pgi.

If you plan to use an MPI library, you must module load the appropriate MPI library. (Exception: On the Deepthought clusters if you are using Intel compilers and wish to use the Intel MPI libraries, the module load for the intel compilers gets you both. For the MARCC/Bluecrab, you would need to load both the compiler (module load intel-ics) and the MPI library (module load intel-mpi)). The MPI library chosen must match your compiler choice, or bad things might happen. Use of the MPICH and LAM MPI libraries is strongly deprecated. OpenMPI is the recommended choice. You are advised to use the latest; normally you only need to specify openmpi/VERSION in the module load command, and we will figure out which compiler you previously loaded, and configure for that compiler. If you chose an incompatible pair (e.g intel/2011 and openmpi/1.6.5) you will get an error when you load the openmpi module.

The following compiler commands are available on the Deepthought HPC clusters:

Compiler FamilyMPI library C compilerC++ compilerFortran77 compiler Fortran90 compiler
GNUnonegccg++ gfortrangfortran
GNUOpenMPImpiccmpic++ mpifort (for OpenMPI>1.7)
mpifort (for OpenMPI>1.7)
GNUIntel MPI*** Illegal combination ***
Intelnoneiccicc ifortifort
IntelIntel MPImpiiccmpiicc mpiifort (NOTE the doubled i) mpiifort (NOTE the doubled i)
IntelOpenMPImpiccmpic++ mpifort (for OpenMPI>1.7)
mpifort (for OpenMPI>1.7)
PGInonepgccpgc++ pgf77 pgf90
PGIOpenMPImpiccmpic++ mpifortmpifort
WARNING: If you are using the Intel MPI libraries, there are commands mpicc, mpif77, and mpif90 defined, but these will NOT WORK. These attempt to compile the code using GNU compilers and Intel MPI, a combination which is NOT supported on these clusters.
WARNING: When using recent OpenMPI libraries and the Intel Fortran Compiler, the preferred (by OpenMPI authors) command to use is mpifort (note: the i is NOT doubled). This is unfortunately very similar to mpiifort (with the i doubled); be careful to use the correct one. The latter (mpiifort, with the doubled i) also exists, and will compile with the Intel Fortran Compiler and the Intel MPI libraries, which is not what you want.

Note that if you are planning to run the code on a different machine than it was compiled on, there can be issues with the instruction sets recognized by the compilers on the two systems. This is not much of a problem on a mostly homogeneous cluster like Deepthought2, but can be problematic on a heterogeneous cluster like the original Deepthought. The PGI compilers in particular have a tendency to use instructions specific to the processor being compiled on by default, and since the DT1 login nodes are newer than some of the compute nodes, this can cause problems. If you are getting Illegal instruction errors at runtime, you may need to specify a more generic target processor. See the PGI compiler suite page for more information on dealing with that with the PGI compilers.

With the release of OpenMPI 1.7, the commands mpif77 and mpif90 have been deprecated in favor of mpifort (NOTE: the i is NOT doubled). The mpif77 and mpif90 commands still exist as of OpenMPI 1.8.6 (and simply are aliases for mpifort), but might be dropped in future versions. The choice of name is regrettably similar to the command for compiling with the Intel Fortran Compiler and MPI libraries (namely, mpiifort, with the doubled i); users of Intel Fortran compilers intending to compile with OpenMPI libraries are encouraged to use care; as in this case both mpifort and mpiifort are defined, the the former using OpenMPI libraries and the latter using Intel MPI libraries.

Also note that not all MPI libraries/versions are not built for all compiler versions. The Intel MPI libraries are currently only supported for the Intel compiler suites (within the same version of the compiler suite). For the OpenMPI libraries, the command module avail openmpi/ will list the compatible combinations, both as openmpi/OPENMPI_VERSION/COMPILER_FAMILY/COMPILER_VERSION and as openmpi/COMPILER_FAMILY/COMPILER_VERSION/OPENMPI_VERSION. For your convenvience, the following table also attempts to list the compatible versions. The boxes marked with an "X" indicate a compatible pair:

Compiler MPI library
GNU 4.6.1 X        
4.8.1 X X X    
4.9.3     X    
6.1.0       X  
Intel 2013 X X X   X
2015     X   X
2016       X X
PGI 17.3       X  

For the MARCC/Bluecrab cluster, use the command module avail to see what compiler and MPI library combinations are available. The compiler names should be the same as in the table for the Deepthought clusters above.

If you have any external libraries you need to use, you need to module load or tap these as well. Some libraries have specific versions compiled with and for a specific compiler/MPI library combination; in such cases you need to pick a version which matches what you are using. Not all combinations exist; if yours does not you can submit a help ticket requesting that combination. We generally try to avoid doing this for old versions of compilers/packages/etc. unless there are extraordinary reasons, so you are generally advised to try to use the latest versions available on the system. Fortran90 codes are particularly sensitive to this, and the *.mod files between different versions of the same compiler might not be compatible (and are definitely not compatible across compilers).

For packages which are libraries used by other packages (e.g. LAPACK, NETCDF, etc), the module load command is not enough. The package being compiled needs to know where to find the libraries, and the way to inform the package of that depends on the build system used by the package.

Doing a module load or tap for an external library generally only defines some environmental variables for you, you still need to instruct the compiler where to find any needed include files and where to find the libraries. Generally, "module help MODULENAME" or "tap TAPNAME" will give a brief usage summary giving the names of these variables. Typically there is a variable with a name like "*INC" or "*INCDIR", and another one with a name like "*LIB" or "*LIBDIR". E.g. the netcdf package defines NETCDF_INCDIR and NETCDF_LIBDIR. The package fftw/3 on the Deepthought clusters defines FFTWINC and FFTWLIB.

The "INC" variable provides the location of the include directory for the package. You will generally want to either add arguemnts with these variables preceded by the -I flag to your compilation command. The -I flag takes a single path, so you should repeat it if you have multiple library packages that you are using.

login-1:~: gcc -c -I$NETCDF_INCDIR -I$FFTWINC my_netcdf_code.c
or to the CFLAGS and FFLAGS variables in your Makefile, e.g.

The "LIB" variables work similarly, except these are needed when the compiler/linker creates the executable. For small codes, the compilation and linking usually occur in a single step; larger codes, especially those with makefiles, usually break this into separate steps. Also, the dynamic linker needs to know where the libraries files are when you run the code, so it is usually easiest to set the "rpath" during compilation. (Otherwise you will need to set the environmental variable LD_LIBRARY_PATH each time before you run the code). To tell the compiler where to find the library for linking, you provide the "LIB" variables as arguments to the "-L" flag. To set the rpath, you provide them as arguments to the "-Wl,-rpath," flag. Both the -L and -Wl,-rpath, take a single path, so you should repeat these flags for each library package.

For the simple case, you would do something like

login-1:~: ifort -o mycode -I$NETCDF_INCDIR -I$FFTWINC 
	-Wl,-rpath,$NETCDF_LIBDIR  -Wl,-rpath,$FFTWLIB my_netcdf_code.f90

Since this compiles and links my_netcdf_code.f90 in a single step, we need to provide both the compile stage (-I) flags and the link stage (-L and -Wl,-rpath) flags.

More complicated cases typically use makefiles, and here you typically will just do something like:


Here we have the CFLAGS and FFLAGS definitions from above (which will be used in the compilation stage), and we put the -L and -Wl,-rpath flags in the LDFLAGS variable (we did this in two steps to make it more readable).

If you opt not to go the "rpath" route, and instead compile the code with something like


(note that the -Wl,-rpath arguments are missing in the above), then before running the resulting binary (which we will call myprog) you will need to set LD_LIBRARY_PATH appropriately. This (and the module loads which precede it) will need to be done in every interactive or batch session which plans to run the code. E.g.

login-1: echo $LD_LIBRARY_PATH
LD_LIBRARY_PATH: Undefined variable.
login-1: ./myprog
./myprog:  error while loading shared libraries: cannot open shared object file: No such file or directory
login-1: module load fftw/2.1.5
login-1: module load netcdf
login-1: ./myprog
Program runs successfully

If you do NOT use the "rpath" arguments shown earlier, every time you run the program the variable LD_LIBRARY_PATH must be properly defined to point to the library locations, or you will get an error like shown above. In general, you will need to load the modules and issue the setenv command once in every interactive login session in which you will use it, and once in every batch script. And you MUST set the directories correctly; if you, e.g., give the $FFTWLIB path for a different version of FFTW than the one the code was compiled with, the binary might run, but it might crash with a difficult to debug error at some seemingly arbitrary place. Or perhaps even worse, it might run to a seemingly successful conclusion but produce erroneous output.

We strongly recommend that for your own code, or for code that you are compiling, that you use the rpath arguments shown earlier. The LD_LIBRARY_PATH option is, in our opinion, best used when you do not have the option of compiling the code with the rpath settings.

Optimization of Code

Since you are using a High Performance Computing cluster, you most likely have long, compute intensive jobs. These can generally benefit greatly from various optimization techniques. The general topic of code optimization is quite broad, far too large to give more than a cursory discussion here. You are encouraged to look at the myriad of resources on the topic that exist on the web. Here we just discuss some specific details related to the UMD clusters.

A major performance feature of modern processors is the ability to vectorize certain operations. In this mode, an instruction is decoded once and operates on a stream of data. This allows for significant performance boosts. There are various vectorization command sets available on Intel processors, known by names like SSE and AVX (with numerical suffixes to specify the version of the command sets). Selecting the correct optimization level can be tricky, as different processors support different vectorization command sets.

Things are relatively simple on the Deepthought2 and Bluecrab clusters, since these clusters are fairly homogeneous. All nodes on the Deepthought2 cluster support SSE3, SSSE3, SSE4.1, SSE4.2, and AVX. This includes the login nodes. All nodes on Bluecrab support all of those as well as AVX2 since they are a bit newer than the DT2 nodes.

Things are trickier on the original Deepthought cluster, as it grew over time and is quite heterogeneous. All of the original Deepthought cluster nodes support SSE3 and SSSE3, but some support one or more of SSE4.1, SSE4.2 and AVX. In particular, the login nodes have newer processors (and therefore support more vectorization command sets) than some of the compute nodes, which can cause issues if the Makefiles, etc. try to optimize aggressively based on the capabilities of the compilation node --- the resulting code might not work on all of the DT1 compute nodes. For best results, you might wish to throttle optimization to the least capable processors on the cluster (e.g. SSE3 and SSSE3 only).

The following table shows the breakdown by cluster:

ClusterProcessor TypesNode count SSE3 ?SSSE3 ? SSE4.1 ?SSE4.2 ? AVX ?AVX2 ?
Bluecrab E5-2680v3 (Haswell-EP) ~700 YYYYYY
E5-2690v4 (Broadwell) ~125 YYYYYY
E7-8857v2 (IvyBridge-EP) ~50 YYYYYY
Deepthought (DT1) E5130 (Woodcrest) ~40 YYNNNN
E5150 (Woodcrest) ~89 YYNNNN
E5345 (Clovertown) ~39 YYNNNN
E5410 (Harpertown) ~81 YYYNNN
E5520 (Gainestown) ~16 YYYYNN
E5650 (Westmere-EP) ~4 YYYYNN
E5670 (Westmere-EP) ~78 YYYYNN
E5-2660 (Sandybridge-EP) ~22 YYYYYN
E5-2670 (Sandybridge-EP) ~8 YYYYYN
Deepthought2 (DT2) E5-2680v2 (Ivybridge-EP) ~490 YYYYYN
E5-4640v2 (Ivybridge-EP) ~5 YYYYYN

As previously stated, it is sometimes useful to specify the processor type and/or vectorization command sets to compiler/optimize for. When specifying processor type, we recommend that one uses an option which generates code which will run on any processor but is optimized for the expected processor type (instead of code which is optimized for a specific processor but might not run on other processors). I.e., use -mtune instead of -march.

Some useful optimization options:

Optimization TypeGCC FlagsIntel Flags
SSE3 -msse3 -mSSE3
SSE4.1 -msse4.1 -mSSE4.1
SSE4.2 -msse4.2 -mSSE4.2
AVX -mavx -mAVX
AVX2 -mavx2 -mAVX2
Optimize for sandybridge
(but run anywhere)
Optimize for ivybridge
(but run anywhere)
Optimize for broadwell
(but run anywhere)

OpenMP is an API for shared memory parallelization, i.e. for spliting a code into multiple threads running on the same node. Because the parallelization is limited to a single node, it is less powerful than other APIs (e.g. MPI) which can span multiple nodes, but is also much easier to code for.

Indeed, OpenMP is implemented by the compiler, and is generally invoked through various compiler directives and/or pragmas. E.g., if you have a C code with a for loop, you can add a pragma just before the start of the for loop which will tell the compiler to try to split the loop into multiple threads each running in parallel on a different processor core.

If you have a code with OpenMP directives, you need to tell the compiler to compile it with the OpenMP extensions enabled. The exact mechanism is compiler dependent:

Compiler FamilyFlag to use OpenMP
GNU compiler suite-fopenmp
Intel Compiler Suite-openmp

NOTE: If you are using the Intel compiler suite and the Math Kernel Libraries (MKL), some of the MKL routines might use OpenMP even if you did not request OpenMP in the main compilation. You can set the environmental variable OMP_NUM_THREADS to 1 to effectively disable that if really desired.

By default, OpenMP will attempt to use as many threads as there are cores on the system. This can be problematic in some cases. At runtime, you can set the environmental variable OMP_NUM_THREADS to an integer to control the maximum number of threads that OpenMP will use.

NOTE: Be sure to use the -openmp or -fopenmp flag (as appropriate for the compiler using) on all; of the compilation AND link stages for your code.