Globus is a cloud-based software-as-a-service providing file transfer, sharing, and data publication functionality. It is supported by most High Performance Computing clusters in the world and is designed for efficiently moving very large (many TBs) of data. Globus has automatic support for using multiple streams (thereby speeding up transfers) as well as restarting failed transfers without manual intervention.
Globus allows for transferring files and data between any pair of "end points". As stated earlier, most HPC clusters have Globus "end points", allowing you to transfer data efficiently between HPC clusters. Also, you can use a web application to temporarily make your desktop/workstation into an "end point" so that you can transfer files to and from it as well.
Globus Connect Personal
Globus Connect Personal allows you to set up your workstation as a Globus endpoint which you can later use to transfer files to and from. If you are only interested in using Globus to transfer files between existing endpoints (e.g. HPC clusters), you will not need to use this.
To setup your workstation as an endpoint, you will need to install Globus Connect Personal on it. This needs only be done once. To do this, open a web browser on your workstation and:
- Visit https://www.globus.org/app/endpoints/create-gcp.
- In the field labeled
Display Name, enter the name for this endpoint. This name is private to you, and will be used as the name of the endpoint which you will select during file transfer. So something like "My Desktop" or "My Desktop at Lab" would suggested.
- Click the
Generate Setup Keybutton. This will return a a setup key (a long string of hexadecimal digits with occasional hyphens). You will need this in the next step. Select the key with the mouse and save to your clipboard.
- You will need to download and install Globus Connect Personal. Click the
appropriate button for the OS on your desktop. This will download a file
containing the application.
- For windows: See the Globus web site for more detail, but basically you just need to double click on the downloaded file and it should start to install Globus Connect Personal. Click "Yes" to allow the program to make changes to your computer. Confirm the installation directory and click "Install". Click the "Run Globus Connect Personal" checkbox and click finish.
- For Mac: See the Globus site for more detail, but basically you just need to drag the downloaded file into your Applications folder, the double click on it. Click "Open" when asked to confirm if it is safe to run this application.
- For Linux: See the Globus site for more detail, but basically
you just need to untar the download tarball then run
- Because this is the first time you are running Globus Connect Personal on this system, when it comes up it will ask for the setup key. Copy the key you generated in the previous step, and click OK. Globus Connect Personal is now running and you can use this endpoint to transfer files to or from this workstation.
- You should exit Globus Connect Personal when you are not transferring or intending to transfer files to/from this workstation. You should start it up again when you plan to transfer files (you will not need the setup key after the first startup).
Using the Globus Web Application to transfer files
This section discusses how to transfer files with the Globus Web Application. If you wish to transfer files to or from you workstation, you will need to configure Globus Connect Personal first; that is not needed if you just wish to transfer between existing endpoints (e.g. most HPC clusters).
- Open a web browser on your workstation and go to https://globus.org/login
- You will be requested to log in. There will be a dropdown listing
various organizations. To login with your UMD username and passowrd, select
University of Maryland College Parkas your organization, and you will be redirected to the campus login page.
- After logging in, you should reach the file transfer page. On this page
you will get two file browser windows, one on the left and one on the right.
Each window will have an "Endpoint" field and represent the two systems
between which you wish to transfer files (once set up, you can transfer files
in either direction). Select your endpoints (you can generally just starting typing in any part of the name to search, e.g. "Deepthought" for Deepthought2 or "MARCC" for bluecrab):
- For the Deepthought2 cluster, choose
University of Maryland - Deepthought2
- For MARCC and/or the Bluecrab cluster, choose
- If you want to transfer data to/from your workstation, you will need to have previously configured your workstation for Globus Connect Personal. The Globus Connect Personal application will also need to be running on your workstation. Enter the name you gave your personal endpoint when you set it up. You should also be able to find it under the "Administered by Me" tab.
Most endpoints will request an username and password for you to access them. Login in, and you should get a list of files and directories on that system.
- For the Deepthought2 cluster, choose
- Once you have selected and logged into both endpoints, the file browser screens will display the filesystems on the two endpoints. You can then select files and/or directories by selecting on one system and then selecting the appropriate arrow to initiate the transfer.