Useful Globus Commands
Globus Online
Globus Online orchestrates data movement between two gridFTP servers or between a GridFTP server and the user's local machine(with one-click Globus Connect installed). It moves data rapidly and reliably with very minimal intervention.
Globus Online offers an intuitive , easy to use web interface that doesn't rely on knowledge of Globus commands. For steps on how to use this system with the web interface, see the Globus Online Tutorial part 2.
In addition to the web interface, Globus Online provides a command line interface so users can script their data movement tasks. The following example illustrates a command-line for moving data between 'keele' cluster and 'manchester' cluster.
$ ssh your-go-username @cli.globusonline.org scp ngs#keele-ngs cluster1:#/myfile ngs#manchester-ngs2 vidar:#/
If the command prompts you for a username and password, provide your MyProxy username and password.
Note: In order to do the above, you must have an account on GO and load your ssh keys as described in the Globus Online tutorial Part 3.
You can also use gsissh:
$ gsissh your-GO-username @cli.globusonline.org scp -g ngs#keele-ngs cluster1:~/myfile ngs#manchester -ngs2 vidar:~/
Note: In order to do the above, you must have an account on GO and load your certificate as described here in the Globus Online tutorial Part 3.
The following Globus commands maybe useful if you are working with Globus on the command line
globus-url-copy (GridFTP).
The globus-url-copy command instructs two GridFTP servers to transfer a file between them. This command is for use when you want to transfer large amounts of data around quickly and efficiently. You need to know your userid for both the remote and local machines so you can get the path arguments right in the command. Executing the following command results in uploading the myfile.tgz file.
Note replace 'xxxx' with your userid on the resource and note that they may be different on different resources.
globus-url-copy -vb file://localhost/home/ngsxxxx/myfile.tgz \
gsiftp://ngs.leeds.ac.uk/home/ngsxxxx/myfile.tgz
globus-job-run.
The globus-job-run is a command that is used to run jobs on the NGS that are short and interactive in nature. To demonstrate the command we will use the Linux command /bin/hostname -f as typical of the type of jobs to be run however this could be any job that fulfills the short and interactive requirements. On Linux the /bin/hostname -f prints the hostname and domain of the machine the command ran on to the stdout. The globus-job-run will block user input until the job has been completed and returns. Running the command /bin/hostname -f on the Leeds head node produces the output in shown below. From the outputted hostname it is clear that the Linux command has been executed directly on the head node as a Linux system command.
Submit to head node
globus-job-run ngs.leeds.ac.uk /bin/hostname -f
Submit to Batch System
globus-job-run ngs.leeds.ac.uk/jobmanager-pbs /bin/hostname -f
globus-job-submit.
The globus-job-submit command is very similar in syntax and structure to the globus-job-run except the job does not block the user's shell and returns control immediately to the user. On return the command places a contact string or URN (Uniform Resource Name) on the stdout which is used as the point of contact for the job. A typical scenario is depicted in below.
The job is submitted using globus-job-submit
globus-job-submit ngs.leeds.ac.uk/jobmanager-pbs /bin/hostname -f
globus-job-status
its status is then checked using globus-job-status using the URL returned from above
globus-job-status https://ngs.leeds.ac.uk:64173/12336/1255956948/
globus-job-get-output
If status is "DONE" the standard output of the job is the obtained by using globus-job-get-output
globus-job-get-output https://ngs.leeds.ac.uk:64173/12336/1255956948/
And this job returned "node004.beowulf.cluster"
globus-job-clean To clean up temporary files:
globus-job-clean https://ngs.leeds.ac.uk:64173/12336/1255956948/
The examples so far have involved running a standard system program (hostname). The next stage is therefore to submit a simple custom program. The first program to use is "myjob.sh" This program just prints the present working directory, the hostname again and the date. Run the following commands to view and run this script on your local machine
cat myjob.sh
#!/bin/sh
echo $PWD
hostname -f
date
By default the globus-job-submit (and globus-job-run) assume that the program is local to the node the job will run on. The script "myjob.sh" only exists on the local machine and hence it is necessary to send "myjob.sh" to the execute node as part of the job. This process is known as "staging". Run the job with the command (note the extra -s):
globus-job-submit ngs.leeds.ac.uk/jobmanager-pbs -s ./myjob.sh
and as before use
globus-job-status "uid"
globus-job-get-output "uid"
globus-job-clean "uid"
