Connecting Infrastructure, Connecting Research

Globus Online Tutorial

This tutorial provides instructions for setting up a Globus Online account, using the system to move files to/from any resource, and adding a resource as a transfer endpoint in the system.

  1. Account Creation and Login
  2. File Transfer with the Web Interface
  3. File Transfer with the Command Line Interface (CLI)
  4. Endpoint Creation: Adding New Resources (Servers, Laptops etc.) as Transfer Endpoints

Note: A transfer 'endpoint' is the file source (where the file is located now) or destination (where you are moving it to).

Part 1: Account Creation and Login

To get started using Globus Online with NGS, first sign up for a free Globus Online account and then associate that account with your NGS identity (Note: the below are one-time actions only):

  1. Complete the brief form on https://www.globusonline.org/SignUp
  2. Look for a confirmation email and click to validate your email address
  3. Sign into your account and go to 'My Profile' in the 'Go to:' drop-down menu (top of page)
  4. Click 'Manage Identities' / 'Add External Identity'
  5. Select 'UK NGS' as account provider and provide your myproxy username and password, then click 'Add Account'.

Once these steps are done, UK NGS members can sign in to Globus Online using their NGS credentials and all NGS resources will automatically be available for transfer.

To sign in using your UK NGS identity:

  1. Go to https://www.globusonline.org/SignIn and click the 'External Identity' tab
  2. Select 'UK NGS' as the Identity Provider and login with your UK NGS username and password

Of course, you can always sign in to Globus Online using your Globus Online login and password as well, but you will then need to authenticate on the NGS resources you want to access.

Part 2: File Transfer with the Web Interface

To move files using the Globus Online GUI at www.globusonline.org :

  1. Log in at https://www.globusonline.org/SignIn using your NGS identity
  2. Click the 'Start Transfer' link on the Dashboard (under 'File Transfer'), or select 'Start Transfer' from the 'Go To:' menu in the top bar.
  3. Locate the file you want to move
    • View the list of available endpoints in the 'Endpoint' drop-down box (left side of window).
    • NGS endpoints will appear with 'ngs#' at front of name - you can also type in the 'Endpoint' field and use the auto-complete feature to find the desired endpoint.
    • Select the file(s) or directory(ies) that you want to transfer - you can navigate the file system by double-clicking on directories to view their contents.
  4. Select the destination endpoint for your transfer (right side of window) and navigate to the desired location.
  5. Click on the highlighted 'arrow button' to initiate the transfer.

Part 3: File Transfer with the Command Line Interface (CLI)

PREREQUISITES

To use the CLI you must have a Globus Online account with ssh access enabled. Please see Part 1 for a walkthrough on creating a Globus Online account. To enable ssh access, do the following:

  1. Sign into your account and go to 'My Profile' in the 'Go to:' drop-down menu (top of page)
  2. Click 'Manage Identities' / 'Add SSH Public Key'
  3. Paste the contents of your SSH public key into the 'SSH Public Key' field (you can optionally enter any name you like in the 'Alias' field) and click on 'Add SSH Key'. It will take a minute or two for the SSH key to become active for use.

CLI STRUCTURE

There is no need to install custom client software. CLI users interact with Globus Online via secure shell:

If desired, gsissh can be used in place of ssh:

To enable gsissh access, do the following:

  1. Sign into your account and go to 'My Profile' in the 'Go to:' drop-down menu (top of page)
  2. Click 'Manage Identities' | 'Add X.509 Certificate'
  3. Paste the contents of your X.509 certificate into the 'X.509 Certificate' field (you can optionally enter any name you like in the 'Alias' field) and click on 'Add X509 Certificate'.

FILE TRANSFER

Globus Online provides two commands for moving files: transfer and scp. The scp command offers a subset of transfer's functionality but is packaged in a way that should be familiar to users of the standalone scp command.

All Globus Online accounts are provisioned with two endpoints for exploratory use, so as soon as you have an account you should be able to transfer /share/godata/file1.txt from endpoint go#ep1go#ep2, as demonstrated here by user lcc:

ssh lcc@cli.globusonline.org scp go#ep1:/share/godata/file1.txt go#ep2:~/myfile.txt

Contacting 'myproxy.tutorial.globusonline.org'...

Activating 'ep2'

Activating 'ep1'

Task ID: 19029d64-ecec-11df-aa30-1231350018b1 \
  
[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 1/1 0.00 mbps

ssh lcc@cli.globusonline.org ls go#ep2/~/

myfile.txt

ONLINE HELP

You can execute the help command to view the current Globus Online CLI command set:

ssh lcc@cli.globusonline.org help
Type '<command> -h' for basic help on a command.

Type 'man <command>' for detailed help.

Task Management:	cancel			modify
			details			status
			events			wait

Task Creation:		scp			transfer

Endpoint Management:	endpoint-add		endpoint-modify
			endpoint-activate	endpoint-remove
			endpoint-deactivate	endpoint-rename
			endpoint-list		diagnose (beta)

Other:			help			mkdir
			hostory			profile (deprecated)
			
ls			versions
			man

In addition, <command> -h displays a command syntax summary and man <command> displays the command’s manual page.

ENDPOINT ACTIVATION

Though it happens automatically with go#* endpoints, transfers involving other endpoints require that you activate them. Activation is Globus Online’s endpoint user authentication mechanism; it enables endpoint owners to determine who is transferring files. Transfers will not go through unless both the source and destination endpoints are activated.

You can practice by activating the Globus Online endpoints:

ssh lcc@cli.globusonline.org endpoint-activate go#*
Contacting 'myproxy.tutorial.globusonline.org'...

Credential Subject : /O=Grid/OU=GlobusTutorial/CN=lcc

Credential Time Left: 12:00:00

Activating 'go#ep1'

Activating 'go#ep2'

In the following example user lcc activates her TeraGrid endpoints using the ssh -t option to prevent her TeraGrid passphrase from being echoed to stdout.

Note that even if an endpoint is activated transfers will fail if the user is not authorized to use it. Access to a given endpoint is entirely within the endpoint owner's control, so you must contact owners directly to obtain permission to use their endpoints.

ssh -t lcc@cli.globusonline.org endpoint-activate tg#*
Contacting 'myproxy.teragrid.org'...

Enter MyProxy pass phrase:

Credential Subject : \
  /C=US/O=National Center for Supercomputing Applications/CN=Lisa Childers

Credential Time Left: 12:00:01

Activating 'tg#abe'

Activating 'tg#bigred'

Activating 'tg#cobalt'
Activating 'tg#condor'

Activating 'tg#ember'

Activating 'tg#frost'

Activating 'tg#hpss'

Activating 'tg#kraken'

Activating 'tg#lincoln'
Activating 'tg#lonestar'

Activating 'tg#longhorn'

Activating 'tg#nstg'

Activating 'tg#pople'

Activating 'tg#queenbee'

Activating 'tg#ranch'
Activating 'tg#ranger'

Activating 'tg#spur'

Activating 'tg#steele'

Connection to cli.globusonline.org closed.

Because frost and ranger have been activated using her TeraGrid credentials and the frost and ranger owners have authorized her to access their endpoints, lcc is able to transfer a file:

ssh lcc@cli.globusonline.org scp tg#frost:~/frost-file.txt tg#ranger:~/to-ranger.txt
Task ID: 6356aa16-ed20-11df-aa30-1231350018b1
 \
  [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 1/1 0.00 mbps

lcc activates the NERSC endpoints using her NIM password:

ssh -t lcc@cli.globusonline.org endpoint-activate nersc#dtn -U childers
Contacting 'nerscca.nersc.gov'...

Enter MyProxy pass phrase:

Credential Subject : \
  /DC=gov/DC=nersc/OU=People/CN=Lisa Childers 47849

Credential Time Left: 11:59:58

Activating 'nersc#dtn'
Connection to cli.globusonline.org closed.

And OLCF using her RSA SecurID:

ssh -t lcc@cli.globusonline.org endpoint-activate olcf#dtn -U childers
Contacting 'myproxy1.ccs.ornl.gov'...

Enter MyProxy pass phrase:

Credential Subject : \
  /DC=org/DC=doegrids/OU=People/CN=Lisa C Childers 319818/CN=1577622648/CN=574492166/CN=1930388809

Credential Time Left: 12:00:00

Activating 'olcf#dtn'

Connection to cli.globusonline.org closed.

An endpoint can also be activated using grid-proxy-init and gsissh:

grid-proxy-init
Your identity: /DC=org/DC=doegrids/OU=People/CN=Lisa C Childers 319818

Enter GRID pass phrase for this identity:

Creating proxy ..................................................... Done

Your proxy is valid until: Fri Nov 12 02:17:27 2010
gsissh lcc@cli.globusonline.org endpoint-activate -g ci#pads
Credential Subject : \
  /DC=org/DC=doegrids/OU=People/CN=Lisa C Childers 319818/CN=529754051/CN=1462231917

Credential Time Left: 11:59:54

Activating 'ci#pads'

Endpoints can also be activated inline by specifying the -g option with the transfer and scp commands.

FOR MORE INFORMATION

The Globus Online CLI is quite powerful and includes many features not mentioned above (e.g., file synchronization, monitoring, endpoint management, etc.). You can learn more about the full CLI feature set by reading the online Globus Online man pages and the 'Beyond the Basics guide'.

Please send all questions to support@globusonine.org.

Part 4: Endpoint Creation: Adding New Resources (Servers, Laptops etc.) as Transfer Endpoints

Sometime you might need to move files to/from a resource that is not listed as an endpoint in the Globus Online system, such as a campus server or home computer/laptop. In such cases, you can easily add this resource as a Globus Online endpoint using the Globus Connect feature.

(Note there are hundreds of public endpoints currently defined at various supercomputer centers and campuses, and new endpoints are being added daily. Check the Globus Online endpoint list to see if the endpoint you need is there. If your server has GridFTP in place but has not yet been enabled, visit http://www.globusonline.org/enable for more information.) To add a new endpoint using Globus Connect, an easy process that shields you from the complex details of the GridFTP install, follow these steps:

  1. Log in to Globus Online at https://www.globusonline.org/SignIn
  2. On the main Dashboard, click the 'Globus Connect' button
    • If you are not already on the Dashboard, select 'Dashboard' in the 'Go To' drop-down box at top of page.
  3. Install and run Globus Connect following the instructions for your operating system (https://www.globusonline.org/globus_connect/)
    • When you run Globus Connect for the first time, it will ask you for a setup key. On the Globus Online Dashboard page, click 'Globus Connect
    • Click 'Generate Setup Key'
    • Copy the setup key and paste it into the Globus Connect window to complete the setup process.

You should now see your machine in the list of endpoints ('Start Transfer' screen) identified as '#' on Globus Online. You can select it to view the contents of your machine and transfer files to and from it as explained in Parts 2 and 3.

Learn More

Navigation

Poll

How easy is it to find the information you are looking for on the NGS website?:

©NGS