This box searches only documentation in the Data Store space. To search the entire CyVerse wiki, use the box at the upper right.


For overview information, see Data Store or Manage Data on the CyVerse website.

DATA STORE USER MANUAL
 

 

Skip to end of metadata
Go to start of metadata
IMPORTANT NOTE ABOUT UPGRADING TO 4.1.9

iCommands 4.1.9, the latest supported version for CyVerse services, has been released by iRODS but we are currently testing it for compatibility with CyVerse services. Instructions will be available on this page by September 5, 2016. For a list of affected services and testing done, see Upgrading to iRODS 4.1.9 (work in progress).

About anonymous access

CyVerse's commitment to open science means you can download many files in our Data Store—the repository of all data files at CyVerse—that have been shared by CyVerse users for those without a CyVerse user account. Additionally, it is possible for the owner of a data file to send a public link to a file so that anyone with the link can download that file; if you received such a link, simply click to download the file. For more information on how CyVerse users share their data, see Storing Your Data with CyVerse and Accessing that Data.

There are several different access points you can use to download data without an account:

  • CyVerse Data Commons Community Contributed and Curated Data repositories: Easiest to use for anonymous users, the Data Commons repositories create web-based access that allow users to download files up to 2 GB from the repositories. Requires no installation or configuration.
  • iCommands: Command-line for Linux and Mac. Best method for downloading large files of 2-100 GB each or for bulk file transfers of many small files. Requires installation and configuration on each user's local machine or server.
  • Cyberduck: GUI-based for Mac OS X and Windows (Linux users should use iCommands or may also wish to use iDrop Desktop). Used for downloading large files or bulk downloads (>10 GB). Requires installation and configuration on the user's local machine.
  • iDrop Desktop: A Beta release, we recommend that you use Cyberduck instead of iDrop Desktop. If you choose to use iDrop, it is a GUI-based application for Mac, Linux, and Windows that establishes multiple connections between the source and destination, and allows parallel data transfers of greater than 10 GB. Requires installation and configuration.

Want access to more data files?

You may want to register for a CyVerse user account to gain full download and upload access to all files shared with CyVerse users.

 

Using the Community Contributed Data and CyVerse Curated Data sites

The easiest way to download files up to 2 GB in size from a web browser (size cutoff due to speed limitations) without a CyVerse account is using the CyVerse Data Commons repositories, Community Contributed Data or CyVerse Curated Data. It is also possible to download without an account using iCommands or Cyberduck. No installation or configuration is needed for the DCR.

For files larger than 2 GB, you will need to use iCommands or Cyberduck (Mac OS and Windows), which require installation and setup.

  1. Go to CyVerse Data Commons at http://datacommons.cyverse.org/. No login is required.
  2. Click either Community Released to access data that has been provided by community collaborators for public access, or click CyVerse Curated to access data that has been curated and given a permanent identifier by CyVerse curators.

Downloading a file from the Data Commons

Files up to 2 GB can be downloaded directly from Data commons repositories, while files larger than 2 GB must be downloaded using another mode.

  1. In either repository, click the folder name with the file to download.

  2. Click the filename.

  3. Depending on the file size, do one of the following:

    • If the file is smaller than 2 GB:
      1. Click Download.
      2. Click the checkbox for I'm not a robot.
      3. Change the path if needed, click Save, and click OK.
    • If the file is larger than 2 GB:
      1. Click Download Options to view the other download options (or read the information below on this page).
      2. Copy the path to the file and note the filename.
      3. Continue to your preferred method, following the instructions in the sections below:
      • To learn more about other methods for managing and using data, see Manage Data on the CyVerse website.
      • To learn more about downloading files from the Discovery Environment as a registered CyVerse user, see Downloading Files and Folders in the Discovery Environment manual.
  4. Change the download location if needed and click Save.
  5. Repeat for each file to download.
  1. Click the link you received.
  2. Download the file to your system.

Using iCommands

iCommands is a collection of commands for Linux and Mac OS operating systems that are used in the iRODS system to interact with the CyVerse Data Store. Many commands are very similar to Unix utilities. For example, to list files and directories, in Linux you use ls, but in iCommands you use ils. While iCommands are great for all transfers and for automating tasks via scripts, they are the best choice for downloading large files (2-100 GB each) and for bulk file transfers (many small files) without a CyVerse account.

iCommands must be installed and configured for your system. Once set up, you can use iCommands to download files that have been shared by other users. If you have a CyVerse account, you have access to all public files in the DE Community Data folder, but even if you don't have an account, you can download public files using iCommands. For more information, see Using iCommands.

Setting up iCommands to download files from Community Contributed Data and CyVerse Curated Data

iCommands must be set up and configured on each user's local machine or server before you can download data from either of the CyVerse Data Commons repositories using iCommands. Before you begin, you may want to watch a CyVerse video about iCommands.

Step 1: Download iCommands for your operating system

Because CyVerse products are interdependent with iRODS iCommands and iRODS products such as FUSE, the instructions for downloading iCommands vary according to your OS:

IMPORTANT! -- BEFORE YOU UPGRADE

Icon

Before upgrading to the 4.1.9 version, see Upgrading to iRODS 4.1.9 to view the list of services and testing done on the affected CyVerse offerings.

The most recent supported version for use with most CyVerse services is iRODS 4.1.9 (9/26/16 release).

 Not sure which version you have?

To find the version you have installed, use ienv to find the client version or imiscsvrinfo to find the server version.

Linux

Icon

This Linux installer was tested for iCommands 4.1.9 compatibility with CentOS 6 and Ubuntu 14.04 only.

If you are using Linux and have the ability to install software through a package manager such as yum or apt, you may install the irods-icommands package obtainable from the iRODS Download page

If you cannot use one of the packages from iRODS, we provide some installers for CentOS 6 and Ubuntu 14.04 that don't require special privileges to use. To do so, do the following.

  1. Download the correct installer: CentOS 6 Installer or Ubuntu 14.04 Installer. 
  2. From a terminal window, run the installer with sh.
    For example, if you were installing iCommands on an Ubuntu 14.04 system as the user cyverse-user, you would do something like the following example. The iCommands would end up being placed in the directory /home/cyverse-user/icommands.

Icon

iCommands can be installed into any directory without the requirement for elevated permissions. You must have write permission to the directory (such as /home/cyverse-user in the above example). If you encounter difficulties, you may want to seek assistance from your local IT support staff.


Tip: If using iCommands in an HPC environment, which has many systems with iCommands installed, run the module load irods command to get access to iRODS iCommands.  

    3, Continue to Step 2 to configure iCommands.  

Mac OS

Icon

This Mac OS installer was tested for iCommands 4.1.9 compatibility with Mac OS 10.9 to 10.11 only.

iRODS does not currently offer a 4.1.9 download for Mac OS. However, CyVerse has created a custom build of the iRODS 4.1.9 iCommands. To obtain these, do the following.

  1. Download the Mac OS X iCommands 4.1.9 installer. 
  2. Find the downloaded package, hold down the control button, and select the downloaded package.
  3. Click Open to give permission for Mac to open the package.
  4. Use the installer and follow the instructions.
  5. Open a new terminal window, if necessary, and run ienv to check that the new version was installed.
  6. Continue to Step 2 to configure iCommands.

Microsoft Windows

iRODS does not currently offer a 4.1.9 download for Windows users. If you already had version 3 of iCommands for Windows installed, the main functionality should work, but it has not been tested by CyVerse. If necessary, you can use Cyberduck instead.

Step 2: Initialize/start the iRODS connection and configure the settings (One-time only or after changing your configuration)

The first time you use iCommands, or after you change a configuration detail such as resetting your CyVerse password, you must initiate the connection to iRODS.

  1. In a terminal window, enter iinit to initialize iCommands and your Data Store connection. For example, here's what you would do if your user name were cyverse-user:

     Example: iinit
    Terminal window displays:
  2. For both platforms, enter the following:

    For this line:

    Enter this:Comments
    Host:
    • irodsHost (for iRODS)
    • Host (for iDrop)
    data.iplantcollaborative.org 
    Port1247If required

    User:

    • irodsUserName (for iRODS)
    • User (for iDrop)

    (your CyVerse username) 
    If no CyVerse account or only accessing public data sets, use anonymous.

     

    Zone:

    • irodsZone (for iRODS)
    • Zone (for iDrop)
    iplant 

    Password:

    • Current iRODS password (for iRODS)
    • Password (for iDrop)

    (your CyVerse password) 
    If using the anonymous username, no password is required.

     
    Default Resource(leave blank)Defaults to whatever is available
    About Firewalls

    Transfers to the Data Store require firewalls to allow TCP port 1247 and TCP and UDP ports 20000-20399 to be open. For more information on firewalls and connecting to the Data Store, see Configuring Your Firewall to Connect to the Data Store.

    If you notice that long running transfers are terminated abruptly: Some institutional firewalls have policies governing the duration allowed for idle connections that may interfere with data transfers. 

    CyVerse staff can provide additional networking and subnet information for firewall policies upon request. For more information, contact CyVerse Support at support@cyverse.org.

  3. When done, log out of the Data Store (especially important if on a shared or public computer) by entering iexit. (To log back in on the same device, enter iinit.)
    After you are successfully connected, you do not need to reenter your password to use the CyVerse Data Store from that computer unless you log out.

Advanced configuration settings (for power users only)

FOR POWER USERS ONLY

The following settings allow power users to apply more advanced configuration settings for iRODS, but may be safely ignored by others.

A full list of iRODS environment variables that may be set can be determined from ~/.irods/irods_environment.json section of the official iRODS documentation. The names environment variables are the all capitals version of the corresponding json field. Only IRODS_HOST, IRODS_PORT, IRODS_USER_NAME, and IRODS_ZONE_NAME are required use the iCommands.

Setting up Bash autocomplete for iCommands

If using the Bash shell, you can enable tab auto-complete when using iCommands:

  1. Download i-commands-auto.bash.
  2. In your home directory, rename i-commands-auto.bash to .i-commands-auto.bash

  3. In your .bashrc or .bash_profile, enter the following:
    source .i-commands-auto.bash

Next steps

To learn more about using iCommands and some common commands to use, see Using iCommands.


Cyberduck

Cyberduck is a free cross-platform open source file transfer program that supports multiple transfer protocols (FTP, SFTP, WebDAV, Cloud files, Amazon S3, etc.) for Mac OS (OS X 10.6 on Intel 64-bit) and Windows (XP, Vista, Windows 7, Windows 8). LINUX users should use iCommands or iDrop Desktop. For an overview of using Cyberduck at CyVerse, see Using Cyberduck for Uploading and Downloading to the Data Store. Instructions for installing for anonymous login are below.

Installing and configuring Cyberduck

Step 1: Install or update Cyberduck

If Cyberduck is already installed, check if you need to update:

  1. Click the Cyberduck menu.
  2. Click Check for Updates.
  3. If an update is available (you must be at version 4.7.1 or later), click Install Update.

To install Cyberduck for your operating system for initial installation:

  1. Go to the Cyberduck installation page at https://cyberduck.io/.
  2. Follow the steps for your OS (not available for Linux users):
    • For Mac OS:
    1. Click Download Cyberduck-4.7.1.zip (or later version than 4.7.1).
    2. Move the downloaded file (either a zip file or the unzipped application file, depending on your browser) to your Applications folder. If the zip file is listed, unzip the file in your Applications folder.
      IMPORTANT: The file must be located in your Applications folder.
    • For Windows:
  1. Click Download Cyberduck-Installer-4.7.1.exe (or later version than 4.7.1, if displayed).
  2. Locate the downloaded file and double-click to begin installation.
  3. Go through the install process.

Step 2: Configure Cyberduck for use with the Data Store (one-time only)

These instructions are for configuring the iRODS Connective Profile for the Data Store, both for CyVerse user account access and for anonymous access, if you do not have a CyVerse user account.

Icon

This configuration is for the CyVerse main Data Store. To connect Cyberduck to a federated or dedicated resource server, please contact your network administrator.

  1. Click Cyberduck app icon to open Cyberduck.
    See the Cyberduck Preferences Help page on the Cyberduck website for more information on installation.
  2. Click this link to download the Connection Profile, which contains preconfigured settings for using Cyberduck with the CyVerse Data Store.
  3. Click to open the downloaded CyVerse Data Store.cyberduckprofile file.

  4. Verify CyVerse Data Store is displayed in the first field.
  5. In the Nickname field, enter data.cyverse.org – iRODS.
  6. Create the connection:
    1. In the Server field, enter data.cyverse.org.
    2. In the Port field, enter 1247.
    3. Click the Anonymous Login checkbox.
    4. Verify anonymous is displayed in the Username field.
      (To connect using your CyVerse user account, enter your CyVerse user name in the Username field, and then verify your username is added to the URL field, as shown above. Example: irods://yourusername@data.cyverse.org.
      The remaining fields are populated.

7. Click in the Transfer Files drop-down list and select Open multiple connections.

8. Close the window.

Using Cyberduck

About these steps

These instructions give you only the main steps for downloading files to the CyVerse Data Store using Cyberduck. For more information on how to use Cyberduck, please see the Cyberduck Help manual or the Cyberduck FAQs, or contact Cyberduck Support.

Logging in to Cyberduck with anonymous login (no user account)

  1. If necessary, click  to open Cyberduck.
  2. In the Cyberduck window, click the bookmark to use and then click Open Connection.
  3. In the drop-down list, click the repository to use.
  4. Click the Anonymous Login checkbox.
  5. Click Connect.

Downloading a file or folder from the Data Store (CyVerse users and anonymous users)

  1. Click the file or folder to download.
  2. Click the Cyberduck File menu and then click New Download. The file is downloaded to your default download folder.
    • To change your default download folder, click , click Download Folder and select the new default download folder.
  3. To navigate to a different folder (for example, to download a file from the Community Data folder):

    Do not click iplant/home or iplant/

    Icon

    Clicking cyverse/home or cyverse/ will cause your system to hang.

    1. Click the Cyberduck Go menu and then click Go to Folder:
    2. For path, enter /iplant/home/shared (as shown above).
    3. Click Go.

iDrop Desktop

iDrop Desktop allows anonymous users to download multiple large files (50-100 GB or larger) and directories from the CyVerse Data Store to your local machine in a cross-platform, high-throughput data transfer application. Please note that we recommend that Mac OS and Windows users use Cyberduck instead; LINUX users should use iCommands or iDrop Desktop.

To install and configure iDrop Desktop for your system, as well as to see troubleshooting for iDrop, please see Using iDrop Desktop.

Once installed, use the following to log in for anonymous download using iDrop Desktop:

  • User name: anonymous
  • Password: anonymous
  • Press Return.