This box searches only documentation in the Data Store space. To search the entire CyVerse wiki, use the box at the upper right.

For overview information, see Data Store or Manage Data on the CyVerse website.

DATA STORE USER MANUAL
 

 

 

 

 

 

 

Skip to end of metadata
Go to start of metadata
IMPORTANT NOTE ABOUT UPGRADING TO 4.1

iCommands 4.1.10, the latest supported version for CyVerse services, has been released by iRODS. For a list of affected services and testing done, see Upgrading to iRODS 4.1.

About iCommands

iCommands is a collection of commands for Linux and Mac OS operating systems that are used in the iRODS system to interact with the CyVerse Data Store. Many commands are very similar to Unix utilities. For example, to list files and directories, in Linux you use ls , but in iCommands you use ils .

While iCommands are great for all transfers and for automating tasks via scripts, they are the best choice for large files (2-100 GB each) and for bulk file transfers (many small files). For a comparison of the different methods of uploading and downloading data items, see Downloading and Uploading Data.

iCommands must be installed and configured for your system. Once set up, CyVerse users can use iCommands to access files that have been shared by other users, as well as manage metadata, change permissions on data files and folders you own, and more.

Even if you don't have an account, you can download public files using iCommands. For more information, see the iCommands section on the Downloading Data Files Without a User Account page. For information on using iCommands, see Using iCommands.

For common Linux distributions, there are NetCDF iCommands available. Please see the NetCDF iCommands section.

For common Linux distributions, there are NetCDF iCommands available. Please see the NetCDF iCommands section for more information.

Setting up iCommands

iCommands must be set up and configured on each user's local machine or server, as outlined below, before you can transfer data to (with a CyVerse user account) and from the CyVerse Data Store using iCommands. Note that iCommands cannot be used to upload files into Data Store via URL from other sites (ftp, http, etc.). Without a CyVerse account, you can download public data from the Data Commons Repository (DCR).

For account users, you can download data from an external site using iCommands. To do so, download the file to a local machine using wget or a similar mechanism and then use iput to upload it to the Data Store. If you have a significant amount of external data to import from an external site, you may want to perform the transfer using a utility, such as Import by URL in the DE (not available using iCommands), or an Atmosphere instance.

Before you begin, you may want to watch a CyVerse video about iCommands.

Helpful Links

On this page:

Related Pages:

Step 1: Download iCommands for your operating system

Because CyVerse products are interdependent with iRODS iCommands and iRODS products such as FUSE, the instructions for downloading iCommands vary according to your OS.

IMPORTANT! -- BEFORE YOU UPGRADE

Icon

Before upgrading to a 4.1 version, see Upgrading to iRODS 4.1 to view the list of services and testing done on the affected CyVerse offerings.

The most recent supported version for use with most CyVerse services is iRODS 4.1.10.

 Not sure which version you have?

To find the version you have installed, use ienv to find the client version or imiscsvrinfo to find the server version.

Linux

Icon

If using iCommands in an HPC environment, which has many systems with iCommands installed, run the module load irods command to get access to iRODS iCommands.

Installation via Package Manager

If you are using Linux and have the ability to install software (that is, you have sudo or root access) through a package manager such as yum or apt, you may install the irods-icommands package obtainable from one of the following links.

The CentOS packages require the the yum be configured to access the EPEL repository. The Ubuntu 14 package appears to also work for Ubuntu 16.04. Run the package installation commands appropriate for your package manager, replacing the filename as appropriate for your distribution version, as in the following examples.

alternate packages with performance enhanced FUSE client

Icon

If you intend to heavily use the iRODS FUSE client, we have a performance enhanced version. The enhancements were made to the 4.1.9 version. You many install the package obtainable from one of the following links.

 

For Debian/Ubuntu:

Code for Debian/Ubuntu

For RHEL/CentOS:

Code for RHEL/CentOS

Next , continue to Step 2 to configure iCommands.

Installation for Unprivileged Users

If you cannot use one of the packages from iRODS, we provide some installers for CentOS and Ubuntu that don't require special privileges to use. To do so, do the following.

iCommands can be installed into any directory without the requirement for elevated permissions. You must have write permission to the directory (such as /home/cyverse-user in the above example). If you encounter difficulties, you may want to seek assistance from your local IT support staff.

  1. Download the correct installer.

    alternate installers compatible with performance enhanced FUSE client

    Icon

    If you intend to heavily use the iRODS FUSE client, it requires version 4.1.9 of the iCommands to be installed. Here are the installers for this version.

  2. From a terminal window, run the installer with  sh .

    For example, if you were installing iCommands on an Ubuntu 14.04 system as the user cyverse-user, you would do something like the following example. The iCommands would end up being placed in the directory /home/cyverse-user/icommands.

    Important: To make the changes take effect in the current shell, you will need to source your .bashrc file by entering either `. ~/.bashrc` .

    If you aren't using bash, or if your .bashrc file can't be edited, the environment variable IRODS_PLUGINS_HOME will need to be set to $INSTALL_BASE/icommands/plugins/ , where $INSTALL_BASE is the directory you provided to the installer. It is also recommended that you add the iCommands to your path. Since setting the environment variable and modifying your path is the last thing the installer does, you don't need to run the installer again.


  3. Open a new terminal window, if necessary, and run ienv to check that the new version was installed.

     

  4. Continue to Step 2 to configure iCommands.

Mac OS

Icon

This Mac OS installer was tested by CyVerse for iCommands 4.1.9 compatibility with Mac OS 10.9 to 10.11 only, users have reported success with 10.15.2

iRODS does not currently offer a 4.1 download for Mac OS. However, CyVerse has created a custom build of the iRODS 4.1.9 iCommands. To obtain these, do the following.

  1. Download the   Mac OS X iCommands 4.1.9 installer .
  2. Find the downloaded file in the finder window, hold down the control button, and select the downloaded package.
  3. Click Open to give permission for Mac to open the package.
    Note: if you try to directly click and install the downloaded file, privacy and security setting in new release of Mac OS will prevent the installation and warn that "CyVerse is a unknown publisher", it is important to follow step #2 and 3 above.
  4. Use the installer and follow the instructions .
  5. Open a new terminal window, if necessary, and run ienv to check that the new version was installed.
  6. Continue to Step 2 to configure iCommands.

Microsoft Windows

iRODS does not currently offer a 4.1 download for Windows users. If you are running Windows 10, iCommands can be run on the Linux subsystem. See  https://docs.microsoft.com/en-us/windows/wsl/install-win10 for information on installing the Linux subsystem. Then follow the iCommands for Linux instructions to install iCommands.

Step 2: Initialize/start the iRODS connection and configure the settings (One-time only or after changing your configuration)

The first time you use iCommands, or after you change a configuration detail such as resetting your CyVerse password, you must initiate the connection to iRODS.

In a terminal window, enter iinit to initialize iCommands and your Data Store connection. For example, here's what you would do if your iRODS user name is cyverse-user:

 

 Example: iinit
Terminal window displays:

 

Here are the values to enter.

For this line:

Enter this:Comments
Host:
  • irodsHost (for iRODS)
  • Host (for iDrop)
data.cyverse.org 
Port1247If required

User:

  • irodsUserName (for iRODS)
  • User (for iDrop)

(your CyVerse username) 
If no CyVerse account or only accessing public data sets, use anonymous.

 

Zone:

  • irodsZone (for iRODS)
  • Zone (for iDrop)
iplant 

Password:

  • Current iRODS password (for iRODS)
  • Password (for iDrop)

(your CyVerse password) 
If using the anonymous username, no password is required.

 
Default Resource(leave blank)Defaults to whatever is available
About Firewalls

Transfers to the Data Store require firewalls to allow TCP port 1247 and TCP and UDP ports 20000-20399 to be open. For more information on firewalls and connecting to the Data Store, see Configuring Your Firewall to Connect to the Data Store.

If you notice that long running transfers are terminated abruptly: Some institutional firewalls have policies governing the duration allowed for idle connections that may interfere with data transfers. 

CyVerse staff can provide additional networking and subnet information for firewall policies upon request. For more information, contact CyVerse Support at support@cyverse.org.

Icon

If you are using Atmosphere and get an error that includes the following:

You may need to define an environment variable IRODS_PLUGINS_HOME to resolve the error. Execute the following command and then try running iinit again (and add this line to your $HOME/.bashrc):

Once iinit has finished, use ils to check that the iCommands is working. You should see your home directory at /iplant/home/your_user_name

When done, log out of the Data Store (especially important if on a shared or public computer) by entering iexit full. (To log back in on the same device, enter iinit.)
After you are successfully connected, you do not need to reenter your password to use the CyVerse Data Store from that computer unless you log out.

Advanced configuration settings (for power users only)

FOR POWER USERS ONLY

The following settings allow power users to apply more advanced configuration settings for iRODS, but may be safely ignored by others.

A full list of iRODS environment variables that may be set can be determined from ~/.irods/irods_environment.json section of the official iRODS documentation. The names environment variables are the all capitals version of the corresponding JSON field. Only IRODS_HOST, IRODS_PORT, IRODS_USER_NAME, and IRODS_ZONE_NAME are required for using the iCommands.

Setting up Bash autocomplete for iCommands

If using the Bash shell, you can enable tab auto-complete when using iCommands:

Download i-commands-auto.bash.

In your home directory, rename i-commands-auto.bash to .i-commands-auto.bash.

In your .bashrc or .bash_profile, enter the following:
source .i-commands-auto.bash

NetCDF iCommands

For the Linux distributions CentOS 6 and 7 and Ubuntu 12.04 and 14.04, there are three extra iCommands that support common NetCDF operations.  inc performs data operations on a list of NetCDF files. incarch archives a open ended time series data. incattr performs operation on attributes of NetCDF files. Each of these commands accepts the -h command line option. When a command is called with this option, it displays the command's help documentation.  Please see this help documentation for more information.

Installation

It is assumed that the core iCommands are already installed on your system. To install the NetCDF iCommands, root or sudo access is required.

Step 1 - Install iRODS Run-time Library

Before the NetCDF iCommands can be installed, the version 4.1.10 of the iRODS run-time library needs to be installed. Please install the appropriate version of the irods-runtime-4.1.10. The distribution specific packages can be found on  RENCI's website.

Step 2 - Install iRODS NetCDF API

Once the run-time library is installed, the iRODS NetCDF API library needs to be installed. Please use the appropriate link to the download the installation package and install it. The package installer will likely warn that irods user and/or group don't exist, and that it will be using root instead. These warnings are harmless, since the package contents should be installed with root ownership.

Step 3 - Install iRODS NetCDF iCommands

Once the run-time and API libraries are installed, the NetCDF iCommands can be installed. Please use the appropriate link to download the installation package and install it. The package installer will likely warn that irods user and/or group don't exist, and that it will be using root instead. These warnings are harmless, since the package contents should be installed with root ownership.

Next steps

To learn more about using iCommands and some common commands to use, see Using iCommands.

  • No labels

2 Comments

  1. Chris Martin

    I published an Ansible role which installs iCommands (including irodsFs) for CentOS 5+ and Ubuntu 12+. This is designed to be modular and consumable by anyone, at CyVerse or elsewhere.

    https://github.com/CyVerse-Ansible/ansible-irods-icommands

    Whoever maintains these instructions, feel free to work this in as an installation option.

  2. Andrew Antaya

    I found that the on Windows Subsystem for Linux using Ubuntu 16.04, the setup code above for Debian/Ubunutu did not work for my machine. However I found that I first had to use 'wget' to fetch the iCommands package from the web and then use 'dpkg' to install the iCommands package. Here's my code:

    wget https://files.renci.org/pub/irods/releases/4.1.10/ubuntu14/irods-icommands-4.1.10-ubuntu14-x86_64.deb

    sudo dpkg -i irods-icommands-4.1.10-ubuntu14-x86_64.deb

    I hope this may be useful to anyone running Windows but still wanting to use iCommands.