This box searches only documentation in the Data Store space. To search the entire CyVerse wiki, use the box at the upper right.


For overview information, see Data Store or Manage Data on the CyVerse website.

DATA STORE USER MANUAL
Maintenance Notice for July 23, 2019

On Tuesday, July 23, CyVerse will be performing a Ceph storage upgrade from 11:00 AM to 5:00 PM MST. This is not expected to cause disruptions to the affected services. The services to be upgraded include:
- CyVerse authentication (CAS), which provides authentication to CyVerse web applications, e.g., the Discovery Environment and User Portal
- Atmosphere
- Any "Powered by CyVerse" systems or other hosted systems on Tombstone Cloud (please ask if you are unsure)
- Hosted Jupyter Notebooks on jupyter-notebook.cyverse.org

To view the maintenance times for your location: https://bit.ly/2xT1eo7

Please contact support@cyverse.org with any questions or concerns.

 

 

 

 

Skip to end of metadata
Go to start of metadata

About file and folder names

Icon

File names and folder names that are stored using the CyVerse Data Store should not use spaces in the name. Instead, use either a hyphen ( - ) or an underscore ( _ ).

Helpful Links
On this page:

 

WebDAV, according to WebDAV.org, "stands for "Web-based Distributed Authoring and Versioning". It is a set of extensions to the HTTP protocol which allows users to collaboratively edit and manage files on remote web servers." WebDAV provides an easy way to manipulate remote files. CyVerse has added support for WebDAV to the Data Store. This means a users can access their home and public folders in the CyVerse Data Store from their local computers use web browsers and other WebDAV enabled applications like common operating system file managers. With WebDAV, a user can copy files between her local computer and the Data Store as easily as if she were copying them between two folders on her local computer.

Limitations

WebDAV works best for small files or small collections of files. There is no hard size limit for files, but it is recommended to not work with files over 1 GiB in size. However, 10 GiB files have been downloaded from the CyVerse WebDAV service using a web browser with decent performance. The iCommands still out perform WebDAV.  For better ways to download large files or large sets of files, please see Data Store Intro.

Accessing the WebDAV Service

There are two ways to access the WebDAV service. It provides anonymous read-only access through URLs rooted at https://data.cyverse.org/dav-anon/. All data that can be seen by the anonymous user can be accessed anonymously through this service excluding the immediate contents of /iplant/home and the immediate contents of /iplant/home/<username> where <username> is any CyVerse login. The service also provides authenticated access through URLs rooted at https://data.cyverse.org/dav/. Once a user has authenticated with his CyVerse credentials, he can access any file or folder with the permission level he has on that file or folder.

User Data

A user with a CyVerse login of <username> would use the WebDAV link https://data.cyverse.org/dav/iplant/home/<username> / to access his data.

Project Data

To access project data stored in iRODS at /iplant/home/shared/<project> , the link https://data.cyverse.org/dav/iplant/projects/<project>/ would be used.

Community Released Data

To access data released by the CyVerse community into the Data Commons, the following link can be used: https://data.cyverse.org/dav-anon/iplant/projects/.

CyVerse Curated Data

To access the data curated by CyVerse in the Data Commons (that is, datasets with DOIs), the following link can be used: https://data.cyverse.org/dav-anon/iplant/commons/cyverse_curated/.

Common Ways to Access the WebDAV Service

Web Browser

Since WebDAV is an extension of HTTP, any web browser can be used to browse and download data through the service.

File Manager

Most common operating systems come with a file manager application that can interface with a WebDAV service and can mount a WebDAV folder into the local file system. This allows other applications running on the same computer to access WebDAV data as if it were local.

Mac OS X Finder

The following instructions demonstrate how to mount a Data Store folder using Finder.

  1. Open a Finder window.
  2. Click the Go menu at the top left and then click Connect to Server (or type command k):
  3. In the Server Address field, enter the URL for the folder to access and click Connect.
  4. Enter your CyVerse username and password if prompted. If you want to have your username and password saved and automatically used the next time you connect to the Data Store using WebDAV, click Remember this password in my keychain.
  5. Once connected, the chosen Data Store folder appears under the Shared section in Finder, and you may now interact with it as if it is a local or network storage device.

Microsoft Windows File Explorer

Windows 10

The following instructions demonstrate how to mount a Data Store folder in Windows 10 using the File Explorer.

  1. Open File Explorer (file and folder browser).
  2. Right click on This PC.
  3. Click on Map network drive....
  4. In the What network folder would you like to map? window, click on Connect to a Web site that you can use to store your documents and pictures.
  5. In the Welcome to the Add Network Location Wizard, click Next button
  6. In Where do you want to create this network location?, click on Choose a custom network location.
  7. In Specify the location of your website, enter the WebDAV URL for the desired Data Store folder. Then click Next.

    Examples:
    1. If you want your home folder and your username is myusername, use https://data.cyverse.org/dav/iplant/home/myusername/.
    2. If you want a community folder named shareddata, use https://data.cyverse.org/dav-anon/iplant/projects/shareddata/.
  8. Windows will ask you to authenticate if the folder is restricted. Enter your credentials and check the box Remember my credentials, if desired. Click OK.
  9. In the What do you want to name this location?, you can click on Next to accept the default name or create your own name.
  10. In the Completing the Add Network Location Wizard, click on Finish.
  11. The folder should automatically open (if you left the checkbox checked). Otherwise, you can find the folder in File Explorer, under the Network icon.

Windows 7

Icon

These instructions are for Microsoft Windows 7. Other versions of Microsoft Windows should be similar, but have not been tested.

Problems have been reported using WebDAV with Microsoft Windows 7, especially the 64-bit version. This is an operating system issue — not a CyVerse issue. See the Microsoft Answers website for more information. An alternate setup for Windows 7 64-bit is listed below.

The following instructions demonstrate how to mount a Data Store folder in Windows 7 using the File Explorer.

  1. Click the Windows logo at the bottom left of the screen to open the Start menu.
  2. Click Computer from the list at the right:
  3. To add a network connection, right-click in the Computer window and then click Add a network location:
  4. Enter your Data Store information.
    1. Click Choose a custom network location and then click Next.
    2. Enter the WebDAV URL for the desired Data Store folder, and click Next.
    3. Enter your CyVerse username and password if prompted. If you want to have your username and password saved and automatically used the next time you connect to the Data Store WebDAV service, click Remember my credentials, and click OK.

Once connected, the chosen Data Store folder appears in Windows Explorer, and you may interact with it as if it were a local or network storage device.

Windows 7 Alternate

This setup circumvents the issues caused by the built-in Microsoft WebDAV Mini Redirector.

  1. Go to NetDrive to download and install the NetDrive WebDAV client.
  2. Start NetDrive.
  3. Click New Site.
  4. Enter a name for the connection.
  5. Click Server Type and then click WebDAV.
  6. In the Site IP or URL field in NetDrive, enter the WebDAV URL for the desired Data Store folder.
  7. If authentication is required, click the Account field and enter your CyVerse username.
  8. If authentication is required, click the Password field and enter your CyVerse password.
  9. Click Advanced.
  10. In the Advanced window:
    1. Click Use HTTPS.
    2. Click OK. The port should change to 443.
  11. Select a drive letter.
  12. (Optional) Click Automatic login when system starts or Automatic login when NetDrive starts.
  13. Click Save.
  14. Click Connect.

Gnome Desktop on Linux

Icon

These instructions were written using Ubuntu 18.04. Other Linux distributions with a desktop environment will be similar.

The following instructions demonstrate how to mount a Data Store folder in the Gnome Desktop.

  1. Open Files (the file manager).
  2. Select Other Locations in the Places sidebar.
  3. In the Connect to Server footer, enter the WebDAV URL for the desired Data Store folder. Note: Files identifies TLS encrypted WebDAV URLs with the scheme  davs . This means the base for CyVerse URLs is  davs://data.cyverse.org/  instead of https://data.cyverse.org.
  4. Click the neighboring Connect button.
  5. Provide your CyVerse username and password if prompted.

Linux Terminal

Privilege Escalation Requirement

Icon

This method requires root or at least sudo access.

The following instructions demonstrate how to mount a Data Store folder into the file system through a terminal.

  1. Ensure that davfs2 is installed, e.g., for Ubuntu, sudo apt install davfs2.
  2. Create a directory where you want to mount the desired Data Store folder, e.g., mkdir /tmp/data.
  3. Mount the data as root, i.e., sudo mount -o gid=<you>,uid=<you> -t davfs <link> /tmp/data, where <you> is your username on the Linux machine and <link> is the WebDAV URL to the desired Data Store folder.
  4. Provide your CyVerse username and password if prompted.
  • No labels

1 Comment

  1. It looks like there's a problem with installing the davfs2 package in Ubuntu 18.04 (and related distros like Mint 19.1).  The package depends on libneon27 which requires a version of libssl that the distro doesn't have.  There's an open bug at https://bugs.launchpad.net/ubuntu/+source/neon27/+bug/1826063.  You can work around this issue by using an older version of libneon as described at https://askubuntu.com/questions/1137592/davfs2-depends-libneon27-but-it-is-not-going-to-be-installed