IOPro

IOPro loads NumPy arrays (and Pandas DataFrames) directly from files, SQL databases, and NoSQL stores–including ones with millions of rows–without creating millions of temporary, intermediate Python objects, or requiring expensive array resizing operations.

IOPro provides a drop-in replacement for the NumPy functions loadtxt() and genfromtxt(), but dramatically improves performance and reduces memory overhead.

The current (and final) version of IOPro 1.9.1 was released on March 1, 2017.

The open source successors to IOPro

All of the code in IOPro was released in 2017 under a BSD open source license. As part of that release, the code was refactored into its several component libraries, and version numbers of each were set to 2.0.0 to indicate they supersede the code under the IOPro name.

The several libraries are available on GitHub at:

These various packages are or will be available as conda packages (using lowercase versions of their names), mostly in the conda-forge channel. Maintenance, improvements, and packaging will be a community-led project.

How to get IOPro

IOPro is included with Anaconda Workgroup and Anaconda Enterprise subscriptions.

To start a 30-day free trial just download and install the IOPro package.

If you already have Anaconda (free Python platform) or Miniconda installed:

conda update conda
conda install -c anaconda-extras iopro=1.9.1

If you do not have Anaconda installed, you can download it.

For more information about IOPro please contact sales@anaconda.com.

Requirements

  • Python 2.7 or 3.4+
  • NumPy 1.10+

Optional Python modules:

  • Boto (for S3 support)
  • Pandas (to use DataFrames)

What’s new in version 1.9?

The documentation has been substantially updated for version 1.9.0. Numba has been removed and the code has been cleaned up, but no other features were added or removed. Some refactoring was done that didn’t change functionality. We recommend that users not use older versions. See Release notes for additional detail.

Getting started

Some of the basic usage patterns look like these. Create TextAdapter object for data source:

>>> import iopro
>>> adapter = iopro.text_adapter('data.csv', parser='csv')

Define field dtypes (example: set field 0 to unsigned int and field 4 to float):

>>> adapter.set_field_types({0: 'u4', 4:'f4'})

Parse text and store records in NumPy array using slicing notation:

>>> # read all records
>>> array = adapter[:]

>>> # read first ten records
>>> array = adapter[0:10]

>>> # read last record
>>> array = adapter[-1]

>>> # read every other record
>>> array = adapter[::2]

Previous Versions

This documentation is provided for the use of our customers who have not yet upgraded to the current version.

NOTE: We recommend that users not use older versions of IOPro.

Was this helpful?