General functions

Installing conda packages

Install conda packages on the cluster nodes using adam scale conda install.

EXAMPLE: To install NumPy and pandas and their dependencies into the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda install numpy pandas
All nodes (x4) response:
{
  "actions": {
    "LINK": [
      "mkl-11.3.3-0 /opt/continuum/anaconda/pkgs 1",
      "openssl-1.0.2h-1 /opt/continuum/anaconda/pkgs 1",
      "sqlite-3.13.0-0 /opt/continuum/anaconda/pkgs 1",
      "zlib-1.2.8-3 /opt/continuum/anaconda/pkgs 1",
      "python-2.7.11-5 /opt/continuum/anaconda/pkgs 1",
      "conda-env-2.5.1-py27_0 /opt/continuum/anaconda/pkgs 1",
      "numpy-1.11.0-py27_2 /opt/continuum/anaconda/pkgs 1",
      "pycosat-0.6.1-py27_1 /opt/continuum/anaconda/pkgs 1",
      "pytz-2016.4-py27_0 /opt/continuum/anaconda/pkgs 1",
      "pyyaml-3.11-py27_4 /opt/continuum/anaconda/pkgs 1",
      "requests-2.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
      "ruamel_yaml-0.11.7-py27_0 /opt/continuum/anaconda/pkgs 1",
      "setuptools-23.0.0-py27_0 /opt/continuum/anaconda/pkgs 1",
      "six-1.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
      "wheel-0.29.0-py27_0 /opt/continuum/anaconda/pkgs 1",
      "conda-4.1.4-py27_0 /opt/continuum/anaconda/pkgs 1",
      "pip-8.1.2-py27_0 /opt/continuum/anaconda/pkgs 1",
      "python-dateutil-2.5.3-py27_0 /opt/continuum/anaconda/pkgs 1",
      "pandas-0.18.1-np111py27_0 /opt/continuum/anaconda/pkgs 1"
      ],
      "PREFIX": "/opt/continuum/anaconda",
    },
    "success": true
  }

Creating conda environments

Create multiple conda environments on the cluster nodes using adam scale conda create.

EXAMPLE: To create a conda environment called py3 with Python 3 on a cluster named cluster:

$ adam scale -n cluster conda create -n py3 python=3
All nodes (x4) response:
{
  "actions": {
    "LINK": [
      "openssl-1.0.2h-1 1 None",
      "readline-6.2-2 1 None",
      "sqlite-3.13.0-0 1 None",
      "tk-8.5.18-0 1 None",
      "xz-5.2.2-0 1 None",
      "zlib-1.2.8-3 1 None",
      "python-3.5.1-5 1 None",
      "setuptools-23.0.0-py35_0 1 None",
      "wheel-0.29.0-py35_0 1 None",
      "pip-8.1.2-py35_0 1 None"
    ],
    "PREFIX": "/opt/continuum/anaconda/envs/py3",
    "SYMLINK_CONDA": [
      "/opt/continuum/anaconda"
    ],
  },
  "success": true
}

Updating conda packages

Update conda packages on the cluster nodes using adam scale conda update.

EXAMPLE: To update pandas and its dependencies in the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda update pandas
All nodes (x4) response:
{
  "actions": {
    "LINK": [
      "pandas-0.18.1-np111py27_0 1 None"
    ],
    "PREFIX": "/opt/continuum/anaconda",
    "SYMLINK_CONDA": [
      "/opt/continuum/anaconda"
    ],
    "UNLINK": [
      "pandas-0.18.0-np111py27_0"
    ],
  },
  "success": true
}

Removing conda packages or environments

Remove conda packages on the cluster nodes using adam scale conda remove.

EXAMPLE: To remove pandas from the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda remove pandas
All nodes (x4) response:
{
  "actions": {
    "PREFIX": "/opt/continuum/anaconda",
    "UNLINK": [
      "pandas-0.18.1-np111py27_0"
    ],
  },
  "success": true
}

Remove conda environments on the cluster nodes using adam scale conda remove.

EXAMPLE: To remove an environment called py3 on a cluster named cluster:

$ adam scale -n cluster conda remove -n py3 --all
All nodes (x4) response:
{
  "actions": {
    "PREFIX": "/opt/continuum/anaconda/envs/py3",
    "UNLINK": [
      "openssl-1.0.2j-0",
      "pip-8.1.2-py35_0",
      "python-3.5.2-0",
      "readline-6.2-2",
      "setuptools-27.2.0-py35_0",
      "sqlite-3.13.0-0",
      "tk-8.5.18-0",
      "wheel-0.29.0-py35_0",
      "xz-5.2.2-0",
      "zlib-1.2.8-3"
    ]
  },
  "success": true
}

Listing conda packages

List conda packages on the cluster nodes using adam scale conda list.

EXAMPLE: To list conda packages in the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda list
All nodes (x4) response:
+-----------------+---------+--------+
| Package         | Version | Build  |
+-----------------+---------+--------+
| numpy           | 1.11.0  | py27_2 |
| conda-env       | 2.5.1   | py27_0 |
| tk              | 8.5.18  | 0      |
| pip             | 8.1.2   | py27_0 |
| readline        | 6.2     | 2      |
| setuptools      | 23.0.0  | py27_0 |
| openssl         | 1.0.2h  | 1      |
| pycosat         | 0.6.1   | py27_1 |
| python-dateutil | 2.5.3   | py27_0 |
| sqlite          | 3.13.0  | 0      |
| pytz            | 2016.4  | py27_0 |
| mkl             | 11.3.3  | 0      |
| yaml            | 0.1.6   | 0      |
| ruamel_yaml     | 0.11.7  | py27_0 |
| zlib            | 1.2.8   | 3      |
| pyyaml          | 3.11    | py27_4 |
| requests        | 2.10.0  | py27_0 |
| wheel           | 0.29.0  | py27_0 |
| python          | 2.7.11  | 5      |
| conda           | 4.1.4   | py27_0 |
| pycrypto        | 2.6.1   | py27_0 |
| six             | 1.10.0  | py27_0 |
+-----------------+---------+--------+

Getting conda information

View information about the conda installation on the cluster nodes using adam scale conda info.

EXAMPLE: To view information about the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda info
All nodes (x4) response:
Current conda install:

             platform : linux-64
        conda version : 4.1.4
    conda-env version : 2.5.1
  conda-build version : not installed
       python version : 2.7.11.final.0
     requests version : 2.10.0
     root environment : /opt/continuum/anaconda  (writable)
  default environment : /opt/continuum/anaconda
     envs directories : /opt/continuum/anaconda/envs
        package cache : /opt/continuum/anaconda/pkgs
         channel URLs : https://repo.anaconda.com/pkgs/free/linux-64/
                        https://repo.anaconda.com/pkgs/free/noarch/
                        https://repo.anaconda.com/pkgs/pro/linux-64/
                        https://repo.anaconda.com/pkgs/pro/noarch/
          config file : None
         offline mode : False
    is foreign system : False

Or to view a list of conda environments on the cluster nodes:

$ adam scale -n cluster conda info -e
All nodes (x4) response:
# conda environments:
#
py27                     /opt/continuum/anaconda/envs/py27
py35                     /opt/continuum/anaconda/envs/py35
r                        /opt/continuum/anaconda/envs/r
root                  *  /opt/continuum/anaconda

Managing conda with environment definition files

Manage conda environments on the cluster nodes using adam scale env:

$ adam scale -n cluster env
Usage: adam scale env [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help  Show this message and exit.

Commands:
  create  Create conda environment
  list    List conda environments
  remove  Remove conda environment
  update  Update conda environment

This functionality uses conda-env and conda environment environment definition–YAML–files.

EXAMPLE: The following environment.yml file describes a conda environment named my-env with Python 2.7 and additional packages:

name: my-env
channels:
- defaults
dependencies:
- python=2.7.*
- bokeh
- dask
- distributed
- pandas

Use adam scale env create to create a conda environment based on the above conda environment definition file environment.yml:

$ adam scale -n cluster env create -f environment.yml
All nodes (x4) response:
{
  "actions": {
    "EXTRACT": [
      "msgpack-python-0.4.8-py27_0",
      "numpy-1.11.2-py27_0",
      "psutil-5.0.0-py27_0",
      "pytz-2016.7-py27_0",
      "tblib-1.3.0-py27_0",
      "python-dateutil-2.6.0-py27_0",
      "zict-0.1.0-py27_0",
      "pandas-0.19.1-np111py27_0",
      "tornado-4.4.2-py27_0",
      "bokeh-0.12.3-py27_0",
      "dask-0.12.0-py27_0",
      "distributed-1.14.1-py27_0"
    ],
    "FETCH": [
      "msgpack-python-0.4.8-py27_0",
      "numpy-1.11.2-py27_0",
      "psutil-5.0.0-py27_0",
      "pytz-2016.7-py27_0",
      "tblib-1.3.0-py27_0",
      "python-dateutil-2.6.0-py27_0",
      "zict-0.1.0-py27_0",
      "pandas-0.19.1-np111py27_0",
      "tornado-4.4.2-py27_0",
      "bokeh-0.12.3-py27_0",
      "dask-0.12.0-py27_0",
      "distributed-1.14.1-py27_0"
    ],
    "LINK": [
      "mkl-11.3.3-0 1",
      "openssl-1.0.2j-0 1",
      "readline-6.2-2 1",
      "sqlite-3.13.0-0 1",
      "tk-8.5.18-0 1",
      "yaml-0.1.6-0 1",
      "zlib-1.2.8-3 1",
      "python-2.7.12-1 1",
      "backports-1.0-py27_0 1",
      "backports_abc-0.4-py27_0 1",
      "click-6.6-py27_0 1",
      "cloudpickle-0.2.1-py27_0 1",
      "futures-3.0.5-py27_0 1",
      "heapdict-1.0.0-py27_1 1",
      "locket-0.2.0-py27_1 1",
      "markupsafe-0.23-py27_2 1",
      "msgpack-python-0.4.8-py27_0 1",
      "numpy-1.11.2-py27_0 1",
      "psutil-5.0.0-py27_0 1",
      "pytz-2016.7-py27_0 1",
      "pyyaml-3.12-py27_0 1",
      "requests-2.11.1-py27_0 1",
      "setuptools-27.2.0-py27_0 1",
      "six-1.10.0-py27_0 1",
      "tblib-1.3.0-py27_0 1",
      "toolz-0.8.0-py27_0 1",
      "wheel-0.29.0-py27_0 1",
      "chest-0.2.3-py27_0 1",
      "jinja2-2.8-py27_1 1",
      "partd-0.3.6-py27_0 1",
      "pip-9.0.1-py27_0 1",
      "python-dateutil-2.6.0-py27_0 1",
      "singledispatch-3.4.0.3-py27_0 1",
      "ssl_match_hostname-3.4.0.2-py27_1 1",
      "zict-0.1.0-py27_0 1",
      "pandas-0.19.1-np111py27_0 1",
      "tornado-4.4.2-py27_0 1",
      "bokeh-0.12.3-py27_0 1",
      "dask-0.12.0-py27_0 1",
      "distributed-1.14.1-py27_0 1"
    ],
    "PREFIX": "/opt/continuum/anaconda/envs/my-env",
    "SYMLINK_CONDA": [
      "/opt/continuum/anaconda"
    ],
    "op_order": [
      "RM_FETCHED",
      "FETCH",
      "RM_EXTRACTED",
      "EXTRACT",
      "UNLINK",
      "LINK",
      "SYMLINK_CONDA"
    ]
  },
  "success": true
}

NOTE: The command adam scale env create is compatible only with conda dependencies and ignores any pip or other non-conda dependencies that are specified in the conda environment definition file.

Use adam scale env list to list conda environments on the cluster:

$ adam scale -n cluster env list
All nodes (x4) response:
# conda environments:
#
my-env                   /opt/continuum/anaconda/envs/my-env
root                  *  /opt/continuum/anaconda

Use adam scale env update to update packages in an existing conda environment based on a conda environment definition file:

$ adam scale -n cluster env update -f environment.yml
    All nodes (x4) response:
    {
      "message": "All requested packages already installed.",
      "success": true
    }

Use adam scale env remove to remove a conda environment from the cluster:

$ adam scale -n cluster env remove -n my-env
All nodes (x4) response:
{
  "actions": {
    "PREFIX": "/opt/continuum/anaconda/envs/my-env",
    "UNLINK": [
      "mkl-11.3.3-0",
      "numpy-1.11.2-py27_0",
      "openssl-1.0.2j-0",
      "pandas-0.18.0-np111py27_0",
      "pip-9.0.1-py27_0",
      "python-2.7.12-1",
      "python-dateutil-2.6.0-py27_0",
      "pytz-2016.7-py27_0",
      "readline-6.2-2",
      "setuptools-27.2.0-py27_0",
      "six-1.10.0-py27_0",
      "sqlite-3.13.0-0",
      "tk-8.5.18-0",
      "wheel-0.29.0-py27_0",
      "zlib-1.2.8-3"
    ]
  },
  "success": true
}