Upgrading Anaconda Enterprise Notebooks (AEN 4.1.2)#

Summary¶

IMPORTANT: These instructions are for upgrading Anaconda Enterprise Notebooks (AEN) to the current version 4.1.2 from 4.1.1 ONLY. Each version must be upgraded iteratively from the previous version. Do not skip versions.

Upgrade instructions to previous versions:

For upgrades from versions before those listed, please contact your enterprise support representative for help.

NOTE: Named Service Account functionality is available with Anaconda Enterprise Notebooks 4.0.0+ for new installations only. It is not available for upgraded installations. Contact your enterprise support representative for more information.

An upgrade to the Anaconda Enterprise Notebooks Platform requires all instances of the three service categories to be upgraded individually:

AEN Server
AEN Gateway
AEN Compute

An installation of Anaconda Enterprise Notebooks typically consists of several Compute instances running on independent Compute Nodes, and multiple Gateways if there is more than one Data Center tied into AEN. More details can be found in the Overview.

The upgrade process requires that all AEN service instances be stopped, upgraded, and then restarted. These instructions describe how to perform this process.

Note: Any commands that call for the root user can also be done using sudo.

If you encounter any difficulty during the process, please refer to the Troubleshooting Guide which provides guidance on:

processes
configuration files
log files
ports

In the event you are unable to resolve an installation or upgrade problem, please contact your enterprise support representative.

Pre-Flight Check¶

CAUTION: Make a tested backup of your installation before starting the upgrade. Upgrading to a higher version of AEN is not reversible. Any errors during the upgrade procedure may result in partial or complete data loss and require restoring data from backups.

IMPORTANT: TERMINATE ALL AEN APPS AND STOP ALL PROJECTS before starting the upgrade process.

Before upgrading each service on each host, perform the following operation:

Suspend the services on each of the nodes with the following commands:

sudo service wakari-server stop
sudo service wakari-gateway stop
sudo service wakari-compute stop

Set the AEN Functional ID (NFI) and AEN Functional Group (NFG), which are described in the installation instructions.

The NFI is the username of the AEN Service Account which is used to run all AEN services and is also the username of the AEN Admin account.

The NFG is the Linux group that includes the AEN Service Account, so all files and directories that have the owner NFI also have the group NFG.

When upgrading AEN, set the NFI and NFG to the NFI and NFG of the current installation.

The default NFI is “wakari”, and “aen_admin” or any other name may be used instead.

Set the NFI and NFG with these commands, replacing “wakari” if you are using another name:
```
export AEN_SRVC_ACCT="wakari"
export AEN_SRVC_GRP="wakari"
```
Install wget:
```
yum install wget
```

AEN Server¶

NOTE: If you are using LDAP-based authentication, back up the configuration file:

/opt/wakari/wakari-server/etc/wakari/wk-server-config.json

After the Server has been upgraded, copy that file back into the same location as before the upgrade.

The following operations are all done on the Server host:

Stop the ElasticSearch service:
```
sudo service elasticsearch stop
```

Remove any previous index:

sudo rm -rf /var/lib/elasticsearch/*

NOTE: You can choose to keep the old index but if you detect any issues with the search capabilities after the upgrade, you will need to perform the following steps to start with a clean index:

sudo service wakari-server stop
sudo service elasticsearch stop
sudo rm -rf /var/lib/elasticsearch/*
sudo service elasticsearch start
sudo service wakari-server start

Upgrade the Server:

pushd /tmp
wget http://j.mp/aen-server-update-4-1-2

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                 -p /opt/wakari/wakari-server          \
                 --file aen-server-update-4-1-2

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                 -p /opt/wakari/wakari-server          \
                 --no-deps                             \
                 wakari-enterprise-server-conf-update=2.0.3
popd

Start ElasticSearch:
```
sudo service elasticsearch start
```
Or, if you do not want to use the search features, edit your server’s config.json:
```
/opt/wakari/wakari-server/etc/wakari/config.json
```
adding the line item: "SEARCH_ENABLED": false.
Start the Server:
```
sudo service wakari-server start
```
Check that the Server is running properly:
```
sudo service wakari-server status
```
Finally, use your web browser to connect to AEN Server, using the correct protocol (http or https), hostname, and port number.

AEN Gateway¶

Repeat the following operations on each Gateway host.

Upgrade the Gateway:

pushd /tmp
wget http://j.mp/aen-gateway-update-4-1-2

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                 -p /opt/wakari/wakari-gateway         \
                 --file aen-gateway-update-4-1-2

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                 -p /opt/wakari/wakari-gateway         \
                 --no-deps                             \
                 wakari-enterprise-gateway-conf-update=2.0.3
popd

Start the Gateway:
```
sudo service wakari-gateway start
```
Check that the Gateway is running properly:
```
sudo service wakari-gateway status
```
Finally, use your web browser to connect to the Gateway, using the correct protocol (http or https), hostname, and port number.

AEN Compute¶

Repeat the following operations on each host where an AEN Compute service is running.

The processes killed with killall are run by the $AEN_SRVC_ACCT user, so they can be killed as root with sudo killall or killed as the $AEN_SRVC_ACCT user with sudo -u $AEN_SRVC_ACCT killall. These example commands show the sudo killall option.

Check for any wakari-indexer processes running:
```
ps aux | grep wakari-indexer
```
NOTE: If you stopped all the projects, you will not see any wakari-indexer processes running.
Terminate any remaining wakari-indexer processes:
```
sudo killall wakari-indexer
```
Check for any AEN apps (gateone, workbench, viewer, terminal, notebook) processes running:
```
ps aux | grep wk-app-gateone
ps aux | grep wk-app-workbench
ps aux | grep wk-app-viewer
ps aux | grep wk-app-terminal
ps aux | grep jupyter-notebook
```
NOTE: If you stopped all the projects, you will not see any AEN app processes running.

Terminate any remaining AEN apps processes that you find:

sudo killall wk-app-gateone
sudo killall wk-app-workbench
sudo killall wk-app-viewer
sudo killall wk-app-terminal
sudo killall jupyter-notebook

Verify the contents of /opt/wakari/anaconda/.condarc. Modify it to contain the following entries (and possibly others if you customized the .condarc file). Be sure to remove the auto_update_conda: false because we are going to upgrade conda in the following steps.

NOTE: Modify the file as the AEN_SRVC_ACCT user (or be sure to keep the same ownership).
```
channels:
  - https://conda.anaconda.org/t/<TOKEN>/anaconda-nb-extensions
  - r
  - https://conda.anaconda.org/wakari
  - defaults

create_default_packages:
  - anaconda-client
  - ipykernel
```
NOTE: Contact your enterprise support representative to get your <TOKEN> for the Anaconda channel referenced above.

NOTE: Replace <TOKEN> with the actual token from your enterprise support representative.
Upgrade conda in the root environment:
```
sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                -p /opt/wakari/anaconda conda=4.3.14
```
After the successful conda upgrade, modify the /opt/wakari/anaconda/.condarc file and re-add the line auto_update_conda: false removed in the previous step.

Upgrade Anaconda in the root environment:

pushd /tmp
wget http://j.mp/aen-anaconda-update-4-1-2

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                -p /opt/wakari/anaconda              \
                --file aen-anaconda-update-4-1-2

popd

Initialize the root environment to prime the package cache:

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                -p /opt/wakari/testenv \
                --clone root

and test the offline cloning step with:

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                -p /opt/wakari/testenvoffline \
                --clone root --offline

finally remove those test environments with:

sudo rm -rf /opt/wakari/testenv
sudo rm -rf /opt/wakari/testenvoffline

Upgrade each Compute service:

pushd /tmp
wget http://j.mp/aen-compute-update-4-1-2

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                 -p /opt/wakari/wakari-compute        \
                 --file aen-compute-update-4-1-2

sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                 --no-deps                            \
                 -p /opt/wakari/wakari-compute        \
                 wakari-enterprise-compute-conf-update=2.0.4
popd

NOTE: From AEN 4.1.0 and above, wakari-app-gateone is deprecated and no longer supported.

NOTE: When upgrading the wakari-compute environment you may see ImportError warnings with some nbextensions. As long as the Validating message is OK, the ImportError warnings are harmless (just a consequence of the post-link presence on those packages).

Run the upgrade_412.sh script to make your 4.1.1 projects compatible with the 4.1.2 codebase:

pushd /tmp
wget -O upgrade_412.sh http://j.mp/upgrade_412
chmod a+x upgrade_412.sh
./upgrade_412.sh
popd

Install necessary dependencies:

Note: Skip this step if you already have these dependencies installed from previous installations.
```
sudo yum groupinstall "X Window System" -y
sudo yum install git -y
```
Note: If you don’t want to install the whole X Window System, you need to install, at least, the following packages to have R plotting support:
```
sudo yum install -y libXrender libXext libXdmcp libSM libICE libXt \
dejavu-sans-fonts dejavu-serif-fonts dejavu-fonts-common \
fontpackages-filesystem
```
Start the Compute service:
```
sudo service wakari-compute start
```
Verify the Compute service is running properly:
```
sudo service wakari-compute status
```
Repeat the upgrade procedure for all the Compute Nodes in your Data Center.

Post-Flight Check¶

You can now start the projects and start using AEN apps.
If you have a customized default environment, you may choose to upgrade it depending on the needs of your users.

NOTE: Upgrading the default environment at /opt/wakari/anaconda/envs/default does NOT automatically upgrade the default environment in the users pre-existing projects. For pre-existing projects, the upgrade (if it is requested) should be done on a per-user basis.

If you did not stop all your projects before upgrading, then the first time you start an app you will see an error page requesting you to restart the app. Restart the app to complete the upgrade.
If you still see old apps or the old IPython logo in the notebook app, reload the page to reset the browser cache.