Updating Anaconda Enterprise Notebooks (AEN 4.0)¶
Summary¶
These instructions are for upgrading Anaconda Enterprise Notebooks Platform 0.10.0 or later. For help updating earlier versions, please contact your enterprise support representative.
NOTE: The Named Service Account functionality of Anaconda Enterprise Notebooks 4.0 is available only via a full installation of Anaconda Enterprise Notebooks. The upgrade process does not add this new functionality. For full installation support, please contact your enterprise support representative.
An update to the Anaconda Enterprise Notebooks Platform requires all instances of the three service categories to be updated individually:
- AEN Server
- AEN Gateway
- AEN Compute
A typical installation of Anaconda Enterprise Notebooks likely consists of several Compute instances running on independent Compute Nodes, and possibly multiple Gateways if there is more than one Data Center tied into Anaconda Enterprise Notebooks. More details can be found in the Overview.
The update process requires that all of Anaconda Enterprise Notebooks service instances be stopped, updated, and then restarted. These instructions describe how to perform this process.
Note: Any commands that call for the root
user can also be done
using sudo
.
If you encounter any difficulty during the process, please refer to the Troubleshooting Guide which provides guidance on:
- processes
- configuration files
- log files
- ports
In the event you are unable to resolve an installation or upgrade problem please contact your enterprise support representative.
Pre-Flight Check¶
CAUTION: Make a tested backup of your installation before starting the upgrade. Upgrading to a higher version of AEN is not reversible. Any errors during the upgrade procedure may result in partial or complete data loss and require restoring data from backups.
Prior to updating each service on each host, you should perform the following operations:
As the
root
user, check for pre-existing.condarc
configuration files:ls ~root/.condarc ls ~wakari/.condarc
If they exist, inspect these files for any special configuration not related to Anaconda Enterprise Notebooks. These settings need to be transferred to
.condarc
in the root environment (/opt/wakari/miniconda/.condarc
or/opt/wakari/anaconda/.condarc
)If there is an anaconda.org token string (similar to a UUID) in one of channel entries embedded in the path or URL, make a note of it.
Once this has been completed, remove the following files:
~root/.condarc
~wakari/.condarc
The Anaconda (or Miniconda) install’s conda configuration file
.condarc
needs to be verified. This file is in the top level directory of the Anaconda or Miniconda installation. It should exist and have at least the following entries (and possibly others):channels: - https://conda.anaconda.org/wakari - defaults
The
.condarc
file should not have any other references to specific versions of Anaconda Enterprise Notebooks. If found, remove such entries from the.condarc
file.Suspend the services on each of the nodes with the appropriate command. For Anaconda Enterprise Notebooks versions 0.10.0 and later:
sudo service wakari-server stop sudo service wakari-gateway stop sudo service wakari-compute stop
Versions of Anaconda Enterprise Notebooks prior to 0.10.0 used
crontab
entries to restart the services after a reboot. These have been replaced by init scripts. Remove thecrontab
entries from all nodes by runningcrontab -e
and removing the@reboot
lines.
AEN Server¶
NOTE: If you are using LDAP-based authentication, you need to backup the configuration file:
/opt/wakari/wakari-server/etc/wakari/wk-server-config.json
And then after the Server has been updated, copy it back in place. The current update process will overwrite this file. This is a known issue scheduled to be resolved in an upcoming release.
The following operations are all done on the Server host.
To take advantage of the new search capabilities, install ElasticSearch and a Java Runtime Environment (JRE 8).
Stop the ElasticSearch service:
service elasticsearch stop
and remove any previous index (if exist) with:
rm -rf /var/lib/elasticsearch/*
Ensure that Anaconda Enterprise Notebooks administrative account
wakari
owns all files in the Server installation, by running the following asroot
chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/miniconda /opt/wakari/wakari-server
Update the Server:
pushd /tmp wget http://j.mp/aen-server-update-4_0_0 sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \ -p /opt/wakari/wakari-server \ --file aen-server-update-4_0_0 sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \ -p /opt/wakari/wakari-server \ --no-deps \ wakari-enterprise-server-conf-update=2.0.0 popd
Fix file ownership and permissions, by running the following as
root
chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/miniconda /opt/wakari/wakari-server chmod -R a+rX /opt/wakari/miniconda chmod -R o-w /opt/wakari/miniconda chmod -R go+rX /opt/wakari/wakari-server chmod -R o-rwx /opt/wakari/wakari-server chmod o+rX /opt/wakari/wakari-server chmod -R o+rX /opt/wakari/wakari-server/share
Start ElasticSearch:
service elasticsearch start
Or, if you do not want to use the search features, edit your server’s
config.json
:/opt/wakari/wakari-server/etc/wakari/config.json
adding the line item:
"SEARCH_ENABLED": false
.Start the Server:
service wakari-server start
Check that the Server is running properly:
service wakari-server status
Finally, use your web browser to connect to AEN Server, using the correct protocol (http or https), hostname, and port number.
AEN Gateway¶
The following operations are all done on each Gateway host.
Ensure that Anaconda Enterprise Notebooks administrative account
wakari
owns all files in the Gateway installation, by running the following asroot
chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/miniconda /opt/wakari/wakari-gateway
Update the Gateway:
pushd /tmp wget http://j.mp/aen-gateway-update-4_0_0 sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \ -p /opt/wakari/wakari-gateway \ --file aen-gateway-update-4_0_0 sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \ -p /opt/wakari/wakari-gateway \ --no-deps \ wakari-enterprise-gateway-conf-update=2.0.0 popd
Fix file ownership and permissions, by running the following as
root
chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/miniconda /opt/wakari/wakari-gateway chmod -R a+rX /opt/wakari/miniconda chmod -R o-w /opt/wakari/miniconda chmod -R go+rX /opt/wakari/wakari-gateway chmod -R o-rwx /opt/wakari/wakari-gateway
Start the Gateway:
service wakari-gateway start
Finally, use your web browser to connect to the Gateway, using the correct protocol (http or https), hostname, and port number.
AEN Compute¶
The following operations are all done on each host where an AEN Compute service is running.
Check if there is any
wakari-indexer
processes running with:ps aux | grep indexer
And terminate any remaining
wakari-indexer
processes, if you find them, with:killall wakari-indexer
Ensure that the AEN administrative account
wakari
owns all files in the compute installation, by running the following asroot
:chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/anaconda chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/wakari-compute/{bin,conda-meta,config,etc} chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/wakari-compute/{include,lib,lib64,pkgs,share,ssl}
Verify the contents of
/opt/wakari/anaconda/.condarc
, as described above. In addition, it should contain the following entries (and possibly others):channels: - https://conda.anaconda.org/wakari - https://conda.anaconda.org/t/<TOKEN>/anaconda-nb-extensions - r - defaults create_default_packages: - python - ipython-we - pip
NOTE: Contact our sales representatives to get your <TOKEN>
for
the Anaconda channel referenced above.
Install the
R
dependencies:yum groupinstall "X Window System" -y
Note: The
groupinstall
should be enough to cover all the R needs but in case you have problems with R-based plotting in the notebook, please also run:yum install libXrender libXext libSM -y
Update the Compute service:
sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \ -p /opt/wakari/anaconda/ \ conda=3.19.3 conda-env=2.4.5 pushd /tmp wget http://j.mp/aen-compute-update-4_0_0 wget http://j.mp/aen-anaconda-update-4_0_0 sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \ -p /opt/wakari/wakari-compute \ --file aen-compute-update-4_0_0 sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \ --no-deps \ -p /opt/wakari/wakari-compute \ wakari-enterprise-compute-conf-update=2.0.0 sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \ -p /opt/wakari/anaconda \ --file aen-anaconda-update-4_0_0 popd
Fix file ownership and permissions, by running the following as
root
chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/anaconda chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/wakari-compute/{bin,conda-meta,config,etc} chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/wakari-compute/{include,lib,lib64,pkgs,share,ssl} chmod -R a+rX /opt/wakari/anaconda chmod -R o-w /opt/wakari/anaconda
Initialize the root environment to prime the package cache
sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \ -p /opt/wakari/testenv \ --clone root rm -rf /opt/wakari/testenv
Pin conda to version 3.19.3 (and compatible conda-env) to avoid automatic self-updates on the anaconda environment
sudo echo -e "conda ==3.19.3\nconda-env ==2.4.5" > \ /opt/wakari/anaconda/conda-meta/pinned
With the latest Anaconda Enterprise Notebooks release (4.0.0) we have introduced new features such as Functional IDs and no root access during normal operations. You must execute the
upgrade\_400.sh
script to make your projects compatible with the new codebase:pushd /tmp wget http://j.mp/upgrade_400 chmod a+x upgrade_400.sh ./upgrade_400.sh popd
Start the Compute service:
service wakari-compute start
Verify the Compute service is running properly:
ps auxef | grep bin/wk-compute service wakari-compute status
Versions of Anaconda Enterprise Notebooks prior to 0.10.0 used ipython version 2.2.0. If you detect that version with
conda list
, you should:sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda remove ipython sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install ipython-we --force sudo -u $AEN_SRVC_ACCT /opt/wakari/wakari-compute/bin/conda remove ipython sudo -u $AEN_SRVC_ACCT /opt/wakari/wakari-compute/bin/conda install ipython-we --force
NOTE: Existing AEN Projects may have their own Conda environments. If so, those environments are not automatically updated by this process. In this case, updates can be done by the users themselves with the following commands:
conda remove ipython conda install ipython-we --force
Alternatively, an administrator can update the users conda environments, with the following commands:
# Update the conda environments for the user $USERNAME working in AEN Project $PROJECTNAME source activate /projects/$USERNAME/$PROJECTNAME/envs/default sudo -u $USERNAME conda remove ipython sudo -u $USERNAME conda install ipython --force source deactivate
If you need
R
in a pre-existing project, use the following command to install the neededR
packages:source activate /projects/$USERNAME/$PROJECTNAME/envs/default sudo -u $USERNAME conda install r-base r-essentials -c r source deactivate
NOTE: In order to start using the new jupyter extensions you’ll need to remove the
.jupyter_$USERNAME
and.ipython_$USERNAME
folders.If you have a customized default environment, you could update it depending of your users needs.
NOTE: Updating the default
environment at
/opt/wakari/anaconda/envs/default
do not automatically update the
default
environment in the users pre-existing projects. For
pre-existing projects the update (if it is rquested) should be done on a
per-user basis.
Project Permissions¶
Since version 0.10.0 Anaconda Enterprise Notebook uses POSIX.1e access control lists to manage shared content, rather than plain POSIX user groups. If you are upgrading from an AEN version earlier than 0.5.0, please execute the project_permissions.py script on every Compute node:
/opt/wakari/wakari-compute/bin/python project_permissions.py /projects
NOTE: In some deployments, the project directory may be different
from /projects
.
Then, on the AEN Server host, run repair_team_members.py:
/opt/wakari/wakari-server/bin/python repair_team_members.py
These scripts will read all the Project and Team information from the AEN database, and re-add individuals to all the Project on all Compute nodes.