Mirroring an Anaconda repository with Anaconda Enterprise 4 Repository#

Before you start#

You need to have already installed and configured your Repository instance. Due to the size of Repository, it is important that you have configured a file storage location with sufficient disk space. If necessary, see the requirements for the file storage location.

A full Anaconda mirror requires approximately 650 GB.

You will also need to install cas-mirror as it is the recommended mirroring tool.

NOTE: The anaconda-mirror tool has been deprecated and will not be updated any further.

Mirroring all packages#

You can mirror some or all of the contents of the Anaconda repository using the cas-sync-api-v4 command:

$ cas-sync-api-v4 --help
usage: cas-sync-api-v4 [-h] [-f FILENAME] [--config] [--version]
                       [-l LOG_LEVEL] [-v]

Updates an Anaconda repository instance

optional arguments:
  -h, --help            show this help message and exit
  -f FILENAME, --file FILENAME
                        Configuration file location (Defaults to:
                        /home/abarto/.cas-mirror or /etc/cas-mirror)
  --config, --show-config
                        Show running configuration and exit
  --version             Print version and exit
  -l LOG_LEVEL, --log-level LOG_LEVEL
                        Set the log level (CRITICAL, ERROR, WARNING, INFO,
                        DEBUG)Default: INFO.
  -v, --verbose         Shorthand for --log-level DEBUG

The cas-sync-api-v4 command leverages the functionality exposed by the anaconda-client package to import the remote packages into an existing site. It mirrors all of the packages from the default repo.anaconda.com channels into the anaconda user account. These default channels are:

  1. Associate the URL of your Anaconda Enterprise 4 Repository with a site using the anaconda command:

    anaconda config --set sites.myrepo.url "http://your-anaconda-repo/"

    NOTE: Replace your-anaconda-repo with the URL to your installation of Repository.

  2. Create a configuration file sync.yaml that tells cas-sync-api-v4 which site to use:

    dest_site: myrepo

  3. Check that the configuration is valid with the --config parameter:

    $ cas-sync-api-v4 -f sync.yaml --config
path: /home/ec2-user/sync.yaml
remote_url: https://repo.anaconda.com/
mirror_dir: /opt/cas-mirror
platforms: ['osx-32', 'osx-64', 'win-32', 'win-64', 'linux-32', 'linux-64', 'linux-armv6l', 'linux-armv7l', 'linux-ppc64le']
fetch_installers: True
repodata_source: False
dir_names: ['archive']
server_log_dir: None
server_port: None
dest_site: myrepo
dest_channel: anaconda
verify_checksum: False
delta: False
delta_dir: None
log_dir: None
log_level: 20
python_versions: []
pkg_list: []
license_blacklist: []
blacklist: []
whitelist: []
channels:
    - https://repo.anaconda.com/pkgs/main/
    - https://repo.anaconda.com/pkgs/free/
    - https://repo.anaconda.com/pkgs/pro/

    With this configuration, cas-sync-api-v4 mirrors the contents of all the default channels into the anaconda account of the myrepo site.

  4. Run cas-sync-api-v4:

    cas-sync-api-v4 -f sync.yaml

  5. Verify the mirroring by opening a browser and loading this URL:

    http://your-anaconda-repo/anaconda/

    NOTE: Replace your-anaconda-repo with the URL to your installation of Repository.

Mirroring some packages#

Alternately, you may not want to mirror all packages. To mirror a subset of the total repository, specify which platforms you want to include, or use the whitelist, blacklist or license_blacklist functionality to control which packages are mirrored, by copying the default configuration file $PREFIX/etc/anaconda-server/mirror/anaconda.yaml to $PREFIX/etc/anaconda-server/mirror/anaconda-custom.yaml.

For Repository 2.27 or newer, there are sample yaml config files located at: $PREFIX/etc/anaconda-server/mirror. PREFIX is the install location of Repository, which by default is ~anaconda-server/repo/etc/anaconda-server/mirror.

This command mirrors the repository according to the settings in the configuration file anaconda-custom.yaml:

cas-sync-api-v4 -f anaconda-custom.yaml

For more information, see Customizing mirrors.

Offline mirroring#

Offline mirroring is done by using both cas-sync and cas-sync-api-v4. First download all the packages onto a host with Internet access.

EXAMPLE:

To download the packages, create a configuration file named export.yaml:

mirror_dir: /opt/mirror/export/
platforms:
  - linux-64
  - win-64
python_versions:
  - 2.7
  - 3.6
fetch_installers: false
pkg_list:
  - ca-certificates
  - certifi
  - libedit
  - libffi
  - libgcc-ng
  - libstdcxx-ng
  - ncurses
  - openssl
  - pip
  - python
  - readline
  - setuptools
  - sqlite
  - tk
  - wheel
  - xz
  - zlib

This example downloads only a subset of the default channels.

Run cas-sync:

cas-sync -f export.yaml

After it finishes, a conda repository for each platform is created in the directory /opt/mirror/export/pkgs. We can now take the contents of the directory /opt/mirror/export/ to the air-gapped environment.

To import the packages, create a config file named import.yaml:

dest_site: mysite
dest_channel: anaconda
channels:
  - file:///opt/mirror/export/pkgs/
platforms:
  - linux-64
  - win-64
python_versions:
  - 2.7
  - 3.6

Make sure these requirements are true:

  • The files mirrored onto the connected box have been put in the /opt/mirror/export directory.

  • An anaconda site named mysite is properly configured and the user has logged into it using the anaconda command.

  • An “anaconda” account exists in the “mysite” site and the logged in user has access to it.

After these requirements are true, run cas-sync-api-v4:

cas-sync-api-v4 -f import.yaml

Mirroring with the deprecated anaconda-mirror command#

The anaconda-mirror command is deprecated but still in use at some installations.

It is used with the command anaconda-mirror sync, or with a configuration file such as anaconda-custom.yaml with the command anaconda-mirror --config-file anaconda-custom sync.

Resetting packages#

Use the --reset option to reset the previously mirrored packages:

anaconda-mirror --config-file anaconda-custom sync --reset

This resets the “last sync” time for the repository, so anaconda-mirror requests all packages, not just those changed or added since the last sync. As the requests are processed, anaconda-mirror still automatically downloads only those files that differ from the files currently in the repository.

Exporting a mirror#

To generate a mirror archive:

anaconda-mirror export mirror.tar

This command dumps the packages, according to the configured settings, into the file mirror.tar.

This mirror can be used in an air gapped environment.

Importing a mirror#

To mirror the Anaconda repository in an air gapped environment, point anaconda-mirror to the exported mirror archive.

Mount the USB drive and then run:

anaconda-mirror import $USB/mirror.tar

This command mirrors the contents of the local Anaconda repository to your Anaconda Enterprise 4 Repository installation under the username “anaconda.”

Filtering#

If you want to update the filters on your mirror—for example, to exclude additional licenses—running anaconda-mirror sync again retrieves new packages that match this filter, but it does not remove existing packages that no longer match the filter.

To see which packages no longer match your filter:

anaconda-mirror clean --dry-run

To remove these packages from your mirror:

anaconda-mirror clean

Mirroring additional channels#

If mirroring from an air gap archive, the channel in the following configuration points to a local directory to which the archive is expanded.

In addition, if a platform-specific archive is downloaded, then the config file needs the platforms section. The examples in the following sections assume x64-repo-mirrors-\*.tar <airgap-archive-mirrors> is expanded to $MIRRORS_ARCHIVE.

Similarly, for an online system, the channel points to Anaconda.org. The platforms are optional and limit the mirrored conda packages to the specified platforms.

Mirroring R channel#

  1. Create the yaml config file.

    EXAMPLE: The following is a config to mirror from an air gap archive containing only x64 packages:

    cat $PREFIX/etc/anaconda-server/mirror/r.yaml

channels:
  - file://$MIRRORS_ARCHIVE/r/pkgs

# The platforms should correspond to the platforms contained in
# the archive. Omit if the archive contains conda packages for all platforms.
platforms:
  - linux-64
  - osx-64
  - win-64

    EXAMPLE: The following is for an online system:

    cat $PREFIX/etc/anaconda-server/mirror/r.yaml

channels:
  - https://conda.anaconda.org/r

  2. Mirror the packages to r-channel:

    anaconda-server-sync-conda --mirror-config \
    $PREFIX/etc/anaconda-server/mirror/r.yaml --account=r-channel

Mirroring Wakari channel for AEN#

  1. Create the yaml config file.

    EXAMPLE: The following is a config to mirror from an air gap archive containing only x64 packages:

    cat $PREFIX/etc/anaconda-server/mirror/wakari.yaml

channels:
  - file://$MIRRORS_ARCHIVE/wakari/pkgs

# The platforms should correspond with the platforms contained in
# the archive. Omit if the archive contains conda packages for all platforms.
platforms:
  - linux-64
  - osx-64
  - win-64

    EXAMPLE: The following is for an online system:

    cat $PREFIX/etc/anaconda-server/mirror/wakari.yaml

channels:
  - https://conda.anaconda.org/t/<TOKEN>/anaconda-nb-extensions
  - https://conda.anaconda.org/wakari

    NOTE: Replace <TOKEN> with the token for the anaconda-nb-extensions channel that you should have received along with your Repository license.

  2. Mirror the packages to the Wakari channel:

    anaconda-server-sync-conda --mirror-config \
    $PREFIX/etc/anaconda-server/mirror/wakari.yaml --account=wakari

Mirroring anaconda-adam channel for cluster management#

  1. Create the yaml config file.

    EXAMPLE: The following is a config to mirror from an air gap archive containing only x64 packages:

    cat $PREFIX/etc/anaconda-server/mirror/anaconda-adam.yaml

channels:
  - file://$MIRRORS_ARCHIVE/anaconda-adam/pkgs

# The platforms should correspond with the platforms contained in
# the archive. Omit if the archive contains conda packages for all
# platforms.
platforms:
      * linux-64
      * osx-64
      * win-64

    EXAMPLE: The following is for an online system:

    cat $PREFIX/etc/anaconda-server/mirror/anaconda-adam.yaml

channels:
  - https://conda.anaconda.org/anaconda-adam

  2. Mirror the packages to anaconda-adam channel:

    anaconda-server-sync-conda --mirror-config \
    $PREFIX/etc/anaconda-server/mirror/anaconda-adam.yaml --account=anaconda-adam

Configuring conda#

Having created the mirror, you still need to configure conda to search for packages here rather than on the default Anaconda repository. You can do that by editing your ~/.condarc file to add the appropriate channel:

channels:
    - http://<anaconda.repo.ipaddress>:<port>/conda/anaconda/

NOTE: Replace <anaconda.repo.ipaddress> with the URL to your installation of Repository.

NOTE: This configuration change can be made at the user level or via an administrative conda file, to force all internal users to use your local Anaconda mirror rather than querying the Anaconda repository.

NOTE: Users can download Anaconda installers that are pre-configured to search your Repository from http://<anaconda.repository.addr>/downloads. To learn how to generate these installers, see Customizing installers.