Mirroring a PyPI repository#

Before you start#

You need to have already installed and configured your Repository instance. Due to the size of Repository, it is important that you have configured a file storage location with sufficient disk space. If necessary, see the requirements for the file storage location.

The full PyPI mirror requires approximately 120 GB.

Running the PyPI mirror command#

To create a PyPI mirror:

anaconda-server-sync-pypi

This command loads all of the packages on pypi.python.org into the ~pypi binstar user account.

Verify that the command ran successfully by opening your browser to http://your-anaconda-repo/pypi/~pypi.

NOTE: Replace your-anaconda-repo with the URL to your installation of Repository.

Customizing the mirror#

It is possible to customize the mirror behavior by creating a configuration file such as $PREFIX/etc/anaconda-server/mirror/pypi.yaml and using the --mirror-config option:

anaconda-server-sync-pypi --mirror-config /etc/binstar/mirrors/pypi.yaml

The following configuration options are available:

Name

Description

user

The local user under which the PyPI packages are imported. Default: pypi.

pkg_list

A list of packages to mirror. Only packages listed are mirrored. If this is set, blacklist and whitelist settings are ignored. Default: [].

whitelist

A list of packages to mirror. Only packages listed are mirrored. If the list is empty, all packages are checked. Default: [].

blacklist

A list of packages to skip. The packages listed are ignored. Default: [].

latest_only

Only download the latest versions of the packages. Default: false.

remote_url

The URL of the PyPI mirror. /pypi is appended to build the XML RPC API URL, /simple for the simple index and /pypi/{package}/{version}/json for the JSON API. Default: https://pypi.python.org/.

xml_rpc_api_url

A custom value for XML RPC URL. If this value is present, it takes precedence over the URL built using remote_url. Default: null.

simple_index_url

A custom value for the simple index URL. If this value is present, it takes precedence over the URL built using remote_url. Default: null.

use_xml_rpc

Whether to use the XML RPC API as specified by PEP381. If this is set to true, the XML RPC API is used to determine which packages to check. Otherwise the scripts falls back to the simple index. If the XML RPC fails, the simple index is used. Default: true.

use_serial

Whether to use the serial number provided by the XML RPC API. Only packages updated since the last serial saved are checked. If this is set to false, all PyPI packages are checked for updates. Default: true.

create_org

Create the mirror user as an organization instead of a regular user account. All superusers are added to the “Owners” group of the organization. Default: false.

private

Save the mirrored packages as private. Default: false.

EXAMPLE:

whitelist:
  - requests
  - six
  - numpy
  - simplejson
latest_only: true
remote_url: http://pypimirror.local/
use_xml_rpc: true

Configuring pip#

To configure pip to use this new mirror, edit /etc/pip.conf as follows:

[global]
index-url=https://pypi.anaconda.org/pypi/simple
On this page