Homebrew and Python

Python is a powerful and beautiful interpreted language.

Overview

This page describes how Python is handled in Homebrew and how to write formulae that install Python bindings.

Homebrew should work with any CPython, in particular with OS X system's Python. Further, Homebrew provides formulae to brew a more up-to-date Python 2.7.x. (and 3.x). However, if you choose to use another Python than these two alternatives (system Python or brewed Python), the Homebrew team can only provide limited support.

Brew Python

We recommend to brew install python because:

• It is newer
• Comes with pip (and distribute)
• Uses readline (better than libedit)
• Sqlite with loadable extensions
• Use newer OpenSSL (optionally)
• Use newer Tcl/Tk (optionally)
• Python (distutils) finds brewed software (includes, libs), knows about the compiler and flags even if the command line tools for Xcode are not installed.
• No need to set the PYTHONPATH for Homebrew bindings.
• No need to work-around the sudo-is-needed-for-easy_install issue, described there --> Gems, Eggs and Perl Modules.

Python 2.x or Python 3.x

Homebrew provides a formula for Python 2.7.x and one for Python 3.x. They don't conflict, so they can both be installed. The executable python will always point to the 2.x and python3 to the 3.x version. But which version should I use?

Since June 2013, brew supports to build bindings for both major Python versions, 2.x and 3.x, however each formula has to be adapted and tested (some project just don't yet provide 3.x support). As of June 2013, formulae --with-python3 support are Sip, PyQt, Shiboken, PySide and Swig. More to come. Note, Python 2.x bindings are installed, too, so you get both, unless you say --without-python.

Your help on adding 3.x support for formulae is very welcome!

Distribute, Pip, etc.

The Python formula installs Pip and Distribute, the former also provides easy_install (not recommended because it doesn't support uninstall).

Distribute can be updated via Pip, without having to re-brew Python:

pip install --upgrade distribute

Similarly, Pip can be used to upgrade itself via:

pip install --upgrade pip

Note, pip install --user is disabled for brewed Python. This is a bug in distutils, because Homebrew write a distutils.cfg that sets the prefix for packages.

The site-packages and the PYTHONPATH.

The site-packages is a directory to contain Python modules, especially bindings installed by other formulae. Homebrew creates such a directory at $(brew --prefix)/lib/pythonX.Y/site-packages for example /usr/local/lib/python2.7/site-packages for python2.7. The reasoning is that for (minor) upgrades or reinstalls of Python, your modules are not lost. And a rather strict Homebrew policy is not to write stuff outside of the brew --prefix, so we don't spam your system.

A brewed Python 2.7 also searches for modules in

• /Library/Python/2.7/site-packages and in
• python -c "import site; print(site.USER_SITE)" => /Users/<your_name>/Library/Python/2.7/lib/python/site-packages

Homebrew's site-packages directory is first created if any Homebrew formula with Python bindings is installed or if brew install python. A brewed Python already knows about this dir.
For other Pythons you'll have to add that dir to your PYTHONPATH environment variable. You may want to append a line like so to your hidden configuration file .bash_profile in your home dir:

This is not needed if you use a brewed Python!

touch ~/.bash_profile
echo export PYTHONPATH=\\"$(brew --prefix)/lib/python2.7/site-packages:\$PYTHONPATH\\" >> ~/.bash_profile
source ~/.bash_profile
echo $PYTHONPATH

Homebrew provided Python bindings

Some formulae provide python bindings. Sometimes a --with-python or --with-python3 option has to be passed to brew install in order to build the python bindings. Check with brew options <formula>.
If you have a brewed python, then the bindings are installed for that one. But if you don't have a brewed Python, Homebrew basically just uses the first python (and python-config) in your PATH. Check that by which python.

Warning, Python may crash (see Common Issues) if you import <module> in a different python interpreter than the executable that was used during the brew install <formula_with_python_bindings>. Therefore, if you decide to switch between a brewed and system python, then (re-‍)install all formulae that provide python bindings (such as pyside, wxwidgets, pygtk, pygobject, opencv, vtk, boost to name just a few).

Other - non-homebrew - Python bindings

Our policy is that these should be installed via pip install <x>. To discover, you can use pip search, the new http://crate.io or http://pypi.python.org/pypi. Note, system Python does not provide pip but you can easy_install pip to fix that.

For a brewed Python, modules installed by pip or with python setup.py, will be put into the before-mentioned $(brew --prefix)/lib/pythonX.Y/site-packages directory, too. Executable python scripts will be in $(brew --prefix)/bin. (To behave more standard conform, brew no longer puts Python scripts into share/python/$(brew --prefix))

Further, pip (or more precisely distutils) knows which compiler flags to set in order to build bindings for software installed in Homebrew (Find the includes, the libs, the compiler and so forth). In contrast to that, system's Python does not know about the the correct flags, so you may need to

CFLAGS=-I$(brew --prefix)/include LDFLAGS=-L$(brew --prefix)/lib pip install <package>.

Virtualenv

A brewed Python works nicely with Virtualenv. However, when you brew install formulae that provide Python bindings, you should NOT be in an active virtual environment. Activate the virtualenv after you have brewed or, alternatively, brew in a fresh Terminal window. The Python interpreter from that virtual environment would be used by Homebrew but the Python modules would still be put into Homebrew's site-packages and not into the virtual environment's site-package. Virtualenv has a switch to allow "global",i.e. Homebrew's, site-packages to be accessible from within the virtualenv.

How to create a formula with nice Python bindings

Just add the special dependency to the formula:

depends_on :python

You may want to give the user the possibility to opt-out:

depends_on :python => :recommended

or opt-in (here for 3.x):

depends_on :python3 => :optional

Options, such as --with-python3 or --without-python (in the examples above) are automatically generated. In the formula you can check, as usual, via build.with? 'python'. To depend on a specific version, instead use depends_on PythonInstalled.new("2.7").

If the software provides a setup.py

Just calling system python, "setup.py", "--prefix=#{prefix}" should work.

We want to achieve that the Python bindings are installed into the Cellar for that specific formula ($(brew --prefix)/Cellar/<formula>/<version>/lib/python2.7/site-packages) and when brew link is run (automatically at the end of brew install), the Python modules should be linked into $(brew --prefix)/lib/pythonX.Y/site-packages and the scripts should go into $(brew --prefix)/bin. Only then can we unlink/link/uninstall cleanly.

And one of the most important features is the python-block which creates the python.site_packages dir and already sets the PYTHONPATH correctly. This block is executed once for each python requirement. So in the next snippet, first the block will be run for python 2.x and then the block will be run for 3.x:

  ...
  depends_on :python => :recommended
  depends_on :python3 => :optional
  ...
  def install
    ...
    python do
      # In order to install into the Cellar, the dir must exist and be in the
      # PYTHONPATH. This is already done inside this `python do` block.
      system python, "setup.py", "install", "--prefix=#{prefix}"
    end
  end

  def caveats
    python.standard_caveats if python
  end

If the setup.py is older, it may need two additional arguments to avoid writing an easy-install.pth file which will conflict with the easy-install.pth already installed by pip/distribute. So if you get a brew link problem mentioning this file, you'll need to add to setup.py args:

"--single-version-externally-managed", "--record=installed.txt"

If the formula uses cmake

python do
  # Cmake picks up system Python on OS X
  args << "-DPYTHON_INCLUDE_DIR='#{python.incdir}'"
  args << "-DPYTHON_LIBRARY='#{python.libdir}/lib#{python.xy}.dylib'"
  # Set the prefix for the python bindings to the Cellar
  args << "-DPYTHON_SETUP_ARGS:STRING='--prefix=#{prefix} --single-version-externally-managed --record=installed.txt'"
  args << ".."
  system 'cmake', *args
  system 'make'
  system 'make install'
end

You may want to have a look at the homebrew/science/vtk.rb formula.

If the formula uses configure/make

Mostly configure is able to find python or python-config and/or look at the PYTHON var (both are set up by Homebrew) and figure out how to properly install the bindings. Sometimes we have to patch a software to use our prefix for the python bindings. Often a --with-python or similar flag can be given to configure. Check with ./configure --help.

The python object contains support methods

In the Formula class there will be a python object available (python2 and python3, too), that supports some convenience methods which do the complicated work of figuring out the information no matter which python version or brewed vs. python from OS X:

python.site_packages # the site-packages in the Cellar
python.global_site_packages # homebrew's global one
python.binary # the full path to the python binary. For brewed python: HOMEBREW_PREFIX/opt/python/bin/python2
python.prefix
python.version
python.version.major
python.version.minor
python.xy # => e.g. "python2.7"
python.incdir # includes of python
python.libdir # the python dylib library
python.framework # The dir that contains the Python.framework
python.pkg_config_path # used internally by brew
python.from_osx?
python.framework?
python.universal?
python.pypy?
python.standard_caveats # The usual text to set PYTHONPATH that is only shown for python.from_osx?
python.if3then3 # => "" for 2.x and to "3" for 3.x.

Instead of using python, there is also python2 and python3 to explicitly refer to 2.x and 3.x resp..

Executing code only for 2.x or 3.x

if python2
  # do something specific for python 2.x
end

if python3
  # do something specific for python 3.x
end

unless python
  # do something if there is no python (or python support was disabled --without-python --without-python3)
end

Technical details for maintainers

A site-packes/sitecustomize.py is written in order to

• allow other non-brewed python to parse .pth files in our site-packages
• filter out dirs from the PYTHONPATH, starting with /System/...
• remove the hard coded site-package location in the Cellar (where we installed Python to) so that pip uninstall works.