Improved Python Support

Wercker is a Docker-Native CI/CD Automation platform for Kubernetes & Microservice Deployments

Jacco Flenter
Jacco Flenter
November 8, 2013

A few days ago there was a support request over at the #wercker IRC channel, requesting support for wheels. The python stack on wercker could use some brush up so, yesterday we released some new and upgraded boxes and steps.

New menu bar

Let’s summarize first what has changed on the python-box:

  • setuptools, pip and virtualenv are upgraded to the latest versions.
  • wheel is globally installed for all python versions.
  • python 3.2 is installed.
  • the default python wercker.yml now uses virtualenv.

What is wheel?

Wheel is relatively new and basically is designed to contain all the files for an install in a way that is very close to the on-disk format. Making the install process quick and painless. In more detail, wheel is (According to the documentation):

A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension. It is designed to contain all the files for a PEP 376 compatible install in a way that is very close to the on-disk format. Many packages will be properly installed with only the “Unpack” step (simply extracting the file onto sys.path), and the unpacked archive preserves enough information to “Spread” (copy data and scripts to their final locations) at any later time.

The end result is that some packages can benefit greatly from being installed from wheel. This is particularly true for packages which have C bindings and take a long time to build/compile (such as numpy). It doesn’t help the first time packages are being installed, in fact, it may slow things down a bit. Subsequent builds however can see significant decreases in build time.

How to use

Let’s look at the new default wercker.yml for python:

box: wercker/python
# Build definition
build:
  # The steps that will be executed on build
  steps:
    # A step that sets up the python virtual environment
    - virtualenv:
        name: setup virtual environment

    # # Use this virtualenv step for python 3.2
    # - virtualenv
    #     name: setup virtual environment
    #     python_location: /usr/bin/python3.2

    # A step that executes `pip install` command.
    - pip-install

    # This pip-install clears the local wheel cache
    # - pip-install:
    #     clean_wheel_dir: true

    # A custom script step, name value is used in the UI
    # and the code value contains the command that get executed
    - script:
        name: echo python information
        code: |
          echo "python version $(python --version) running"
          echo "pip version $(pip --version) running"

A lot of comments, but basically there are 3 steps:

  1. virtualenv
  2. pip install
  3. a custom step, which displays some basic information.

 The virtualenv step

Virtualenv is something, we hope, everybody is familiar with (and uses). In short it allows a person to create a separate python environment where you can install packages. This way you can have a different version of the same package in each environment, it also allows you to install packages even when you have superuser privileges. Both points are probably not that important on the wercker platform, since you have superuser privileges and are probably testing only one application. It is however a good practice to test in an environment that is as close to production as possible and that’s where virtual environments make more sense.

The virtualenv step on wercker does however, more than just setting up and activating the virtual environment. It also configures some environmnent variables, which allows pip to use a local cache (which is stored for 14 days and shared between builds/deploys) and installs the wheel package.

The pip-install step

The pip install step, by default, tries to install all packages from therequirements.txt file in the root of your project (and will fail if it is not found). You can however configure it to install just packages, not use requirements.txt or different file.

Because we want to speed up our subsequent builds with wheels, the pip-install step actually performs two actions:

  1. if wheels are enabled via environment variables, run pip wheel.
  2. run pip install (with or without sudo, depending on whether we use a virtual environment or not).

This is the main reason to use our step, and not just run pip install via a custom script step.

The benefits

As a test I added numpy to my requirements.txt and looked at the pip install time:

django==1.6.0
gunicorn
dj-database-url
psycopg2
geopy
numpy

It went from 3 minutes 14 seconds without wheel (but still using the download cache) to 24 seconds with wheel enabled (the wheel files were already build and in the cache).

Summary

Although technically everybody could already use wheel, by upgrading the pip library, setuptools etc. Wercker now suggests for python a better way of running and testing (via the virtual environment) and allows people to enjoy the advantages of wheel without the need to know in details what it is exactly.

Have fun!

Earn some stickers!

Let us know about the applications you build with wercker. Don’t forget to tweet out a screenshot of your first green build with #wercker and we’ll send you some @wercker stickers.

Signing up for wercker is free and easy.