Advanced: Extending Wercker - Part 1

Wercker is a Docker-Native CI/CD Automation platform for Kubernetes & Microservice Deployments

Jacco Flenter
Jacco Flenter
October 11, 2013

Wercker is an open platform and can be extended and enhanced by utilizing easily reusable steps and boxes.

You can find them in the wercker directory. What if you can’t find what you are looking for? Create your own. So let’s see how to create a step in this in-depth article.

wercker steps

Introduction

While creating the series on android development I found that I wanted wercker to set the version numbering for me. Each release should increment some counter. In this in-depth article, we will explore a solution using an external website (running on a free Heroku dyno) which can supply us with a build number. A working version of the step on wercker can be found here

The basics

What is a step? We envision steps as basic blocks of functionality which are contained and might be reused for different applications. Steps are executed in either the build or deploy pipeline, as explained on our devcenter. So instead of using the script step for everything, you can create a step for actions that are a bit more complex or that may use certain options/switches you otherwise have to copy/paste from that other project. Typically steps do one of the following:

Steps and how to create them are explained as well on our devcenter.

Our step will create an environment variable that we can use later during the build and update whichever file we need to for our specific language.

The pipeline

Whenever you start a build or a deploy there are a couple of basic steps that are set in stone in the wercker pipeline:

  • get code. During builds this is a git clone, during deploy the build output is retrieved.
  • setup environment. During this step the boxes are started. Based on the wercker.yml
  • environment variables. This step defines the default wercker environment variables as well as any defined in the deploy target or in settings / pipeline.
  • saving build output (builds only)

In our wercker.yml, there are two opportunities in the pipeline to insert our own logic, which are defined in the wercker yaml as:

  • steps. This is run right after environment variables
  • after-steps. They are run after a build/deploy is finished. Even when a build is not succesful.

The steps are executed in order and if one fails, subsequent steps are not executed. The after-steps are different, they basically run outside of the build flow and are meant for steps that need to be run, no matter the outcome of the build or deploy (or specifically when a build/deploy is failed). This is particularly useful for sending notifications.

The Wercker directory

There are a couple of things that are good to know about the wercker directory.

First of all, applications that will be deployed as steps or boxes have to be public. Deploys of steps and boxes to the wercker directory can not be removed at the moment. The reason for this is that users of wercker should be able to rely on both boxes and steps to be there and not have suddenly disappeared. You can alwayscontact us if you feel a step or box should be deleted by us. Finally: versioning is based around semantic versioning and it is a good idea to follow those specifications.

note: you can lock versions or specify versions of steps and boxes by adding @ for a specific version (i.e. bundle-install@0.9.1) or specify ranges (i.e. bundle-install>=0.9.2)

Helpers and definitions

As defined in the introduction of this article, we want to increment some counter and optionaly want to do this per branch. Wercker provides a lot of information by default that we can use in our steps. So let’s look at a couple of them.

VARIABLE NAME EXAMPLE VALUE PURPOSE
WERCKER_GIT_OWNER wercker The owner of the repository
WERCKER_GIT_REPOSITORY step-bundle-install The name of the repository
WERCKER_GIT_BRANCH master The branch name
WERCKER_GIT_COMMIT ef306b2479a7ecd433 7875b4d954a4c8fc18 e237 The commit hash
WERCKER_SOURCE_DIR $WERCKER_ROOT/src The path to the directory of the source code
WERCKER_CACHE_DIR /cache The path to the cache directory. This directory will be stored after the pipeline completes and restored when the pipeline runs again
WERCKER_STEP_ROOT /wercker/steps/wercker /bundle-install/0.9.1 The path to the working directory of the step that is currently executed. It contains the full content as deployed to the wercker directory
WERCKER_STEP_ID 9c182f44-e12d-4daf-91eb-a48d0540cc10 The unique idenfier for the step, unique for each build/deploy.
WERCKER_STEP_NAME bundle-install The name of the step as specified by the step inwercker-step.yml
WERCKER_REPORT_MESSAGE_FILE $WERCKER_REPORT_DIR/ $WERCKER_STEP_ID/ message.txt The location of a file you can use to output additonal information about the step.

We have all of the variables we need to determine our build number: the branch, the commit hash and the application name. Do we need an external server for getting this build number? Can we not create a database or simple text file and store it in the wercker cache? The answer to that, yes. It would however not behave exactly as we want. The biggest problem is, that the cache of a build is only stored on a succesful build. Treating the cache as persistant storage that we can access each build might also not be the best idea. Fortunately there’s a simple django application calledversioning_service which can help us get the correct build numbers.

Communicating with the service

The django application provides a RESTful interface which allows a user to query for a build number based on the application, branch name and commit hash. An example version is running on: buildnr.herokuapp.com

Back to our rest call, what does a user of the step need to do? The django app requires a user to register, so random people can’t increment your build numbers artificially. It also requires a user to create an application.

In detail we need to know a number of things in order to get our build number:

  • app id
  • the branch name
  • commit hash
  • user credentials: versioning_service application allows for a username and api_key to be specified for api calls.

The definition of a step

Defining a step is done in the wercker-step.yml file. There we can specify the name, version and parameters of the step:

name: generate-version
version: 0.0.1-alpha
description: Generate build number
keywords:
  - version
  - build number
properties:
  api_key:
    type: string
    required: true
  username:
    type: string
    required: true
  for_app:
    type: number
  ignore_branches:
    type: boolean
    default: false
  base_url:
    type: string
    default: http://buildnr.herokuapp.com

Since commit hash and branch name are already environment variables defined earlier in the pipeline, the steps’ user doesn’t need to specify them. There are however two optional properties defined which make the step a bit more flexible: ignorebranches and baseurl. Ignore branches basically will set the branch name to a fixed string, so the build count goes up no matter which branch the user is working on. Base_url is to allow a user to specify his/her own instance.

The properties specified in our wercker-step.yml will be available in our step as$WERCKER_[step name]_[property name]. In our case this means our step can expect$WERCKER_GENERATE_VERSION_API_KEY to be a string. If can see a couple of conversions happened:

  • dashes are converted to underscores.
  • all characters are transformed to uppercase.

All right, now that we know the name of the step and our properties we can start writing logic. The core of a step is run.sh. At wercker we prefer to implement as much of our steps in bash. Since not every machine may have the language of Ourrun.sh however will call a small python script fetch.py to retrieve the json and return a number. Let’s look at the first part of the python script, which contains the handling of the command options:

#!/usr/bin/env python

from optparse import OptionParser
import urllib2
import sys


parser = OptionParser()

parser.add_option("-U", "--url", type="string", dest="url")
parser.add_option("-u", "--user", type="string", dest="username")
parser.add_option("-k", "--key", type="string", dest="key")
parser.add_option("-a", "--appId", type="string", dest="app")
parser.add_option("-b", "--branch", type="string", dest="branch")
parser.add_option("-c", "--commit", type="string", dest="commit")

(options, args) = parser.parse_args()

if options.url.endswith("/") is False:
    options.url += "/"

url = "{url}api/v1/branch_version?format=json&".format(url=options.url)

url += "username={user}&api_key={key}".format(
    user=options.username,
    key=options.key)

url += "&for_app={app}&for_branch={branch}&commit_hash={commit}".format(
    app=options.app,
    branch=options.branch,
    commit=options.commit
)

The second part retrieves the content and returns either a string or a non 0 exit for the bash script to handle.

try:
    f = urllib2.urlopen(url)

    status_code = f.getcode()

except urllib2.HTTPError as e:
    status_code = e.code
except urllib2.URLError as e:
    sys.exit("""Error: Failed to reach server.
Reason: {reason}""".format(reason=e.reason))

if status_code == -1:
    sys.exit("Unable to connect to {url}".format(url=url))
elif status_code != 200:
    sys.exit(
        """Server couldn't fulfill the request.
url: {url}
status code: {code}""".format(
        url=url,
        code=status_code
    ))
else:
    import json

    data = json.load(f)
    meta = data.get("meta")
    if meta and meta.get("total_count") == 1:
        obj = data.get("objects")[0]
        version_number = obj.get("version_number")
        print version_number
    else:
        sys.exit("Unexpected return data: " + data)

A step is by default executed in either the $WERCKERSOURCEDIR. And this means that for our run.sh to call fetch.py we need to refer to it’s full path (or change directory). There is a $WERCKERSTEPROOT environment variable. So our run.shwill look like:

#!/usr/bin/env bash

if [ "$WERCKER_GENERATE_VERSION_IGNORE_BRANCHES" = "false" ]
then
    GENERATED_BUILD_NR=$($WERCKER_STEP_ROOT/fetch.py -U $WERCKER_GENERATE_VERSION_BASE_URL -u $WERCKER_GENERATE_VERSION_USERNAME -k $WERCKER_GENERATE_VERSION_API_KEY -a $WERCKER_GENERATE_VERSION_FOR_APP -b $WERCKER_GIT_BRANCH -c $WERCKER_GIT_COMMIT)
else
    GENERATED_BUILD_NR=$($WERCKER_STEP_ROOT/fetch.py -U $WERCKER_GENERATE_VERSION_BASE_URL -u $WERCKER_GENERATE_VERSION_USERNAME -k $WERCKER_GENERATE_VERSION_API_KEY -a $WERCKER_GENERATE_VERSION_FOR_APP -b master -c $WERCKER_GIT_COMMIT)
fi

if [[ "$GENERATED_BUILD_NR" = "" ]]; then
    echo ""
    echo "Failed to get build number" 1>&2

    return 1 2>/dev/null || exit 1
else
    echo "\$GENERATED_BUILD_NR: $GENERATED_BUILD_NR"
fi

This is the minimum amount of code for our wercker generate-version step. To get them in the wercker directory, you need to:

  1. add the files to a repository and please add a README.md.
  2. create a public application on wercker.
  3. Push your code.
  4. Add a ‘Wercker directory’ deploy target
  5. Go to the build and click deploy (to wercker directory).

After this you can use your step by adding following code to an apps’ steps:

        - your_user_name/generate-version:
            api_key: your_api_key
            username: your_username
            for_app: app_id

It may be a good idea to make this step the first step in the build pipeline, so failing steps won’t prevent the build number from being incremented.

Have fun!

Earn some stickers!

Let us know about the applications you build with wercker. Don’t forget to tweet out a screenshot of your first green build with #wercker and we’ll send you some @wercker stickers.

Signing up for wercker is free and easy.