Continuous Deployment with Wercker: A Tale of Python Bots

Wercker is a Docker-Native CI/CD Automation platform for Kubernetes & Microservice Deployments

Sneha Inguva
Sneha Inguva
May 18, 2017

It started with a Twitter parody account, inspired by @BoredElonMusk...

 

Bored Elon Musk

 

I decided to throw together a bit of Python code to analyze a person’s tweets and then subsequently generate new tweets. The hacked-together result is the following: https://github.com/si74/twitter-parody-bot. The underlying code grabs a body of texts from Twitter, builds a Markov model, and subsequently uses the model to generate new tweets. But I wondered—Could this be updated somehow?

This is where Wercker comes in. Wercker is a container-native continuous integration and deployment tool. As a result, it can be easily integrated and used with many common cloud or Container-as-a-Service providers.

Using Wercker, I can deploy a containerized version of the twitter-parody-bot that is redeployed as the Markov model is improved. Typically, I would need to create a Dockerfile in order to build my image before pushing it to either the Docker Hub or a private registry. Using Wercker, however, my Dockerfile will essentially be incorporated into my wercker.yml file. (For more on how this works, check out Wercker’s guide to getting started with Python and the Wercker documentation.) For the twitter-parody-bot in particular, my plan is to create a deployment process such that changes in the GitHub repository trigger a build and deploy of the twitter-parody-bot image to a DigitalOcean Droplet (for more on how this works, click here).

 

Step 1. Clean Up Twitter-Parody-Bot

Before setting up Wercker, I first needed to clean up the GitHub repository. Currently, the repository is separated into two main Python scripts: one to grab the tweets of my selected Twitter account to parody, and another which is actually the bot code. Next, to simplify the Wercker deployment pipelines for this demo, I’ll grab a corpus of tweets and add it to an outputs.txt file.  

For this particular example, I will be parodying the @DeepakChopra Twitter account. (Note: In the future, grabbing the corpus itself could be added as part of the Wercker pipeline step.)

 

Step 2. Set Up Accounts

After grabbing the necessary @DeepakChopra tweets, I needed to create a parody Twitter account and generate the necessary API keys which would allow the script to tweet on behalf of that user.

I created an account entitled @BotDeepakChopra, verified the associated email address and phone number, and then navigated to apps.twitter.com. I created an application entitled parodyBot, and then created the necessary keys and access tokens.

 
create an application

Fig.1: Going to apps.twitter.com to create an application

 


Generating keys and access tokens for parodyBot

Fig. 2: Generating keys and access tokens for parodyBot

 

Done!

In order to utilize Wercker to deploy a containerized twitter-parody-bot, however, I also needed to create a Wercker account and a Docker Hub account.

I created an account on Docker Hub and created a parody bot repository to which to push the parody Bot image:

 The parodybot private Docker repository

Fig 3. The parodybot private Docker repository

 

Creating a Wercker account was easy enough—I signed up using my email.  

In order to deploy a Wercker application, however, I will need to be able to pull code from GitHub, create an image, and push/pull code from the Docker Hub to my DigitalOcean repository. As a result, when creating my application, I will also need to:

  • Generate an SSH keypair for GitHub and add the key to my GitHub account
  • Add my Docker Hub username and pw as environment variables
  • Generate an SSH keypair to access my DigitalOcean Droplet

Generating the SSH keypair for GitHub is easy enough: it’s actually part of the application creation process! All I needed to do was grab the public key and add it to my GitHub account.

 

Generating the SSH keypair to be used with GitHub

Fig. 4: Generating the SSH keypair to be used with GitHub. (Note the public key will need to be manually added to my GitHub account.)

 

I then headed over to the environment variable tab to add my Docker username and password:

 

Adding the Docker username and password as environment variables.

Fig 5: Adding the Docker username and password as environment variables.

 

Finally, I also generated the SSH keypair for use with DigitalOcean in the environment variable tab. The private key will eventually be referenced in my wercker.yml file, and the public key will be added to my DigitalOcean account.

 

Step 3. Create a DigitalOcean Droplet

Given that Wercker is a “container-native” CI/CD tool, a very common choice is to use a Wercker pipeline in conjunction with Google Container Engine, or Amazon EC2 Container Service, etc. However, given that I already have a DigitalOcean account, I elected to utilize Wercker to deploy the parody bot container to a DigitalOcean Droplet (a.k.a virtual server).

Creating the Droplet is a simple matter of using the command-line tool, doctl. Note that it is also entirely possible to utilize the DigitalOcean cloud interface. In order to communicate with the Droplet (i.e. pull the twitter-parody-bot Docker image and start the container), we will need to SSH into the Droplet. We have already generated the SSH keypair in the aforementioned step. All we need to do now is add the public key to the DO account, and utilize it when generating the Droplet (it can also be manually added to the ~/.ssh/authorized_keys on an already existing Droplet; see DigitalOcean’s docs about SSH keys for more information).

Given that we will also be running a Docker container on the DigitalOcean Droplet, it’s essential to also install Docker on the Droplet, or ensure that the Droplet created has Docker pre-installed. Find out how to do that here.

 

Step 5. Prepping the Wercker Pipelines

We can now delve into the nitty-gritty and add our wercker.yml file to our repository. Note that the Wercker file has a base-box, upon which the image is based, as well as three pipelines—dev, build, and deploy. Each pipeline has steps, self-contained tasks to install packages, run scripts, or run other commands. It is also possible to create your own customized Wercker steps. 

The wercker.yml file we will be using is below:

 

 box: python:2.7.13
dev:
  steps:
    - pip-install
    - internal/watch:
      code: |
        python bot.py
      reload: true
build:
  steps:
    - pip-install
deploy:
  steps:
    - pip-install
    - script:
      code: |
          env
    - internal/docker-push:
      username: $DOCKER_USERNAME
      password: $DOCKER_PASSWORD
      repository: $REPO_NAME
      # note: wercker adds all source files to the subdirectory below
      cmd: /bin/bash -c "cd /pipeline/source && python bot.py"
    - add-ssh-key:
      keyname: DO
    - add-to-known_hosts:
      hostname: $HOSTNAME
    - script:
      # using a private registry so need to authenticate
      name: authenticate to docker
      code: ssh root@$HOSTNAME docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
    - script:
      name: pull latest image
      code: ssh root@$HOSTNAME docker pull $REPO_NAME:latest
    - script:
        name: stop running container
        code: ssh root@$HOSTNAME docker stop parodybot || echo ‘failed to stop running container’
    - script:
        name: remove stopped container
        code: ssh root@$HOSTNAME docker rm parodybot || echo ‘failed to remove stopped container’
    - script:
        name: remove image behind stopped container
        code: ssh root@$HOSTNAME docker rmi $REPO_NAME:current || echo ‘failed to remove image behind stopped container’
    - script:
        name: tag newly pulled image
        code: ssh root@$HOSTNAME docker tag $REPO_NAME:latest $REPO_NAME:current
    - script:
        name: run new container
        code: ssh root@$HOSTNAME docker run -d -e "consumer_key=$consumer_key" \
          -e "consumer_secret=$consumer_secret" \
          -e "access_token_key=$access_token_key" \
          -e "access_token_secret=$access_token_secret" \
          --name parodybot $REPO_NAME:current

 

Note that we are leveraging the Wercker environment variables in our yaml file when running any set of commands on our DigitalOcean Droplet. Furthermore, we are also utilizing the environment variables at runtime when starting the container.

I commit the wercker.yml file to my local repository and push to GitHub. I immediately notice that this kicks off the build pipeline by navigating to the Runs tab in Wercker:


A Wercker build pipeline that failed

Fig. 6: A Wercker build pipeline that failed

 

Unfortunately, this build failed (minor yaml syntax error initially) but once fixed, I realized that I needed to also explicitly add the deploy pipeline such that it will be triggered upon the build pipeline successfully completing. I can do this in the Workflow tab:

 
Setting up the Wercker deploy pipeline

Fig. 7: Setting up the Wercker deploy pipeline

 

Step 5. Let’s try it!

 Voila! We are now ready to go. Making a minor change in my repo and after pushing it to GitHub, I navigate to the Runs tab and see that both my build and deploy pipelines have run successfully:

 A successful build and deploy-prod pipeline

Fig. 8: A successful build and deploy-prod pipeline

 

I can also confirm this bot is SSH’ing into my DigitalOcean Droplet and confirming that my Docker container is running:

 

Image 2017-04-21 at 5.48.27 pm.png

Fig. 9: Currently running containers

 

 And checking Twitter, I see that @BotDeepakChopra is happily tweeting away:

 

Currently running containers

 

Step 6. Next Steps

This Python bot and pipeline were simple for the purpose of demonstration, but hopefully you can now see how easy it is to use Wercker to deploy your containerized Python applications!

If I were to continue with this project, however, I would look into:

  • Running a cron job to append new texts to outputs.txt daily. This would allow the model to constantly improve.
  • Separate building of the model and bot into separate steps. This would prevent the bot from hanging, were the model to take some time to generate.
  • Improving security around secrets. Currently our API keys are being injected into the Docker container at runtime. However, we could restrict access by using pipeline-specific environment variables, among other solutions.

 

About the Author

Sneha Inguva is an enthusiastic software engineer currently working on building developer tooling at DigitalOcean. She has worked at a variety of startups in the last few years, and has a unique perspective on building and deploying software in eclectic verticals (education, 3D printing, and casinos, to name a few). When she isn’t bashing away on a project or reading about the latest emerging technology, she is busy rescuing animals or practicing martial arts.

 

Like Wercker?

We’re hiring! Check out the careers page for open positions in Amsterdam, London and San Francisco.

As usual, if you want to stay in the loop follow us on twitter @wercker or hop on our public slack channel. If it’s your first time using Wercker, be sure to tweet out your #greenbuilds, and we’ll send you some swag!       

 

Topics: Containers, Tutorials