The software stack 10 years ago was simple. Requests started at the client side, and they could be answered from a web server itself, or sent for additional processing to an application server or MySQL database.Meanwhile, the LAMP stack (Linux, Apache, MySQL, PHP) worked because it standardized components across the spectrum. Virtual machines improved server utilization. IaaS vendors made VMs easily accessible and cheap.
As significant as these technological developments of the past were, they couldn’t keep pace with new challenges presented by trends like mobile, big data and microservices. Over the past decade, the software stack has been disrupted for the better by foundational shifts in technology. Below, we’ll take a look.
Configuration management tools—Setting the stage for DevOps
It started with the realization that the deployment process for applications was broken. Working on a major update for months, and then having it fail because of configuration mismatches resulted in a loss of revenue for businesses, and stress for software teams. Enter configuration management tools like Chef and Puppet that could automate the creation of virtual machines in a datacenter or through a cloud provider. But that was just the start. The bigger change was the start of the DevOps movement with continuous integration and continuous delivery.
Continuous integration tools—The first steps in DevOps
To release code more frequently and reliably, a change upstream in the way applications were built was required. Now, developers need to write code that is deployment-ready. This is what continuous integration (CI) is all about.
Continuous integration involves using a modern versioning tool like Git, and automating builds to a single repository. Building this quality in from the start requires not just developers, but also vital input from the QA team on the feasibility of adding new features. Selenium has transformed QA over the past decade, moving QA teams from manual testing to automated testing in the cloud on web and real mobile devices.
Jenkins has become the most popular CI server, bringing powerful integration between Git, Selenium, and other deployment tools downstream.
Docker containers—The foundation for microservices
Virtual machines were chained to the infrastructure that powered them. They consisted of a heavy hypervisor that extended their time to startup. Docker changed all this when they introduced their containers that left out the hypervisor, and focused on packaging an application's code and dependencies, providing a layer of abstraction from the underlying infrastructure. It brought portability to application code across teams and environments.
With this key piece of the puzzle in place, teams started to find ways to isolate each component of the stack from others. This led to the microservices model, where applications are not a single monolithic unit, but instead numerous interconnected services. Each service is isolated from the failure of other services, and can be deployed and managed independent of other services.
Containers need an orchestration tool
Spinning up a few containers on your laptop and packaging a few services for a simple app takes just a few minutes. Getting started has never been easier. However, for real-world applications that span hundreds of services, and numerous containers per service, orchestration is a major challenge.
Chef and Puppet worked for VMs that were fewer in number and had a longer lifespan. But for containers, it was determined that a different kind of orchestration was needed—One that didn’t focus on the infrastructure, but on the application code itself. One that enabled portability across cloud vendors. One that was container-native, purpose-built for automating the creation and management of containers, not VMs. And one that could manage a vast complex cluster of containers as a single powerful supercomputer. That’s what Kubernetes is.
Docker was a year or two late introducing its container orchestration solution, Swarm, in mid-2016. In the meantime, Kubernetes, which was launched in 2014, had gotten a big head start, and was already being used by many to orchestrate containers in production. Today, with widespread adoption, strong open source support, and commercial support from many vendors, Kubernetes is the favorite container orchestrator of many Docker shops.
However, Kubernetes is still a maturing platform, and while it has strong defaults for pod replication and cluster management, managing Kubernetes on your own can be a challenge. It needs to be well integrated into your CI/CD workflows, and various parts of your infrastructure like AWS instances and storage. And networking needs to be planned. To offload this management tax, today there is a growing need for Kubernetes management tools.
The software stack we know and love today is comprised of components that didn’t exist or weren’t widely used a decade ago. Chef and Puppet, Git, Selenium, Jenkins, Docker, and Kubernetes are some of the most prominent tools that make up the modern software stack. Many of them are active open source projects with outstanding commercial support from vendors in the community.
It’s a safe bet that technology will continue evolving to meet new challenges. But in order to understand what will come next, you need to know how we got here.
About the Author
Twain Taylor began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.
We’re hiring! Check out the careers page for open positions in Amsterdam, London and San Francisco.
As usual, if you want to stay in the loop follow us on twitter @wercker or hop on our public slack channel. If it’s your first time using Wercker, be sure to tweet out your #greenbuilds, and we’ll send you some swag!