"The deployment pipeline takes value from development and delivers it to clients."
This is one of the better summations of a development pipeline that I have heard, and it came from Dan Barker (@barkerd427) during his recent All Day DevOps conference talk, "Becoming a Plumber: Building Deployment Pipelines." Dan's talk focuses on the value of pipelines, how to build them well, and how to build both development and infrastructure pipelines.
The traditional pipeline: development-->test-->production servers is a good foundation, and, like the plumbing in your home, is helpful to generally get us where we want to go. However, as anyone who has raised boys knows, a toilet doesn't guarantee 100% compliance. The user isn't perfect.
It is similar to basic development pipelines. Things don't go as planned. As Dan mentioned in his talk, he strides to keep his code all in dev, and then push to test, and then push to production. However, that doesn't always happen. And there are other inconsistencies that inadvertently crop up (such as tab vs. space), despite everyone's best intentions.
Dan is a chief architect and works in highly-regulated industries -- financial services, health care, and insurance -- so precision, compliance, audit trails, etc. are all the more important. He architects pipelines to ensure human error is minimized.
But why all the fuss over pipelines? Well, first Dan touched on the values pipelines provide:
Abstract audit and compliance
Trivialities eliminated (spaces vs. tabs)
Security checks occur early/often
Tests all the things
Nimble security
Common artifact repositories
Standardized approval system
Apps become secure by default
But how do you architect a pipeline to deliver these values?
In the illustration below, you can see Dan's pipeline architecture, where he offers key points that all of us should consider applying:
Production data is separated from development data. You can only get there through the pipeline.
You interact with the Platform, and it configures the levels below it.
You can duplicate containers as much as you want to scale horizontally, and can replicate that in production.
The configuration must come from the environment.
As he dug in, he started with covering the advantages of two types of pipelines, recognizing you need to select this first:
Scripted:
Very Groovy
More powerful
Provides the greatest flexibility
Declarative:
Only a little Groovy
Simpler to maintain
Easier to read and understand
Dan then dug into the technical nitty-gritty, such as shared libraries, setups, software to manage pipelines, fabric8, Kubernetes, Yahoo! Screwdriver, etc., and then made the observation that he thinks deployment pipelines have fallen behind. In his real-life example, he and his team decided to take a more holistic approach to their pipeline (think: the first way of DevOps) so they could build and deploy apps, and also build and deploy the infrastructure.
They built a system that is especially applicable to those in highly-regulated industries, and they hope to open source it soon. Keep an eye on @barkerd427 for more info. In the meantime, he describes in detail the process it follows, software and libraries it uses, etc.
Yearning for more? You can watch any of the 2017 All Day DevOps sessions free-of-charge here.