“If you really think DevOps is just tooling, and just continuous delivery, you are not going to be successful in a DevOps transformation story.”
Projects & Presentations
J. Paul Reed has over fifteen years experience in build engineering, release management, software configuration management, and process design, with a focus on discovering business requirements and integrating those needs to get products shipped faster and more reliably.
In this Innovator's Journey, we talk with Paul about how he got started, his work within the DevOps community and what he hopes to leave as his legacy.
Mark Miller: I want to start off with the basics, were you interested in technology as a child?
Paul Reed: Yes I was. There's a photo of me on a blog post where I'm staring at an Apple 2E. I think I'm in elementary school, and the look on my face is just this one of wonderment and like just a whole new world opening up.
I learned to program Basic in 5th grade. Actually it's funny that I'm doing consulting now. I was actually doing programming and consulting in high school, too. I've had a long love of technology and the internet. In that regard, when I went off to school I always knew I was going to get a computer science degree. I didn't have a lot of the confusion that sometimes people have about "Yo what am I gonna do with my life?"
Mark Miller: Where did you go to school?
Paul Reed: I went to school at Cal Poly in San Luis Obispo right near the beach, wonderful weather, great campus, great computer science department. In fact I was just down there last week guest lecturing for the software Capstone Classes. These are the students that are finishing up their final software engineering projects and they're going to be graduating.
It was actually a lot of fun to go give them advice, return to the campus see what's changed because it has grown a lot, but give them some advice as well. That was a lot of fun.
Mark Miller: When did you graduate?
Paul Reed: I graduated in 2003.
Mark Miller: That was after the main movement to the internet, with the major push in the late '90s. You came in after the transition.
Paul Reed: Yeah. I can tell you it's kind of funny. I was telling a story with a little known fact; the first year at Cal Poly I actually got kicked out because of academic probation. Then later when I came back to campus I was on the Dean's list 3 or 4 times in a row.
The problem was that, this was back when I arrived in the late '90s, where the dorms had just been wired with ethernet and we would just stay up all night playing Doom and just fooling around on the internet. I was coming from modems, 56k modems and so having that at your fingertips was a little distracting. It was right at the beginning of when I was in school, pets.com and actually that whole first internet bubble occurred.
Mark Miller: When you graduated from school, was there a job waiting for you? Did you move right into tech?
Paul Reed: There wasn't a job waiting for me. I actually spent a summer in Washington State up across the Puget Sound from Seattle in a little town called Port Townsend. I did that for the summer, I finished off a couple of final reports that I had due. Got all that done and I ended up looking for a job.
My first role was actually at VMware. I worked on their installer for their ESX product. It's actually their enterprise level Hypervisor. The installer for it was installing an operating system. That was my first role and that's how I got into installers and Release Engineering.
I had had experience with Release Engineering right out of high school. I worked with the Mozilla Project as an intern the summer between high school and college on the Release Engineering team. I really fell in love with Release Engineering.
I got to return to it in my first job out of school. I've been doing that ever since. I really love that functional role within the organization but I've also enjoyed seeing what that has turned into, especially since DevOps has changed that conversation about what is Release Engineering. It's been interesting to watch the evolution in that role, and those functions within the software delivery process.
Mark Miller: When did you first hear about DevOps? Do you remember hearing about it?
Paul Reed: I do. It's actually funny I was just reminiscing about this the other day.
I started a podcast ... I've been a consultant for about 4 years now and I started a podcast about 3 years ago. In fact we just wrapped up the last episode of the podcast ever, the final episode. Coincidentally, we did the episode on burnout, the podcast was called The Ship Show. You can Google for it to find the archives.
One of our first episodes was DevOps Release Engineering for Web 2.0. My co-host at the time brought up this idea of DevOps. Sascha Bates, who was a consultant at the time in Minneapolis and really big in the Minneapolis operations community, introduced me to the community and the term.
That episode of the podcast I was at first pretty skeptical about this whole DevOps thing. I was really lucky to have Sascha there with me, with us to talk through all of that, mostly because her background was very operations heavy.
I had operations as a Release Engineer in the context of build farms and keeping all of those things running. I’m still a Linux nerd. I'd been running Linux since I was 15 or 16, back when Linux was still very new. I understood Linux, I understood some of the operational consideration that Ops people had to go through, but I'd never really been an Ops person. I initially was a little skeptical that it was anything new. A lot of the things we talked about with DevOps, using version control and managing your artifacts, those are things we did in Release Engineering. "This is not new."
I think over time I began to realize not only is there something to the whole DevOps ethos and movement, but it actually is not just Release Engineering. I think there's a huge component in terms of the activities and patterns, especially if you look at continuous delivery. Organizations that are really successful at continuous delivery actually really invest in Release Engineering.
From that perspective I think Release Engineering is as important as it ever was but DevOps brought a bunch of other things to the table. I'm now firmly on the side that DevOps is more than just Release Engineering. It's an important aspect to our industry because it's highlighting things that are really important and that need to be addressed if we want to deliver better software faster, with less security issues and all of these other things we hear about the DevOps unicorns doing.
Mark Miller: There seems to be two camps. One is a technology camp and one is a change management transformation camp. Where do you fit into that?
Paul Reed: That's actually a really interesting parsing. I have a lot of mixed feelings about that. I think it's a totally accurate parsing.
From a technology perspective, I started as a consultant doing in the trenches Release Engineering, make files and packaging and all of that stuff. That is a very DevOps tools technology-centric task. Working with teams on that level. I still do that and I love doing it. I think it's really important to be in the trenches and still get that perspective.
This was not a plan, I fell into it. I realized about two years into consulting that we were starting to do more, the conversations we were having were higher up the stack. They were with VPs in C-levels and they were talking about organizational structure, silos and how we reorganize people, how we get people to work together.
For me I think I tend to actually stay on both of those camps, in terms of the organizational change, the cultural aspect. I certainly started in the technology, and the tooling space and I still have deep roots in that space, I still work in that space.
This year I just started my master’s degree in human factors and systems safety. It's kind of interesting. A lot of my classmates are nuclear power engineers and pilots, and air traffic controllers, and doctors, and people like that. And then there's me, the lone web operations person.
If you look at that and you look at organizational transformation in that area, a lot of that stuff that we talked about like Three Mile Island and Chernobyl, you dig into what actually happened there. It's very applicable in the web operations space, and of course that's not really a technology play at all. That's a “how do humans interact with that system?” How do they manage that system at a huge scale? How do they deal with those problems when they occur?
Mark Miller: It's interesting that you bring up the idea of “at scale”. It's one of the things that Gene Kim is trying to work through the community; the idea that this is now an enterprise play, not just a small shop play.
Paul Reed: It certainly is. One of the problems that the DevOps community has and that I think it struggles a lot with is this definition of DevOps. I've done some presentations where I actually have quoted Jez Humble's definition, and Gene Kim's definition. I actually did a presentation called The Aesthetics of DevOps, where I surveyed all of those people and asked them, “What words come into your mind when we talk about DevOps?” I asked them a few questions that were all around the definition of DevOps.
That presentation is interesting because you actually see different threads of these DevOps thought leaders, what they care about. It’s interesting to look at the definitions and where they intersect and where they actually don't at all. The one thing that you do see repeatedly, a thread that you do see is this idea of “at scale”. Part of the thing that we found out with cloud infrastructure and architecture is that you can do these things at scale with smaller teams.
In some sense startups were running into these scaling problems in many of the same ways that the enterprises are. What is fascinating to me about that is those organizations were still struggling with the same scaling problems, but because they didn't have other constraints they were able to really solve some of the problems that they were having using some of these DevOps ethos that we talk about, these cultural patterns, these tooling patterns. I think to Gene's point, that really demonstrated that you can have the scale. If you look at the problem a little differently you can apply the same patterns and have the same successes.
Mark Miller: As you're looking now and the community is growing, if somebody, if a technician, wants to get in and be part of this community, what kind of skillsets should they be putting together?
Paul Reed: I recently did a presentation on What does Release Engineering look like in a DevOps World, because that's my community. You hear this a lot with operations people, I see this a lot with operations people at client sites, this idea that the entire industry is moving from people that do the thing to people that build the thing that does the thing.
What I mean by that is Release Engineers, what we used to do a lot of the time is we would sit down and we might have a build process, and then to release the product we might get different components from different teams, even sometimes in large products, different build teams. There would be a release team that would assemble all those components and they would take the time to do that. They would put it all together and then they would ship it.
If you're doing something like continuous delivery, and really the business value there is being able to move faster, we see that business value being espoused by DevOps.
The reason people want to do DevOps is to move faster. You could not hire enough Release Engineers because there's just not enough to do that job at the speed you want to do it, at the scale that you would have to do it when you're talking about something like Netflix, where you're deploying into all these instances.
What the shift is, is now we see people building continuous delivery pipelines. This is the same story for Ops engineers and configuration management automation. It's the same for QA engineer. If you're doing continuous delivery you cannot manually test it. There's just not enough hours of the day or people you could hire to do it.
We're moving again from people doing the thing to people building the thing that does the thing. The biggest hurdle for people that are used to doing the thing and not willing to let go of doing the thing is they're fretting on that problem.
I always am worried when I run into someone like that. It doesn't happen too often. I think people are starting to get it, but those roles are going away. If all of their energy is invested on holding on to them being the person that does that one thing, the business is going to work on trying to automate that thing. Whether that person gets the automated, or someone else automates their way around them, it's going to happen. I've seen it with Release Engineers, I've seen with QA, I've seen it with Ops.
I am not one of those people that believes everybody needs to code when we talk about education. Certainly if you're in this industry and you've got a pile of shell scripts that aren't version controlled, that you don't really have a testing methodology for, or you're like "Ah I don't wanna learn Ruby, because I don't wanna do shell for Puppet, like I'm not doing that." I think the industry is going to be an increasingly tougher place to exist in.
Mark Miller: I had a discussion with people in London and Stockholm last week along this line, that a lot of people think of DevOps as NoOps. What we are more towards is it's either NewOps or TransformedOps. Your Ops people have to start thinking about managing the platforms themselves, not the applications.
Paul Reed: Yeah definitely. It's interesting that we're seeing the same things.
There was a whole big NoOps explosion a couple of years ago. Part of it came from how Netflix described that they have no Ops teams. What was interesting was the big backlash against that. Then people who talked about NoOps said "Well, we actually don't mean no operations engineers, it just means where we put that expertise is different."
There was a great discussion about what NoOps is and is it actually NoOps. I think what you were saying or the conversations that you were having is much more accurate reality than we don't have Ops people. We have people that operations experience that are contributing to the software delivery stream in a different way than "Ops".
What's interesting is I think you're seeing that exact same discussion happening with serverless right now. Serverless architecture, this idea that the developers don't even need servers, so of course you wouldn't need Ops. I don't have an opinion yet. I'm still trying to watch what's going on there.
Charity Majors just presented at the Serverless Conference. She did a great write up where she was talking about the whole serverless thing as a rinse and repeat of NoOps in certain ways. "You don't need to freak out" was her thesis.
Mark Miller: When you're looking back on the work you've done so far in your career, what are you most proud of?
Paul Reed: I have some open source projects that I open sourced as a college student and they're still in use. There's a professor ratings engine that a bunch of students just worked on giving a new UI to, a new face to. They were working on that and that's still in use, even though it's code I wrote in 2012.
There are certain technical projects that I am proud of.
Mark Miller: Hold on, you did a professor “hot or not”?
Paul Reed: Yeah. You can check it out. It's at polyratings.com. It was just for Cal Poly. It's funny because it hasn't been updated. If you go and look at it right now, the UI is very old, the dates on it refer to 2012. It's very old. I've been with students as their senior projects to update the UI for it. I'm actually excited that it's having new life breathed into it.
I've done a lot of technical projects that I am proud of. In a previous life I was a journalist. When I was in high school I actually worked for the local newspaper and I had a number of stories that were on the front page. I did some investigative journalism for the newspaper at the time.
I've always been very interested in people's stories, people's narratives. The narratives of communities, how DevOps started and it became a thing, like the back story of that is fascinating. The cast of characters if you will is always certainly very fascinating.
I did a report for O'Reilly called DevOps in Practice, collecting those stories, telling them, discussing them with others. When we talk about cultural change, talk through what Target did, or what Nordstrom did, or what Netflix did and be able to tell the stories of people that I know at all of those places, that I've actually become friends with. I have been privileged to be able to listen to their stories and look at their narrative thread. I think that is the thing I'm most proud of; being able to be in the role to get some of these threads and be able to tell those stories about the technology industry.
Mark Miller: As you move through the community now and you think about what you would like to leave behind, what are you aiming for?
Paul Reed: Some of the work I enjoy doing the most is working with teams on hard non-technical problems. Because I have roots in technology, I can have the conversations and have a deep dive in technical conversation if we need to have one. I'm lucky to have that experience. Solving cultural problems is why I find this area about human factors and systems safety so interesting. I think it is very early times in operations and software, and the organizations that use those things to operate their infrastructure to a business end.
It's still early times for them and figuring out what the patterns are and what the anti-patterns are, and all of those things. The interesting thing about that is it doesn't matter whether you're NoOps or you're serverless, or what your architecture is. If you are operating a site or a service, ultimately there's a saying in the monitoring community, "You need to own your availability." These operational problems are in many ways complicated, complex problems. In some cases they may be wicked problems, if you're familiar with that terminology.
Mark Miller: Yes.
Paul Reed: If we look at other industries, the nuclear industry, the power plants, air traffic control, aviation, they still wrestle with these problems. Those are industries that have been around forever. I think it's early times for us. Organizations that are just coming to DevOps and understanding that and realizing that are still reorganizing people on the chess board and figuring out how to move faster.
I think one of the outcomes of moving faster is moving faster at a larger scale. We've seen this in every other industry. You have more industry accidents and incidents. People deal with those. Organizations deal with those. Computers don't deal with those.
Figuring that out, I think it's still early times. It's an area I'm super interested in. It's an area that I'm learning stuff about everyday because I have to, but also because I find it interesting. That's what I would like to leave as a legacy, patterns in the industry that makes that easier.
Mark Miller: It's interesting of all the discussions I've had over the years, you're the first one that I think has brought up the concept of wicked problems. For those of us that have read on it, it makes so much sense. Can DevOps actually deal with wicked problems?
Paul Reed: I was introduced to the concept my roommate, who is a city planner. We have a bunch of friends that are planners and they talk a lot about wicked problems because of the complexities of local politics and zoning regulations, and people that have agendas that want things built or not built, and everyone thought it might be better for the area or even the region. We were going through this with high-speed rail in San Francisco and in the State of California. I was introduced to it through that way.
Wicked problems are so named because they are difficult to solve.
I think one of the things that DevOp's done right tries to make apparent is the idea that we live in complexity. We live in a complex space. Very prescriptive, rigid models for the way we do things, when we talk about best practices, that's just not a world we exist in. We don't exist in a world where there's a singular, superlative practice in every case.
I think that's actually one of the things that make DevOps so hard for people to get their heads around, is that they want a recipe book. They want a cookbook. I think of all the unicorn organizations that succeed at that, just acknowledge that's not the way it works. There's a growing understanding that decentralized results in better business outcomes than centralized. You see this from an architectural perspective of microservices.
Microservices is all about having a decentralized infrastructure and architecture so teams can move at the pace that's right for them, and they can figure that out and can still have something that is a running whole at the end. All of that is complexity informed.
I don't know that I would say I think DevOps can solve wicked problems necessarily. I do think DevOps forces us to confront the reality of the environment that we're in, and then offers some patterns we've seen that have been successful because companies have used them to address some of these wicked problems. I don't want to say manageable, but even approachable or even in a space where we can try to run experiments to deal with them.
Mark Miller: I had an interesting statistic thrown at me a couple of weeks ago in a discussion. They said only 5% of companies know DevOps exist. Not necessarily even are working with DevOps, but actually know that DevOps is something. 95% of the companies have never even heard of it. Does that sound right to you?
Paul Reed: Oh man. My reaction is no, it doesn't sound right.
Having said that every DevOpsDays that I've been to, pretty much ever, and every event that I've been that had where we're talking about DevOps, a lot of times the question gets asked to the audience in the introductory remarks, "Raise your hand if this is your first DevOpsDays." Even at events like Silicon Valley where you would assume that people would be in the know about it, you still find to this day half of the audience raising their hand. They've never been to a DevOpsDays, they wouldn't consider themselves part of the community. They're just figuring it out. I find that number hard to believe but given the anecdotal data I think it's possible.
Mark Miller: I've seen the same thing. When you and I were at RSAC doing the DevOps track there when we polled the audience I'd easily say 50-60% were there as complete novices just trying to figure out what it was.
Paul Reed: Right. One of the problems is that it's difficult for people coming to the space ... even now people ask "What's your elevator pitch for DevOps?"
I give the answer "Man I don't know. What do you think it is?" The reason I actually do that, I don't give them an answer on purpose is because I actually want to understand their framing of the problem. Where are they approaching DevOps elephant?
There's that famous photo of all the people touching the elephant and the person touching the trunk, and they're all blindfolded. The person touching the trunk thinks it's a snake, and the person touching the leg thinks it's a tree, and the person touching the tail thinks it's a pig. There's that one where somebody took that photo and put little thought bubbles and they're all saying "it's DevOps. It's DevOps."
That is one thing I think is actually perilous within the DevOps community. In fact I have a thesis that DevOps may actually be disintegrating. I don't mean disintegrating as like exploding or anything violent like that. I actually mean it in the root form of that word, disintegration.
We have refused to define it (DevOps). We now have groups that DevOps means different things and they are promoting that particular view of DevOps. There's nothing wrong with that. I'm not saying any definition is right or wrong. What I'm saying is that makes it difficult for newcomers asking, "Can you just explain it to me? Can I get a basic definition?" They'll ask 5 people at a DevOpsDays and they'll get 7 definitions. I think that's problematic, especially if we're looking at bringing people into the fold. I think that's frustrating.
One thing I noticed at DevOps Enterprise Summit, both years I was there, is that a lot of the organizations in the enterprises that are there will talk about DevOps ever so briefly and then they'll talk about continuous delivery, or they'll talk about a specific monitoring stuff they do. What's interesting to me about that is they kind of talk of about DevOps. They use the language but then they immediately go to practices and patterns because those are tangible things. That's especially, if you're trying to do it at scale, I think very useful to do.
The only concern there is that's a really easy way to optimize the culture part of DevOp's tools and culture just out of the equation. I think it's widely understood that if you don't include the cultural aspect or the organizational structure aspect sometimes as part of it as well, you're not going to be successful. If you really think it's just tooling and just continuous delivery, you're not going to be successful in a DevOps transformation type story.