"Building software that never goes to deployment doesn't count. The software engineer has to be responsible for building things that run in production."
Making Operations Visible
As CTO/Founder of Signal Sciences, Nick Galbreath focuses on web application security defense and security monitoring. His main concern is the wall between security and CI/CD deployments because there is no good story for the security people. Nick didn’t realize he was doing DevOps until Mike Brittain from Etsy asked him what he thought of this DevOps "thing".
In this “Innovator's Journey” profile, we talk with Nick about his personal motivation for working in a DevOps environment, his desire to incorporate security as a critical part of any DevOps initiative, and his pride in building “fantastic” teams.
Mark Miller: What are you working on now? What are you doing?
Nick Galbreath: Right now I'm the CTO and co-founder of a company called Signal Sciences here in Los Angeles with my co-founder Zane Lackey and Andrew Peterson. This company came out of the work that Zane, Andrew and I did at Etsy about making security visible.
The biggest blocker we've seen like CI and CD deployments is there's no good story for the security people. They don't get the cool metrics. They don't get the cool dashboard. We're working on building that out for them. They have a buy-in for just everyone going faster. The answer of like, "Hey, we're going faster. Security, what do you think about that?" The answer can't be “No”, but at the same time they need some tools in order to get to the yes. We're working on building out a whole bunch of infrastructure that allows security teams to move just as fast as everyone else.
Mark Miller: Where did the name come from?
Nick Galbreath: We didn't necessarily want security in its title and we thought that signals applies to DevOps and operations just as much as it does security. It just sort of came in one of those brainstorms and Zane doing some ferocious hacking on some of the domain providers.
Mark Miller: What's your background?
Nick Galbreath: I got started in the software engineering in the mid-90s. I actually dropped out of a PhD program because I was playing around with a website around '94. I was like, “I think this internet thing is going to be big. I didn't really know what I was going to do with the math PhD and I want to try something else”, and I got really lucky with joining the startup very early in Boston called Open Market. The main goal or main point of that was it set me on a trajectory doing software engineering work at a number of really fantastic companies and so I don't actually consider myself a DevOps person.
DevOps is just a part of good software engineering and I've never really known it any other way and I'll tell you the story on how the word even first came to me.
I was kind of working in a vacuum and I was interviewing at Etsy and Mike Brittain who's now the VP of engineering he's like, "Hey, so what do you think of this DevOps thing?" I was like, "I don't know what that is."
"Oh, well, it's like where operation is like automated code. It's to deploy infrastructure and developers are writing systems to deploy their code and being responsible for it.”
I was like, "Well, yeah, that's what I do," and we can talk more about those details later.
Mark Miller: There really wasn't a trajectory or starting point where you said I'm going to do DevOps. It was just you were doing this thing and then somebody put a label on it.
Nick Galbreath: Yeah, pretty much. The full background is quite interesting.
Before Etsy me, and John Allspaw actually worked at Friendster of all places together which was very traditional waterfall. Everyone was doing a fine job but it had a whole bunch of well known problems. We both left to do different things. John went to Flicker and I went to a company in New York called Right Media which was doing an online advertising.
At Right Media, it was extremely high scale ad serving for the time. It was extremely fragile and brittle. The round trip times if you deploy code to when an ad is viewed, to when it's clicked, to when it's converted, to when it's reported on, to when the human logs in is a matter of days. Any change would take a couple of days even for anyone to know about it.
Just in that environment, and it's real money, the ops people really had no idea how to manage this thing because nothing was standardized yet. The ad server group really had to take responsibility for operations of the product. We did all sorts of interesting things around that… sending UDP packets to an aggregator to make graphs, sort of like what people now call stats D. Writing our own tools to deploying things, writing our own tools to display metrics so developers could see exactly when they deployed something what exactly was going on.
After that I end up going to Etsy where John was VP Ops, CTO at that time and that's how we reconnected and had some discussions like, "Wow, we sort of came to the same conclusion but from two very different angles."
Mark Miller: One of the interesting things that is starting to happen now is people are concerned about security as part of that DevOps cycle. Did you guys worry about security at that time?
Nick Galbreath: At Right Media, we were doing deploys daily. and daily was the exact right amount of time, just because changes took a while to even know happened. So we had to do deploys daily. The nice thing is we knew how to deploy, we could deploy anytime we wanted to. Yeah, we had some security and fraud problems and we knew we're able to deploy changes very quickly. That was the first case, "Hey, this deploying quick thing solves more than just production problems." It helps solve security problems.
This was followed up again at Etsy with me and Zane Lackey.
The famous story is we were at a conference in Italy and all of a suddenly graphs started spiking. We got an alert and while we're on lunch break we were able to push the fix that actually shut down across the cross-site scripting. A guy wrote in the next day, "Hey, I think I found something but don't throw off my account, I don't know what happened." We were able to push the fix before it could even be exploited.
Zane was a little skeptical,” I can't use deployment.” That's cool but after seeing I could actually do my job better because I'm able to make changes and fixes faster, it's a real eye opener. It's very counter-intuitive to most security people, that faster actually can be better.
Mark Miller: When you're describing what you're doing here and the way that you got started, do you perceive yourself as part of the DevOps community or is the DevOps community just somebody that reaches out to you periodically? How do you perceive yourself?
Nick Galbreath: Good question. I guess I just view DevOps as a slightly polarizing word simply because it creates another group. What are DevOps people doing? They're writing codes, they're deploying code maybe to help others deploy code, to build systems. To me, that's software engineering.
I don't want the dichotomy or polarization of “that person is in DevOps”, “I'm in Dev”,” I'm a real engineer”, that person is doing DevOps, that person is doing ops. They're doing software engineering on a different focus.
What I like about DevOps and the way I view it is it's solving real problems. It's making people lives better and it's getting things done.
I certainly talk at DevOps conferences frequently so I think it's a great community and a great audience. I like applying software engineering to all the other stuff we're doing, all the operational stuff, all the deployment stuff. I think it makes everyone better.
Mark Miller: It's interesting that you say that. Yesterday I was talking in a session and somebody came up with the idea that everybody's talking about going fast in CI and CD and he said, "You can go as fast as you want but if you're delivering crap it really doesn't matter." He thought that DevOps was the part of quality in continuous delivery. I thought that was fascinating.
Nick Galbreath: It's a two-edged sword. The way I like to describe continuous deployment is , “Do you have a mechanism that on demand you're able to deploy?“ What the policy is changes company to company. You have a high risk system, you're doing PCI, you're doing compliance. Okay, you're probably not pushing once a day. You're probably pushing at a much shorter time frame but at least it's regular and it's not with a lot of ceremony.
If you're doing a traditional marketing website, there's very little risk in deploying faster. People would certainly debate that, but it really depends on the nature of your job. Having the mechanism to deploy fast is absolutely critical I think.
Mark Miller: What's your motivation? What gets you up in the morning? Why are you doing what you're doing?
Nick Galbreath: I love building things. I like making things. I think every engineer, good engineer, who's trying to build great products and great things have found that traditional models for deployment and delivery and infrastructure are just too slow and it's frustrating for everyone. It's frustrating for customers, it's frustrating internally. It's frustrating for the engineers, of not having control over what's going in. What motivates me is making things, building things better, making people's lives better, making better product.
No engineer likes building something waiting for it to go to QA, coming out who knows when, and then sometimes not even knowing when it goes to production. It's like it's a recipe for fail. Make the engineer responsible for their code, what happens to it in production.
It's just a different model now with the web and we're slowly catching up on what it actually means for building software. Even that word, “building” software is an outdated concept. Building software that never goes to deployment doesn't count. The software engineer has to be responsible for building things that run in production with everything that comes with it.
Mark Miller: The next generation of engineers come up to me and asks, "What do I have to do to get started here?” How can someone get started in the community we are building here? What would you say to them?
Nick Galbreath: There are two things. It's almost the same thing I say to anyone in software engineering or the security field or software engineering.
The first thing is to write about problems you encounter and how you solve them. If you don't have a blog, whatever that means nowadays, Snapchat, twitter, Medium Feed that doesn't describe sort of what you do, that's going to be difficult for employers to evaluate what's going on.
What is that? Use Google and go find something? Go write a quick article. Hey, I had this problem, here's how I solved it, here's how it worked. It's a much better view of what you actually do than your resume.
Certainly, if you want to pick something in particular, more to DevOps would be to pick a problem, take it, see it through to its conclusion and iterate it.
If your company is having deployment issues, what's the smallest thing you can do to make that better? Write about it. A lot of the problems are not necessarily technical. It's about building relationships. It's getting buy-in to changes and just learning even no matter how small it is, building those networks, building those bonds, building those human connections is just as important as the technical part of it.
If you're a young engineer and you don't actually know how your software is deployed, that's a good place to start. Go talk to the ops guys or the CD guys or the DevOps guys, whoever is in charge of that. Go find out how that works. It's really just being curious and going after it and not being afraid to talk to people.
There's a funny story. I am a fairly guilty offender of keeping my blog up to date but for a while I wasn’t. I've never really professionally programmed Ruby, but I had to do something with it one time. I couldn't figure how to get, eMax or VI to do something and so I wrote “here's how I did Ruby to work on my Mac “with so and so. It's a pretty niche topic, but what's funny is a year or two later I found out that if I type in Ruby eMax spellcheck or reformat, guess what is the number one post? It's like you just don't know what people find useful until you actually put it out there.
No one's going to be following your blog but you know what? Your problems are not unique. Other people will bump into them. The process of writing makes everyone better. It clears your thoughts. It's a way better resume than the traditional one pager. It doesn't take much time either so go forth and blog about what you do.
Mark Miller: That's a sage advice from a young man like you. If you could name a title that you would want to be with what you're doing, what you're associated with, what would it be?
Nick Galbreath: I'm going to punt on that one. That's something I've never thought of. I've never really put too much thought into titles to be quite honest. Most of the time at startups I normally have the conversation with new hires and early hires, “We're not doing titles right now because we're too small.” It's our job is to build great product.
Mark Miller: Then I'll turn it on its head… if you could be a superhero, what would it be, because I give our superhero stickers at all the conferences?
Nick Galbreath: That one I do have an answer on and that is a “Universal Translator”. If my superpower could read and write and comprehend more people automatically, that would be awesome.
Mark Miller: When you look back, what are you most proud of with the things you’ve done?
Nick Galbreath: That's really easy to answer. At some point in my life I decided to move from individual developer to management and the proudest is actually building fantastic teams. The people I've worked with and the teams we've built together is easily the most memorable thing. The technology comes and goes, people stay.