Over the past couple of years, American Express has gone through a DevOps journey. A transformation that Tim Klever was kind enough to share with us at this year's Nexus User Conference. From draconian rules to a state where the company was able to use the right tools to solve problems, there is a lot of interesting insights that we can all learn from.
Tim shared both failures and successes in automation and tooling; and that’s just the beginning. The journey isn’t over!
Some Background on AmEx and Their Rules
American Express has been around for 170 years, starting as a competitor to the US Postal Service. Over time, they’ve also built a massive catalog of software.
In the beginning, there were rules. Manual reviews and processes would result in forms being sent off to lawyers and approvers. This was
- Slow
- Expensive
- Full of friction
So they brought in a tool—Sonatype! They expected quick success.
At the time - some of their ecosystem wasn't covered, and getting organic adoption of the tool wasn't happening as quickly as expected.
Giving Developers a Second Job Instead of Supporting Them
Next, American Express had to determine why this wasn’t working. Surely, they’ve done all they can. The developers don’t worry enough about security! Why aren’t they helping?
But, when going through conversations like these, Klever and others realized that something was wrong. They found themselves guilty of what he calls “laissez-faire DevOps.” Leadership threw tools at teams but were missing the partnership.
If the developers are brilliant people but still not using the tools they’re given, what’s the problem?
Well, instead of helping, these new tools increased developer cognitive load and gave the dev teams additional work to do while providing no support. They failed to understand the needs of their customers.
At that point, leadership went back to the drawing board. Who should they be? What can DevOps do? How can it build a bridge or community?
After some brainstorming, they discovered how many aspects they wanted to be part of this identity and came up with a laughably long final version:
That last one is most reflective of their values, but it was time for a concise summary. Who should the tech division of AmEx be?
The Path Forward
After going through that identity crisis, they decided that they needed to focus on service first: in fact, not only serving colleagues but partnering with them to aid delivery.
Guidelines that developed out of this were around empowerment, not enforcement. When looking at regulatory and governance needs, the American Express team looked for ways to automate the process. And finally, they focused on not blocking the development teams as much as possible. Stopping the deployment process is a tactic that should only be used for the most critical issues.
Next, it was time to execute. The team parallelized non-critical issues and instead focused on building a solid community. By far and away, the largest wins were gained from these communities and feedback channels.
And finally, they built a suite of tools that delivered insights and answers to the development teams.
The Results
Coverage went from a tiny sliver of the organization to going across their whole ecosystem. They created an active and supportive community that cared about security and compliance.
Developers got the answers and insights they needed when they needed.
To sum it up, the teams were able to deliver secure software with less friction from a collaborative engaged community of technology professionals.
The Learnings
Tim and his team learned that great tools were not enough. They needed to explore developer focused integrations that provided data where the developers needed it most. All of this is about people, not just code and computers.
What’s Next?
Tim's team is constantly looking to see where they can go next. They look to add integrations in more areas of the development ecosystem. And they hope to put further focus on whatever else the American Express developer community needs.
