*Note: Join us live and online for the 2019 Nexus User Conference on June 12.
"DevOps is Agile on steroids - because Agile isn't Agile enough."
So says Jim Bird, the CTO for BiDS Trading, a trading platform for institutional investors. Jim continued, "DevOps teams can move really fast... maybe too fast? This is a significant challenge for operations and security. How do you identify and contain risks when decisions are being made quickly and often by self-managing delivery teams? CABs, annual pen tests, and periodic vulnerability assessment are quickly made irrelevant. How can you prove compliance when developers push their own changes to production?"
Jim was presenting at the 2018 Nexus User Conference on continuous delivery. Pulling on his 20+ years of experience in development, operations, and security in highly regulated environments, Jim laid out how and why continuous delivery reduces risk, and how you can get some easy wins toward making it more secure.
As Jim points out, it reduces risk because:
Breaking work down and delivering it incrementally reduces delivery costs and risks
Continuous delivery takes care of the critical "last mile" of production
Change becomes routine
Change becomes cheap, because teams can afford to make smaller bets, iterate, and learn
Change becomes fast. Teams can quickly push fixes to respond to failures or security threats.
To ensure continuous delivery is secure without getting in the way of the delivery team, Jim outlines a few key points:
Understand the team's workflow and tool-chain - work the way they work
Address real risks and threats - problems that everyone agrees are important and relevant
Take an incremental, iterative approach
Provide useful, quick, and continuous feedback to the team and to management
Minimize friction and delays
Look for controls and checks that are easy and cheap to implement
With the case made, Jim provides five quick wins for continuous delivery.
Why? Because you often make changes, it is easier to make mistakes. You need simple, fast checks that find them before the bad guys. This makes auditors happy, because they can see that policies are being enforced.
How? Add tests/assets for common mistakes to the end of the automated build/deploy pipeline in test and production. This includes: default/missing credentials; SSL and HTTPS settings; known critical vulnerabilities; file and user permission, and open ports.
Tools and frameworks available include:
Netflix Security Monkey, Conformity Monkey
Gauntit
OWASP ZAP Baseline Scan
OSQuery
InSpec
Cloud services like AWS Inspector
Why? Teams are making decisions quickly, so you want them to do this safely and look for potential new risks upfront before coding starts. You get security/compliance stories into the backlog so they can be planned for, and so you can find the right tools, libraries, and frameworks. It also makes the risks visible to the teams and management.
How? Check out Mozilla's Rapid Risk Assessment. It's a good example.
Why? These prove vulnerabilities are being fixed correctly. They teach developers how to "step off the happy path" in their testing, and it will catch/prevent regressions. Normal tests prove it works, not that it might not work, and developers need to write happy tests, sad tests, and bad tests.
How? As you find vulnerabilities, write a test first, so you can prove it exists. Then, have developers fix it and then run the test again. They can use tools such as JUnit, NUnit, and Jasmine.
Why? Third-party code injects risk and security vulnerabilities and technical debt: known security vulnerabilities; abandoned and out-of-date code; and license risks. It is relatively easy, there is some lifting at the front, but once you do it, you have a nice control set going forward.
How? Get a tool to automatically catalog dependencies and check against vulnerability databases. Integrate it into the continuous delivery pipeline, and fail builds on serious problems.
Tools include:
Retire JS
Anchore Engine
Licensed
GitHub Security Alerts
Why? Automated static analysis scanners can catch common coding mistakes and bad practices. This is the only way to cover large code bases, and it backstops code reviews, because reviewers won't waste time on superficial and subtle coding problems. It fits naturally into workflows, and results can be directly fed back to developers.
How? Focus on high-risks and high-fidelity rules. Accuracy and speed are more important than completeness in continuous integration/continuous delivery.
Tools include:
Java - SpotBugs + FindSecurityBugs plugin + FBContrib plugin
.NET - Puma Scan, Roslyn Security Guard, and Microsoft's DevSkim
SonarLint IDE plugin for various languages
Ruby on Rails - Brakeman
Docker - Hadolint
Git Pre-commit checks to catch secrets, etc. on check in: Talisman, Yelp's pre-commit, git-secrets
Git tools to scan reports for secrets: TruffleHog, git-all-secrets
While Jim's quick wins reduce risks and get the ball rolling, he emphasizes that "deep, real change to culture and how people work takes a l-o-n-g time, patience, and reinforcement."