DevOps transformation can require a major shift in organizational culture and "the way things are done." This can be difficult in any organization, but gets incrementally more difficult as the organization grows. When you reach the size of the U.S. government, implementing DevOps can seem insurmountable. But, it can be done.
For many organizations, especially a government agency, security controls can be a chief focus. This is one of the "reasons not to implement DevOps," because security professionals mistakenly believe that handing control over to others or automating the process will weaken security.
Done well, it makes applications more secure. But the hurdles can be high to implement the necessary changes, especially in a bureaucracy where process and hierarchy trump efficiency.
Janek Claus and Svetlana Yazhuk work for General Dynamics Information Technology, implementing DevOps for clients. Svetlana is a DevOps engineer supporting a platform for continuous deployment of containerized applications for a large U.S. government agency. Together, they share challenges overcome and the platform they are using during their presentation at last year's All Day DevOps conference.
Janek introduced several challenges to improving application security with DevOps. They are applicable to any large organization, with some specifics to the U.S. government.
Janek references a U.S. government report on the challenges of hiring and retaining enough security specialists, noting that 74% of agencies are "at risk" or higher, according to the report. Of course, this is true across many other industries and organizations. Governments also see recruiting and retention challenges, because employees want to work in modern environments. Training employees is also difficult, because - as we all know - it interrupts the work everyone wants to do.
Some solutions Janek covers include:
Bake automated security into the pipeline
Give developers and testers security tools
Use continuous security
Gamify training, such as code bashing
If you spent at least 10 minutes learning about DevOps, you heard about the need to breakdown silos in the organization. Silos separate teams and hand-offs complicate the process. Two solutions include:
Modeling your organization with the principles of Conway's Law - structure an organization that resembles your goals
Creating a platform team that offers DevOps-as-a-service
People naturally hate change. It is a law of nature. You can't change that, but you can show people the value of the change, so the desire to change overcomes natural resistance. Changing culture can be challenging, but it must be done to properly implement a DevOps culture. As the famed business strategist Peter Drucker said, "Culture eats strategy for breakfast."
What can you do to change the culture:
Ensure sustained executive management support and a long-term vision
Provide training, coaching sessions, etc. on tools, concepts, and concrete implementation steps, but ensure they are provided in a way that offers the least interruption to the normal workflow.
The Authority-to-Operate (ATO) is a specific term for the U.S. government, but the concept can be found across organizations. In essence, security needs to approve an application, or even a subset of a larger application, to operate on the system.
For instance, you might need to get an ATO for a library you want to use in your larger application. This can be a time and resource consuming process that can take months or even longer. It also hampers the availability of the best tools for the solution. If you can leverage cloud solutions or, even better, DevOps-as-a-Service, the ATO for these services can cover many of the items you need. This eliminates the need for separate ATOs for each application.
Don't forget that your DevOps tool chain needs to be secured too. With dozens of tools being used, each with their own privileges and credentials, the security risk grows. Employing secrets management and solutions that offer integrated credentials management across the DevOps pipeline helps ease the burden and tighten your security posture.
Janek notes these days that 80-90% of an application's code may not be your own code. Commercial and open source components help make application development efficient, but you need to take steps to ensure that code's vulnerabilities are managed continuously. Ensure you have tools that scan your libraries before they are used, and continuously monitor applications during development, testing, deployment, and production. Also, leverage existing open-source intelligence and use tools that inject human intelligence into scans.
Janek and Svetlana are implementing DevOps-as-a-service to solve the aforementioned challenges. It involves a team and platform to continuously increase the number and scale of self-service capabilities.
Svetlana dove into the platform, Monarch, during the presentation. Monarch is a platform that GDIT created for their government client. It is a self-service platform that development teams can use to build Docker containers and deploy them into AWS Elastic Container Service.
Currently, the monarch has:
Early vulnerability detection
Quick remediation response
Frequent updates in production are encouraged
Multiple parties involved in security auditing
Next steps including providing code quality monitoring via Sonarqube.
You can listen to Janek's and Svetlana's entire presentation, including a more detailed explanation of Monarch here.
Register now for the next All DayDevOps, November 6, 2019.
Photo: RawPixels