What is DevSecOps?

DevSecOps is an extension of DevOps that integrates security throughout the entire software development life cycle (SDLC) rather than only during the final stages.

So, what does DevSecOps mean exactly? The “Sec” in DevSecOps stands for security. The inclusion of “Sec” highlights the importance of security in an organization’s software development operations and the changing impact it will have.

For many organizations, implementing a DevOps mindset involves “bridging the gap” or “removing silos” between software development and IT operations teams, often aiming to release software faster and with greater stability.

DevSecOps is an extension of the DevOps mindset and is often presented with the tagline of “shifting security left” (i.e., earlier) in the SDLC rather than tackling security reviews at the end of the cycle, when any findings requiring mitigation are more complex and costly to implement.

The principle behind DevSecOps — push security considerations to the earliest stages of development — is a key tenet of The Second Way (Principles of Feedback) of DevOps as discussed in The DevOps Handbook:

“In complex systems, adding more inspection steps and approval processes actually increases the likelihood of future failures. The effectiveness of approval processes decreases as we push decision-making further away from where the work is performed. Doing so not only lowers the quality of decisions but also increases our cycle time, thus decreasing the strength of the feedback between cause and effect, and reducing our ability to learn from successes and failures.”

W. Edwards Deming, an engineer, professor of statistics, and management consultant often credited with the teachings that helped launch the Total Quality Management methodology in the United States, put forth the same idea as the third (of his 14) key management principles for transforming business effectiveness:

“Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.”

Principles do not necessarily equal practice

The Shift Left principle is not new, and it seems pretty straightforward in theory. But the reality is that in practice, many organizations are not operating this way.

If an organization prioritizes security, the aim is often to achieve the minimum criteria for some sort of compliance, usually with an understaffed security team.

In his DZone article “10 Tips for Integrating Security Into DevOps,” author and entrepreneur Gene Kim describes this challenge:

“The ratio of engineers in Development, Operations, and Infosec in a typical technology organization is 100:10:1. When Infosec is that outnumbered, without automation and integrating information security into the daily work of Dev and Ops, Infosec can only do compliance checking, which is the opposite of security engineering — and besides, it also makes everyone hate us.”

Kim’s novel The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win illustrates one telling example of this scenario. In the story, the not-so-popular binder-carrying Chief Information Security Officer (CISO) character, John, goes through an existential crisis when his company passes a SOX-404 audit without ever needing to rely on his process-heavy security controls.

After a period of wallowing and soul-searching that all his hard work wasn’t really adding value to the company, CISO John “rises from the ashes,” humbled, and seeks to understand how his role can help the company achieve its business objectives, working more cooperatively with the software development and IT operations departments to understand how he can help make their jobs easier.

Soon the Dev, Ops, and Infosec triad start working together to tackle those common business objectives and become much more agile in the process, adding automation to reduce effort and security risks wherever possible.

A cultural shift in both directions

One of the founders of DevSecOps, Shannon Lietz of Intuit, echoes that sentiment for a cooperative system of security in her “What is DevSecOps?” blog post on devsecops.org, saying “…with the change of DevOps afoot, traditional security is no longer an option.”

According to Lietz:

“The purpose and intent of DevSecOps is to build on the mindset that ‘everyone is responsible for security’ with the goal of safely distributing security decisions at speed and scale to those who hold the highest level of context without sacrificing the safety required.”

When you think of DevSecOps as a collaborative discipline, the key ingredient for a successful approach often comes down to a shift in the cultural mindset of the organization as you shift left.

The authors of The DevOps Handbook further support shifting left as they state:

“By doing this, we truly make quality everyone’s responsibility as opposed to it being the sole responsibility of a separate department. Information security is not just Information Security’s job, just as availability isn’t merely the job of operations.”

When you integrate security into all stages of your SDLC, you prioritize tasks such as the following:

Scan and evaluate the open source software (OSS) component risks in both new and existing legacy applications.
Block “bad” OSS components from ever entering your ecosystem in the first place.
Continuously monitor all applications in production, automatically alerting development teams when vulnerabilities arise that affect their applications.

The (increasingly scary) state of the software supply chain

(Open source) software is eating the world

In our current “software is eating the world” climate, developers are constantly “on the hook” to release software — with more and more complex features — faster than ever.

This mandate often involves reliance on OSS components. After all, why would you “reinvent the wheel” by writing your own proprietary code if there’s a freely available widely used component that provides the same functionality needed in your application?

In fact, OSS components make up more than 90% of most modern applications.

With open source becoming an increasingly prominent part of most software development, the role of developers is also shifting from the more narrow definition of “coder” to a broader one. This expanding role includes assembly and integration of code to both build software platforms and achieve business requirements tied to building platforms.

And the software development process itself begins to look more and more like a traditional manufacturing supply chain — a concept we at Sonatype termed “software supply chain,” illustrating the similarities.

Traditional manufacturing-based supply chains work with trusted suppliers who build components used in the final assembly of the finished product. In the case of a software supply chain, open source projects supply some of the most common components to be used in the application being developed.

Warehouses = Component repositories

Traditional supply chains streamline logistics and sourcing by using large, centralized warehouses to store and distribute their physical goods with more security and efficiency.

Similarly, software developers now often rely on sourcing OSS components from public repositories (with their corresponding programming languages) such as:

Maven Central (Java)
RubyGems (Ruby)
PyPI (Python)
npm (JavaScript)
NuGet Gallery (.NET)

Just as warehouses store and manage physical goods in traditional supply chains, code repositories serve as digital warehouses for software components in the software supply chain, enabling efficient storage, versioning, and distribution.

Manufacturers = Software development teams

To continue the metaphor, manufacturers are analogous to software development teams sourcing and consuming OSS components used as building blocks in applications.

Just as manufacturers source and consume raw materials to create their products, software development teams source and consume OSS components to build their applications. Software developers, like manufacturers, play a vital role in assembling these components and transforming them into full-fledged end products.

Assembled product = Software applications

Just as assembled products pass quality checks and validation before reaching customers, software applications undergo testing processes to ensure they meet desired standards of functionality, reliability, and security.

These tested final builds represent the culmination of the SDLC and the end products, ready to be made available to customers for their use. They serve as the tangible outcomes of the software supply chain, delivering value and addressing specific needs or requirements of the end-users, much like assembled products do in a traditional supply chain.

Open source breaches on the rise (and increasingly sophisticated)

Here’s where it gets a little scary. There is a common perception that using OSS components is safer, or they’re less likely to be compromised than proprietary software possibly because of the “many eyes” approach of contributing to and reviewing OSS projects.

However, at least one known security vulnerability was found in 29% of the most popular (the top 10%) Java, JavaScript, Python, and .NET projects.

In addition, the number of attacks via OSS components is increasing.

One example of a sophisticated attack via an OSS component would be the Spring4Shell zero-day vulnerability (CVE-2022-22965). Ilkka Turunen chronicled the story of the attack through one of the most popular open source Java frameworks.

Ilkka described how the sequence of events unfolded in spring 2022 as follows:

On a Wednesday morning, rumors appeared on the internet (via distributed blogs and Twitter screenshots) about a new remote code execution (RCE) flaw affecting the Spring Framework.
Later that day, the rumors were anecdotally confirmed when a proof-of-concept (PoC) exploit surfaced on an original GitHub repository, shared with a Sonatype researcher by a source. The PoC leveraged a previously unknown method to achieve RCE. Also present in the GitHub repository was a PDF with detailed exploit instructions.
Further investigation revealed the Spring4Shell zero-day vulnerability stemmed from a previously exploited issue (CVE-2010-1622) in Spring that was patched in the past. But an update to JDK9 caused the once-patched vulnerability to reemerge.
Spring released patched versions 5.3.18 and 5.2.20 as well as remediated versions of Spring Boot, and Sonatype recommended immediate upgrade. At the time, other known forms of attack also seemed to leverage running Spring applications on Tomcat. This showed that as understanding of the vulnerability evolved, so did forms of attack with increasing scope.

This vulnerability is very noteworthy as a concerted attack via open source because 60% of developers use Spring for their Java applications, so tons of applications were and still are potentially affected.

The Spring4Shell zero-day vulnerability serves as a prime example of an open source breach, with the exploit surfacing online and posing a significant risk to the multitude of applications that rely on the Spring Framework. As the understanding of such vulnerabilities evolves, it is crucial for developers to promptly upgrade their systems and remain alert to evolving forms of attack.

How are you managing your open source software components?

When integral security processes are at odds with developers’ mandates to accelerate the pace of software delivery, tension develops.

According to Enterprise Strategy Group (ESG), “companies use a variety of application security testing tools,” but the top challenges were personnel-based. Survey respondents listed top problems such as:

Developers lacking the knowledge to mitigate the identified issues (29%).
Developers not using tools the company invested in effectively (24%).
Security testing tools adding friction and slowing down development cycles (26%).

In the evolution of the software development community, high-friction legacy-infused processes have created a divide between security and development functions. What may be controversial, however, is to follow the chasm that’s been created down into the trenches, looking at potential (if inadvertent) outcomes that may creep in when legacy processes begin slowing down developers.

For example, if a developer has no easy automated way of choosing a “good” OSS component over a “bad” component, they’re forced to make a subjective choice.

If that developer happens to make a “bad” choice (in a “scan and scold” type of environment), will there be any incentive for them to try again? Have they been equipped with the information necessary to make an informed choice on their OSS component selection?

Or what if they happen to make a “good” choice instead? Why not stick to what worked, then? Is there any incentive to upgrade in that scenario without specific information around vulnerabilities in other component versions?

Finally, what if the approval process takes so long that the developer can’t do anything about it when the determination from security finally comes back? The rework required to choose a new component late in the game may be enough of a roadblock that they’re forced to stick with a “bad” component.

In this way, the lack of an integrated and automated open source governance strategy could inadvertently encourage developers to do the “wrong” thing (while stifling innovation).

According to Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, there is a better way:

“When building security into software is part of the daily work of developers, and when infosec teams provide tools, training, and support to make it easy to do the right thing, delivery performance gets better. Furthermore, this has a positive impact on security.”

The business case for DevSecOps

One of the most widely publicized open source breaches happened in 2017, when hackers gained access to millions of consumers’ personal information via the Equifax website, which relied on the popular Apache Struts open source web framework component.

It drew so many headlines because it affected so many people, and Equifax’s reputation suffered greatly.

Gaining (and keeping) customers’ trust is not an easy task for any company, and minimizing the risk of lost trust is at the top of many organizations’ priority lists in this climate of security breaches and privacy-related issues.

The average time to exploit continues to decrease dramatically, so organizations have a much narrower window to address these security breaches.

Unfortunately, the impetus for some organizations to prioritize security may only be in the wake of their own security breach. But wouldn’t a better strategy be prevention, so that customers’ trust — in addition to security — is never breached in the first place?

Security as a competitive differentiator

In her All Day DevOps talk, “Why Does Security Matter for DevOps?,” Caroline Wong discussed how security is not only becoming table stakes for enterprise-level, compliance-minded organizations, but also a competitive differentiator:

“For DevOps companies, securing software can actually become a competitive differentiator. So, security is a feature that’s marketed, and it promotes necessary trust with customers and partners. So if your peers and your competition are focused on application security and you’re not, you might actually risk falling behind.”

Development cost savings

It may seem counterintuitive at first, and even possibly at odds with developers’ abilities to maintain velocity, but integrating automated governance of OSS components earlier in the SDLC can translate to cost savings overall.

Consider the following as you decide whether or not to integrate automated open source governance:

Developers will spend less time researching and vetting OSS components if they have automated tooling that does much of this legwork for them.
When security-related code reviews happen at the end of the SDLC, any findings that involve re-work or technical debt incurred at that stage will be more costly in the long run than if the security issues were found earlier in the cycle.
If a security vulnerability is found in your application after it’s already in production, the costs associated with the “all hands on deck” approach to remediation can be astronomical.

According to the authors of Accelerate, remediation time is significantly reduced with the adoption of a DevSecOps workflow:

“We found that high performers were spending 50% less time remediating security issues than low performers. In other words, by building security into their daily work, as opposed to retrofitting security concerns at the end, they spent significantly less time addressing security issues.”

As bad actors increasingly “poison the well” of OSS components in software supply chains, your risks continue to increase as you seek to consistently:

Retain customer trust;
Remain ahead of your competitors; and
Keep costs from running out of control.

With these considerations in mind, the business case for DevSecOps shifts clearly into focus.