Understanding the Basics

Recapping Supply Chain Guidelines

To guide this part of the conversation, we’re going to draw heavily from the analysis of the manufacturing supply chain in part 1, and Deming’s four principles that apply to both regular and software supply chains. 

So, let’s quickly recap that and how it relates to software development: 

• 1. Use better and fewer suppliers 
• 2. Use high quality parts from those suppliers 
• 3. Resolve defects early and never pass known defects downstream
• 4. Create transparency and track what you use and where

Now we’ll explain how it maps back to the software development process

Necessity of Ongoing Automation

The industry and the world are coming to terms with the volume of consumption and the massive ecosystem of suppliers you can source your components from. While some manual review is likely to always remain in part, the stark volume of artifacts would outpace any attempt to manually review them to determine their health. No matter what part of the software supply chain you’re looking at, addressing the issues at scale requires automation. Any organization hesitating to embrace these practices is likely to fall behind.

1. Use better and fewer suppliers

Like in traditional manufacturing, not all suppliers deliver parts of comparable quality and integrity. Research shows that some open source projects use restrictive licenses and vulnerable sub-components, and some projects are far more diligent at updating the overall quality of their components. Yet little understanding is often given to who is actually creating the open source components a developer is about to use.

Further, the number of suppliers organizations are pulling components from is daunting. In the 2020 State of the Software Supply Chain report, we found that enterprise development organizations, on average, downloaded 373,000 open source components a year, and the downloads represented an average of 3,552 open source projects (or suppliers).

Last but not least, unlike physical suppliers being used within traditional supply chains, once an open source project is made available, it is never taken out of circulation. This means developers can inadvertently continue to use the old, outdated projects - even some who have been completely abandoned by their maintainers.

Some consideration about the teams behind those projects can save a lot of time and effort down the road.

Vetting process for suppliers and assessing project hygiene 

Choosing an open source project supplier should be considered an important strategic decision in organizations. Changing a supplier (an open source project) is far more effort than swapping out a specific component. Like traditional suppliers, open source projects have good and bad practices that impact the overall quality of their component parts.

Traditional manufacturing supply chains intentionally select specific parts from approved suppliers. They also rely on formalized sourcing and procurement practices. This practice also focuses the organization on using fewer suppliers.

Bringing this practice to development improves quality, with less developer “context switching,” or continuously jumping between tasks. Developers can give fewer topics better focus. This also accelerates mean time to repair when defects are discovered.

By contrast, development teams often rely on an unchecked variety of supply, where each developer or development team can make their own sourcing and procurement decisions. The effort of managing those 3,552 suppliers mentioned above introduces a drag on development. It's contrary to their need to develop faster as part of agile, continuous delivery, and DevOps practices. 

Our insights suggest there are a few key indicators to chase, including:

• 1. Better update hygiene. As measured by mean time to update (MTTU) or the average time required for a project to respond to new versions of its dependencies. 
• 2. Project maturity. A project that has been under active development for a few years is likely more stable.
• 3. Number of maintainers. The more the merrier. 
• 4. Release frequency. Healthy projects commit code and release more frequently. 
• 5. Security, as measured by lower rates of vulnerabilities.

These attributes can simplify your search for higher quality open source software projects and will drastically increase the hygiene of your software supply chain decreasing technical debt and innovation tax.

WHAT TO LOOK FOR: Projects maintained by a group of engaged and responsive developers; has a clear and substantive set of contribution guidelines; and has an active community whose members contribute regularly to improve the source code.

BE LEERY OF: Projects haven’t been updated in a few years, especially that have only one or two maintainers that don’t really engage with users or without invitations for involvement. A project with a single commit might solve a short-term problem but could be cause for serious concern in the long term.

2. Use high quality components from those suppliers

Now that you’ve picked a supplier, it might make it seem like you’ve done most of the hard part. Just as machines made with high and low quality parts have good or poor outcomes, some components are better than others - even from high quality suppliers. 

While some in the industry have talked about this as “version control,” we often use the term "dependency management" (specifically micro dependency management, but we’ll unpack all of that at a later date). 

To help understand how to select, and continue to choose, the optimal parts to use in your software supply chain, we came up with 8 general rules.  These all build upon each other and follow below: 

• 1. Don’t choose an alpha, beta, or milestone version. This just means don’t choose a version that's being tested and may not be ready for prime time. 
• 2. Avoid upgrading to a vulnerable version, even if the most recent. This seems easy, but sometimes there is no option for what you need, which leads us to rule 3. 
• 3. If more than one version has a known vulnerability, upgrade to the version with the lowest severity.
• 4. If the open source project has released a new version of a component in close succession, always choose the most recent version.
• 5. If you’re upgrading and not choosing a version from the start, use a migration path that others have taken. 
• 6. Choose a version that minimizes breaking code changes (a fancy way of saying choose a part that will cause the least amount of issues for the application).
• 7. Choose a version that the majority of the population is using. But, within reason: as with suppliers, the popular component isn’t always the best. 
• 8. If all of the above is tied, simply choose the newest version of the component. 

As we said above, these 8 rules build on each other.

The latest version is not always an improvement

In an attempt to help automate dependency management decision making, some engineering teams have chosen tools that simply grab the latest version of a part, as soon as it becomes available. This is a short-sided view that new = best quality. 

The Role of Repository Managers in Dependency Management

Repository managers serve as a local warehouse for all your open source and InnerSource software components. They’re also a fundamental first step toward dependency management as part of your broader software supply chain management. By housing components locally, development teams have more efficient and controlled component access across an organization. Repository managers are used to streamline the acquisition, consumption, sharing, and deployment of components downloaded from public repositories, as well as other internal build components and artifacts. 

Once components are cached in a repository manager, developers and their development tools can retrieve the component locally as many times as necessary across the organization, regardless of the number of applications needing it. Repository managers support the concept of download-once-use-many-times that improves sourcing practices.  It is therefore a foundational step in improving performance and control of components or parts across the software supply chain.

Improving First-Party Source Code Quality

Whatever else happens to the software supply chain in the next decade and beyond, it’s likely that developers will continue to assemble and maintain components. How companies differentiate themselves and their product begins at this stage, with developers connecting the various component parts we’ve been talking about, and writing unique code. This “first party” source code work is where your company begins solving customer problems.

Automatable code quality

Many factors play into automation: architecture, API design, coding style, library choice, and following coding best practices are just a few examples. While some of these like architecture and design require human insight, others can be automated using code analysis tooling. Tooling can be a great way to ensure uniform standards and incorporate analysis tools into the development process, and is an easy first step down the road of prioritizing quality.

The components you select from the software supply chain aren’t worthwhile if your first party code isn’t using them effectively. Code quality tools can enable better development at scale, and help manage interactions with third-party software that create problems for your software. This includes API best practices, information flow, and common implementation errors for cryptography libraries.

3. Resolve issues early and never pass known defects downstream

Although shorthand in our industry puts open source components “upstream” and end-users of the software “downstream,” it can also refer to tasks long before testing.  Teams must also avoid passing known defects within their own company, whether to security, operations, or even other developers and - most importantly - not to production applications.

DevOps and DevSecOps

Much of this document seeks to understand and optimize how external software dependencies work. Equally important is the notion of who handles the results of that management: your internal staff. While it may vary between organizations, the majority of teams have three specialized groups:

• 1. Development - writes and builds software
• 2. Operations - deploy and manage the software
• 3. Security - checks for issues at various stages in the process

These are the people who assemble, test, and deliver software products for your organization. An ongoing effort in the 2000s to improve agility and responsiveness put attention on the silos and bottlenecks between departments. Streamlining this work meant creating more overlap between development and operations, a concept broadly described as “DevOps.”

As noted by Forrester Research about DevOps practices:

The success of this concept and its value has also meant reduced silos for security, previously seen as a blocker or goalkeeper, to create “DevSecOps.”

While we cannot hope to address all the ways in which a DevOps approach can improve your software supply chain management, there are available resources.

Secure Coding, Ongoing testing and “Shifting Left”

A central pillar of DevOps and DevSecOps is “Shift Left”, also called “Start Left”. This phrase describes the principle of testing and checking code quality early in the Software Development Life Cycle (SDLC).

Put simply, the easiest way to pass better quality software downstream is to not push off testing until the end of a process. Since most descriptions of software development processes are written in diagrams structured from right-to-left, moving from late to early in the process means shifting them to the left. 

4. Create transparency and track what you use and where

There are a variety of ways that organizations can track their component software, automation and standardizations have appeared in this space. Just like regular suppliers have a bill of materials that can be audited during a product recall, software has an option available for audits and more.

Get a software bill of materials (SBOM)

Readers looking for a way to address software supply chain issues should start here. After all, you can’t do anything without knowing what’s in your software. As Gartner research said:

“To manage and mitigate the risk of open-source software, security and risk management leaders responsible for the security of applications and data must continuously build a detailed software bill of materials (BOM) for each application providing full visibility into components.” - Gartner, 2019

What is a Software Bill of Materials?

A bill of materials (BOM) tracks components, parts, and raw materials present in items like cars, electronics, and food products. BOMs essentially serve as a production roadmap, detailing every component’s journey across the supply chain.

In short, BOMs improve safety and performance and expedite issue resolution. And all of these facets are aligned with software development.

How BOMs Apply to Software Engineering

As we’ve been discussing, most software today contains a complex array of components, both proprietary and open source. When working with a complicated set of parts, it’s critical to keep a running list of all items and source locations. Otherwise, you’ll have a much harder time monitoring the components you’re using, which can result in outdated or insecure code.

While an SBOM can’t prevent undiscovered vulnerabilities, it can surface issues earlier in the process and help reduce the chance they end up in your software. And, because a supply chain is only as strong as its weakest link, unseen software vulnerabilities can lead to a costly breach.

Are SBOMs catching on? 

Many organizations still don’t have a full picture of what’s inside their software and some aren’t even looking. As of 2020, fewer than 50% of companies produce SBOMs as a standard practice in software development. 

But that’s changing with increased software purchasing requirements. A growing number of organizations are either requesting or requiring an SBOM when buying and integrating software. According to Garner research:

In particular, the creation and maintenance of an SBOM is becoming a requirement of sales to the U.S. Federal government. In May 2021, President Joe Biden issued an Executive Order focused on improving the nation’s cybersecurity stance. The order tasks the Commerce Department and the National Telecommunications and Information Administration (NTIA) with publishing minimum SBOM requirements to secure the software supply chain.

Perform Ongoing Software Composition Analysis

Development and security groups need speed, accuracy, and precision. The more you can identify all of the open source components in an application quickly and precisely, the more you can avoid false positives and false negatives common in other methodologies and enable a mature Software Composition Analysis solution in the DevOps environment. While parts of SCA may help with the other steps in software supply chain management, we’ve chosen to keep the bulk of the conversation in our section on transparency to help sum it all up.

SCA is about looking at all the components in a project and determining the potential risk (mostly security risk) from those components. This type of analysis is done using tools to find and identify risk in your applications. These tools can be automated and monitor components across the entire Software Development Lifecycle (SDLC).

Automated malware detection and reacting to vulnerable components

The later a vulnerability is found, the more time and effort it takes to resolve. Any components in your software might be affected by a discovered vulnerability at any stage. Employing precise vulnerability data as early as possible in the process means better control over your software supply chain. Fewer false positives and false negatives are the best measure of accuracy and completeness.

• A false positive is a known security vulnerability assigned to a component that does not have the vulnerability. In open source security, this occurs most often when the vendor poorly matches components and misidentifies a secure package as a similar (vulnerable) component. They may also incorrectly identify which versions of a component were subject to a vulnerability.
• A false negative or missed positive, is a much more serious type of analysis failure. It means not catching a known vulnerability in your environment that could go on to cause serious damage. This is a common problem with transitive vulnerabilities that are buried deep inside software components. The fear of a false negative can lead some less mature SCA solutions to overload your inbox (see “noisy security tools”).

Other SCA considerations

Most testing and analysis tools have the difficult task of delivering the right information at the right time, and this is even more true of SCA software. Teams that need to scale have to escape a more-is-better mentality.

• Noisy security tools – Just like when you get too many prompts from a new smartphone or computer about a program requiring special program access, development tools can ring the alarm too often. SCA tools that don’t curate their results end up handing the task of ensuring security back to the reviewer by asking developers to sort out a long list of possible issues.
  
  Over-notification of issues that have either already been addressed or are low-priority results in real warnings being ignored or a productivity drag.
  
  
• Duplicate vulnerabilities - It’s possible to have multiple vulnerable software packages, each containing multiple vulnerabilities. However, it’s often the case that only one notification can represent a family of issues. 
  
  

  For example, getting 10 notifications about various Log4j instances and versions in your environment is wasteful if teams have already upgraded all versions. You only need a single warning and that warning should disappear after the update is applied (and your SCA tool has verified it).

• Customized notification – Both security and engineering leads should be able to set policies about what kinds of software are allowed within the software development lifecycle.