On December 4th, 2014, U.S. Congressional Representatives Ed Royce (R-CA) and Lynn Jenkins (R-KS) introduced H.R. 5793, the “Cyber Supply Chain Management and Transparency Act of 2014.” The legislation will ensure all contractors of software, firmware or products to the federal government provide the procuring agency with a bill of materials of all third party and open source components used, and demonstrate that those component versions have no known vulnerabilities.
“As a house is only as strong as its foundation, it’s no wonder cyber attacks are on the rise with reports showing 71 percent of software contains components with critical vulnerabilities,” said Rep. Royce in a press release from his office. “This bill protects our nation’s cyber infrastructure by ensuring the building blocks that make it up are secure and uncompromised.”
In light of this new legislation, I thought it would be worthwhile to revisit a set of discussions I started earlier this year focused on changes in software development, the prolific use of open source components today, and our need to embrace software supply chain management principles. Here is Part 2:
A Closer Look at Today's Software Supply Chain
In part one of my blog, It's Just the Way Software is Made, I discussed the realities of how software is made, the birth of agile development, and the advent of component-based software development. Today, we will drive down the software supply chain to understand where your software is really coming from. I’ll also discuss why it’s important for us to instill high quality standards and governance policies in our “parts” ecosystem.
The Next Wave: A Fly In The Ointment of Supply Chain Management
Building quality cars requires sourcing high quality components. But the best quality components don’t simply arrive on the receiving dock and work as anticipated. Toyota understood this to an extreme. They placed expectations on their suppliers to deliver high quality parts, they tested those components to ensure they delivered the proper outcomes, and they empowered people on their production lines to report any concern or suggest any improvements to those parts or the assembly process. It was not just a policy that Toyota’s manufacturing executives put on paper, they made sure that the policy met practice all the way down to each worker on the production line, each component used in the cars, and every process undertaken to deliver their vehicles to market.
While software development practices have begun to mimic Toyota’s lean manufacturing, a closer look at software’s supply chain management practices, reveals a fly in the ointment. By comparison to Toyota’s supply chain management practices, modern software supply chain management is, for the most part, complete chaos. Developers are selecting open source software components (“parts”) based on arbitrary criteria, most likely whatever they or their buddy used the last time.
Human-Speed Monitoring Can’t Keep Pace With Development
The parts of organizations that are trying to assert control over the development practices (e.g., Open Source Review Boards, Enterprise Architects, Application Security teams) are doing so with manual review processes and paper-based policies that are too far removed or out-of-step with developers and the continuous delivery expectations many of them are driven by.
The sheer volume and variety of open source software components sourced into the typical software supply chain is enormous. What the review boards, risk and compliance groups, and application security teams are coming to realize is that their manual review practices, approval policies, and observations have no earthly hope of keeping pace with the volume, complexity, and pace of change in the open source ecosystem.
As U.S. Representatives Ed Royce (R-CA) stated when introducing the Cyber Supply Chain Management legislation, “This year, it is expected that open source components will be downloaded more than 21 billion times, for use in software applications. Half a dozen years ago, roughly one billion were downloaded. The scope of the issue of open source component supply chain integrity is becoming more important as open source component use in software development explodes.”
For example, if a developer selected the open source application server, WildFly, they would not only be using that artifact, but all of the dependencies that come along with it. In this case, that means relying on 754 unique open source components that make up the WildFly foundation.
Similarly, if a developer selected the Apache CXF services framework, they would be relying on 172 unique open source software components (including all of its dependencies). This means that any application developed might rely on hundreds or thousands of open source components -- with each of them updated an average of four to five times a year, in order to improve functionality, quality, and security.
Let’s bring the scenario more down to earth. Dealing with thousands of new open source components being sourced by a given developer each year would be analogous to every production worker on Toyota’s line sourcing each tiny car part they need with little governance over those choices. Imagine Toyota letting their assembly line workers choose whatever suppliers they prefer. Imagine Toyota letting them use outdated parts that were known to be defective.
No Chance for the Software Equivalent of a Recall
To my amazement, most organizations don’t even keep a bill of materials for the applications that they deploy. Without one, critical security (or performance, or stability, etc.) disclosures go ignored. There is no chance for the software equivalent of a recall, because the organization hasn't kept track of the parts (including version numbers) being used in production. While not perfect, the auto industry has clear visibility and records for every part used in their vehicles so they can organize industry-wide recalls when necessary.
U.S. Representatives Ed Royce (R-CA) is agreement along similar lines stating, “Just like when you find out your car’s brake lines need to be replaced, when an open source components are found to have a vulnerability or defect, it needs to be replaced.” The Cyber Supply Chain Management and Transparency Act of 2014, he and Lynn Jenkins (R-KS) introduced would allow those patches to be applied.
Today’s open source community can identify vulnerabilities in components and post notice of those issues in a central database, but the proliferation of communications, practices to stop using those faulty parts, and the ability to manually track down which components have been deployed is nearly impossible in today’s software industry. The word-of-mouth and manual practices used today, like those recently employed to alert businesses of the April 2014 Heartbleed Open SSL bug, are far from comprehensive and cannot keep up with the current pace of application development. Automation within the software supply chain management is long past due.
In part three of my blog, I will share a real world story on how thousands of corporate and government organizations sourced one “bad part” last year. Even worse, they sourced that open source component five years after the vulnerability had been disclosed.
Read Part 3 of Wayne's blog series.
