Last week I wrote about how important it is to pay attention to the security of the OSS projects you depend on. This isn’t just a one-time responsibility when you are trying to choose which component to depend on, this is an ongoing requirement. Even if you use the most secure OSS projects out there, if you don’t pay attention to security updates, it is all for nothing. Staying secure requires constant vigilance.

In this post, I’m going to talk about OSS project security. Since we’ve been paying a lot of attention to OSS security, I wanted to lay out some guidelines for evaluating an OSS project’s security. There’s a wide range of approaches to security from OSS projects: on one end of the spectrum, a one-person OSS project on Github won’t have a formal approach to security; on the other end of the spectrum, a project that is at the center of a billion dollar commercial ecosystem (like Apache httpd or Tomcat) will have a dedicated security team.

This post focuses on the secure end of the spectrum. Projects like Tomcat and Apache httpd that have dedicated security teams. Here are some of the baseline requirements for an OSS security teams. If you are maintain an open source project and you want to let the end-user know you take security seriously, you should consider starting a security team and following the guidelines in this post. If you are consuming open source, you should look for the following signs that this project has a mature approach to security in place:

• A low-volume general, public announcements list – Every OSS project should have a “announcements” list which only contains release announcements or critical security announcements, no more, no less. Having a low-volume announcement list and being disciplined about what you send to this list increases the value of having an announcement list. The best case is a project that has a separate security announcement list. Users can define filters and flag these messages as important guides for security. The worst case is a project that has a noisy list that is a mixture of discussions and announcements. Keep the noise out of security announcements.
• A private security list – If you run an open source project and someone notifies you of a vulnerability, you’ll want a private place to discuss the potential impact and any proposed fixes. If your project is especially large (Linux, httpd, Tomcat) you want to limit this list to a few trusted members of the project.
• One or more PGP Keys – This is a critical requirement, if someone identifies a security vulnerability in your software they need some assurance that the vulnerability report is being delivered to the right people. This is critical because (as a hypothetical) if I were going to compromise Tomcat, I might also attempt to compromise the accounts of the people who maintain Tomcat. Email is, in general, unencrypted over the public internet, you shouldn’t put anything sensitive into a plaintext email that you wouldn’t want broadcast to the entire world.
• How to Report a Vulnerability – Every project has different requirements, but if someone is reporting a vulnerability in a project like Tomcat, the security team will likely want to know some basic common details: what JVM was being used? What version of Tomcat was vulnerable? Is there any exploit code that can test the vulnerability? Also important, who else is aware fo the vulnerability? How long have you known about the vulnerability? Are you aware of any successful attacks using this vulnerability?
• A Description of the Security Process – This is especially important because Security teams are one part of an OSS project that is very opaque. While the public has visibility into almost all other aspects of a collaborative open source project, the security team is often working in secret to address identified vulnerabilities (possibly for months before they are generally known). To reduce friction between the transparency and the need for secrecy make sure that the public is aware of the security process. Consider retroactive transparency for discussions once the exploit has been published.

Here are a few examples of projects with mature security teams:

• SpringSource Security Team – http://www.springsource.com/security
• Apache Security Team – http://www.apache.org/security/ (Focuses mostly on APR and HTTP)
• Apache Tomcat Security Team – http://tomcat.apache.org/security.html
• Apache Struts Security Team – http://struts.apache.org/security.html

These projects have enough developers to have created a critical mass of both end-users and developers. All of these projects also have a strong commercial interest that can sustain continuous investment in a security team. As I’ve been surveying open source security, I’ve been impressed at the speed with which most open source projects react to security vulnerabilities. In general, projects that are attached to a respected forge (like Apache and Eclipse) are associated with a process and procedure for making sure that end-users have an interface to a security team. On the other hand, I see a very large list of projects that don’t present any interface for security other than a public developer’s list.

If we’re going to start taking application security seriously, every open source project should take the time to satisfy these minimum standards for presenting a secure interface to end-users.

In case you missed it, we published the results of our Developer Survey as a PDF. One of the things we did this year was post some comparisons to last year’s survey, specifically the changing attitudes toward OSS license compliance and policy. Here’s a statistic that caught my attention:

These two ends of spectrum – no standards vs. total lock down – had huge movement between 2011 and 2012, and I predict that we’re going to see the same sort of movement in next year’s survey. Open source compliance is top of mind for a few reasons, but I think that the trend can be explained by the timing of corporate adoption of OSS over the last decade and the average lifecycle of enterprise development.

My general sense about open source adoption is that it didn’t hit the mainstream for Java developers until the beginning of the last decade. 2001 saw an explosion of activity at Jakarta (Struts, Tomcat, Maven, Ant), each subsequent year showed a constant increase in open source usage (particularly in the Java space), but larger business didn’t really start moving toward wide-scale OSS adoption until the last half of the decade (Springsource and JBoss). While mainstream open source Java is more than a decade old at this point, larger business only made the jump to OSS five to seven years ago.

Couple this with average lifecycle of an enterprise application. Larger companies tend to invest in an application, architect a system and watch it mature over 5-10 years. This means that applications that were once relying on proprietary components are coming up for redesign just now. Every year there’s a new crop of applications coming up for redesign. Enterprises that embark on new application development now have a rich array of open source components to choose from and OSS in the business has matured to include rigorous compliance efforts.

Compliance is top of mind for businesses these days. With security incidents and IP litigation making front-page news almost every week, it is one of the first questions management asks when people are starting to use OSS software. What licenses are we using? Do we have a process for identifying our exposure to security risk? As a developer, you can decide to integrate tools like Nexus Professional 2.0 and take the lead in compliance reporting, or you can wait around for your lawyers to dictate your technology adoption process. This responsibility is still evolving and developers have an opportunity to choose to either lead through action or be led by someone else taking responsibility.

Read our survey results and you can draw your own conclusions.

Let’s talk about security. You may have seen that Sonatype released research on the security of some of the most commonly used open source components. To be honest, the results surprised me. However, now that we are aware of the realities, it’s important to be practical about this.

Join me for 30 minutes at 11:00AM EDT (GMT-0400) on Thursday, April 12, when I will be sharing some of our findings and my thoughts on how we can build a more healthy open source ecosystem.

Register here

Hope you can make it,

Jason

At RSA 2012, Wayne Jackson gave a short presentation focused on the security aspects of Sonatype Insight and the newly released Repository Health Check in Nexus Professional. This five minute overview gives you a sense of the magnitude of the problem we are trying to solve.

Here are some of the highlights from Wayne’s presentation followed by the video of his talk and his slide deck:

• “The benefits of ‘many eyeballs’ in open source does create better software but you can only leverage that if you know about it. That’s particularly troubling in the context of the fact that more than 80% of the modern software application is [comprised of] open source and the components that are used to build those applications are surprisingly complex.”
• “That complexity is compounded by the fact that when issues arise their implications are viral and the big problem is that when those issues are resolved in the root components the solutions are not [similarly viral] . Spring Beans 2.5.6 compromised 1400 open source components and God knows how many downstream applications. When Spring Beans 2.5.6 was fixed, none of the others were fixed.”
• “You can imagine the ripple effect of compromising open source. And the combination of things like the lack of notification infrastructure and the complexity of open source componentry is how you get situations like this. 6,982 organizations including the Dept of Homeland Security and several financial institutions are still using a 3 year old crypto library with an “as bad as it gets” Level 10 flaw that has known exploit code.”
• “Sonatype is creating an extraordinary infrastructure for finding out everything knowable about a given component. So that when flaws are discovered, we can know and we have the ability to deliver that knowledge into the tools that developers are using every day. This family of technologies is called Insight.”
• “Critical to that is the Central repository. Central houses hundreds of thousands of components from nearly every open source project in the world and it is used by tens of thousands of organizations.”

This article is another in a series of articles associated with our Executive Brief. To access the executive brief, “Addressing Security Concerns in Open-Source Components,” visit www.sonatype.com/securitybrief. You can follow the conversation on Twitter using the hashtag #OSSsecurity.

I just wanted to reiterate the key point of yesterday’s security brief which is: “You and everyone else in the world are likely downloading vulnerable components.” If you don’t believe me, then take a look at this graph:

First, note the logarithmic scale – downloads over an entire year. Then, take a look at the left-side of the chart. See anything familiar? GWT, Spring, Struts, CXF, Xerces? If you use these components, you should try to identify which versions are affected by widely known CVE vulnerabilities. It’s that simple, if you use these components it would be a good idea to browse the CVE database, or to take a look at Nexus Professional’s Repository Health Check.

Really, attackers aren’t going to go to the trouble…

Developers, you might be thinking, “an insecurity in GWT or Xerces, who’s going to trouble of doing that much research? Who’s really going to hack into Megabank via some obscure AJP vulnerability in a Tomcat connector?” And if you are asking these questions as a way to shuffle this all under the rug, I understand. There’s enough work in the pipeline already and you don’t need another thing to worry about. As developers we’re not going to turn into security professionals overnight, but we can start using tools like Nexus Professional to help identify vulnerable components and isolate us from deploying known security problems to production.

It isn’t the likelihood that someone will hack GWT that is the issue, it is the idea that deploying any code with a known security vulnerability needs to be identified as a disqualifier. The idea that if you get compromised and someone realizes that it was a known vulnerability (for years): developers need to be motivated to avoid this embarrasing situation. The point I’ve tried to make on this blog is that we (developers) are not really paying attention to this problem because we just assume that it is someone else’s problem.

Ignoring Security: It isn’t a question of if you’ll get hacked, it’s when

The issue of data and systems security has repeatedly been front-page news time and time again over the past year. Groups like Anonymous and Lulzsec made a public sport in 2011 of hacking into serious organizations and making every effort to embarrass and ridicule them for lax security. The last few years have been pretty embarrassing years for a lot of security departments at large corporations and a few governments. 2012 promises to be even more active with McAfee predicting the reorganization of Anonymous, but focusing on these high-profile, news-generating events ignores the scope of the problem. It isn’t about volume, it is about your exposure to this risk.

I’ve seen some recent attacks in action. Attacks on both Java-based web architectures and PHP-based architectures. While it’s true that PHP-based applications present a much larger and more insecure surface area to attack, it has to be said that Java-based web applications and .NET present a much more lucrative target. An attacker can compromise all the two-bit Drupal instances in the world without stumbling upon anything worth intruding, or they can focus on a multi-month strategy of social engineering and direct attacks to compromise one the Global 100 financial institutions that are downloading insecure dependencies every day.

Welcome to the Security Theater

If you are banking on the fact that attacking Struts 2 or Log4J is just too esoteric for most hackers to do, you are participating in something Bruce Schneier calls Security Theater, and that’s really what I’m taking away from this study. Some of these institutions are so invested in presenting an image of trust and security that they will spend millions on Super Bowl ads and marketing efforts to purchase customer trust. But, at the end of that day they continue to download vulnerabilities. It doesn’t match up, we need a change of culture in development and security needs to be top of mind.

It’s time for developers to start taking security seriously. You could choose to be proactive about the problem and use tools like Nexus Professional to automatically correlate CVE vulnerabilities from CERT with your artifacts, or you can wait until someone replaces your company website with a funny picture and lose the ability to download artifacts from Central altogether. The choice is yours.

Today we published a paper with Aspect Security, and it’s a shocking look at how few people are paying attention to application security. If you consume dependencies from the Central Repository and you don’t want to get hacked, I’d suggest reading the report and understanding some of the challenges, I’d also check out some of these statistics. Here are three that jumped out at me:

• Global 500 organizations downloaded more than 2.8 million insecure components in one year.
• Financial services firms are the most exposed: Global 100 financial services firms alone downloaded more than 567,000 insecure components in one year.
• 48% (a little under half) of organizations don’t have an inventory of Open source software used in production. (If there’s a new vulnerability discovered in something like GWT, who knows if we have that in production.)

To access the executive brief, “Addressing Security Concerns in Open-Source Components,” visit www.sonatype.com/securitybrief. You can follow the conversation on Twitter using the hashtag #OSSsecurity.

NOTE: Now, Developers, I know what you are thinking, you see the word “Executive Brief” and immediately dismiss this as C-level corporate-speak. Sure, there’s a little bit of that, but you’ll also learn how to own any unpatched Struts 2 application with a known vulnerability. If you use Struts, maybe you should read this report before your boss uncovers a vulnerability in your application?

Choosing components with appropriate licenses is critical to ensuring you realize the benefits and avoid the risks when developing with open source components. But, how well do you know your licenses?

• Can you describe the differences between permissive, weakly protective and copyleft licenses?
• Do you understand the ramifications of including copyleft licensed components in your commercial applications?
• Do you know how component dependencies affect your application’s licensing?

If you want to brush up on your knowledge, please check out our short paper on open source licensing available here.

With our launch of Insight, we’ve been talking to a lot of customers and prospective customers about effective management of open source-based development.  At this point, we’ve heard it all.  But some trends have emerged.  One thing is clear — virtually everyone wants to use more open source in their development processs, but realizes the need to effectively manage its use.  With thousands of components in use across their organizations, many people struggle with where to start.  With this in mind, we’ve put together a ‘top 10 list” to get things started.  You’ll find a summary of the entire list here.

We’ll be exploring each item in more depth through a series of five blog posts.   But for now, let’s start at the beginning with understanding your current usage of open source components.

1.  Understand where you are.

• Analyze your consumption to understand what and where components are being downloaded. By doing this you’ll know how open source components are used throughout your organization for development and which development groups are the heaviest users. Many IT organizations are surprised at the level of open-source penetration that has occurred “under the radar”.
• Identify problematic components currently being used in development projects. If your organization doesn’t yet have an open source policy, or at least one that’s followed, then it’s likely at least one group is inadvertently using components with license or security issues. Completing this review will give you a good idea of how well (or poorly) your organization is doing.
• Classify existing projects based on business importance to establish the role of OSS within the enterprise’s existing software portfolio. You’ll typically want to introduce new processes to a few projects at a time before rolling them out to the whole enterprise.  As any change can be disruptive, you may want to first try out your new procedures on less critical projects to avoid disruption. Once proven, you’ll want to quickly implement them on your most important projects to realize the benefits.

2.  Analyze your key production applications for security vulnerabilities and licensing issues

You need to first understand where you stand before you can make informed decisions as to what changes, if any, need to be made. We recommend the following steps:

Java dependencies are complicated and can introduce hidden risks

• Examine the complete bill of materials for your applications, not just first level dependencies. Java’s building block approach makes it fast and easy to build new applications using open source.  But at the same time, this makes it difficult to identify all the dependencies as these may be nested several levels deep. Unfortunately, what you don’t know could indeed hurt you as illustrated in the figure.
• Analyze new and existing applications  – it’s never too late to find and fix issues. Your existing applications, finished before you had a complete open source governance program in place, are likely to contain hidden risks.  New vulnerabilities are being discovered all the time, so even if your development team carefully vetted every component before choosing them, it’s likely that new issues have been discovered since you put the application in production.

While its possible to manually implement these tips, you’ll really want automated tools, especially if you are part of a large organization.  While you could cobble together your own tools, we recommend you check out Sonatype Insight, a set of tools and services we developed to help our customers address these very issues.

Once you implement these two tips you’ll have a really good idea of where you stand in terms of open source governance.   In the next post we’ll provide tips on how to start up your governance program.

Sonatype makes it easy to add your projects to the Central Repository with a free, public hosting service called OSSRH. We first blogged about this back in 2009, but given the growth in the community, we thought some of you may not have seen that post, so we decided to update it.

At Sonatype, we want to make synchronizing and publishing your artifacts to Central easier and to improve the quality of repository metadata for everyone at the same time. To facilitate this, we offer a dedicated instance of Sonatype Pro for Nexus at http://oss.sonatype.org specifically to host the artifacts of open source projects. In this post, I talk about the process of creating a repository for your open source projects and publishing artifacts so that they will be available from the Central Repository.

This service has been available since 2009 and includes many projects such as Plexus, Jetty, Google Guice, Spring and Ehcache (Greg wrote about his experience with migrating to oss.sonatype.org). We have tooling in place to make it easy for us to process a larger set of requests, so we invite everyone to use this resource. As of October, 2011, we have over 1,500 projects using this repository on a daily basis.

To get the process started, go here. We’ll setup a release and snapshot repository for your project, along with the appropriate configuration to allow you to use the staging features for your releases. If you have an existing repository somewhere, we can migrate that for you too. We’ll even help you add artifacts to Central that you use, but don’t necessarily own — assuming of course that it doesn’t violate the projects license.

The system allows customizable rules to be run during the staging process, which allows us to automatically check things like valid pgp signatures and correct POM parsing. This will ensure that your users have the best experience possible when using your artifacts, and relieve some of the manual validation on your side — a win for everyone.

On the technical details, this instance gets its network connection via Contegix‘s high availability network, the same one running Central, Codehaus.org and Atlassian.com. New Relic has donated monitoring services to help us monitor and tune this instance of Nexus. Since OSSRH is hosted on the same infrastructure as the Central Repository, we are able to frequently synchronize the repositories.

Next time you need to add a project to the Central Repository, you’ll know how.