As part of our launch of Sonatype Nexus Repository 2.0 and the Repository Health Check, we're telling some stories about security and how security affects working developers. As developers, we're not always focused on security, but as attacks grow more complex, we're more aware of platforms like Java and .NET, and more capable of affecting custom application code, security will play an increasingly important role in development. In this post, I talk about a security incident I watched unfold last year, and identify some lessons I walked away from.
But first, a message from our sponsor: Sonatype Nexus Repository 2.0 offers a Repository Health Check (RHC). It's not an answer to security by itself, but it can play a critical role in a larger, organization-wide approach to security. If you are developing applications with Tomcat, Spring, and other well-known components, you'd be surprised at the vulnerabilities floating around in production. Once you can run a detailed RHC report in Sonatype Nexus Repository, you can remove these components from applications, iterate, and get rid of known vulnerabilities.
To Most Non-Technical People, Technology Is Magic...
Last year, I spent a lot of time dealing with a complex security incident. First, let me be clear, this incident had nothing to do with Sonatype. It was a company with which I shared office space, and to protect the innocent, I won't name the company. Let's just say this is a company that depended on the internet to keep working. If the site went down, the business stopped until the site came back up. It's as simple as that.
While the business relied on technology, none of the direct employees were very technical. They outsourced hosting and application development to contractors, and they didn't give technology another thought. To most non-technical people, technology is magic: websites just work, databases just sit there and "database" all day. If you start talking about different types of hosting servers and implementation languages, you can't expect most people to get into the technology.
Developers Are Flexible
That's why companies pay people like you and me. I'm a developer, I understand most things you'd expect a developer to know about a particular technology domain. For example, everyone who develops web applications should know at least the basic of TCP/IP and HTTP protocol. Basic stuff, things that you'd expect to be asked in an initial interview. Beyond that basic collection of facts we're all expected to know, there are extras. For example, some developers know a little more about operations than others, some developers can double as DBAs, and other developers do a good job pretending to be managers.
Developers are good at adapting, and of everyone in the IT department, we're the most likely to have multiple responsibilities or responsibilities that span job descriptions. As a good developer, you have to be able to understand abstractions and model processes, you have to be able to adapt. But, despite the flexibility of our profession, I've noticed one glaring gap - security.
Yeah, I Don't Know, We've Got Security People for That...
Back to this security incident. A web hosting company had deployed some seriously custom application logic full of problems. For starters, they created a website that didn't accept last names with apostrophes. "Why?," I asked. "Well, the web doesn't work apostrophes." Ignoring this was wrong, the SQL error produced by the last name "O'Brien" was also evidence of a SQL injection vulnerability. As a good neighbor, I notified the CEO of the company of the vulnerability and told him to ask his contractors to fix it.
Their response, "Yeah, our ISP has security people, I'm sure it's secure."
Weeks passed, and the site was finally hacked. Someone had found a way to perform a cross-site scripting hack using a vulnerable plugin. Overnight, my colleague's business went from making money to selling Viagra via a hacked Google Sitemap. Not just that, the machine was owned, repeatedly hit by several IP addresses in Russia and the Philippines, and it was triggering some Level 1 SMTP blacklists.
We're Not Security People, Do You Have Any Security People?
This is when it started to get fun. My neighbor, the CEO, initially got some feedback that everything was fixed and not to worry. A week passed, and the attack expanded. Instead of just a single page selling ED medication, the entire site was compromised. The server was now being flagged as a Level 2 SMTP blacklist threat (which means the entire ISP was being compromised). No customers were returning his calls, and the business was dead in the water.
While the application developers had identified several modified files in the application and were confident the problem had been isolated, my colleague had lost confidence in his developers. The fact that they initially sidestepped responsibility for security was what bugged him the most.
If You Don't Think You Are Responsible for Security, Think Again.
I wasn't working for either the company or contractor at the time, but I was there as a sort of second opinion, a sounding board, and the lesson I took away from the experience is that non-technical people don't care about the nuanced differences between developers, sysadmins, DBA, and security people. The other lesson to take away from this is that we're all security people. Unless you work behind seventeen national-security level firewalls, you'd better be focused on building secure systems.
You'd better not use plugins and software that have known vulnerabilities and the big secret, there's no such thing as "A security person." The big news flash here is that you are the security person, and if you are writing any code whatsoever, you should think about vulnerabilities and how to avoid them.
When you suffer through an attack (and it's going to happen, eventually), there’s no excuse. There's especially no excuse if you already knew about a vulnerability.
Today's Attacks: New Tricks Up Sleeves
Today's attacks are complex. They are "multi-modal." Instead of just compromising a single system-level component like BIND, you'll find that today's attacks start to involve custom application code and system-level hacks. That single web application compromised set the stage for an attack on other systems, because someone's SSH password was the same as a password stored, unencrypted in a MySQL database. This means that even if you clean up the affected application code, there's no telling what else was compromised. In this way, it's become impossible to clean up a compromised system. More often than not, the best answer when you suffer a complex attack is to start over (as crazy and impractical as that sounds).
That, I guess, was one of the biggest lessons I learned from watching this unfold. People are doing this these days, and they are taking advantage of the same levels of automation that we are. Everything is scripted and automatic. Once they find one way in, they'll compromise several vulnerabilities simultaneously. Once they leave so many back doors into your system, there's no securing it once it's broken.
Pay attention to security. Even if you don’t think it is your job, it is.
Tim is a Software Architect with experience in all aspects of software development from project inception to developing scaleable production architectures for large-scale systems during critical, high-risk events such as Black Friday. He has helped many organizations ranging from small startups to ...
Explore All Posts by Tim OBrienTags
Discover a Better Way to SCA
Forrester evaluated 10 SCA providers and recognized Sonatype with the highest possible scores. Learn why Sonatype was named a leader in Forrester Wave™ for SCA.