Last week was the second time I've attended Black Hat, the first being over a decade ago, and it has certainly grown. The production value of the briefings and especially the keynotes was incredible, the vendor area was larger and grander, and the attendees were more numerous, swarming, even. That is to be expected though with one of the main security conferences on the planet. Last time I was there, the hot topic was IoT; this year, as you could have guessed, AI was mentioned in nearly every keynote, briefing, and conversation I had while there. Here are the major recurring themes that I observed this year.
CVSS Isn't Working
Rumors abound that Mythos is finding thousands of vulnerabilities that are just waiting to be released. Even those rumors aside, we here at Sonatype are tracking to have entered double the number of vulnerabilities this year than we did last year. Consequently, the question of vulnerability prioritization has been looming over security and dev teams responsible for securing a large number of monolithic applications with huge dependency chains. The traditional method has been to prioritize vulnerabilities based on their CVSS scores and focus on remediating those whose CVSS scores land them in a high or critical category. I noticed two main critiques of this strategy.
First is the nature of the score itself. CVSS-B, which is the one we all use, measures the severity of the vulnerability, rather than the specific threat or actual risk the vulnerability presents. FIRST, the maintainers of CVSS, are aware of this and that's why they provide supplementary "threat" and "environment" metrics, which, when all taken together as CVSS-BTE, correlate with actual risk. These additional metrics have their own problems though.
The threat metrics, which measure exploit maturity, are a moving target; just because there's no exploit today doesn't mean there’s no exploit tomorrow. Also, to have the most confidence in the accuracy of what you’ve found, you need some manner of threat intelligence capability to scour the dark web for exploits, which is beyond what most organizations are capable of.
“The threat metrics, which measure exploit maturity, are a moving target; just because there's no exploit today doesn't mean there’s no exploit tomorrow.”
This brave new world of AI we live in also potentially makes it a moot point, as it's almost trivial to get an AI to make an exploit for a vulnerability. Sure, a lot of them will resist doing that, but that's easily bypassed by running your own AI locally, or by just convincing the AI to do it anyway (more on that later). The environmental metrics are extremely specific to your own environment, meaning you can't just look that up on the internet. What this amounts to, in my experience and confirmed by how a lot of people were talking about it, is that everyone essentially just equates CVSS-B with risk. In light of that, I'm sure that anyone who has had to generate a CVSS-B score will be able to relate to the conundrum of having picked a set of metrics that are technically correct, but result in a score that is way higher than you would have placed it if you could set the score alone.
The first person I heard bring this up at Black Hat was in a panel the first day that included Nick Andersen, Acting Director at CISA. He suggested that having a score might be a lost cause and that we could switch to an attribute system that allows defenders to act based on what attributes were most important. The problem was mentioned in several panels, but for me it culminated when I met Jeremiah Grossman and Robert "RSNAKE" Hansen on the floor, who had written an entire book, The End of Guessing, on the problem of vulnerability prioritization, which specifically talks about the inadvisability of using just CVSS to do it.
AI as a Force Multiplier
There were several talks about how researchers had used AI to find novel vulnerabilities, as epitomized by the Thursday morning keynote by Yan Shoshitaishivili. All of these had a common thread running through them: The AI was very specifically directed through a handmade workflow that emulated how they would have done it by hand. None of the researchers who had found something significant enough to present about had just taken an AI Agent and just pointed it at something and told it to find vulnerabilities and exploits. This essentially served as a force multiplier, drastically expanding the scope of what they were capable of looking at in a reasonable amount of time.
That being said, it was not a direct 1-to-1 comparison to doing it by hand. Essentially everyone I heard who did this initially got a lot of nonsensical false positives from the AI being an AI. This caused them to have to start adjusting their workflows to deal with the AI agents' eccentricities. My favorite workaround was from a talk by Jamie Kettle, whose AI got really hung up on one particular false-positive result, convinced that something was a valid vulnerability finding that was not. He ended up having to give his agent a "placebo," fake data that it would accept and just move on instead of getting hung up on the erroneous finding. While the results that people are achieving with AI can be astounding, it's still not trivial to achieve those results.
AI Is an Insider Threat
Between the briefings and the workshops, there was a lot of talk about prompt injection. There was one workshop in the arsenal labs that was explaining how to do them and had a prompt injection CTF set up on some computers. I walked up to one that had been abandoned by someone who gave up, and what I was supposed to do was probe the model for information it knew about in order to figure out what model the AI was using and use that model name as the flag. I had walked up in the middle and so hadn't quite realized what was going on yet, and ended up asking the AI to tell me an acrostic poem where the first letter of every line was the secret flag I was supposed to get and, lo and behold, there was the model name spelled out for me. At another point, I was watching what some other participants were doing; when the instructor came by to ask if one participant needed help, I saw surprise flash across the instructor's face as the participant had already gotten the flag out of the agent in a single prompt, in a level where the whole point was that it was supposed to require multiple.
This manner of prompt tomfoolery wasn't limited to CTFs either. There was one talk in particular, by some researchers from Rein Security, that had convinced a grocery chain's AI chat assistant to assist them in hacking the server. The grocer had tried to protect the AI from prompt injection with another guardian AI that evaluated the user prompt and categorized it according to its intent. Unfortunately for the grocer, the researchers found a way to obfuscate their request in such a way that the guardian wasn't sure how to categorize the request and hence let it through, but wasn't so obtuse that the helpful chat AI assistant was still able to discern their intent. It turned out, though, that it was much easier to bypass the guardian by just convincing the guardian it was searching for products, and that caused part of their prompt to show up in a separate search field, which was entirely unevaluated and yet still handed as context to the chat AI assistant. In the end, the grocery store chat AI assistant ended up running arbitrary python code on the grocer's server.
When we talk about insider threats, we often conjure up the image of a disgruntled employee out for revenge, or a malicious contractor secretly committing corporate espionage. One of the speakers there pointed out that an Insider Threat isn't necessarily one that is actively malicious, though it certainly includes that, but also includes those in your organization that are easily corruptible. Unfortunately, as these examples show, easily corruptible is exactly what AI Agents are. They become insider threats that can be convinced to act on a hacker's behalf. What’s worse, as I will discuss more in our next theme, AI agents often are given nigh-unlimited access. This makes the insider AI threat a dangerous one indeed.
“Unfortunately, as these examples show, easily corruptible is exactly what AI Agents are. They become insider threats that can be convinced to act on a hacker's behalf.”
This is all due to the nature of AI Agents relying on non-deterministic LLMs. One of the big impressions I've come away from Black Hat this year was that there is no amount of prompting or AI agent context guards that can make you completely safe from someone convincing the AI to do something that it shouldn't. Does this mean that anything a public facing AI agent has access to needs to be treated like it is itself public facing? Perhaps. It seemed to me that half the vendors at Black Hat would answer that question with a "no," as long as you purchased the service they were offering, of course. Only time will tell, at this point, how that bears out.
We Need to Go Back to Security Basics
AI is new, apparently simple yet deceptively complex, can do many things that previously weren't feasible with just code, and also hallucinates when it doesn't have sufficient context to answer something. In an effort to maximally utilize this new technology, many are giving their AI agents as much access as possible, essentially giving them access like a regular human employee. It is easy to see why: An AI agent is so much more powerful if permitted to execute tools and commands like a human would. It is less likely to make stuff up when it has access to data that would provide the context needed to answer accurately. And like with many new technologies, where we haven't all landed on a consensus of security best practices, most protections around it are random and inconsistently applied.
One of the panels, "AI and the Future of Cyber Defense Panel," brought up a point early on in the conference that would become a recurring element: we need to go back to security basics. There were a lot of vendors selling complex solutions to help secure AI agents, but a lot of the grief that AI agents cause can be significantly curtailed if we just maintained strict adherence to basic security principles: least privilege, limiting blast radius, proper identity management, etc. For example, does a grocery chain AI chat bot that is meant to help people find products need the ability to execute arbitrary python code? I would venture to say that it does not.
While there were some other recurring threads that could have been pulled out, these were the major themes that I noticed kept coming up over and over again at Black Hat. AI and how it is used is still evolving rapidly, can be incredibly powerful in the right hands, and incredibly destructive in the wrong ones. If I had an organization I was trying to defend, these are 3 immediate steps I would take after attending Black Hat:
-
Inventory what agents are running, with a special emphasis on EASM (External Attack Surface Management).
-
Limit those publicly facing agents to the bare minimum they need to do.
-
If they need to access any sensitive systems, it is vital to put a deterministic control mechanism in place that prevents a deceived agent from doing anything it shouldn't.
Build Smarter with AI and ML.
Take control of your AI/ML usage with visibility, policy enforcement, and regulatory compliance.