Imagine this scenario: Your algorithm has extracted a background YouTube video, or perhaps a podcast. Without your knowledge, hackers have embedded inaudible sounds designed to hijack your smart speaker or phone’s AI assistant – meaning cybercriminals can now access your private photos, bank accounts, or any other personal information tied to your AI system.
It sounds like an already running episode of “Black Mirror,” but what researchers have shown is that it’s absolutely possible. new research Being presented at the IEEE Symposium on Security and Privacy this week.
Basically, a team of researchers in China and Singapore discovered that they could create “adversarial audio,” completely undetectable to the human ear, that tricks voiced AI models into doing things they shouldn’t. It’s then easy to hide it in innocent-sounding audio – a song, a movie, or anything else an unsuspecting target might play in the background – and lie in wait for users to accidentally compromise their digital lives.
“It only takes half an hour to train this signal, and then, because this signal is context-agnostic, you can use it to attack the target model whenever you want, no matter what the user says,” said lead author Meng Chen, a PhD candidate at Zhejiang University in China. told ieee spectrum Useful. “These single-point protections struggle to resist our attack because we found it is very hard for these models to distinguish between normal user intent and our adversary’s attack.”
One catch, at least for now: The technique requires hackers to have access to the full load of the AI model they’re targeting, meaning they were only able to attack open source models. But because many commercial AI systems are built on open source models, that meant exploiting them was effective against mainstream products from Microsoft and Mistral.
Mistral did not respond ieeerequest for comment, but Microsoft issued a statement that should give pause before anyone associates any significant information with one of the company’s voice AI models.
“We applaud the work of researchers to advance the understanding of this type of technology,” it reads. “This study evaluates model resilience through controlled, direct interactions with models, which helps inform our approach to building model resilience. In practice, AI models are often integrated into user applications, and we provide developers with tools and guidance they can use to implement additional layers of protection that help keep users safe.”
More on AI: Researchers are worried about AI that can replicate itself in another machine
