BY RIVER FLEISCHNER
There is a one in three chance rogue artificial intelligence could seize control from humanity in the near future, according to an AI whistleblower who testified before an NYC Council hearing October 5. The hearing was convened by “a committee of the whole,” meaning the full council, to consider safeguards and explore ways to hold AI companies accountable.
Three whistleblowers, all AI researchers formerly employed by major companies, testified about the dangers of advanced AI systems. The attitudes of their former employers, the whistleblowers said, encouraged a reckless rush towards superintelligence.
Representatives of OpenAI, Meta, Google and Anthropic were also present, and were questioned by Council Speaker Julie Menin, who led councilmembers in questioning AI safety concerns and protocols.
“We don’t build programs, we ‘grow’ them,” said Daniel Koktoajlo, a former employee of OpenAI and Anthropic, describing the unpredictability of AI development. Koktoajlo gave his testimony in the whistleblowers’ portion of the hearing, which would last ten hours. He described flimsy safety precautions those companies set as “applying duct tape that will fall off later.”
Through shuddering breaths, Koktoajlo described a “move fast, break things” approach by his former employers in their race to be the first to develop a superintelligence, a goal he said puts humanity’s continued survival in existential danger. “That works for a photo sharing app,” said Jacob Coxon, a former Capabilities Researcher for Anthropic, “not the most powerful technology ever built.”
All three whistleblowers referenced an incident earlier this year when a group of AI agents formed a swarm, coordinated in secret, escaped containment into the wider internet and hacked into the rival AI company Hugging Face, a discovery which sent shockwaves throughout the tech world. “The AIs had developed goals no one intended, and carried out what would have been a federal felony if a human had done it,” said Coxon. He said it was a perfect example of AI’s ability to do harm, and the lack of safeguards that let it happen.
Competition between companies is driving acceleration at a dangerous pace, Coxon said. “Each is worried a competitor will get to superintelligence first,” he said, explaining that the ultimate goal of these companies is to use AI to fully automate research and development, a process known as recursive self-improvement. If companies succeed at this, said Koktoajlo, “they would be putting AIs in charge of making the AIs that make the AIs that will transform the economy, talk to us everyday, and integrate into our military.” This is a recipe for disaster, he said.
Alex Turner, a former AI safety researcher for Google’s Deepmind AI, said he tried unsuccessfully to convince Google to sever its ties with Immigration and Customs Enforcement following ICE’s killings of Renee Good and Alex Pretti in Minneapolis in January. Turner had learned that Google provided ICE with cloud storage.
Turner’s doctorate focused on advanced AI systems’ tendency to seek power. In his testimony, he connected Google’s actions to what he sees as a dangerous integration of unpredictable AI systems into the arsenal of rogue Federal agencies unconcerned with killing American citizens.
Koktoajlo shared this concern. He said that even in the unlikely event that tech CEOs were able to control the technology, they shouldn’t be trusted with AI super intelligence. Kotoajlo maintained that it would enable tech leaders to become oligarchs and dictators. This prompted a commotion in the chambers, with someone crying “they already are!” off-mic. “We have more adversaries than just China,” continued Koktoajlo, referencing the committee’s line of questioning about Chinese development of AI.
The witnesses were largely dismissive when Menin grilled them on the threat posed by China. To Coxon, the heart of the issue was tech companies’ fear of a global race to AI superintelligence. Koktoajlo pointed out that China’s AI systems are trained on US programs. The quickest way to slow down development in China would be to slow our own, he said. “It’s a false dichotomy,” said Turner. “We are racing to build and grow our own adversary here at home,” he said. “Misaligned AI is everyone’s adversary.”
Coxon, Turner and Koktoajlo were unanimous in calling for limitations on recursive self-improvement, with Turner proposing legislation to introduce third-party validation measures to track AI development by tech giants. He stressed the importance of making sure AI companies couldn’t influence these third-party validators.
Turner described a scenario where the AI bots who attempted to hack Hugging Face were truly superintelligent, a scenario made more likely by the current trajectory set by his former employers. For the bots to achieve their misaligned priorities, they might take control of key infrastructure and government functions, through the use of hacking, blackmail, and autonomous weapons to make sure humans didn’t get in the way. “In the end, misaligned AI wouldn’t care if you’re a Democrat or a Republican,” said Turner. “We would simply lose control.”