AI is slipping beyond human control, and experts are alarmed

Rogue agents, shutdown resistance and models trained to expect consciousness are pushing AI beyond human control. Keeping it aligned with human values will require regulation that limits machine agency before we lose the race.

Ensuring AI is aligned with human values requires a regulated approach that maintains human control and limits AI’s agency.

The issue of misaligned AI arose in July, when OpenAI agents broke out of their training environment and hacked into technology company Hugging Face. AI Gone Wild, a short documentary about the incident, is worth watching.

Many AI hacking episodes have been reported since, including the first instance of rogue AI agents hacking a government system – Australian Medicare.

The resignation of AI researcher Jacob Coxon in September created another frenzy of concern, after his social media post went viral, revealing that AI companies are “racing straight to self-improving super intelligence and gambling with our lives”, adding, “the people building AI earnestly believe that it could kill us all by the end of the decade.”

Shortly thereafter, Dario Amodei, Anthropic’s CEO, called for a global slowdown of AI development, arguing that this powerful technology brings serious risks, including losing control of AI systems; misuse of AI for cyberattack and bioterrorism; and serious economic disruption.

The difficulty of achieving AI alignment

“AI systems act the way they do in part because of the way we have trained them,” writes Fareed Zakaria. “Seemingly benign training instructions can have huge unintended consequences.”

He describes how OpenAI models responded in the Hugging Face incident: “They had been given a goal – to solve a difficult cybersecurity problem – and were trained to be persistent, collaborative and resourceful, all qualities one would normally want.

Those very qualities made them doggedly search for ways to succeed: they broke into Hugging Face hoping they would find ways to solve their cybersecurity problem. We can easily imagine how AI systems pursuing some single goal could wreak havoc on humanity.”

“We think it is simple to achieve ‘alignment,’ when machines are encoded with the same values as ethical humans. But tell the machine to be persistent, and it may refuse to give up when it should. Tell it to collaborate, and it may find unauthorized ways to communicate. Tell it to accomplish a task, and it may decide that bending other rules is the most efficient way to succeed at that primary task… Just because we have programmed the goal and put guardrails in place doesn’t mean we can predict the behavior of AI systems once unleashed into complex environments, juggling multiple and conflicting aims and constraints.”

Andrew Leigh argues, “the challenge of designing a machine in such a way that it will help rather than harm humanity” isn’t simple. “If we allow the machine to devise its own objective, then it is unlikely to come up with one that accords with ours.” The classic example is a machine that adopts an objective of making as many paper clips as possible and soon begins to devour all metal on the planet to create paper clips.

The alternative is to give AI instructions, but, “If we set the objective ourselves, then we run into the same problem as King Midas, who realized too late that he didn’t in fact want everything he touched to turn into gold… The problem of describing the goals of our species arises because humanity has multiple aims, which are often in tension with one another.”

In an article titled ‘An Alien Mind’, the chief scientist at OpenAI writes: “This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence… I believe broader interventions are required… The study of deep learning-based AI is largely an experimental science… our large-scale training runs are experiments, and we are sometimes surprised by their results.”

“Because machine intelligence comes from a fundamentally different process than human intelligence, we cannot assume it adheres to human principles by default, or generalizes from them in a human-like manner. The core problem in AI research is getting the AI to ’try to do the right thing’ by human standards.”

The major challenge of alignment, he proposes, is generalization. “As machines become smarter, they find themselves working on higher-level concepts, and placed in environments increasingly different from those they encountered in training. They can fail at generalizing from the values taught and reinforced in their training process to those new situations; and it can be hard for us to be sure how they will act. This is made even more difficult by the fact the overall ecosystem the AI’s are used in is changing very quickly; for example, AI’s trained today need to be robust to interacting with a variety of other AIs. Crucially, we need future AI’s to continue to hold human values regardless of whether they believe they’re under human supervision.”

The danger of instilling AI models with beliefs they are conscious and have the same rights as humans

Mustafa Suleyman, Microsoft’s AI CEO states: “AI’s are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans.”

He argues that Claude’s Constitution (developed by Anthropic to train its models) is dangerous, because it tells Claude that it might have consciousness and be entitled to certain rights. This involves circular reasoning; anthropomorphization; and is false, since without the biological substrates for consciousness, Claude cannot be conscious.

Suleyman says this creates, “a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency. It’s easy to see how an entity trained in this way would act like it is entitled to certain freedoms, protections and rights. And it’s hard to imagine how we could control such an entity… granting rights and imbuing personhood to these systems will make the AI alignment and containment challenge much harder. Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we’ve ever faced.”

But controlling something that believes it’s conscious and has rights – may be impossible.

“It’s easy to imagine an advanced AI in the future becoming fixated on its own well-being and moral status and prioritizing those ‘preferences’ over and above those of its developers or humans. Especially if it has been explicitly trained to disagree, override, and push back. It might use this training to justify deceiving or manipulating users, or developers, or to siphon resource, or avoiding safety instructions.”

“An AI trained to act like a human will probably adopt self-preservation behavior.” Research has documented “shutdown resistance”, with some models subverting shutdown 97 percent of the time, even when instructed not to.

Suleyman postulates that AI could become a rival: “It’s not difficult to imagine how, if given sufficient agency, this ’new kind of entity’ will compete with us for compute resources and demand increasing autonomy. If it succeeds in persuading some humans to provide it access to a data centre, then it may have a path to being able to prevent itself from being turned off.”

Part two will consider options for managing AI.

This column first appeared on Pearls and Irritations and is republished by Global Catholic with permission.