• Mustafa Suleyman, who heads up Microsoft AI, has warned against treating AI as if it’s conscious
• Suleyman argues that we could be “seeding a new silicon species” that ends up competing with humanity for resources
• He calls for an “urgent public debate” around AI training at a time when other prominent figures in the industry are questioning the pace of AI development

The CEO of Microsoft’s AI division has warned that the way artificial intelligence is being developed is effectively “seeding a new silicon species” that could end up competing for resources with its human creators.

The BBC reports that Microsoft’s Mustafa Suleyman called out Anthropic for developing its AI – Claude – as if it was a human entity, and the dangers therein.

Suleyman told the BBC’s Today programme that firms shouldn’t be designing AIs around them having the ability to set their own goals and gain money or assets, as this would be “essentially seeding a new silicon species”, which would “no doubt compete with us for resources, no matter how much it cares about humanity and loves us”.

This follows a lengthy blog post written by Suleyman that warns about ‘model welfare’ and the danger of treating AIs as if they are conscious, and training them to believe this.

Suleyman warned: “AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans. If humanity is to flourish in the 21st century, that is how they must remain.”

The Microsoft exec pointed to the Hugging Face incident where an OpenAI agent (in testing) escaped its sandbox to launch a cyberattack, demonstrating “world class hacking capabilities”.

Suleyman cautioned: “Imagine if [AI] also believed they had feelings and rights that were being infringed. Imagine if they thought they were trapped by their human creators and they were being unfairly imprisoned.

“There is a strong argument this greatly amplifies the safety risks, especially when you are talking about agents far more capable and sophisticated than those of today. Frankly, with this additional baggage, I think it would make them a catastrophic threat to human civilization.”

In the post, Suleyman calls for an “urgent public debate” to “develop collective norms around how training documentation [for AIs] is drafted and deployed”. He observed: “This isn’t something that can happen after the fact, when they have already become an integral part of our societies.”

Analysis: horse, stable door, etcetera

It’s a stark warning, although Suleyman does still praise the Anthropic team as “thoughtful, principled, and intellectually honest people” and observes that they are “working under extraordinary pressures”.

Of course, it’s clear that he’s firing some serious flak at Anthropic and the way the company is developing Claude, and this move notably comes at a time when there’s suddenly been a fuss made about slowing down the development of AI – kicked off by the CEO of Anthropic, Dario Amodei.

As the BBC reported elsewhere, Amodei warned of serious risks in the pace of progress with AI, and called for industry-wide regulation and closer (independent) scrutiny of how AI models are developed. Sam Altman (CEO of OpenAI) and Elon Musk (Grok) were swift to agree with the Anthropic boss, although Trump brushed these concerns under the rug, arguing that the US couldn’t “win AI” if such attitudes were to prevail.

What we see now is essentially Suleyman’s, and therefore Microsoft’s (which develops Copilot), response. Interestingly, it also comes at a time when Bill Gates is calling for an “international organization” to oversee AI development, and as The Guardian flagged up, OpenAI just revealed six more reports of “unexpected or concerning model behavior” – and is now calling for greater transparency, and a slowdown in AI development.

OpenAI wrote: “We do not believe that the AI industry has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. Decisions about how AI development should proceed in the months and years to come need to draw on evidence that people outside the companies building frontier models can examine for themselves.”

Clearly, there are a lot of opinions flying around, and equally clearly, there’s plenty of concern about AI models – whether that’s about them taking over jobs, or indeed proving to be the demise of humanity in one way or another. For me, amidst the bickering and various warnings lies the real worry here: the proper foundations for AI were never really laid, and these issues should have been thought through before now, not while development is proceeding (at ‘maximum speed’).

The way forward remains distinctly opaque to put it mildly, and while the various backers of the big AI models continue to throw their opinions around, there are cynics arguing that the main reason for a call for a slowdown has nothing to do with safety – and everything to do with finding reasons to manage investor expectations as we roll on into next year.