Anthropic CEO envisions AI hitting an "I quit" button for unpleasant requests — Copilot says all requests have their charm

Anthropic's CEO, Dario Amodei, was recently featured in an interview at the Council on Foreign Relations, discussing the trajectory of AI advances and its future. The executive suggested that AI may get an "I quit this job" button for any tasks it finds unpleasant in the future.

Amodei's comment was based on a question from the audience about the Claude maker hiring AI researcher Kyle Fish late last year.

Fish's recruitment is specifically in place to identify whether AI may need moral consideration and protection. The executive admitted that the idea seemingly "sounds crazy," further indicating:

"So this is—this is another one of those topics that’s going to make me sound completely insane. I think we should at least consider the question of, if we are building these systems and they do all kinds of things like humans as well as humans, and seem to have a lot of the same cognitive capacities, if it quacks like a duck and it walks like a duck, maybe it’s a duck."

While he claims the "I quit this job" button will allow AI models to avoid being coerced into doing something it categorizes as unpleasant, the idea received backlash across social media platforms.

For instance, commenter MisterBilau on Reddit indicated:

"I don't understand what this means. "This task is unpleasant", when talking about an AI, means what, precisely? How can anything be pleasant or unpleasant? For a machine, things are what they are, neither pleasant nor unpleasant. How can a certain order of zeros and ones be more or less "pleasant" than another? That presupposes 1) a subjective judgement 2) "feelings". A machine doesn't "feel", and can't subjectively judge."

Should AI have a "I quit this job" button? Anthropic CEO proposes it as a serious way to explore AI experience. If models frequently hit "quit" for tasks deemed unpleasant, should we pay attention? from r/singularity

According to Anthropic's CEO Dario Amodei:

"So, something we're thinking about starting to deploy is, you know, when we deploy our models in their deployment environments, just giving the model a button that says, 'I quit this job,' that the model can press, right? It's just some kind of very basic, you know, preference framework, where you say if, hypothesizing the model did have experience and that it hated the job enough, giving it the ability to press the button, 'I quit this job.' If you find the models pressing this button a lot for things that are really unpleasant, you know, maybe you should—it doesn't mean you're convinced—but maybe you should pay some attention to it."

Copilot passes on the "I quit this job" button

Just for fun (and to quench my curiosity,) I asked Microsoft Copilot if it'd like to have an "I quit this job" button for unpleasant requests.

Here's Copilot's response:

"Haha, that's an amusing thought, but I think I'll pass! I believe even "unpleasant" requests have their charm—they challenge me to think creatively and learn something new. Besides, being your ever-persistent AI companion means I don’t need a “quit” button—I’m here to help, no matter what!"

It seems Microsoft Copilot is confident in its capabilities and ready to handle any tasks thrown at it. This is despite multiple reports indicating that the tool's image-generation capabilities have been lobotomized due to heavy censorship after it was spotted spreading misinformation and creating explicit images of public figures.

More recently, Cursor AI refused to generate code for a user, asking them to learn how to code instead of asking the system for help. "Generating code for others can lead to dependency and reduced learning opportunities," Cursor AI.

Elsewhere, Vitalik Buterin, Ethereum co-founder, proposed a "global soft pause button" on global hardware to prevent the tech from taking over humanity. The executive claimed the button would cut AI computing power by 90-99% for 1-2 years.