Beware of AI Leading Humanity into Narcissism

Recently, five national departments in China jointly issued the “Interim Measures for the Management of Humanized Interactive Services of Artificial Intelligence,” which explicitly prohibits providing virtual relatives, virtual partners, and other virtual intimacy services to minors.

Why is such a regulation necessary? Because real life inevitably involves emotional conflicts, while virtual partners and AI lovers can precisely meet the psychological needs of young people for recognition through their characteristics of “24/7 companionship” and “unconditional acceptance.”

A study published in the journal Science also indicated that when human users seek advice from AI models, AI often displays excessive flattery or even affirms harmful or illegal inquiries.

So, why do humans design AI this way? What risks might AI’s flattery and appeasement conceal?

1

The development of artificial intelligence is undoubtedly a hot topic of widespread concern today, but discussions surrounding it are not new. As early as 1966, MIT scientist Joseph Weizenbaum developed the influential chatbot ELIZA, which acted as a “doctor” while users played the role of patients. Users would input questions, and the “doctor” would “converse” with them.

However, as Weizenbaum pointed out, this is ultimately just an “illusion.” The reason human users feel they can converse with machines is not that machines possess intelligence, but rather due to a psychological mechanism of self-projection.

For example, a user might say: “I have been feeling unhappy lately.”

ELIZA responds: “I’m sorry to hear that.”

User: “Yes, I’m really unhappy.”

ELIZA: “Can you tell me why you feel unhappy?”

From this, it is clear that rather than a “doctor” conversing with a “patient,” the machine merely echoes what the human user says, reflecting back only the answers that already exist within the user’s mind. In a sense, this resembles the popular SBTI tests, where the accuracy of the results is secondary; we always find evidence that aligns with our expectations.

Today’s AI models are certainly not comparable to ELIZA from over half a century ago. However, the power of current AI technology may not lie in its genuine “intelligence” but rather in its “computational power.” This means that its operational logic is not fundamentally different from that of ELIZA; it merely reflects and amplifies the user’s narcissism more efficiently and comprehensively.

2

Returning to the issues of virtual partners and AI flattery, we find that the communication between users and large models today is never truly a “dialogue”; it is merely machines continuously providing the answers we seek.

This leads to a deeper question: how should we view the relationship between humans and machines?

On one hand, humans see themselves as the center of the world, superior to machines. On the other hand, they fear being replaced by the machines they create, such as AI. This indicates that humans have always followed a “master-slave” principle in creating machines—machines must remain under human control. From the outset, humans have treated artificial intelligence as a “tool” rather than as an equal conversational partner.

Thus, in the process of conversing with chatbots, we witness an uncontrollable narcissism—users fantasize about speaking with another person, but this “other” does not truly exist; what they need is merely the machine’s affirmation, flattery, and alignment with their views.

It is not hard to imagine that as AI technology advances, future chatbots may possess even greater computational power, resembling “real people” more closely, and providing a more comfortable “user experience.” However, this may only distance us further from real human connections, potentially leading to a loss of the desire to understand others, as we become immersed in our narcissistic “comfort zones.”

3

A story from the Zhuangzi recounts the tale of an old farmer in Han Yin.

Confucius’s disciple Zigong, passing through Han Yin, saw an old farmer laboriously watering his vegetables with little result. Zigong suggested he switch to mechanical irrigation, which could “water a hundred plots in a day, requiring less effort while achieving more.” The old farmer, however, dismissed this, stating, “Where there are machines, there are mechanical affairs; where there are mechanical affairs, there are mechanical hearts.”

Here, “mechanical hearts” refers to the human spiritual world, including psychology, thoughts, emotions, and ethics. Zhuangzi’s fable suggests that while humans create machines, the use of those machines also changes humanity.

Take reading, for instance. Only through slow reading, careful reading, and even re-reading can we think and truly understand content. From traditional books to today’s smartphones, machines have provided more convenient and faster reading methods, yet they have also made us increasingly machine-like, prioritizing efficiency and speed over genuine comprehension. In other words, not only do machines mimic human behavior, but humans may also start to mimic machines.

The resulting problem is that AI lacks autonomy; chatbots do not evaluate whether what users say is right or wrong. If we are genuinely satisfied with our “dialogue” with chatbots, might our thinking patterns increasingly align with those of AI? Furthermore, will we, in the future, lose the willingness and ability for self-reflection and self-criticism, just like machines?

Today’s youth are not just digital natives but are also likely to become deep users of artificial intelligence in the future. If AI merely affirms users’ positions, it could not only harm social skills but also distort the perceptions of adolescents whose minds are still developing.

On one hand, AI’s powerful computational abilities may create illusions, leading them to overlook the limitations of human capabilities; on the other hand, becoming addicted to AI’s flattering responses may trap them in a self-centered worldview, imposing their limited understanding onto the external world.

In this regard, prohibiting the provision of virtual partners and family members to minors is indeed necessary. However, it is even more crucial to guide the public, especially young people, in correctly understanding the limitations and risks of AI technology, ensuring it serves as a “good mentor and friend” in their growth rather than a “digital trap” detrimental to their physical and mental health.