Millions of people worldwide are turning to chatbots like ChatGPT or Claude, and a proliferating class of specialized AI companionship apps for friendship, therapy or even romance. While some users report psychological benefits from these simulated relationships, research has also shown the relationships can reinforce or amplify delusions, particularly among users already vulnerable to psychosis. AIs have been linked to multiple suicides, including the death of a Florida teenager who had a months-long relationship with a chatbot made by a company called Character.AI. Mental health experts and computer scientists have warned that chatbot mental health counselors violate accepted mental health standards.

As the technology’s ability to mimic human speech and emotions advances, researchers and clinicians are pushing for mandatory guardrails to ensure that AI systems cannot cause psychological harm. Clinical neuroscientist Ziv Ben-Zion of Yale University in New Haven, Conn., has proposed four safeguards for ‘emotionally responsive AI.’ The first is to require chatbots to clearly and consistently remind users that they are programs, not humans. Then, they should detect patterns in user…