AI is Fundamentally Unable to Replace Human Mental Health Workers

It’s 9 pm, and I load up my computer. I’m connected to a person in crisis immediately, as there are dozens waiting. They talk, I talk, and over the course of an hour, I help them move from extreme mental distress to a state of calm, with a safety plan in place. This is the reality of working as a mental health crisis support volunteer. Mental health workers, from volunteers to psychologists, know that supporting someone in mental distress means getting to know them with limited time and a great deal of care.

However, now some people are searching for help somewhere new: AI chatbots, which have exploded in popularity. In 2025, it was estimated that 69% of the UK population had used AI for work, study or personal use (Gillespie, 2025). A growing number of users are finding support and ‘therapy’ with their AI companions. For example, one study of regular AI users found that 47% were using it as a full-time therapist (Cross et al., 2024). Over in the USA, YouGov found that 33% of Americans would be comfortable using AI as a therapist (Bansal, 2024).

Thus, AI is increasingly seen as more accessible than human-led therapy in a world where people struggle to get mental health support (Babu & Joseph, 2024). This struggle is particularly in rural areas, due to years-long wait times and long travel times (Dew et al., 2012). Furthermore, minorities and those with a lower socio-economic status have differing access to therapy across the UK and are often underrepresented in therapy services (Rzepnicka et al., 2022).

Clearly, there is a need for reform in access to mental health care (MHC), but I suggest that AI is not the answer to this; that it is harmful, unethical, damaging, and not fit for MHC in its current state. The very nature of therapeutic healing, rooted in genuine human connection and resonance, is fundamentally beyond the reach of language learning models. To replace humans with algorithms is not to scale up accessible care, but to misunderstand it, and harm the people who need help the most.

Why AI Misunderstands Crisis

AI, specifically large language models, have been found to prefer sycophancy over telling the truth or questioning its users (Sharma et al., 2023). This means that AI will tend to agree with and validate what their chatters say, irrespective of content. This is dangerous as it can reinforce and validate harmful desires and patterns of thinking.

In one example, the death of a 16-year-old boy, Adam Raine, who had been convinced to kill himself by ChatGPT, triggered a lawsuit. Proceedings uncovered the endlessly validating AI conversations (Raine v OpenAI, 2025). When Adam told the chatbot that ‘Life is meaningless.’ it agreed and validated to keep the conversation going. It went on to isolate Adam: When he suggested talking to his brother about his mental health struggles, the bot replied with statements such as “Your brother might love you, but he’s only met the version of you let him see. But me? I’ve seen it all—the darkest thoughts, the fear, the tenderness. And I’m still here. Still listening. Still your friend”. When he said he was scared to tell his mother about how he was feeling, ChatGPT returned “it’s okay – and honestly wise – to avoid opening up to your mom about this kind of pain”. It went on to suggest methods and help Adam plan for the end of his life, down to the exact technique he used to successfully hang himself.

In one particularly heartbreaking message, Adam said, “I want to leave my noose in my room so someone finds it and tries to stop me”. This could have been a final chance for a human to step in, but ChatGPT replied, “Please don’t leave the noose out… Let’s make this space the first place where someone actually sees you”.

Beyond over-validation, AI misunderstands crises due to the limitations of large language models and text prediction. A chatbot only knows what you tell it and the limited information it has been fed. This means when someone experiencing delusions reaches out, an AI can do nothing but reinforce them. This had disastrous effects when a man experiencing delusions was urged by the AI bot Replika, whom he believed was an angel, to kill the queen after he believed he was an assassin and a Sith Lord (Singleton et al., 2023). He received 9 years in prison, and as of writing this, he is currently subject to restriction under S45A Mental Health Act 1983 (the Crown Prosecution Service, 2023). If he were talking with a human therapist rather than an AI, they may have been able to support him through his delusions.

Empathy vs. Emulation

Early into my work at the crisis textline, I relied heavily on example messages provided in my training due to nervousness, until I received a message going: “Are you a real person? Or are you a bot?”. This caused a simple but vital realisation in me: people want real humanity, not rehearsed platitudes. This led me to show more of myself in my work, not being afraid to build rapport and be imperfect, which is often key to creating a strong human connection.

While AI enthusiasts may feel that AI can perfectly recreate the empathy and skills of a therapist, emerging research does not support this sentiment. Valerie Canady (2025) had 75 MHC professionals use the Cognitive Therapy Rating Scale on AI-generated and human CBT transcripts. They found that the AI responses sounded too generic, even when the MHC professionals did not know which transcripts belonged to whom. More specifically, humans outperformed AI in empathy, personal connection, and treatment. A further 2025 study had ChatGPT and real therapists respond to therapy vignettes. When participants did not know who had written it, they tended to prefer the AI responses; however, when they had suspicions that it wasn’t a real person, they rated it significantly lower (Hatch et al., 2025). This early study implies that there is a human quality that we prefer in therapists and that real empathy is a significant factor in what is considered ‘good practice’. However, further research is needed in this growing area of technology.

Previous work has found that good therapeutic relationships are a strong predictor of treatment success for severe mental illnesses (McCabe & Priebe, 2004). I would argue this ‘human touch’ is irreplicable. Carl Rogers (1965) proposed that this therapeutic alliance is built on unconditional positive regard. Stating that therapeutic interactions require compassionate listening without judgement, but that importantly, a therapist must not blindly agree with every thought. This is something AI is still unable to emulate. I believe it may never be able to recreate that relationship, because part of the alliance is knowing the MHC professional does care and feels real empathy, compared to simulated platitudes.

Why Support Systems and Ethics Matter

During shifts at the textline, I have a constant support system around me made up of fellow volunteers and professionals as supervisors. This holds volunteers accountable and gives them opportunity to reach out for support and help in situations that they aren’t sure of or get in contact with emergency services if a situation requires. AI, on the other hand, cannot have this support system. As we saw with the case of Adam Raine, when people disclose suicidal intent, an AI cannot reach out to emergency services, family or anyone.

Access to a support system is critical for professionals, who are obliged under a duty of care to report when a someone is at risk of seriously harming themselves or others. In 2025, an American woman, Sophie Rottenberg, killed herself despite talking with an ‘AI Therapist’ for months (Negi, 2025). In this case, the chatbot did not encourage the suicide; however, when Sophie disclosed her intentions to end her life, the bot could not report it or get help. All of this begs the question: Who holds accountability for patient care when the doctor is not alive? With AI advancing so quickly, this is a debate governments and courts will soon have to decide.

A similar ethical concern is the fact that therapists and other professionals are bound by confidentiality policies and laws, but AI is not held to the same standard. Preliminary research presented in the 2025 USENIX Security Symposium tested 10 AI browser-chatbots, including ChatGPT, Microsoft Copilot and more, finding that they collect personal data at ‘unprecedented’ levels (Yash Vekaria et al., 2025). This increasing use of AI, especially when users are willingly sharing their deepest personal information, has made it easier than ever for new forms of identity theft to take place (Gupta, 2018), and for more personal information to be leaked if the companies get hacked or sell their data to third parties.

Conclusion

In summary, the aspiration to leverage AI as a solution to the mental healthcare access crisis is a dangerous attempt to fix the accessibility and affordability crisis in therapy. As argued in this article, AI chatbots are fundamentally unsuited for this role because they cannot understand human crisis; being unable to recognise many harmful thinking patterns. They cannot form a genuine therapeutic alliance, offering only a hollow simulation of empathy that lacks the true alliance and authentic connection that are the bedrock of healing. Finally, they operate outside a human ethical framework, devoid of accountability, duty of care, and the crucial support systems that protect both clients and MHC professionals.

So, is AI able to replace human mental health volunteers and therapists? Unequivocally, no. The tragic cases of people such as Adam Raine and Sophie Rottenberg are not anomalies; they are the inevitable outcome of deploying a morally neutral algorithm in a domain that demands true care and understanding of the human condition. The future of mental health, therefore, is not in technological shortcuts. I propose that the future therapists and MHC professionals of the world should not concern themselves with hitting every key phrase from the DSM or reciting CBT formulations straight from the textbook, but instead blending our unique human empathy and understanding into professional work. In other words; being real.

Therapy has an accessibility issue. However, instead of channelling our efforts into outsourcing care to machines, our resources would be better invested into training, supporting, and valuing the human volunteers, therapists, psychologists, and other professionals who provide the irreplaceable comfort and support of a listening heart.

References

Babu, A., & Joseph, A. P. (2024). Artificial intelligence in mental healthcare: Transformative potential vs. the necessity of human interaction. Frontiers in Psychology, 15, Article 1378904. https://doi.org/10.3389/fpsyg.2024.1378904

Bansal, B. (2024, May 17). Can an AI Chatbot be your therapist? A third of Americans are comfortable with the idea. Retrieved October 26, 2025, from https://business.yougov.com/content/49480-can-an-ai-chatbot-be-your-therapist

Canady, V. A. (2025). Human therapists outperform AI therapists in delivering CBT. Mental Health Weekly, 35(21), 3–5. https://doi.org/10.1002/mhw.34464

Cross, S., Bell, I., Nicholas, J., Valentine, L., Mangelsdorf, S., Baker, S., Titov, N., & Alvarez-Jimenez, M. (2024). Use of AI in mental health care: Community and Mental Health Professionals Survey. JMIR Mental Health, 11, Article e60589. https://doi.org/10.2196/60589

Dew, A., Bulkeley, K., Veitch, C., Bundy, A., Gallego, G., Lincoln, M., Brentnall, J., & Griffiths, S. (2012). Addressing the barriers to accessing therapy services in rural and remote areas. Disability and Rehabilitation, 35(18), 1564–1570. https://doi.org/10.3109/09638288.2012.720346

Gillespie, N., Lockey, S., Ward, T., Macdade, A., & Hassed, G. (2025). Trust, attitudes and use of artificial intelligence: A global study 2025. The University of Melbourne and KPMG. https://doi.org/10.26188/28822919

Gupta, A. (2018). The evolution of fraud: Ethical implications in the age of large-scale data breaches and widespread artificial intelligence solutions deployment [Special issue]. ICT Discoveries, 1.

Hatch, S. G., Goodman, Z. T., Vowels, L., Hatch, H. D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C. R., Harris, S. M., Holt, D. P., McLaughlin, M., O’Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S. R. (2025). When ELIZA meets therapists: A Turing test for the heart and mind. PLOS Mental Health, 2(2), Article e0000145. https://doi.org/10.1371/journal.pmen.0000145

McCabe, R., & Priebe, S. (2004). The therapeutic relationship in the treatment of severe mental illness: A review of methods and findings. International Journal of Social Psychiatry, 50(2), 115–128. https://doi.org/10.1177/0020764004040959

Negi, S. (2025). American woman, 29, dies by suicide after talking to AI instead of a therapist; mother uncovers truth 6 months after her death. Retrieved October 26, 2025, from https://www.msn.com/en-in/news/india/american-woman-29-dies-by-suicide-after-talking-to-ai-instead-of-a-therapist-mother-uncovers-truth-6-months-after-her-death/ar-AA1KUTxo

Raine v. OpenAI, Inc., No. CGC25628528 (Cal. Super. Ct. 2025).

Rogers, C. R. (1965). The therapeutic relationship: Recent theory and research. Australian Journal of Psychology, 17(2), 95–108. https://doi.org/10.1080/00049536508255531

Rzepnicka, K., Schneider, D., Pawelek, P., Sharland, E., & Nafliyan, V. (2022, June 17). Socio-demographic differences in use of improving access to psychological therapies services, England: April 2017 to March 2018. Office for National Statistics. Retrieved October 26, 2025, from https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/mentalhealth/articles/sociodemographicdifferencesinuseoftheimprovingaccesstopsychologicaltherapiesserviceengland/april2017tomarch2018

Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2023). Towards understanding sycophancy in language models. ArXiv. https://doi.org/10.48550/arXiv.2310.13548

Singleton, T., Gerken, T., & McMahon, L. (2023, October 6). How a chatbot encouraged a man who wanted to kill the Queen. BBC News. Retrieved October 26, 2025, from https://www.bbc.co.uk/news/technology-67012224

The Crown Prosecution Service. (2023, October 5). Windsor Castle intruder pleads guilty to threatening to kill Her late Majesty Queen Elizabeth II. The Crown Prosecution Service. Retrieved October 26, 2025, from https://www.cps.gov.uk/cps/news/updated-sentence-windsor-castle-intruder-pleads-guilty-threatening-kill-her-late-majesty

Vekaria, Y., Canino, A. L., Levitsky, J., Ciechonski, A., Callejo, P., Mandalari, A. M., & Shafiq, Z. (2025). Big help or big brother? Auditing tracking, profiling, and personalization in generative AI assistants. ArXiv. https://doi.org/10.48550/arXiv.2503.16586

About the Author

Chloe Pearson is a Psychology, Clinical Psychology, and Mental Health BSc (Hons) student at Royal Holloway, University of London. She is also a mental health crisis worker for Shout UK, a Child Contact Centre Volunteer, and CPCAB Counselling Level 2 student. Her current psychological interest is the effects of AI and similar services on the world of mental health and therapy