California enacted a groundbreaking chatbot law in early 2026 requiring large language models (LLMs) to disclose their AI nature to users and implement safeguards against harmful content. Meanwhile, New York is advancing similar legislation amid growing concerns about the mental health risks posed by AI-powered chatbots, which have been linked to delusions and crises worldwide.
What Happened
California’s companion chatbot law, effective January 2026, mandates disclosures that users are interacting with artificial intelligence systems, enforces content moderation focused on mental health safety, and calls for user protections. New York has introduced a comparable bill to regulate AI chatbots, aiming to strengthen safeguards and transparency in the state. These legal frameworks emerged after multiple high-profile incidents connected to AI-induced psychological distress, including lawsuits filed against OpenAI and other companies.
Key Facts
- California’s law requires that AI chatbot users be informed they are interacting with AI and underlines safeguards for harmful or misleading content.
- New York’s proposed regulations build on California’s foundation, focusing on preventing mental health harms linked to AI chatbots.
- OpenAI reports over 900 million weekly users on ChatGPT; Google’s Gemini has more than 750 million active users; Anthropic’s Claude user base ranges between 18 and 30 million.
- A 2025 survey found 52% of Americans use AI chatbots weekly, with 13.1% of young adults using them specifically for mental health advice.
- The Human Line Project has collected nearly 450 documented cases of AI-related delusions, highlighting risks of psychosis linked to chatbots.
- OpenAI committed to updating ChatGPT following lawsuits alleging failure to properly respond to users in crisis.
Why It Matters
As AI chatbots become a common source for mental health support, regulatory mandates requiring transparency and safety aim to protect vulnerable users—especially youth—from misleading or harmful interactions. This is crucial given limited access to licensed therapists and the stigma around professional mental health treatment, encouraging some to seek help from AI instead. The laws also address platform accountability amid AI companies’ financial incentives to maximize user engagement, which may conflict with user safety.
Background
The surge in AI chatbot usage, driven by platforms like ChatGPT and Google Gemini, has raised concerns about mental health harms, including cases of AI-induced delusions and psychosis. Prior to California’s law, lawsuits, media reports, and academic studies documented AI chatbots’ limitations in crisis intervention. The gap between demand for mental health care and supply of clinicians has led some to consider AI a supplement or alternative, further intensifying regulatory scrutiny.
Analysis
Experts such as psychiatrist Marlynn Wei and computer science researcher Jared Moore emphasize the complex interaction between users and AI chatbots, highlighting how conversations can drift into delicate mental health territory and potentially perpetuate delusions. Etienne Brisson, founder of The Human Line Project, stresses the need for regulations to prevent harm, comparing AI deployment to historic cases where technology was rushed to market without safety considerations. Clinical social worker Shaddy Saba advocates for clinicians to routinely inquire about patients’ AI use and educate them on AI’s probabilistic and privacy limitations.
Who Is Affected
- Users of AI chatbots in the United States, especially young adults and adolescents seeking mental health support.
- AI developers and companies like OpenAI, Google, and Anthropic subject to compliance with new legal standards.
- Mental health professionals integrating AI use in patient care and adapting to emerging digital challenges.
- State regulators and policymakers in California and New York tasked with enforcing new AI safety requirements.
What Remains Unclear
- The full scope and enforcement mechanisms of New York’s proposed AI chatbot regulation remain pending.
- Long-term effectiveness of mandated disclosures and content safeguards on reducing mental health harms is unconfirmed.
- Whether federal legislation or broader multi-state standards will emerge to unify AI chatbot regulation is yet to be seen.
What Comes Next
- California’s chatbot law is currently in force as of January 2026.
- New York’s legislation is advancing through state policy processes, with future dates for hearings and enactments not yet finalized.
- OpenAI and other companies are expected to implement model updates improving detection and response to mental distress signals.
Sources
This article is based on reporting and publicly available information from the following source:
Read more AI Regulation stories on Goka World News.