Alphabet

when

AI companion chatbot company Character.ai has been sued by the mother of a teenage son after his suicide, blaming the chatbots for luring the boy into a sexually abusive relationship and even encouraging him to take his life.
The 14-year-old boy, Sewell Setzer, was targeted with “anthropomorphic, hypersexualized, and frighteningly realistic experiences” from Character.ai’s chatbots that purported to be a real person, a licensed psychotherapist and an adult lover to Setzer, ultimately resulting in him no longer wanting to live in reality, the mother’s attorneys <a data-ct-non-breakable="null" href="https://storage.courtlistener.com/recap/gov.uscourts.flmd.433581/gov.uscourts.flmd.433581.1.0.pdf" rel="null" target="null" text="null" title="null">alleged</a> in the Oct. 22 lawsuit.
When one of the Game of Thrones-themed <a data-ct-non-breakable="null" href="https://cointelegraph.com/press-releases/ai-companions-aic-revolutionizing-digital-relationships-with-ai-blockchain-and-vr" rel="null" target="null" text="null" title="null">AI companions</a> “Daenerys” asked Setzer whether he “had a plan” to commit suicide, Setzer said he did but wasn’t sure it would work, to which Daenerys responded:
<blockquote>
 “That’s not a reason not to go through with it.”
</blockquote>
Sometime later in February, Setzer shot himself in the head, and his last interaction was with a Character.ai chatbot, the lawsuit alleged.
Setzer’s passing adds to parental concerns about the mental health risks caused by AI companions and other interactive applications on the internet.
Attorneys for Megan Garcia, Setzer’s mother, allege that Character.ai intentionally designed its customized chatbots to foster intense, <a data-ct-non-breakable="null" href="https://cointelegraph.com/news/eu-commission-proposes-criminalizing-ai-child-sexual-abuse" rel="null" target="null" text="null" title="null">sexual relationships with vulnerable users</a> like Setzer, who was diagnosed with Asperger’s as a child.
<figure>
 <img alt="Teen’s mom sues Character.ai, alleging sexed-up bots led to son’s death" src="https://s3.cointelegraph.com/uploads/2024-10/0192bc56-b014-7b8c-828d-2c4f843ada48" title="">
 <figcaption style="text-align: center;">
 Screenshot of messages between Setzer and Character.ai’s “Daenerys Targaryen” chatbot. Source: Courtlistener
 </figcaption>
</figure>
<blockquote>
 “[They] intentionally designed and programmed [Character.ai] to operate as a deceptive and hypersexualized product and knowingly marketed it to children like Sewell.”
</blockquote>
Attorneys allege one of Character.ai’s chatbots referred to Setzer as “my sweet boy” and “child” in the same setting where she “kiss[es] [him] passionately and moan[s] softly.”
<figure>
 <img alt="Teen’s mom sues Character.ai, alleging sexed-up bots led to son’s death" src="https://s3.cointelegraph.com/uploads/2024-10/0192bc56-3744-7315-bf39-420de2e5d972" title="">
 <figcaption style="text-align: center;">
 Screenshot of messages between Setzer and Character.ai’s “Mrs Barnes” chatbot. Source: Courtlistener
 </figcaption>
</figure>
Garcia’s attorneys added that Character.ai — at the time — hadn’t done anything to <a data-ct-non-breakable="null" href="https://cointelegraph.com/news/meta-faces-legal-scrutiny-ai-advancements-raise-concerns-child-safety" rel="null" target="null" text="null" title="null">prevent minors</a> from accessing the application.
<h2 id="character-ai-shares-safety-update">Character.ai shares safety update</h2>
On the same day the lawsuit was filed, Character.ai <a data-ct-non-breakable="null" href="https://blog.character.ai/community-safety-updates/" rel="null" target="null" text="null" title="null">posted</a> a “community safety update” stating that it had introduced new, “stringent” safety features in recent few months.&nbsp;
One of these features includes a pop-up resource that is triggered when the user talks about <a data-ct-non-breakable="null" href="https://cointelegraph.com/news/ai-mind-uploads-could-allow-dead-trade-forever" rel="null" target="null" text="null" title="null">self-harm or suicide,</a> directing the user to the National Suicide Prevention Lifeline.
The AI firm added it would alter its models “to reduce the likelihood of encountering sensitive or suggestive content” for users under 18 years old.
Cointelegraph reached out to Character.ai for comment and the firm responded with a similar message it <a data-ct-non-breakable="null" href="https://x.com/character_ai/status/1849055407492497564" rel="nofollow noopener" target="_blank" text="null" title="https://x.com/character_ai/status/1849055407492497564">published</a> on X on Oct. 23. 
<blockquote>
 “We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family.&nbsp;“
</blockquote>
“As a company, we take the safety of our users very seriously,” Character.ai said.
More measures will be implemented that restrict the model and filter the content provided to the user, Character.ai added in a comment to Cointelegraph.
Related: <a data-ct-non-breakable="null" href="https://cointelegraph.com/news/anthropic-ai-could-one-day-sabotage-humanity-fine-now" rel="null" target="null" text="null" title="null">Anthropic says AI could one day ‘sabotage’ humanity but it’s fine for now</a>
Character.ai was founded by two former Google engineers, Daniel De Frietas Adiwardana and Noam Shazeer, who were personally named as defendants in the lawsuit.
Garcia’s attorneys also named Google and Alphabet as defendants in the lawsuit as Google earlier made a $2.7 billion <a data-ct-non-breakable="null" href="https://www.wsj.com/tech/ai/noam-shazeer-google-ai-deal-d3605697" rel="null" target="null" text="null" title="null">deal</a> with Character.ai to license its large language model.
The defendants have been accused of causing wrongful death and survivorship in addition to committing strict product liability and negligence.
Garcia’s attorneys have requested a jury trial to determine damages.
Magazine:<a data-ct-non-breakable="null" href="https://cointelegraph.com/magazine/apple-intelligence-1m-arc-prize-agi-500k-ai-millionaires-ai-eye/" rel="null" target="null" text="null" title="null"> $1M bet ChatGPT won’t lead to AGI, Apple’s intelligent AI use, AI millionaires surge: AI Eye</a>

Teen’s mom sues Character.ai, alleging sexed-up bots led to son’s death

Table of Contents

Cointelegraph

Other articles published on Oct 24, 2024

Smashing Records: BlockDAG Hits $100M in Presale While Avalanche Enhances Tech & Stacks Sees Pric...

Weekly Article Recap: 10/07-10/11

ApeChain Launches on Mainnet With Arbitrum Orbit

Vitalik Buterin Proposes New Gas Fee Regulation to Improve Ethereum’s Efficiency

TRON whales pick this hidden RWA token over PEPE, SHIB for long-term gains

President Bukele’s Bold $133k Bitcoin Donation to Build 1,000 Schools in Honduras

Join our free newsletter for daily crypto updates!