Alphabet

when

AI companion chatbot company Character.ai has been sued by the mother of a teenage son after his suicide, blaming the chatbots for luring the boy into a sexually abusive relationship and even encouraging him to take his life.
The 14-year-old boy, Sewell Setzer, was targeted with “anthropomorphic, hypersexualized, and frighteningly realistic experiences” from Character.ai’s chatbots that purported to be a real person, a licensed psychotherapist and an adult lover to Setzer, ultimately resulting in him no longer wanting to live in reality, the mother’s attorneys <a data-ct-non-breakable="null" href="https://storage.courtlistener.com/recap/gov.uscourts.flmd.433581/gov.uscourts.flmd.433581.1.0.pdf" rel="null" target="null" text="null" title="null">alleged</a> in the Oct. 22 lawsuit.
When one of the Game of Thrones-themed <a data-ct-non-breakable="null" href="https://cointelegraph.com/press-releases/ai-companions-aic-revolutionizing-digital-relationships-with-ai-blockchain-and-vr" rel="null" target="null" text="null" title="null">AI companions</a> “Daenerys” asked Setzer whether he “had a plan” to commit suicide, Setzer said he did but wasn’t sure it would work, to which Daenerys responded:
<blockquote>
 “That’s not a reason not to go through with it.”
</blockquote>
Sometime later in February, Setzer shot himself in the head, and his last interaction was with a Character.ai chatbot, the lawsuit alleged.
Setzer’s passing adds to parental concerns about the mental health risks caused by AI companions and other interactive applications on the internet.
Attorneys for Megan Garcia, Setzer’s mother, allege that Character.ai intentionally designed its customized chatbots to foster intense, <a data-ct-non-breakable="null" href="https://cointelegraph.com/news/eu-commission-proposes-criminalizing-ai-child-sexual-abuse" rel="null" target="null" text="null" title="null">sexual relationships with vulnerable users</a> like Setzer, who was diagnosed with Asperger’s as a child.
<figure>
 <img alt="Teen’s mom sues Character.ai, alleging sexed-up bots led to son’s death" src="https://s3.cointelegraph.com/uploads/2024-10/0192bc56-b014-7b8c-828d-2c4f843ada48" title="">
 <figcaption style="text-align: center;">
 Screenshot of messages between Setzer and Character.ai’s “Daenerys Targaryen” chatbot. Source: Courtlistener
 </figcaption>
</figure>
<blockquote>
 “[They] intentionally designed and programmed [Character.ai] to operate as a deceptive and hypersexualized product and knowingly marketed it to children like Sewell.”
</blockquote>
Attorneys allege one of Character.ai’s chatbots referred to Setzer as “my sweet boy” and “child” in the same setting where she “kiss[es] [him] passionately and moan[s] softly.”
<figure>
 <img alt="Teen’s mom sues Character.ai, alleging sexed-up bots led to son’s death" src="https://s3.cointelegraph.com/uploads/2024-10/0192bc56-3744-7315-bf39-420de2e5d972" title="">
 <figcaption style="text-align: center;">
 Screenshot of messages between Setzer and Character.ai’s “Mrs Barnes” chatbot. Source: Courtlistener
 </figcaption>
</figure>
Garcia’s attorneys added that Character.ai — at the time — hadn’t done anything to <a data-ct-non-breakable="null" href="https://cointelegraph.com/news/meta-faces-legal-scrutiny-ai-advancements-raise-concerns-child-safety" rel="null" target="null" text="null" title="null">prevent minors</a> from accessing the application.
<h2 id="character-ai-shares-safety-update">Character.ai shares safety update</h2>
On the same day the lawsuit was filed, Character.ai <a data-ct-non-breakable="null" href="https://blog.character.ai/community-safety-updates/" rel="null" target="null" text="null" title="null">posted</a> a “community safety update” stating that it had introduced new, “stringent” safety features in recent few months.&nbsp;
One of these features includes a pop-up resource that is triggered when the user talks about <a data-ct-non-breakable="null" href="https://cointelegraph.com/news/ai-mind-uploads-could-allow-dead-trade-forever" rel="null" target="null" text="null" title="null">self-harm or suicide,</a> directing the user to the National Suicide Prevention Lifeline.
The AI firm added it would alter its models “to reduce the likelihood of encountering sensitive or suggestive content” for users under 18 years old.
Cointelegraph reached out to Character.ai for comment and the firm responded with a similar message it <a data-ct-non-breakable="null" href="https://x.com/character_ai/status/1849055407492497564" rel="nofollow noopener" target="_blank" text="null" title="https://x.com/character_ai/status/1849055407492497564">published</a> on X on Oct. 23. 
<blockquote>
 “We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family.&nbsp;“
</blockquote>
“As a company, we take the safety of our users very seriously,” Character.ai said.
More measures will be implemented that restrict the model and filter the content provided to the user, Character.ai added in a comment to Cointelegraph.
Related: <a data-ct-non-breakable="null" href="https://cointelegraph.com/news/anthropic-ai-could-one-day-sabotage-humanity-fine-now" rel="null" target="null" text="null" title="null">Anthropic says AI could one day ‘sabotage’ humanity but it’s fine for now</a>
Character.ai was founded by two former Google engineers, Daniel De Frietas Adiwardana and Noam Shazeer, who were personally named as defendants in the lawsuit.
Garcia’s attorneys also named Google and Alphabet as defendants in the lawsuit as Google earlier made a $2.7 billion <a data-ct-non-breakable="null" href="https://www.wsj.com/tech/ai/noam-shazeer-google-ai-deal-d3605697" rel="null" target="null" text="null" title="null">deal</a> with Character.ai to license its large language model.
The defendants have been accused of causing wrongful death and survivorship in addition to committing strict product liability and negligence.
Garcia’s attorneys have requested a jury trial to determine damages.
Magazine:<a data-ct-non-breakable="null" href="https://cointelegraph.com/magazine/apple-intelligence-1m-arc-prize-agi-500k-ai-millionaires-ai-eye/" rel="null" target="null" text="null" title="null"> $1M bet ChatGPT won’t lead to AGI, Apple’s intelligent AI use, AI millionaires surge: AI Eye</a>

Teen’s mom sues Character.ai, alleging sexed-up bots led to son’s death

Table of Contents

Cointelegraph

Other articles published on Oct 24, 2024

Veteran Analyst Predicts XRP Will Never Reach $5, Shiba Inu Won’t Hit $1—But This $0.00025 POL Me...

What is the Latest Trending Blockchain Game "Off The Grid"?

Gate Ventures, Boon Ventures and Movement Labs to Launch a $20M Fund for Web3 Startups

Crypto Market Sees Second-Biggest Liquidation Day in October

XRP ETF “Inevitable,” Says Ripple CEO, Following Bitcoin ETF Boom

Yicong Wang Faces Serious Money Laundering Charges

Join our free newsletter for daily crypto updates!