Meta said Thursday it will notify parents if their teen discusses suicide or self-harm with the company’s Meta AI chatbot. The company also said it is developing a way to contact emergency services if conversations suggest someone may be at imminent risk of suicide.
The changes came as Meta and other tech companies faced scrutiny from regulators and parents over how artificial intelligence chatbots respond to users in crisis, especially teenagers.
Meta said it built a dedicated AI system to identify conversations in which a teen makes a clear reference to hurting themselves.
“We understand how distressing these alerts may be for a parent to receive,” Meta wrote in a blog post. “That’s why, as we continue to improve our detection, all chats flagged by our AI will be manually reviewed before an alert is sent. If a teen’s intent is ambiguous, we’ll err on the side of caution and alert the parent. While that means we may sometimes notify parents when there may not be real cause for concern, we feel this is the right starting point, and we’ll continue to monitor to help make sure we’re in the right place.”
Meta said the alerts are live for parents using Instagram Parental Supervision in the U.S., U.K., Australia, and Canada, and will roll out globally by the end of the year.
Meta also said its “Limited Content” setting, which lets parents place teens in a more restrictive Instagram experience, now applies to Meta AI. Meta said the chatbot is already trained to avoid sexual, romantic, or alcohol-related conversations with teens, and that the setting broadens the range of prompts it will decline.
Meta also said it will contact emergency services if a conversation with Meta AI, involving either an adult or a teen, suggests someone is at risk of suicide. Meta said it already takes that step when posts on Facebook or Instagram suggest someone may be at risk.