In a nutshell: Roblox is unveiling an artificial intelligence system it claims can detect early signs of potential child exploitation in online chats, a move that comes amid mounting criticism and legal challenges over the platform's safety practices. The tool, called Sentinel, is free to use and will be open-sourced, allowing other platforms to integrate it into their own moderation systems.
Roblox, which reports more than 111 million monthly active users, says the AI has already helped flag hundreds of potential child exploitation cases to law enforcement. Matt Kaufman, Roblox's chief safety officer, told The Associated Press that while the company has long used "filters … for doing things like blocking profanity and blocking different types of abusive language," those protections were limited to "what is said in a single line of text or within just a few lines of text."
"But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time," Kaufman said.
Roblox says Sentinel is designed to detect patterns in conversations over time, rather than flag isolated words or phrases. The system analyzes one-minute snapshots of the roughly 6 billion chat messages exchanged daily on the platform, evaluating them in context.
To achieve this, engineers created two separate indexes: one containing examples of harmless chats, and another made up of messages that violated child safety guidelines. According to the company, new material is continuously added to both indexes to help the AI model improve over time.
"That index gets better as we detect more bad actors, we just continuously update that index," said Naren Koneru, vice president of engineering for trust and safety at Roblox. "Then we have another sample of what does a normal, regular user do?"
Koneru explained that the system monitors a user's ongoing activity to determine whether their behavior is trending toward safe interactions or risky conduct. "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," she said.
If the AI flags a user for closer review, human moderators examine the individual's full chat history, list of friends, and the games they've played. When necessary, Roblox says it escalates cases to law enforcement and the National Center for Missing and Exploited Children.
The platform's safety measures are being announced amid high-profile legal challenges. A lawsuit filed in Iowa last month alleges that a 13-year-old girl was contacted by an adult predator through Roblox, then abducted and trafficked across multiple states. The suit accuses the company of creating "easy prey for pedophiles" through its game design.
Roblox states that it prohibits the sharing of personal information, images, and videos in chat, and restricts direct messaging for users under 13 unless parents give explicit permission. Chats are not end-to-end encrypted, which allows the company to monitor conversations for safety violations.
The company maintains that no system can guarantee complete protection but argues that AI advancements like Sentinel significantly improve the odds of early detection.