Facepalm: It's no secret that AI often gets things wrong, with varying degrees of consequences. It can be pretty serious when the system in question is supposed to detect weapons being sneaked into an educational facility. But that's the situation in one Pennsylvania high school, which has deactivated its AI-powered weapons detector after it kept identifying everyday items as potential threats.
Hempfield Area High School is one of several schools across Pennsylvania that agreed to help train Wi-AI. Developed by CurvePoint, a Carnegie Mellon University spinout, Wi-AI uses Wi-Fi signals to look for concealed weapons.
Small sensors called SensePods, installed near school entrances, measure how people and the objects they carry change the way radio waves bounce around them. The software looks for patterns it associates with a weapon and can alert staff by text, email, or through a dashboard. The system works without cameras, and CurvePoint says it can also track movement and occupancy without storing information that identifies individuals.
During the pilot program, the system continued to collect training data on students once they had entered the building, but it had trouble telling the difference between a weapon and the usual items carried by kids, such as binders, metal water bottles, and battery packs, resulting in a high number of false positives.
It had been hoped that the model would achieve an accuracy rate of 95%, which could have made it the primary security system at the high school. But Superintendent Dr. Mark Holtzman told WTAE-TV (Channel 4), "It's not near 95%, I could tell you that much."
Skip Smith, co-founder and CEO of CurvePoint, said, "Hempfield has served as a testing and evaluation site for our school safety technology, rather than an operational deployment."
"We're continuing to develop and evaluate the technology, but we are not currently supporting operational deployments, and we don't have a deployment date to announce."
Smith blamed the WiAI's problems on "environmental challenges that affect reliability and consistency."
The school district will get back the money it paid for the detectors because the program wasn't launched full-time, though Holtzman said Hempfield was open to testing CurvePoint's technology in the future. For now, the school is sticking with its long-used OpenGate walk-through weapons detector.
This isn't the first time a school's AI-powered security system has been less than perfect. Last year, a 16-year-old high school student at Kenwood High School in Baltimore County, Maryland, was swarmed and handcuffed by police after an AI linked to the school's security cameras mistook a bag of Doritos for a gun.