The officer went to where students were hiding in the band room. He found the culprit – a student wearing a military costume for a themed dress-up day – and the “suspected weapon”: a clarinet.

The gaffe occurred because an artificial-intelligence-powered surveillance system used by Lawton Chiles Middle School mistakenly flagged the clarinet as a weapon, according to ZeroEyes, the security company that runs the system and contracts with Lawton Chiles’ school district.

Like a growing number of school districts across the United States, Seminole County Public Schools has turned to AI-powered surveillance to bolster campus security.

ZeroEyes sells a threat-detection system that scans video surveillance footage for signs of weapons or contraband and alerts law enforcement when they are spotted.

The appetite for such systems has grown in an era of frequent, high-profile school shootings – such as the attack at Brown University on Saturday that killed two students and injured nine.

Some school safety and privacy experts said the recent incident at the Florida middle school is part of a trend in which threat detection systems used by schools misfire, putting students under undue suspicion and stress.

“These are unproven technologies that are marketed as providing a lot of certainty and security,” said David Riedman, founder of the K-12 School Shooting Database.

Riedman was employed by ZeroEyes as a director of industry research in September 2023, and his employment ended that year, according to ZeroEyes and Riedman.

ZeroEyes said that trained employees review alerts before they are sent and that its software can make a lifesaving difference in averting mass shootings by alerting law enforcement to weapons on campus within seconds.

At Lawton Chiles, the student flagged by ZeroEyes was holding his musical instrument like a rifle, co-founder Sam Alaimo told the Washington Post.

“We don’t think we made an error, nor does the school,” Alaimo said. “That was better to dispatch [police] than not dispatch.”

Seminole County Public Schools declined to comment on Wednesday’s incident, but it provided a copy of the letter it sent to parents of Lawton Chiles students after the incident.

“While there was no threat to campus, I’d like to ask you to speak with your student about the dangers of pretending to have a weapon on a school campus,” principal Melissa Laudani wrote.

ZeroEyes spokesperson Olga Shmuklyer said its product is used in 48 states and that it has detected more than 1000 weapons in the last three years.

The systems are usually trained to detect a safety risk by reviewing volumes of sample data, such as images of people holding guns, to look for matches in real time.

They have sometimes made mistakes.

In October, parents and officials in Baltimore County, Maryland, called for a review of a different AI threat-detection system after it confused a bag of Doritos chips for a gun and sent an alert that led to a high-schooler being handcuffed.

In 2023, a high school in Clute, Texas, went into lockdown after ZeroEyes falsely alerted that a person was carrying a rifle, according to News 4 San Antonio.

In one case, a different threat-detection system failed to avert a fatal school shooting.

Antioch High School in Nashville was equipped with AI surveillance software to detect guns in January when a 17-year-old student killed a classmate in a shooting, according to CNN. The system missed the shooter because he was too far away from surveillance cameras to detect his weapon, CNN reported.

Other systems that monitor students’ activity on school devices have also been criticised for falsely accusing students and violating their privacy.

In September, students at a Kansas high school sued their school district after a monitoring tool falsely flagged art projects as pornography.

ZeroEyes has worked closely with Seminole County Public Schools since 2021, according to news reports and the company. That year, it held a live demonstration of the ZeroEyes system’s ability to detect guns at Oviedo High School.

“We’ve been very very pleased with the technology,” Seminole County Schools Public Safety Director Richard Francis told Fox 35 News at the time.

Alaimo said the company hires employees with military or law enforcement experience who are “calm under pressure and … very good at identifying guns” to review potential threats flagged by AI,

The image ZeroEyes flagged at Lawton Chiles showed the student appearing to aim his clarinet like a gun at a door and strongly resembled “a shooter about to do something bad”, Alaimo said.

The officer who responded to the alert questioned the student with the clarinet, according to the police report. The student said he was dressed as a military character from the Christmas movie Red One for the school’s Christmas-themed dress-up day.

The student said he was “unaware” he was holding his clarinet in a way that would have triggered an alert, according to the report. Police took no further action.

Chad Marlow, a senior policy counsel at the American Civil Liberties Union who has studied school security systems, said incidents like the one at Lawton Chiles show that systems like ZeroEyes can still be fallible, even with humans reviewing the threats that AI flags.

“If a computer technology is telling a … human evaluator that they see a gun and that literally seconds may be critical, that person is going to err on the side of saying it’s a weapon,” he said.

Amanda Klinger, the director of operations at the Educator’s School Safety Network, added that false reports risk “alarm fatigue” and dangerous situations if armed police respond to a school looking for a shooter.

“We have to be really clear-eyed about what are the limitations of these technologies,” Klinger said.

Alaimo said ZeroEyes – and its partners at school districts – would rather be safe than sorry.

“A superintendent, a school resource officer, a chief of police, a director of security, they’re going to say, ‘Yes, be more proactive, be more inclined to give me the alert if you have a fraction of a doubt,’” Alaimo said. “Because they want to keep people safe.”

Sign up to Herald Premium Editor’s Picks, delivered straight to your inbox every Friday. Editor-in-Chief Murray Kirkness picks the week’s best features, interviews and investigations. Sign up for Herald Premium here.