Schools, parents, police, and existing laws are not prepared to deal with the growing problem of students and minors using generative AI tools to create child sexual abuse material of their peers, according to a new report from researchers at Stanford Cyber Policy Center.

The report, which is based on public records and interviews with NGOs, internet platforms staff, law enforcement, government employees, legislators, victims, parents, and groups that offer online training to schools, found that despite the harm that nonconsensual content causes, the practice has been normalized by mainstream online platforms and certain online communities.

“Respondents told us there is a sense of normalization or legitimacy among those who create and share AI CSAM,” the report said. “This perception is fueled by open discussions in clear web forums, a sense of community through the sharing of tips, the accessibility of nudify apps, and the presence of community members in countries where AI CSAM is legal.”

The report says that while children may recognize that AI-generating nonconsensual content is wrong they can assume “it’s legal, believing that if it were truly illegal, there wouldn’t be an app for it.” The report, which cites several 404 Media stories about this issue, notes that this normalization is in part a result of many “nudify” apps being available on the Google and Apple app stores, and that their ability to AI-generate nonconsensual nudity is openly advertised to students on Google and social media platforms like Instagram and TikTok. One NGO employee told the authors of the report that “there are hundreds of nudify apps” that lack basic built-in safety features to prevent the creation of CSAM, and that even as an expert in the field he regularly encounters AI tools he’s never heard of, but that on certain social media platforms “everyone is talking about them.”

The report notes that while 38 U.S. states now have laws about AI CSAM and the newly signed federal Take It Down Act will further penalize AI CSAM, states “failed to anticipate that student-on-student cases would be a common fact pattern. As a result, that wave of legislation did not account for child offenders. Only now are legislators beginning to respond, with measures such as bills defining student-on-student use of nudify apps as a form of cyberbullying.”

One law enforcement officer told the researchers how accessible these apps are. “You can download an app in one minute, take a picture in 30 seconds, and that child will be impacted for the rest of their life,” they said.

One student victim interviewed for the report said that she struggled to believe that someone actually AI-generated nude images of her when she first learned about them. She knew other students used AI for writing papers, but was not aware people could use AI to create nude images. “People will start rumors about anything for no reason,” she said. “It took a few days to believe that this actually happened.”

Another victim and her mother interviewed for the report described the shock of seeing the images for the first time. “Remember Photoshop?” the mother asked, “I thought it would be like that. But it’s not. It looks just like her. You could see that someone might believe that was really her naked.”

One victim, whose original photo was taken from a non-social media site, said that someone took it and “ruined it by making it creepy [...] he turned it into a curvy boob monster, you feel so out of control.”

In an email from a victim to school staff, one victim said “I was unable to concentrate or feel safe at school. I felt very vulnerable and deeply troubled. The investigation, media coverage, meetings with administrators, no-contact order [against the perpetrator], and the gossip swirl distracted me from school and class work. This is a terrible way to start high school.”

One mother of a victim the researchers interviewed for the report feared that the images could crop up in the future, potentially affecting her daughter’s college applications, job opportunities, or relationships. “She also expressed a loss of trust in teachers, worrying that they might be unwilling to write a positive college recommendation letter for her daughter due to how events unfolded after the images were revealed,” the report said.

In 2024, Jason and I wrote a story about how one school in Washington state struggled to deal with its students using a nudify app on other students. The story showed how teachers and school administration weren’t familiar with the technology, and initially failed to report the incident to the police even though it legally qualified as “sexual abuse” and school administrators are “mandatory reporters.” 

According to the Stanford report, many teachers lack training on how to respond to a nudify incident at their school. A Center for Democracy and Technology report found that 62% of teachers say their school has not provided guidance on policies for handling incidents

involving authentic or AI nonconsensual intimate imagery. A 2024 survey of teachers and principals found that 56 percent did not get any training on “AI deepfakes.” One provider told the authors of the report that while many schools have crisis management plans for “active shooter situations, they had never heard of a school having a crisis management plan for a nudify incident, or even for a real nude image of a student being circulated.”

The report makes several recommendations to schools, like providing victims with third-party counseling services and academic accommodations, drafting language to communicate with the school community when an incident occurs, ensuring that students are not discouraged or punished for reporting incidents, and contacting the school’s legal counsel to assess the school’s legal obligations, including its responsibility as a “mandatory reporter.” 

The authors also emphasized the importance of anonymous tip lines that allow students to report incidents safely. It cites two incidents that were initially discovered this way, one in Pennsylvania where a students used the state’s Safe2Say Something tipline to report that students were AI-generating nude images of their peers, and another school in Washington that first learned about a nudify incident through a submission to the school’s harassment, intimidation, and bullying online tipline. 

One provider of training to schools emphasized the importance of such reporting tools, saying, “Anonymous reporting tools are one of the most important things we can have in our school systems,” because many students lack a trusted adult they can turn to.

Notably, the report does not take a position on whether schools should educate students about nudify apps because “there are legitimate concerns that this instruction could inadvertently educate students about the existence of these apps.”

Earlier this week, it was discovered that the Chicago Sun-Times and the Philadelphia Inquirer had both published an externally-produced "special supplement" that contained facts, experts, and book titles entirely made up by an AI chatbot. There's been a lot written about this (former Chicago Reader editor Martha Bayne's is the best), and I don't need to rehash it all. But the thing that is most disheartening to me is how at every step along the way, nobody cared.

The writer didn't care. The supplement's editors didn't care. The biz people on both sides of the sale of the supplement didn't care. The production people didn't care. And, the fact that it took two days for anyone to discover this epic fuckup in print means that, ultimately, the reader didn't care either.

It's so emblematic of the moment we're in, the Who Cares Era, where completely disposable things are shoddily produced for people to mostly ignore.

AI is, of course, at the center of this moment. It's a mediocrity machine by default, attempting to bend everything it touches toward a mathematical average. Using extraordinary amounts of resources, it has the ability to create something good enough, a squint-and-it-looks-right simulacrum of normality. If you don't care, it's miraculous. If you do, the illusion falls apart pretty quickly. The fact that the userbase for AI chatbots has exploded exponentially demonstrates that good enough is, in fact, good enough for most people. Because most people don't care.

(It's worth pointing out that I'm not a full-throated hater and know people—coders, mostly—who work with AI that do care and have used it to make real, meaningful things. Most people, however, use it quickly and thoughtlessly to make more mediocrity.)

It's easy to blame this all on AI, but it's not just that. Last year I was deep in negotiations with a big-budget podcast production company. We started talking about making a deeply reported, limited-run show about the concept of living in a multiverse that I was (and still am) very excited about. But over time, our discussion kept getting dumbed down and dumbed down until finally the show wasn't about the multiverse at all but instead had transformed into a daily chat show about the Internet, which everyone was trying to make back then. Discussions fell apart.

Looking back, it feels like a little microcosm of everything right now: Over the course of two months, we went from something smart that would demand a listener's attention in a way that was challenging and new to something that sounded like every other thing: some dude talking to some other dude about apps that some third dude would half-listen-to at 2x speed while texting a fourth dude about plans for later.

Hanif Abdurraqib, in one of his excellent Instagram mini-essays the other week, wrote about the rise of content that's designed to be consumed while doing something else. In Hanif's case, he was writing about Time Machine, his incredible 90 minute deep dive into The Fugees' seminal album The Score. Released in 2021, Hanif marveled at the budget, time, and effort that went into crafting the two-part 90 minute podcast and how, today, there's no way it would have happened.

He's right. Nobody's funding that kind of work right now, because nobody cares.

(It's worth pointing out that Hanif wrote this using Stories, a system that erased it 24 hours later. Another victim of the Who Cares Era.)

Of course we're all victims of the biggest perpetrators of this uncaring era, as the Trump administration declares "Who Cares?" to vast swaths of the federal government, to public health, to immigrant families, to college students, to you, to me. As Elon Musk's DOGE rats gnaw their way through federal agencies, not caring is their guiding light. They cut indiscriminately, a smug grin on their faces. That they believe they can replace government workers—people who care an extraordinary amount about their arcane corner of the bureaucracy—with hastily-written AI code is another defining characteristic of right now.

I keep coming back to the word "disheartening," because it all really is.

Without getting into too many specifics, I recently was involved in reviewing hundreds of applications for something. Over the course of reviewing, I was struck by the nearly-identical phrasing that threaded through dozens of the applications. It was eerie at first, like seeing a shadow in the distance, then frustrating, and ultimately completely disheartening: It was AI. For whatever their reasons, a bunch of people had used a chatbot to help write their answers to questions that asked them to draw from their own, unique, personal experience. They had fed their resumes or their personal websites or their actual stories and experiences into the machine, and it had filled in the blanks, Mad Libs-style. I felt crushed.

Until.

Until I read an application written entirely by a person. And then another. And another. They glowed with delight and joy and sadness and with the unexpected at every turn.

They were human.

They were written by people that cared.

In the Who Cares Era, the most radical thing you can do is care.

In a moment where machines churn out mediocrity, make something yourself. Make it imperfect. Make it rough. Just make it.

At a time where the government's uncaring boot is pressing down on all of our necks, the best way to fight back is to care. Care loudly. Tell others. Get going.

As the culture of the Who Cares Era grinds towards the lowest common denominator, support those that are making real things. Listen to something with your full attention. Watch something with your phone in the other room. Read an actual paper magazine or a book.

Be yourself.

Be imperfect.

Be human.

Care.

We’re drawn to activities that invite us to grow, by trying and trying again, because we want to evolve as people. Life is mostly repetitive—wake, eat, work, sleep, repeat—and each day can feel like an unsatisfying circle. But repetition with variation broadens us. It makes our circular days into spiralling journeys. “The spiral is a…

The post Why Even Try if You Have A.I.? Now that machines can think for us, we have to choose whether to be the passengers or pilots of our lives. appeared first on Jerz's Literacy Weblog (est. 1999).