Privacy and Equity Concerns: Questions 4 & 5 – Public Interest Privacy Center

Part II: Privacy and Equity Concerns

4. What harms, such as stigma or discrimination, may stem from sharing of students’ information or flagged status?

Using self-harm monitoring systems raises potential risks of stigma or discrimination. Biases embedded in public perception and media lead to exaggerated fears that students experiencing mental health challenges are prone to violent acts,⁶⁵ even though most people with mental health needs have no propensity for violence.⁶⁶ As a result of such biases, school staff may treat flagged students differently from their peers or subject them to additional scrutiny. The common but false assumption that flagged students may be violent can increase harmful stigmas toward the students who need support and can lead them, especially systematically neglected students, to experience disproportionate rates of discipline.⁶⁷

For example, in 2018, Florida passed a law⁶⁸ requiring schools to collect information from students at registration about past mental health referrals, while the state’s school safety commission proposed that “students with IEPs [Individualized Education Programs] that involve severe behavioral issues” should be referred to threat assessment teams,⁶⁹ which are committees created in the wake of the tragic Parkland school shooting to evaluate whether individual students pose threats of violence to the school community. Policies such as these immediately put students who struggle with mental health on a separate tier of scrutiny and potential disciplinary action due to deeply ingrained societal stigma, without leading to improved support or mental healthcare resources for the students. These policies could also have a chilling effect on disclosure. Parents are likely to worry that if their “children’s mental health history becomes part of their school records, it could be held against them.”⁷⁰ These concerns could result in a loss of trust and an unwillingness to provide schools and districts with sensitive information. In some cases, parents and students may even be disincentivized from seeking mental health treatment for fear that disclosure will harm their future opportunities.

In some cases, even when students seek help on their own, they may experience negative consequences. In one extreme example, the Bazelon Center for Mental Health Law represented a college student who voluntarily admitted himself to a campus hospital after a close friend died by suicide.⁷¹ When he checked in, the campus hospital shared his health information with university administrators. The next day, while still in the hospital, the student received a letter from the university charging him with a violation of the disciplinary code, allegedly for endangering himself. The student was suspended from school, barred from entering the campus (including to see his psychiatrist), and threatened with arrest if he returned to his dormitory. While his father and friends removed his belongings from his dorm room, he was forced to sit in a car with a university official.

Students who may be considering self-harm or who are struggling with their mental health can be disincentivized from seeking help if they fear that all help sought is monitored.

Situations like these demonstrate the potential harms of invasive or disciplinary school responses to information about a student struggling with mental health. Stigma is unfortunately real and should not be underestimated. Punitive responses contradict the goals of self-harm protection programs because they discourage students from seeking the help they need and engaging openly with mental health counselors or other healthcare providers. While monitoring companies claim their products help schools save lives, students may ultimately experience harm by not searching for resources that could help them out of a fear of being identified by school officials. Students who may be considering self-harm or who are struggling with their mental health can be disincentivized from seeking help if they fear that all help sought is monitored. Moreover, students who are identified as “at-risk” may feel like they have a target on their backs, with their personal struggles facing scrutiny in school. Students’ opportunities should not be limited, either by mental health challenges or by violations of their privacy.

The risk of students being unfairly treated or experiencing discrimination as a result of a self-harm flag can be particularly high in schools and districts without enough school-employed mental health professionals⁠—for example, school-based counselors, school psychologists, social workers, and nurses⁠—a shortage that unfortunately afflicts most schools.⁷² For detailed information on the potential discriminatory harms and stigma-related effects that can arise once a student is identified, see Boxes 1 and 2.

Even if schools do not explicitly regard students experiencing mental health challenges as threats or target them for discipline, monitoring can impact students’ natural exploration, academic freedom, or ability to find online communities and resources that are important for their well-being and mental health.
Research has shown that school surveillance can corrode learning environments by instilling an implicit sense that children are untrustworthy.⁹⁵

Many organizations have noted that surveillance technologies such as social media monitoring and facial recognition can harm students by stifling their creativity, individual growth, and speech. The sense that “Big Brother” is always watching can destroy the feelings of safety and support that students need to take intellectual and creative risks—to do the hard work of learning and growing. In one study of Texas high school students whose district monitored their social media accounts, students reported that even if they had nothing to hide, they nonetheless found it chilling to be watched.⁹⁶ A recent national survey found that 80 percent of students who were aware of their schools using monitoring software reported being more careful about what they search online because of knowing that they are being monitored.⁹⁷

Box 1: Monitoring Inflicts Particular Harms on Systemically Marginalized Groups of Students

Beyond understanding the risk of criminalization and potential for referral to law enforcement, school districts should carefully consider the uniquely harmful impacts of monitoring on various systemically marginalized groups of students. Below are some examples of groups of students that may experience unique harms as a result of self-harm monitoring.

Students from Low-Income Backgrounds

Students from Low-Income Backgrounds.

Students from low-income backgrounds may not have a personal computer or internet access outside of the school campus or school-issued devices, leaving students without the ability to engage online free from their school’s monitoring system. Students without personal devices face more monitoring and associated harms; educators report that while 71 percent of schools who use monitoring do so on school-issued devices, only 16 percent monitor students’ personal devices.⁷³ Students without personal devices may be especially uncomfortable using school devices to seek support, if they know that these devices are subject to monitoring. These disparate impacts may be especially pronounced during the COVID-19 pandemic: while learning remotely, students have limited opportunities to seek more information or professional assistance beyond the internet and school-issued devices because they may have limited access to in-person resources. A survey in 2020 found that 8 percent or 4.4 million households do not have a computer always available. In households where a computer was always available, 60 percent received devices from the child’s school or school district.⁷⁴ Similarly, students experiencing homelessness are unlikely to have access to personal devices and may heavily rely on school-issued devices, while especially needing to use them to search for non-academic resources or supports. These contextual factors suggest that self-harm monitoring programs require clear and transparent boundaries, protocols, and appropriate privacy protections. Otherwise, such programs risk harming the students they intend to protect.

Students Experiencing Language Barriers

Almost 5 million students in schools across the country are English Language Learners, comprising 9 percent of all public school students.⁷⁵ Students who are English Language Learners or multilingual, as well as students with disabilities, may be at especially high risk of false, inequitable flagging and of experiencing harm⁷⁶ from being flagged by a monitoring system. Students who are English Language Learners may often use or interact with content in languages that school officials or a monitoring company do not understand or may interpret negatively.⁷⁷ Deeply ingrained biases against students who are English Language Learners can especially influence suspicious and negative interpretations of their writing and activities.⁷⁸ Similarly, students who are English Language Learners may sometimes lack the proficiency or cultural nuance to express themselves as non-English Learners would and may mistakenly use words or phrases that a monitoring program may flag or school officials may misinterpret as a threat to self. As a result, there is a high risk that intent and meaning may get lost in translation, and these students will end up flagged or penalized for innocuous language that school staff fail to accurately decipher. Language barriers or miscommunications and misunderstandings based on differential language use can also surface when monitoring technology scans the content of some students with disabilities.

In addition, monitoring systems may utilize automatic, computerized translations when scanning student content in non-English languages. These computerized translations are frequently inaccurate and fail to account for idiomatic language use or cultural nuance.⁷⁹ For example, direct translation of a phrase meaning, “You’re annoying me,” from Korean to English resulted in widespread use of the phrase, “Do you wanna die?” in Korean-American communities.⁸⁰ These inherent shortcomings of monitoring systems risk disproportionately targeting students who are English Language Learners. For more information on legal protections for students who are English Language Learners, see Legal Implications.

Students with Disabilities

In addition to disproportionate risks of stigmatization and criminalization, students with disabilities may be especially harmed by the ways self-harm monitoring systems analyze student content and writing.

Some students with disabilities may interact with online content or use speech differently than their non-disabled peers and may consequently face risks of disproportionate flagging because of the limitations of these systems in interpreting context. Speech that is a manifestation of a disability may be misinterpreted as a threat to self-harm by the monitoring software or by untrained school staff who are unfamiliar with the intersection of disability and mental health. This misinterpretation often occurs with students who have developmental or learning disabilities.⁸¹ School district leaders should be aware that disparate treatment of students with disabilities, including disproportionately and needlessly flagging them due to typical manifestations of their disabilities, can constitute discrimination and invite potential legal challenges under the Americans with Disabilities Act (ADA). For more information on legal anti-discrimination protections for students with disabilities, please see Legal Implications.

Students of Color and Racial Minorities

In addition to harms stemming from sharing student information with law enforcement and referring mental health-related issues to law enforcement, students of color may disproportionately experience other harms from self-harm monitoring.
For example, natural language processing algorithms, which are used by monitoring systems, have been shown to analyze and interpret Black dialects of English used online less accurately than writing by white individuals online.⁸² Likewise, research at MIT shows many common automated tools that scan online content using natural language processing disproportionately flag writing from Black users.⁸³ These examples demonstrate the technological shortcomings, and inequities, inherent in accurately monitoring online content. Such technological inaccuracies lead to racial disparities in students mistakenly flagged by monitoring systems and can cause students of color to disproportionately experience the harms related to mismanaged and privacy-violative monitoring.

Additionally, low-income youth of color and other vulnerable young people may have a very different relationship with school-based and medical-based systems of formal mental healthcare. These student populations may often look to community-based resources and peer social networks as their preferred sources of care and wellness.⁸⁴ While many monitoring technologies proceed from the assumption that school-based systems of care are best positioned to support young people, that may not be the case for many youth. For many students, state-based systems of mental health screenings and services can trigger harmful episodes where they, or their caregivers, have had to deal with the child welfare system, criminal legal system, juvenile justice system, etc.⁸⁵ School districts should keep the different needs and preferences of various student groups in mind and recognize that a one-size-fits-all approach to responding to student self harm will not equally benefit all students.

Another important consideration is the effect of high-surveillance schools⁸⁶ on the academic outcomes and well-being⁸⁷ of Black students and other structurally disadvantaged racial groups. These students may experience monitoring more as a form of surveillance and control of student behavior than as a mental health support tool, due to the greater prevalence of schools with harsh security and zero-tolerance policies in communities of color.⁸⁸ In these cases, implementing a monitoring system can add to an atmosphere of surveillance and criminalization, thereby compromising students’ sense of comfort and support in their school environment. Research from John Hopkins University and Washington University shows that high surveillance schools can lead to lower test scores and graduation rates for Black students, as well as greater disciplinary disparities.⁸⁹

LGBTQ Students

Besides facing the risks of discipline and criminalization described in Box 2, LGBTQ students face unique additional harms from having their digital activities monitored. These unique harms can be exacerbated depending on students’ school and home environments.

Research shows that LGBTQ students who experience victimization or bullying in school face detrimental psychological outcomes, such as higher instances of depression, low self-esteem, increased isolation, and increased suicidal ideation, compared to non-LGBTQ peers.⁹⁰ The American Psychological Association has reported⁹¹ that 64 percent of LGBTQ students feel unsafe in schools because of prejudice and harassment. Sixty percent of these students did not report these incidents to school officials due to fear the situation would be made worse or that the school would take no action to help them. Self-harm monitoring technologies that flag incidents of harassment and prejudice may result in these very fears for LGBTQ students, particularly in unsupportive school environments or without thoughtful protocols for handling flags.

LGBTQ students have a unique interest in controlling who has information about their sexual orientation and gender identity to prevent incidents of harassment, particularly in situations of unsafe home or school environments.

Nonprofit suicide-prevention organization The Trevor Project reports that about 50 percent of LGBTQ youth selectively and carefully decide which family members and teachers and in which contexts they disclose their sexual orientation or gender identity.⁹² In a national survey conducted by The Trevor Project, less than half of LGBTQ youth had disclosed their identity to an adult at school.⁹³ Research has also found that LGBTQ youth are more likely than their peers to seek identity-related resources and help online.⁹⁴ Monitoring systems may discourage youth from seeking LGBTQ-affirming resources online if they fear surveillance, repercussions, or reporting or being outed to school staff, other students, or even their parents through the monitoring program.

This ability to decide when and how to come out is a critical right that supports mental well-being, particularly when students are in situations where they may feel unsafe or unsupported. This includes school environments where students do not feel confident that their school leaders would support their identities if they were to report bullying or harassment. Consequently, exposing LGBTQ students as a result of monitoring, even with the good intention to help them, can in fact undermine their mental health and safety by damaging this important protective strategy.
School leaders concerned about the mental health of LGBTQ youth should work to create actively affirming and supportive school climates that respect students’ boundaries and privacy, and to provide resources and information in school related to sexual orientation and gender identity, rather than engage in monitoring that would invasively and forcefully expose these students.

5. Who has access to the information identified or flagged, and do they have a legitimate health or educational purpose for accessing it?

Because of the harms that can stem from sharing student information, a key privacy issue involves who can access information about which students have been flagged and the content collected by a self-harm monitoring system. Schools should carefully consider which school staff receive information collected through monitoring technologies and what training and communication is being provided to this staff and limit this access to only those who need it to provide specific mental health-related follow-up and support to the students. Schools must also determine if the information may be lawfully disclosed to these individuals.

Coordination among teachers, parents, administrators, and school-employed mental health professionals regarding identified students could help adults spot warning signs and establish comprehensive support plans for the students. Providing increased attention to students’ mental health from qualified individuals may result in better resources and increased care.

However, simply having information about students’ mental health status, without the skills or capacity to provide specific follow-up or support, could damage teachers’ perceptions of students or negatively affect how the wider school community treats such students. This may be especially true when teachers or school staff receiving this information do not have the training, qualifications, or responsibility for providing mental health-related support to students. Peer-reviewed research demonstrates that teachers do frequently inaccurately identify students as experiencing symptoms of depression and anxiety.⁹⁸ As a result, students may experience the sharing of this information as an invasion of their privacy, resulting in feelings of stigmatization and mistrust.

Simply having information about students’ mental health status, without the skills or capacity to provide specific follow-up or support, could damage teachers’ perceptions of students or negatively affect how the wider school community treats such students.

In addition to considering whether school staff are appropriately equipped to provide mental health-related support to students, schools and districts should ensure that any staff with access to the information identified or flagged are trained on the district’s internal protocol for appropriately handling student information collected through monitoring. Staff must be trained to understand the sensitivity of the information being collected on students, understand appropriate disclosure and use limitations, and be familiar with how and when to appropriately escalate any concerns. They should also be trained on the myriad privacy and equity concerns that arise when students’ online activities are monitored surreptitiously. Finally, staff who may have access to student information collected through monitoring or who may be responsible for following-up with identified students must be trained on the district’s broader mental health policies, including the school’s self-harm prevention and suicide intervention protocols.⁹⁹

Schools should also consider the potential risks and harms that can result from sharing information collected from monitoring with students’ parents. For example, some monitoring software flags terms related to sexual orientation or gender identity (such as “gay” and “lesbian”) as terms that signify potential bullying.¹⁰⁰ If a student is searching for identity-affirming materials and their searches are flagged, what consequences might the student experience if the school shared that information with their parents, to whom the child may not have disclosed these identities? Children in these situations may face serious dangers to their safety and well-being if their home environments are not supportive. More information about this type of harm is presented in Box 1.

Schools and districts should incorporate processes to appropriately ensure that these types of considerations are factored into how, if at all, parents are notified of flags containing sensitive information and what information collected from monitoring is shared with them. These considerations should fall within a broader approach of similar caution that schools must exercise when sharing flagged information with anyone because of the potential negative effects it may have on the identified student. Significant risks arise any time sensitive student information collected through monitoring is shared with any individual who does not directly need access to the information in order to provide mental health-related support.

Previous Questions

Next Question