Proceedings of a Workshop—in Brief

K-12 Behavioral Threat Assessment Efficacy and Implementation Evaluation Research

Proceedings of a Workshop—in Brief

On September 3 and 4, 2024, the National Academies of Sciences, Engineering, and Medicine held a workshop on K-12 Behavioral Threat Assessment Efficacy and Implementation Evaluation Research sponsored by the U.S. Secret Service’s National Threat Assessment Center (NTAC). The workshop was designed to explore key behavioral threat assessment literature findings, examine best practices, and identify future research needs. Franci Crepeau-Hobson (Planning Committee Member) defined behavioral threat assessment (BTA) as “a step-by-step process for evaluating individuals who threaten harm to others, making systematic judgments about the seriousness of the threat based on case-specific and dynamic risk factors, and identifying interventions necessary to support the individuals and reduce risk for harming others.” Kelsey Morris (NTAC) stated that schools have increasingly adopted the threat assessment approach since its inception in 1999; 39 out of 50 states currently either mandate or strongly encourage the use of threat assessment in schools, and 85 percent of public schools report having a threat assessment team in place that has identified a student with concerning behavior. Morris ended by saying that supporting schools to better support students in distress will build the capacity that is needed to better identify students at risk of harming themselves or others and intervene early.

Findings from the Literature: A Systematic Review

Joshua Hendrix (RTI International) shared findings from the literature on BTA research in K-12 schools. Hendrix said that the systematic review summarized evidence-based practices and outcomes associated with BTA, identified factors or conditions for successful implementation, highlighted research gaps, and assessed the implications of BTA research outside of K-12 settings. RTI’s effort included a review of 118 articles published in peer-reviewed journals that focused on behavioral threat assessment in schools, workplaces, institutions of higher learning, and other settings, as well as a more focused and systematic review of 32 essential studies (a subset of the 118 articles initially reviewed) explicitly focused on behavioral threat assessment in K-12 schools and published as of April 2024. They examined implementation and outcome information, as well as findings and key limitations highlighted by the authors, and with additional attention to factors that impacted successful implementation (e.g., organizational support and capacity). They also looked for the use of standard and highly recommended practices, such as multidisciplinary teams that met regularly and tracked case details. Finally, they examined studies for information on the impact of BTA on school discipline and to identify potential disparities in threat classifications or school/law enforcement

responses to threats according to background characteristics of students who had experienced a BTA (e.g., race/ethnicity, enrollment in special education).

More than half of the studies focused on evaluating the Comprehensive School Threat Assessment Guidelines (CSTAG; formerly the Virginia Student Threat Assessment Guidelines) or its predecessor, while the remainder evaluated other approaches to threat assessment or studied threats without identifying the model or approach schools were using. The most common data source used to study BTA in schools was surveys, followed by threat records, said Hendrix, and more than half of studies were based in Virginia schools. Half of the studies were cross-sectional without comparison groups and the remainder used a variety of other study designs. For example, four studies featured cross-sectional study designs and incorporated comparison groups, and two studies featured key components of a randomized controlled trial (e.g., random assignment of schools to treatment or comparison groups).

Hendrix shared key takeaways. First, BTA has been studied from multiple angles, with both qualitative and quantitative evaluations of aspects such as training programs, fidelity to standard practices, and challenges of implementation. Second, Hendrix said that the studies showed several patterns in the nature of threats, classifications, and outcomes. Threats tend to come from boys who target students of the same race and grade level; students who make threats tend to have a history of disciplinary and academic problems; and students receiving special education and Black students are disproportionately represented in threat assessments. Most threats are classified as nonserious; those that are classified as more serious often involve an older student, warning behaviors, and include mention of a weapon. The third takeaway is that programs to train people on threat assessment are very effective at improving skills, knowledge, and attitudes. Hendrix highlighted a study that found that support for zero-tolerance discipline dropped considerably among staff who received training, and another that found that educational programs improved students’ understanding of BTA and willingness to report threats. The review also revealed findings about the impact of BTA on schools and students, said Hendrix: schools that use BTA have lower suspension rates, and students often receive mental health support after experiencing a threat assessment. A few studies also found that schools using BTA have lower levels of bullying and student reports of victimization as well as better perceptions of safety among students or staff. However, said Hendrix, more research is needed to understand the mechanisms that can explain these associations. The evidence on the individual psychological and behavioral impact of BTA on students is mixed, said Hendrix, and more research is needed on how individual students experience BTA and whether and how the experience impacted their behavior, attitudes, and relationships both in the short and long-term following BTA.

Research has generally not identified disparities in threat outcomes by race or ethnicity, but Hendrix suggested that future research should consider additional contextual characteristics (e.g., race of student threatened). The evidence on students receiving special education is mixed, with several studies finding comparable outcomes for special and general education students and others finding that threats from special education students were more likely to be classified as serious or to result in suspension.

More research on BTA is needed, said Hendrix, on implementation practices and context as well as the impact of these practices and contextual characteristics on discipline and safety. Most quantitative studies that examine the impact of BTA adoption on discipline or safety do not account for variation in implementation practices or fidelity to a specific model or approach. Research is needed to understand the range of practices and configurations schools are using for BTA, and whether there are alternative or less traditional approaches that can also be beneficial for reducing exclusionary discipline or improving safety. More qualitative research is needed, he said, to provide a nuanced depiction of implementation, to explore situations in which BTA in practice contradicts formal policies and procedures, and to further demonstrate the complexity and potential pitfalls of BTA. Hendrix said that primary and direct observational research could be useful for gaining insight into implementation, but that strategies need to be developed to collect real-time data.

SOCIAL AND ENVIRONMENTAL CONTEXT OF BEHAVIORAL THREAT ASSESSMENT

Moderated by Eric Madfis (University of Washington and Workshop Planning Committee Member) this session explored how BTA fits into the broader social and environmental context of schools and communities.

Gun Violence in Schools

Sonali Rajan (Columbia University) described the landscape of school violence, noting that there have been noticeable upticks in school gun violence following the COVID-19 pandemic. Since 1999, an estimated 378,000 students have experienced gun violence at school, including witnessing or hearing gunshots, mass shootings, suicides, and unintentional shootings. Current approaches to addressing this problem focused almost exclusively on reactive strategies are insufficient, said Rajan, noting that responses to the school gun violence crisis need to be far more strategic, multifaceted in nature, and address its upstream causes well before a shooting. Rajan argued that BTA is a critical component of a comprehensive school violence prevention plan. Rajan shared a framework¹ that shifts the emphasis away from reactive strategies and broadens the spectrum of what constitutes school gun violence prevention. This approach, she said, “reimagines the prevention of school gun violence as an opportunity to meaningfully invest in schools, its surrounding communities, and by extension its children.” The comprehensive framework includes approaches for primary prevention (e.g., reducing access to firearms), secondary prevention (e.g., BTA), and tertiary prevention (e.g., providing mental health services). While there has been investment in school security and safety, she said, these strategies are largely reactive rather than preventive. On the other hand, there is a rigorous and thoughtful evidence base for BTA. Rather than investing dollars in non-evidence-based, reactive strategies, Rajan suggested that funds would be better spent on infrastructure and services that are known to be beneficial, like expanding access to mental health care by hiring more school counselors and other mental health professionals, training teachers and parents to provide support to students, and connecting K-12 schools with local community-based organizations. Rajan emphasized that an integrated systems approach that coordinates among systems (e.g., schools, healthcare, and social services) is a best practice and is critical to the BTA process.

Role of School Security

Benjamin Fisher (University of Wisconsin-Madison) said school security is largely driven by technologies aimed at “hardening”—adding physical security—to school campuses and other approaches such as behavior management systems. However, there is very little evidence that hardening approaches and technology-based solutions are making schools safer. While it is not feasible to evaluate every new technology, he said, there is a need to look critically at the general approach and assess the evidence base. Such approaches may have implications for equity, said Fisher. For example, data that reflect racial disparities in school suspension or school arrests may be used to “feed” artificial intelligence (AI)-powered safety technologies. When these technologies are used to detect safety threats, such as aggressive behavior, there is potential for reproducing existing inequities. The current environment of school safety, Fisher argued, is not driven by the evidence in the development or uptake of tools. He echoed Rajan in suggesting that the funding currently directed at non-evidence-based, potentially discriminatory tools might be better directed at preventative efforts.

Safe School Initiative

Kristina Anderson Froling (Koshka Foundation for Safe Schools) shared her perspective as a school shooting survivor. Froling was a sophomore at Virginia Tech in 2007 when a fellow student entered Froling’s classroom and shot the professor and every student but one; Froling was shot three times. The shooter killed a total of 32 individuals before taking his own life. Froling said that the “ripples of this trauma are enormous” and lifelong. After of the shooting, Froling founded the Koshka Foundation for Safe Schools and began to learn about school safety. She said that the Safe School Initiative,² a report published by the U.S. Secret Service and the Department of Education in 2002, inspired her and gave her hope that attacks could be prevented. Froling highlighted several key findings of the report: a) incidents of targeted violence rarely were sudden impulsive acts; b) there is no

__________________

accurate or useful “profile” of students who engaged in targeted school violence and c) in more than four out of five attacks, at least one other person knew of the attacker’s plan. Following a shooting, she said that much of the focus is often on the mechanics of the attack and how to improve survivability, but a more fruitful conversation would be about providing the care and resources necessary to avoid the attack. Most states now require or strongly recommend that K-12 schools have a threat assessment team, said Froling, but there is a need to better understand how these teams work: What is the structure of these teams? What is the professional background of the leaders and members? How often do they meet? Do they receive training and if so, what type and how often? Are they compensated for their work?

Children with Disabilities

Meghan Whittaker (U.S. Department of Education) described the landscape of education for children with disabilities, and how students with disabilities are impacted by BTA and other school safety strategies. There are clear disparities in the educational system for children with disabilities, she said. Children with disabilities with an Individualized Education Plan (IEP) under the Individuals with Disabilities Education Act (IDEA) and those served under Section 504 of the Rehabilitation Act of 1973 make up 17 percent of K-12 students, but higher rates of those suspended in school (24%), suspended out of school (29%) and expelled (21%). When students are removed from the classroom, it impacts their ability to learn and leads to increased rates of school dropout, lower performance, and incarceration. These disparities are even more stark for children of color with disabilities, said Whittaker. This context must be taken into consideration when discussing school safety, she said. “Threat assessments are not performed in a vacuum.” BTA systems often flag indicators that are more common among children with disabilities, such as low attendance, history of behavior issues, reduced course completion, and higher rates of bullying or being bullied. It is important to emphasize the human component of threat assessment to ensure that these systems are not reproducing bias, she said. In practice, BTA can be a tool that removes children from learning and contributes to disparities, or it can be a tool that is used for prevention and connects children with the services they need.

There are three phases to think about when a child with a disability is involved in a BTA, said Whittaker. First, what are the supports and services that are already in place for the child leading up to any behavior that triggers a BTA? Second, what supports can be provided to the child and the family if the student is removed from school and what are their rights under IDEA when this happens? Third, what is the plan to get the student back into school or into an appropriate placement? The IEP team is invaluable in this process, she said, because they have the knowledge and experience to know the history of the student, what must be done to protect their rights, and how to best support the student and their family. If the goal is to reduce school violence, this can only be accomplished by meeting student needs upstream. Instead of being the sole strategy, BTA needs to be situated as part of a larger system that involves multi-tiered systems of support to create a safe and supportive learning environment for all children.

Children with Autism and Attention-deficit/hyperactivity Disorder (ADHD)

Frank Straub (Safe and Sound Schools) explained that one in 36 children are on the autism spectrum, about one in nine children have been diagnosed with ADHD, and there is a high rate of co-occurrence of autism spectrum disorder with other psychiatric disorders or mental health conditions. These children can be vulnerable to a whole range of concerns, he said, and schools need to take a multifaceted, multi-disciplined, and respectful approach to understanding their needs. Certain facets of autism may prove challenging in a school setting, including difficulty understanding other people’s perspectives, fixations on specific topics, black-and-white thinking, and difficulty making sense of social interactions. Autistic students may also have a range of mental health conditions, developmental disorders, specific learning difficulties, or other impairments. When assessing a student with autism and ADHD, said Straub, it is important to consider that while autism and ADHD are relevant to the BTA, they are contextual factors rather than a causal factors. It is important to identify which facets of autism and ADHD are relevant and how, and to be specific in the

formulation of interventions. Whether an assessment involves a neurotypical child or a child with autism and ADHD, behavior cannot be taken at face value; instead, schools must understand students and the environments in which they live, including their families, homes, and communities. Straub said that there is a need for further research on the nexus between autism/ADHD and behaviors that cause concern in school settings. Research is needed on whether and how interventions and tools used in this space are appropriate for students with intellectual and developmental disabilities.

IMPLEMENTATION, FIDELITY, AND IMPACT

Moderated by Crepeau-Hobson (University of Colorado Denver and Planning Committee Member) this session continued to explore the evidence base for BTA, looking at issues of implementation, fidelity, and impact.

Research on Comprehensive School Threat Assessment Guidelines (CSTAG)

“Schools do threat assessments all the time”, said Dewey Cornell, University of Virginia. These assessments are often informal evaluations of a student, and schools often take a zero-tolerance approach by removing students from the classroom. The goal of BTA is to move away from this “knee-jerk approach” and to focus on the context and seriousness of a threat, he said. Students make threats for many reasons; they may be joking, seeking attention, or angry or frustrated. Among the many threats made by students, there are a very small number of serious threats that must be identified and addressed, but most threats do not pose a danger. Schools need a structured process to determine whether a threat is serious, and to take protective action if the threat is serious or to attempt to resolve the problem underlying a non-serious threat. CSTAG was developed in 2001 and has been refined over the years to provide this structure. Research evaluating schools that use this model has shown promising results, said Cornell. Schools using the model have resolved thousands of threats without violence, and only a few threats have been attempted. There have been no shootings or fatalities, and 99 percent of threats resulted in no serious injuries. Schools using the model report a more positive school climate, with less bullying and peer aggression. Cornell noted that this research cannot prove that the model prevents school shootings; due to the rarity of shootings, the sample size for such a study would have to be very large. In a study of Florida schools using the CSTAG model, about 80 percent of threats made were easily resolved while about 20 percent required more attention. He noted that using the structured process of assessment saved time and effort that without such a model, many of the students making non-serious threats would be arrested. Among schools using the model, less than one percent of students are arrested, with most disciplinary actions being out-of-school suspension or change in placement. The Florida studies found little or no disparities in disciplinary outcomes or law enforcement actions associated with race/ethnicity or disability. Cornell noted that he and his colleagues are currently conducting a study in about 500 schools in six states, looking at the fidelity of implementation, consistency in case outcomes, academic and behavioral outcomes, and disparities by race/ethnicity or disability status. In closing, Cornell said that there is evidence that it is an approach that can be used very effectively to keep kids in school, to resolve student threats and problems, and to do this work fairly and equitably.

Challenges with Implementation

What happens in real-world settings sometimes differs from the programs that are developed and delivered in training, said Sarah Goodrum (University of Colorado Boulder). Lack of resources, lack of training, and lack of leadership to support efforts are some reasons for lack of fidelity to implementation. Goodrum shared findings from an intensive case study she conducted of a shooting at Arapahoe High School in Colorado, and an evaluation of a program to train 37 schools on the Colorado School Safety Center’s Threat Assessment Process. Goodrum said that three major challenges can impact a school’s implementation of a safety program: readiness, information sharing, and case management. Readiness is essential for the success of any new program. Before implementation, it is important to assess an organization’s capacity and motivation for adopting the program, to evaluate whether the appropriate leadership and resources are in place, and to identify and mitigate any gaps that might hinder success. Goodrum also identified the issue of information sharing as a challenge. There are misconceptions and confusion about who can share information with whom

and how much information can be shared. Not knowing that a student has been the subject of BTA inhibits the ability of school staff to recognize subsequent behavior concerns following a threat assessment. Goodrum argued that sharing information about a student’s BTA results with the staff encountering the student creates a “compassionate circle of support” as well as better management and monitoring of the student. Finally, a lack of best practices for case management is a challenge to successful implementation of BTA programs. Case management is a tailored plan to reduce an individual’s risk of violent behavior, which may include short-term services or long-term intensive approaches. However, said Goodrum, it is not always clear what these approaches look like, and there is a wide variety of strategies discussed in the literature. Threat assessment teams are not always clear on which strategies should be used for which behavior concerns or on what the case management process should look like after BTA.

Cognitive Biases and Decision-making

J. Reid Meloy (San Diego Psychoanalytic Center) spoke about cognitive biases (“shortcuts” that can distort decision making) that can impact how people conduct assessments of threats. Two are particularly relevant to BTA: confirmation bias and availability bias. Confirmation bias is the tendency to avoid information that contrasts or challenges one’s beliefs, and to interpret new information as confirming one’s pre-existing beliefs. Availability bias occurs when a person relies on information that most readily comes to mind, which is often personal, emotional, or most recent, rather than relying on empirical evidence. This bias may impact BTA, or discussions around implementing BTA programs, when people ignore the evidence on BTA and respond from an emotional perspective.

PERSPECTIVES FROM K-12 EDUCATORS AND ADMINISTRATORS

In this session, moderator Jennifer Freeman (University of Connecticut and Planning Committee Member) and panelists explored the perspectives of and challenges facing K-12 educators and administrators.

Training

Freeman asked panelists to describe the training process—who gets trained, what does the training look like, and is there follow-up training? Karen Barnes (Department of Education School Safety Center) trains districts on CSTAG and said training in Maine is multi-disciplinary, with between four and seven disciplines on each team, including mental health, administration, nursing, and special education. Training was originally “one and done,” with no support for implementation. Barnes said this was not successful, so training now includes planned follow-up meetings and implementation resources after initial training. Christina Chester (Montgomery County Public Schools) oversees the district’s BTA process, following the Maryland Model for Behavioral Threat Assessment (a combination of SIGMA, NTAC, and CSTAG). She said that in her district, all staff are mandated to complete a module in BTA, and the threat assessment teams undergo a two-day training that includes practice and coaching. Training on BTA has also been offered to community members, families, and school resource officers (SROs). Courtenay McCarthy (Salem-Keizer Public Schools) serves as lead school psychologist for the student threat assessment system. The district uses the Salem-Keizer/Cascade Model, a two-tiered model of assessment in which most cases are resolved by the school teams and more serious threats are managed by a community-based team that offers resources and support. McCarthy agreed with others that ongoing training and coaching is beneficial, particularly for schools that do not carry out many threat assessments day-to-day. She shared that in the aftermath of the shooting in Arapahoe Colorado, they found that there were often multiple administrators involved in decision-making in each school; this issue of “too many cooks” resulted in loss of information. Now, each secondary building in the district has a designated administrator who acts as lead for BTA and other safety systems and receives more frequent training, consultation, and coaching. This model has made a “huge difference” in maintaining fidelity to the systems in place, said McCarthy.

Reporting Procedures

Freeman asked panelists to discuss how and by whom threats are reported at their schools, and how they ensure referral to BTA is appropriate. In Maryland, said Chester, there are several in-person and anonymous venues for reporting by students, families, community members, and staff. Any information that comes in goes through a triage process that includes the building administrator

and a mental health professional; if it involves a student with a disability, the process must include a person familiar with their Section 504 or IEP plans. Filtering through information and assessing threats is like piecing together a puzzle, she said, with multi-disciplinary professionals working together to understand the entire picture and what supports are needed. McCarthy added that they communicate with students about how to report threats, including through posters, on computers and devices, announcements, and on the back of ID cards.

Protecting Against Bias for Involved Students

Not much is known about the short- or long-term impacts on students involved in the BTA process, said Freeman. She asked panelists to discuss how they guard against bias in the assessment process. Chester replied that she works with members of the equity unit to ensure that training includes conversations about issues such as triggers, implicit bias, and fear. When the assessment process first started in her county, BTA data reflected the same disproportionate impact on students of color as discipline data. This is changing, she said, due to the thorough evaluation that each student gets during BTA and the consideration of other interventions and supports before other more severe measures. She added that the platform they use allows only certain people to access information about a student’s BTA. Protecting this information can allow a student to go about their life in school and the community without the stigma and bias associated with the BTA. A workshop participant in the question and answer period said that there is a need to involve marginalized communities, including students with disabilities, in the discussions around these tools. A model can work well in theory but may result in bias and discrimination when applied to real-life situations. There is a need for collaboration and communication to ensure that school safety programs are both effective and equitable, they said.

Pressures on the BTA Team

Freeman noted that threat assessment teams may feel pressures from parents, the district, school safety tool developers, students, and other groups and asked panelists how this may impact a team’s ability to make decisions. Barnes replied that the best way to cope with pressure from parents and the community is to educate them about the BTA process. “Parents should not find out that [their] school is implementing BTA when their child is the subject of an investigation,” she said. Chester added that there is pressure from external agencies trying to sell their products, particularly after an incident occurs. When deciding whether to bring a new product into schools (e.g., metal detectors), it is essential to involve families, staff, unions, and the community. Most importantly, said Chester, students must be involved in these decisions because they are the people most impacted day-to-day.

BTA WITHIN A SYSTEM OF PREVENTION

This session explored BTA as a prevention strategy, situated within the broader context of school climate and safety.

Mental and Behavioral Health Supports

Data predating the COVID-19 pandemic indicate that approximately 20 percent of school-age youth experience a serious mental health issue, said Catherine Bradshaw (University of Virginia) yet few receive services. The COVID-19 pandemic exacerbated the situation, with increases in depression, anxiety, and suicide attempts. Many schools use the Positive Behavioral Interventions and Supports (PBIS) model to address students’ behavioral and mental health issues and other challenges.³ This model is a multi-tiered system of supports (MTSS) framework that divides interventions into three tiers: Tier 1, universal prevention for all students; Tier 2, selective or targeted intervention for some students; and Tier 3, highly specific intensive intervention for a few students. This framework has been shown to reduce bullying, suspensions, aggressive behavior, and discipline problems, and to improve school climate and classroom management. Bradshaw suggested that if the PBIS model were used with BTA, it could have synergistic effects. For example, optimizing Tier 1 supports could reduce the number of students who need BTA, and an improved school climate could lead to more students disclosing threats because they feel safe and connected; it could also improve the implementation fidelity of BTA. She noted that to implement these models with fidelity, there is a need for training, coaching, and district support. Having structured processes like PBIS and BTA in place can help

__________________

staff make quick decisions about what to do when there is a threat by serving as an “operating system” for the school, said Bradshaw.

CSTAG for Prevention

Jan Neureuther (University of Maine at Presque Isle/University of Maine at Machias/Maine PBIS/Katahdin Area School Districts) continued the discussion about how BTA can fit together with a MTSS/PBIS model. On paper, PBIS provides systems and supports for every single student, but in practice there may be gaps where systems have failed a student. When a student gets to the level of a BTA, it is important to look at the PBIS structure in the school and identify where gaps have occurred and where supports were needed. Neureuther noted that supports for students and their families may come from the school or from other intervention agencies in the community such as social workers and mental health providers. She said she has seen students excluded from school for months because of an “overreaction” from adults about a non-serious threat. This is where BTA is needed, she said, because it gives a process and a system for evaluating threats and for identifying supports. Unfortunately, many schools in rural Maine still take a zero-tolerance stance and favor exclusionary practices over BTA. It costs money and time to implement BTA well, she said, and there is a need for a funding model that supports the implementation, monitoring, and sustaining of BTA in schools.

Integrating BTA into School Systems

Tom Brant (Amity Regional School District No. 5) was a school psychologist in Newtown, Connecticut when the Sandy Hook shooting occurred and now works in administration, implementing best practices in schools. He shared some key takeaways from his work. First, most trained mental health staff working in schools do not learn BTA in graduate school and do not understand it as part of an intervention and prevention strategy. Second, Brant said that there is a misperception that BTA leads to increased arrests, particularly for special education students. Legislators and other decision-makers are working with the best information they have, but they don’t always have accurate data. To counter this, Brant and his colleagues advocate for BTA by sharing research that shows the true impact of BTA programs. There is a need for constituent groups to understand BTA as an evidence-based intervention that is used to identify students at risk, offer support, and monitor and manage the case. Finally, said Brant, schools need to have a system in place for record and data sharing. Brant encouraged schools to develop Memorandums of Understanding with partners including law enforcement, youth and family services, crisis preparedness centers, local hospitals, and other community organizations, and to do so before they are needed.

Team Support

Allison Willemin (Oxford Community Schools) works as the Executive Director of School Safety, Operational Technology, and Student Services, and has been working to develop a comprehensive school safety department focusing on both physical and psychological safety. She noted that in her educational career—from her bachelor’s in education to her Doctor of Education—school safety was never mentioned as part of the curriculum. “We are not prepping our educators for the reality of the world they live in,” she said. While there are any number of tools and technologies designed for school safety, the core of the work is recognizing a child in crisis and gathering a multidisciplinary team to support the child while ensuring the safety of all. There are many reasons why a student might make a threat or engage in threatening behavior, said Willemin, and a team is needed to probe the student’s environment and identify areas where support could be beneficial. Each member of a team brings a different perspective and different ideas; team members need ongoing training, support, and time to work together. Willemin said that in her schools, the threat assessment team meets weekly to talk about students, what supports are in place, and what might be needed. She stressed the importance of good relationships among the team and other people involved in the process. Much of the work isn’t “sunshine and roses,” and it is important that people can trust and respect each other.

BEHAVIORAL THREAT ASSESSMENT TEAMS

Mary Ellen O’Toole (George Mason University and Planning Committee Member) moderated a session on team composition, key contentions, and best practices across different behavioral threat assessment contexts and models.

Team Composition in the Salem-Keizer/Cascade Model

As McCarthy described earlier, the Salem-Keizer/Cascade Model is a two-tiered model of assessment and intervention, set apart from other models by its community-based approach. Level 1 teams, which are school-based, are tasked with assessing and managing situations of concern. Level 2 teams, which are community-based, are brought in for more serious threats to provide expertise, assessment knowledge, management recommendations, and community resources. Level 1 teams always include school administrators, school based mental health staff (counselors, psychologists, or social workers), and law enforcement (sometimes only in a consultative role). Others who know the student well are added as appropriate, such as teachers, special education case managers, or coaches. Schools are often reluctant to involve parents in BTA, said McCarthy, but it is essential to be transparent with parents and involve them. When parents see the BTA process, they come to understand that everyone wants to help their student and support them in the school setting. Level 2 teams are made up of individuals from a broad spectrum of the community, including district-level staff from human resources, safety risk management, student services, or special education; and community agency partners from law enforcement, community mental health, juvenile justice, child welfare, or the district attorney’s office. School safety is not just about schools—it’s about community safety. “We don’t live in a vacuum in schools,” McCarthy said, so it is essential to include community voices, perspectives, and resources in the efforts to keep schools safe.

Team Composition in CSTAG

Benjamin Fernandez (Fairfax County Public Schools) said that in his 28 years of working as a school psychologist in the CSTAG model of BTA, all the teams he has served on have been multidisciplinary. The teams often included school mental health professionals, administrators, nurses, special education professionals, and SROs, BTA team members also served on student assistance teams and provided a range of support to students as a part of their typical responsibilities to the whole school. Within this larger team, smaller groups could break off to focus on specific prevention strategies. For example, mental and physical health professionals may form a group to focus on substance abuse prevention. Everyone on the team was trained in CSTAG and participated in BTA. If a student was removed from school for a period, the same team could offer support to get the student reintegrated back into school. Fernandez said that it is vitally important to make sure that a team maintains its readiness and preparedness, and that good leadership is necessary to help teams stay focused and keep the team together.

Teams for Children with Disabilities

The composition of BTA teams is essential to ensuring we follow the requirements of IDEA and support students with disabilities who are involved in BTA, said Whittaker. For BTAs involving children with disabilities, it is critical to have people on the team who are familiar with federal laws that protect students’ rights, including IDEA and Section 504. If it becomes necessary to remove a student from school, Whittaker said, the threat assessment team needs to be aware of the student’s right to receive a free, appropriate public education, and to be able to identify and provide the support necessary so that the student can return to school as soon as possible. Further, the composition of the team should not be static but should instead include the professionals who know the students best and have knowledge of their existing needs and supports. As other speakers have mentioned, said Whittaker, the BTA is not a one-and-done process; it requires continuous case management and follow-up. The multidisciplinary team must include people who have the capacity and resources to provide long-term support to the student.

State and District Level Threat Assessment

Discussions around BTA tend to focus on the individual and school level after threats have occurred, said Ron Avi Astor (University of California, Los Angeles). Few models have threat assessment team members focused on prevention measures to assess threat risk for an entire school, county, district, or state, interacting across levels and providing resources, he said. Astor shared a model that he and his colleague developed, focused primarily on prevention at the district, county, and state levels. Training for professionals at each of these ecological levels is essential, said Astor, and student, teacher, parent, and community voices as part of comprehensive surveys or qualitative methods serve as the basis of prevention strategies among ecological threat assessment teams. Data from comprehensive surveys and focus groups guide

community, county, state, regional, and district resources to the highest-risk schools in a systemic way. Different types of BTA team members with different levels of authority and skills need to be nested in each of the ecological levels—state, district, and school—to make decisions and direct resources, said Astor.

Role of SROs

Mo Canady (National Association of School Resource Officers) began by defining SRO: “the school resource officer is a carefully selected, specifically trained and properly equipped full-time law enforcement officer with sworn law enforcement authority assigned by the employing law enforcement agency to work in schools utilizing community-oriented policing principles and strategies.”⁴ Canady emphasized that there is a “dramatic difference” between an SRO and a patrol officer working in a school building. The job of an SRO is the “most unique assignment in law enforcement,” he said, so it takes a unique officer to do it. SROs receive special training in mental and behavioral health for children and youth, de-escalation techniques, and adolescent brain development. The key to being a successful SRO, however, is being a good teammate. Relationships are critical to the role of the SRO, who works with a multidisciplinary team—including nurses, mental health specialists, counselors, and administrators—to advocate for student safety. Canady mentioned a program called Project Unite, which is an integrated system for school violence prevention. The program, which is meant to be led by trained multidisciplinary school safety teams, covers issues of culture and climate, bystander reporting and response, information sharing, and BTA and suicide risk assessment.

EMERGING TECHNOLOGIES TO AID BEHAVIORAL THREAT ASSESSMENT

Social media and artificial intelligence have altered the way we communicate, express emotions, and navigate social challenges, and the process of BTA must evolve alongside these changes, said Desmond Patton (University of Pennsylvania and Planning Committee Member). Moderating a session on the role of social media, AI, and other technologies in BTA, Patton noted there are both opportunities and challenges involved. On the one hand, AI can process vast amounts of data and identify potential threats through patterns in language, behavior, and interactions that might otherwise go unnoticed. On the other, AI systems are not infallible—they can misinterpret data, carry inherent biases, and generate false positives that have real consequences for individuals and groups who may be unjustly targeted. This is particularly concerning for communities of color who are often disproportionately impacted by such technologies, said Patton. These technologies hold great promise, but they must be deployed thoughtfully with input from a diverse range of voices and a deep understanding of the social context in which they operate.

Ethical Considerations

Patton asked panelists: “What are the key considerations for ethical implementation of emerging school safety technologies?” Christabel Randolph (Center for AI and Digital Policy) replied that the first ethical question is very simple—does it work and is it scientifically valid? Jeffrey Lane (Rutgers University New Brunswick) added that it can be hard to identify success in this area because the proof is in a counterfactual—a shooting that didn’t occur, or violence that didn’t happen. Second, said Randolph, ethical implementation requires transparency and accountability mechanisms before deployment. Students and parents need to know what data are being collected, who is processing and storing the data, and if the data are being made available to other parties (e.g., law enforcement). A third ethical requirement, said Randolph, is continuous validation and testing, with sufficient resources available past the initial investment. Clarence Okoh (Georgetown Law Center on Privacy and Technology) added that the foundation of any ethical regime is legal compliance such as compliance with disability rights protections. Another ethical consideration, he said, is ensuring that students are co-designers of policies implemented in their schools, and that the impact of a program on students is continuously assessed throughout its lifecycle.

Patton told participants about technologies designed to detect threats made on social media or through device monitoring. He asked panelists if these are useful for BTA. When tragedies happen, Okoh said, people are desperate for solutions. Randolph added that these technol-

__________________

ogies are simply tools for achieving a certain goal and said that there may be other more empirically sound, evidence-based mechanisms for achieving the same goal, such as investing in education or mental health support for students.

LOOKING FORWARD

Panelists shared their thoughts on unmet evaluation needs, potential solutions to methodological challenges, and future research directions. To set the stage for this session, Ali Rowhani-Rahbar (University of Washington, and Workshop Planning Committee Chair) recapped the future research needs presented earlier by Hendrix, including:

• Directly address how behavioral threat assessment affects student behavior, attitudes, and relationships in the short-term and long-term following a BTA;
• Longitudinally examine whether the proliferation of BTA programs has helped to reduce targeted violence in schools over the past few decades;
• Study BTA implementation and outcomes across a variety of models and different parts of the country;
• Prioritize the collection of nuanced implementation data using primary data collection strategies and by conducting longitudinal case studies of schools and districts using BTA;
• Account for variation in implementation practices when studying the impact of BTA on discipline and safety to understand the benefits and challenges of different approaches to BTA;
• Consider intersections or alignments between BTA and other common safety and violence prevention efforts in schools to understand whether BTA is more effective when used simultaneously with other safety approaches;
• Examine theoretically relevant mechanisms that might explain associations between BTA adoption and disciplinary and safety-related outcomes;
• Incorporate the voices of students, including students who have experienced a BTA, who were threatened, parents, teachers, and school staff; and
• Continue to study potential disparities and inequities in threat classifications and outcomes.

Bradshaw began by sharing the statistic that it takes 17 years for an innovation to move from basic research to practice, with the interim years consisting of research, peer review, research synthesis, and guideline development. There is a need to study evidence-based models and provide guidance for practitioners in the field, rather than “just throwing them out there for schools to decide on their own,” said Bradshaw. In addition to selecting good models, ensuring that schools implement them with fidelity is essential. Successful implementation of a program requires funding, training, and support for sustainability; sometimes the longer-term aspects of implementation get forgotten when schools rely on one-time, short-term grants, she said. Moving forward, Bradshaw said that schools need to focus on optimizing implementation, integrating BTA with student behavior and mental health prevention efforts, and investing the time necessary to train staff for effective BTA.

Cornell focused his remarks on the barriers to researching BTA and how these barriers could be overcome. A National Center for Education Statistics survey found that 85 percent of schools report using threat assessment teams, he said, but there are many unknowns about the models they use, the training they conduct, and the outcomes they see. One barrier is that schools are overloaded and understaffed; participating in research on a school safety approach may not be a top priority when staff are already stretched thin. Finally, said Cornell, it is challenging to conduct quality research on BTA because collecting real-time, prospective data is expensive and would face issues with privacy and confidentiality. He offered five suggestions for how to improve BTA research: a) encourage funding for school BTA research; b) emphasize the need for controlled studies and outcome studies; c) clarify what constitutes a BTA model and test core features; d) recommend minimum standards for school records; and e) support training and implementation standards.

Michael Dorn (Safe Havens International) described what he has learned from his work evaluating school safety programs. Analysts at Safe Havens International con-

duct a deep assessment of each school, including interviews with key department heads, controlled scenario simulations, and a review of all policies and procedures. Through this work, Dorn said that he has seen a big gap between what people say they have in place and what they often do in actual practice. The most common problem, he said, is a lack of personnel—there are districts with 25 percent vacancy rates for teachers, social workers, psychologists, and other school-based professionals. Dorn suggested several areas where there are opportunities to improve school safety and the processes used: a) provide tools and training to allow local school officials to conduct fidelity testing; b) provide tools and training to help school districts reduce vacancy rates using improved school climate and safety measures; c) help districts improve student supervision to reduce incidents and as a result, free up personnel to focus on students with the greatest needs; d) for mid-sized and large districts, consider the use of full-time multidisciplinary student assistance teams and threat assessment teams; e) encourage the use of full-time suicide/self-harm prevention teams.

Lea Nepomuceno (Team ENOUGH) urged stakeholders to keep student voices at the center of discussions on school safety. Students are the primary recipients of the educational process, she said, and young people today have “grown up at the front lines of horrific mass shootings.” Nepomuceno said that students play a critical role in building better, safer school systems across the country. Nepomuceno said that it is important to keep in mind that the students who tend to volunteer to serve on boards or participate in programs are often not the students most impacted by the issues being addressed. Given this reality, she said it is important to do the work necessary to ensure as much diversity as possible in activities that involve student input.

Closing Reflections

Rowhani-Rahbar asked each planning committee member to identify takeaway messages to inform future work in this area. These included:

• Including families and students in the BTA process can be challenging, but the benefit is the wealth of information and insight they can provide. (O’Toole)
• There is a need to communicate with and listen to those who are most impacted by the system, and to continuously monitor implementation to better understand short- and long-term impacts. Specifically, this means talking to and listening to students, the disability community, and students and families of color. (Freeman)
• Investing in multi-tiered, data-driven systems of prevention works—these systems can improve outcomes for students and staff and reduce the number of threat assessments that need to be done. There is a need for clear messaging to schools, districts, and states about which strategies are supported by evidence and which are not. (Freeman)
• Schools are working within the reality of society as a whole; BTA teams operate within systems of structural racism (in education and juvenile justice, etc.) and threat assessment teams alone cannot resolve all forms of bias and discrimination. Given that, BTA shows remarkable promise in reducing racial disparities in school safety and discipline. (Madfis)
• In addition to studying examples of successful school safety initiatives, it is critical to analyze the failures as well; looking at what doesn’t work can lead to success in the future. (Rowhani-Rahbar)
• Threat assessment is not a point-in-time activity; it is a continuum that requires looking at what needs and supports already exist, what is happening at the time of a threat, and how a student can be supported moving forward with wraparound care for the student and their family. This cannot be done by schools alone but requires the help of the community and beyond. (Crepeau-Hobson)
• There is a significant gap in qualitative research on BTA; understanding how teams operate, how they are trained, and what is required for sustainability and effectiveness is critical. (Patton)

DISCLAIMER This Proceedings of a Workshop—in Brief was prepared by Erin Hammers Forstag as a factual summary of what occurred at the workshop. The statements made are those of the rapporteur or individual workshop participants and do not necessarily represent the views of all workshop participants; the committee; or the National Academies of Sciences, Engineering, and Medicine.

REVIEWERS To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Jennifer Freeman. We also thank staff member Nancy Whiting for reading and providing helpful comments on this manuscript. Kirsten Sampson Snyder, National Academies of Sciences, Engineering, and Medicine, served as the review coordinator.

PLANNING COMMITTEE MEMBERS Franci Crepeau-Hobson, University of Colorado, Denver; Jennifer Freeman, University of Connecticut; Eric Madfis, University of Washington; Mary Ellen O’Toole, George Mason University; Desmond Patton, University of Pennsylvania; Ali Rowhani-Rahbar, University of Washington.

SPONSORS This workshop was supported by contracts between the National Academy of Sciences and the U.S. Secret Service, National Threat Assessment Center (70RSAT21G00000003). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

For additional information regarding the workshop, visit: https://www.nationalacademies.org/our-work/k-12-threat-assessment-efficacy-and-implementation-evaluation-research-a-workshop

SUGGESTED CITATION National Academies of Sciences, Engineering, and Medicine. 2024. K-12 Behavioral Threat Assessment Efficacy and Implementation Evaluation Research: Proceedings of a Workshop—in Brief. Washington, DC: National Academies Press. https://doi.org/10.17226/27980.

Division of Behavioral and Social Sciences and Education Copyright 2024 by the National Academy of Sciences. All rights reserved.