Transforming Undergraduate STEM Education: Supporting Equitable and Effective Teaching (2025)
Chapter: 9 Role of the Institution in Creating Equitable and Effective Learning Environments
9
Role of the Institution in Creating Equitable and Effective Learning Environments
Prior chapters have described the Principles for Equitable and Effective Teaching and their application in specific teaching and learning settings. Here, we turn to the implementation of the seven Principles at an institutional level. This discussion is grounded in a concept of implementation and change as processes that shape and are shaped by a number of interrelated dimensions within the institution. A goal of continuous improvement throughout the system is needed to work toward equitable and effective teaching and to achieve a student-ready college that values and focuses on student-centered approaches to learning. Institutional leadership at an institution is essential for articulating and prioritizing goals and providing the financial and human resources needed for significant change. That is, achieving equity-minded change in our current system is a challenging endeavor that will likely only succeed through gradual changes to mindset and actors on multiple levels being open to multiple possible interpretations of data (Culver et al., 2021; Equity-Based Teaching Collective, 2024; Kezar & Dizon, 2019; Kezar & Posselt, 2020; Kezar et al., 2008).
Figure 9-1, below, illustrates the various institutional dimensions that we consider in relation to the seven Principles in this report. At their core, the Principles are built around both the socio-cultural and cognitive dimensions that research shows interact in a dynamic way during learning, shown in Figure 9-1 at the center of the diagram. Implementing the Principles at an institution involves the interconnected participation of all levels of the campus: the instructors (e.g., those who are teaching), the academic units (e.g., those who manage the academic mission), and the institutional leaders (e.g., those who make policy and resource decisions that impact how teaching
and learning can take place at the institution); this dimension is illustrated by the middle ring of Figure 9-1. And finally, as shown in the outer ring of Figure 9-1, all of this takes place within the institutional context—including its infrastructure, policies, and practices—which may support or be a barrier to enacting equitable and effective teaching. Moreover, aspects of the institutional context may be deeply entrenched, whether or not they are aligned with the current goals and vision of the institution. Examining these policies, practices, and infrastructure limitations is critical to supporting
NOTE: Inner circle: All learning involves a dynamic interaction between cognitive and socio-cultural processes and the Principles address both. Middle ring: Implementation of the Principles requires the integrated effort of individuals at multiple levels of the institution. Outer ring: Implementation of the Principles also depends on acknowledging the needs and challenges of the particular institutional context. The institutional context also impacts interactions of students, instructors, academic units, and campus leaders (middle ring) in ways that influence student learning (both cognitive and socio-cultural; inner circle).
continuous improvement and requires institutional leaders to be actively involved in changing them.
Higher education scholars often examine institutions from a “systems” perspective to understand and describe their change processes (Austin, 2011; Baecker, 2010; Henderson et al., 2011; Lemke & Sabelli, 2008; Scileppi, 1988). It has been posited that culture change requires multi-level aligned action and leadership from across an institution, including academic unit members and student affairs professionals, as well as deans, provosts, and presidents (Kezar, 2013; Kezar & Holcombe, 2015). It is highly unlikely that sustained and broad implementation of equitable and effective teaching will occur if an institution presumes that the responsibility for change lies only with individual instructors and what they do in their classes. While some particularly motivated and informed individuals may be able to implement change effectively and well, such efforts are unlikely to become the norm at the institution unless there is vertical support and horizontal integration for the Principles for Equitable and Effective Teaching. As implied by the outer ring of Figure 9-1, “Equity-minded teaching must be fostered not only by individuals but supported by the systems and conditions that surround individuals and their efforts” (Bensimon & Harris, 2023; Gonzales & Culpepper, 2024, p. 30; Liera & Desir, 2023).
CULTIVATE A CULTURE OF CONTINUOUS IMPROVEMENT
Improving science, technology, engineering, and mathematics (STEM) teaching at the undergraduate level so that it is more equitable and effective is an iterative process that requires sustained effort over time to ensure the best practices and policies are in place. That is, work to achieve equitable and effective learning environments is a journey and not a destination that can be defined today and reached tomorrow. The concept of continuous improvement, originally developed in manufacturing, can usefully be applied to this type of STEM education reform (Singh & Singh, 2015; Temponi, 2005). Continuous improvement does not focus on continual change but, rather, on evaluating the outcomes of a change and then using the information to guide actions to improve the process (Jha et al., 1996). Continuous improvement approaches can help to address the complexity of culture change and may be more likely to lead to sustainable transformation (Glover et al., 2015; Michela et al., 1996; Mitchell & Sickney, 2019; Reinholz et al., 2019. This process requires iterative cycles of evaluation, reflection, and action. By fostering a culture of continuous improvement, institutions can ensure that efforts to enhance equity and effectiveness in teaching are sustained and refined over time, creating lasting and impactful change in higher education.
Multiple national initiatives have approached teaching and learning through systemic approaches, including the National Academies of Sciences, Engineering, and Medicine’s (National Academies’) Roundtable on Systemic Change in Undergraduate STEM Education.1 Studies have explored the concept of organizational change networks as a lever for systemic change by focusing on the examples of the Advanced Technology Education program of the National Science Foundation (NSF), the Bayview Alliance, the Center for the Integration of Research, Teaching and Learning (see Chapter 8), the network of STEM education centers, the Partnership for Undergraduate Life Sciences Education, and the Association for Undergraduate Education at Research Universities, all of which have been players in the effort to improve undergraduate STEM education (Austin et al., 2024). Recognizing all students, faculty, staff, administrators, and members of the public as valuable and capable, Effective Practices for Physics Programs (EP3)2 provides structured resources informed by practices across the physics community to address challenges and make changes and improvements. Recognizing the challenges of teaching in complex ecosystems, the Equity-Based Teaching Collective, composed of faculty from American University, Florida International University, and the University of Connecticut, has created an ecosystem-based report for understanding the complex roles of different stakeholders and levers in the improvement of college teaching and how they (i.e., institutional leaders, centers for teaching and learning, deans and department chairs, faculty, and students) can interact to improve college teaching and promote equity-based teaching and equitable outcomes for Black, Indigenous, and low-income students. Their work included a large landscape analysis, interviews, and synthesis efforts that informed their recommendations and provided resources for the examples they cite (Equity-Based Teaching Collective, 2024).
Understanding Theories of Organizational Change
Structures, policies, practices, and operating principles of modern higher education replicated those first established by the colonial colleges before the nation’s founding (see Chapter 2). These institutional underpinnings are difficult to change because they form part of the infrastructure of the institution’s operation—they are entrenched and undergird the institutional culture. The changes that are needed now must be carried out intentionally and systemically rather than allowed to evolve slowly, in part because equitable experiences are morally urgent and in part because our interconnected society is changing more rapidly than ever before. The challenges faced
___________________
1 More information is available at https://www.nationalacademies.org/our-work/roundtable-on-systemic-change-in-undergraduate-stem-education
2 More information about EP3 is available at https://ep3guide.org/
by our society today are global and cross-disciplinary challenges. In order to prepare students to be contributors in today’s world, there is a need to prepare a diversity of individuals who can bring their particular forms of creativity and problem solving to the challenges of this century.
Creating an equity-minded institution requires organizational change. Organizational change can be described, broadly, as being one of three types: first order, second order, or third order (Gonazales & Culpepper, 2024). These types are distinguished from one another by the extent to which existing structures are affected. Equity-minded change, generally and more specifically in support of teaching, is most likely to take hold when institutions invest simultaneously in first-, second-, and third-order change. Table 9-1 gives a brief description and example for each type of organizational change.
TABLE 9-1 Types of Organizational Change by Scope
| Type of Change | Scope (e.g., depth, magnitude) | Example |
|---|---|---|
| First-Order Change | Targets, or is meant to be compatible with, existing structures, and thus does not seek to change those structures or their foundations. | Implementing a recruitment effort to diversify the STEM faculty and student body on a campus (see Hrabowski, 2012; Palmer et al., 2013) |
| Does not affect the foundations or core of the organization. | ||
| Second-Order Change | Targets relationships, practices, and norms in phases or in ways that are otherwise limited/contained. | Implementing affinity groups wherein minoritized and marginalized students can build community and share strategies for navigating their disciplines and departments (see Villa et al., 2013) |
| These changes are often more significant than first order change but also do not address the core or foundations of an organization. | ||
| Third-Order Change | Often described as deep, transformative, and implicit. It targets the deepest foundations of the organization, or its core, including mental models. | Ongoing work to design educational and policy initiatives that enhance faculty, staff, and student understanding of the dominant culture-focused foundations of institutions, disciplines, etc. (see Liera, 2023) |
| Third-order change involves or assumes first- and second-order change and requires adjustments at the individual, sub-organization, and whole organization level. |
SOURCES: Adapted from Gonzales and Culpepper (2024, p. 17), informed by ideas from Bartunek and Moch (1987) and Kania et al. (2018).
Most diversity efforts are what organizational theorists would deem first-order changes (Bartunek & Moch, 1987; Kania et al., 2018), which focus on change that can be quickly adopted—e.g., by doing “more” of what an organization does rather than fundamentally altering conditions within the organization. Some of these efforts are performative, such as public statements about the value that an institution places on inclusion. Some are actions, such as an invited speaker or an optional one-time workshop, which are unlikely to lead to any change and may not reach people who are least informed about issues of exclusion, privilege, and implicit bias. David Asai, former Senior Director for Science Education in the Howard Hughes Medical Institute, where his team developed initiatives advancing inclusive STEM learning, recently stated that “when confronted with the truth that STEM lacks diversity, our first impulse is to recruit more persons from the underrepresented groups” (personal correspondence from David Asai to Gabriela Weaver, October 2024). But starting with representation is a mistake, as shown by the failure to make progress in closing the racialized STEM completion gap even after three decades of diversity-centered interventions (Asai, 2020; Bernard & Cooperdock, 2018). A focus on equity and an equitable, inclusive environment can create a space that will be inviting to a diverse group of people—students, staff, and faculty (Griffin, 2020).
Many inclusion efforts tend to fall under second-order changes, as they focus on small and contained alterations to pre-existing practices and relationships (Bartunek & Moch, 1987; Kania et al., 2018). These are important and necessary because they begin to normalize the conversation about equity and provide some ways to act on it. An example of a second-order change that can be helpful is creating a forum where first-generation faculty members can meet with first-generation students and share experiences or provide encouragement about college life. Another example is to provide ongoing PLD for educators and administrators to learn about and recognize their own unconscious biases, and how these may affect their interactions with others and their decision making.
Any effort to make STEM more diverse, inclusive, and equitable demands the deepest and most difficult kind of change: third-order change. This approach requires new or revised policies and practices, long-term reallocation of resources, and ongoing learning and reflection to shift mental models about how academic units, colleges, universities, and/or the profession should work. These underlying models, structures, and culture are what maintain inequity and need to be addressed through transformational change (Kezar, 2018; McNair et al., 2020). As discussed earlier, contributions from all levels of the institution will be needed to achieve this depth of change. By their nature, institutions have nuanced and distributed systems of power and autonomy, and different stakeholders may have competing priorities (e.g., Vican et al., 2020). For this reason, transformation requires
institutional leaders to take an active role in creating change by facilitating and supporting widespread efforts. Researchers have observed that norms that exist within professions tend to change when individuals observe others in their social group exhibiting different behaviors (e.g., individual-level change); when they view these behavior changes being adopted collectively (e.g., departmental and disciplinary change); and when they receive institutional rewards that validate and legitimize the change (e.g., college, university change; Tankard & Paluck, 2016).
Disruptive, third-order change requires institutional equity-mindedness. The presence of diversity (e.g., achieving a designation based on student demographics) is not sufficient to achieve this: institutional culture matters more. Equity-mindedness is necessary because the exclusion of certain classes of individuals from higher education is a systemic problem (Liera & Desir, 2023). Analysis and understanding of the problem are needed before implementing solutions—otherwise the result will be superficial change. Equity-mindedness also requires an awareness of the challenges faced by different populations of students, the practices that affect these students, and the importance of using data (specifically data disaggregated by the population of the students at the institution) to identify and address inequities (Bensimon, 2018; McNair et al., 2020). Success in implementing equity-mindedness requires a sense of shared responsibility and accountability for advancing equity in policies, practices, and norms, meaning equity is everyone’s responsibility (Gonzales & Culpepper, 2024).
CREATE STUDENT-READY INSTITUTIONS
The experience of teaching and mentoring undergraduate students in STEM varies by institutional context, mission, and perhaps, most importantly, across levels of student preparedness to learn the course material. Concerns about underprepared college students are a perennial complaint across post-secondary education, and the COVID-19 pandemic magnified opportunity and preparedness gaps for many students (Bailey et al., 2021; National Academies, 2021). Rather than blaming students and other stakeholders (e.g., parents, high school teachers, the Pre-K–12 education system) or focusing on students’ college readiness, McNair et al. (2016) argue for rejecting the concept of “college readiness,” replacing it with “student-ready”—institutions that meet their admitted students where they are. Institutions ranging from community colleges to highly selective research universities have differing approaches to student admission (see Chapter 2 for more). No matter the process, the students who show up at these institutions have been admitted, and educators and administrators have an obligation to provide a learning environment that supports their education and learning. To create student-ready institutions requires attention to who
has control or influence on which elements in the system. There is variation in the extent that instructors can influence the design of the course or curriculum design, but they usually have control over pedagogical approach and their classroom policies.
Assess Resources and Infrastructure
The creation of an effective and equitable institution that values, promotes, and rewards equitable and effective teaching and learning can be guided by the data and physical infrastructure made available at the instructor, classroom, course, department, college, and university leadership levels. While some instructors may be familiar with institutional graduation and retention data, sometimes disaggregated by demographic and additional variables, many have not had access to data at finer levels of detail. Even fewer have been able to connect instructional revenue and costs with student outcomes.
Physical infrastructure, including items such as classrooms and the pedagogical opportunities they afford, technology capabilities, student study spaces, experimentation spaces, learning laboratories, testing centers and more, and how these spaces are used and allocated for the purposes of teaching and learning, can affect both the costs and outcomes of the educational experiences. These issues are complicated by increasing demand for courses in certain disciplines (e.g., computer science), which has led to difficult choices at some institutions where course sizes have grown, responsibilities of instructors have increased, or student access to courses has been restricted (Alonso, 2023; Nguyen & Lewis, 2020). However, many institutions have worked to make courses more equitable and effective with minimal changes in instructional costs. An equitable and effective teaching approach emphasizes student access and success as metrics that are considered as part of the resource allocation process. A 2024 convening at the Howard Hughes Medical Institute (HHMI), the Financial Alignment with Inclusive Teaching Effectiveness (FAITE),3 brought together institutional financial officers from the National Association of College and University Business Officers with institutional change agents, data analytics experts, and academic leaders. The primary purpose of the workshop was to determine approaches to align financial decisions with inclusive instructional practices and student success metrics. Discussion focused on how data can be used to drive informed decision making that supports institutional improvement, including balancing instructional costs with financial realities, supporting systemic change for increased access and equity, and developing
___________________
3 More information about FAITE is available at https://ascnhighered.org/ASCN/cost_benefit.html
new organizational structures to sustain inclusive teaching innovations. While this type of work is just emerging, it has the ability to have a major impact on equitable and effective teaching since it is difficult to remove financial constraints from what is deemed possible, or desirable, in instructional activities.
The impact of institutional finances on STEM instruction can be seen, for example, when looking at the various roles of introductory STEM courses, both financial and educational. Introductory STEM courses are frequently taught to large numbers of students (and at research institutions are responsible for supporting many graduate students teaching assistants); they are therefore important for institutional revenue (Hatfield et al., 2022; Hopp et al., 2019). There are significant expenses with offering these courses as well, and this sometimes leads to the use of extra “differential” tuition costs for students as institutions try to recoup some of their expenses (Harwell, 2013; Wolniak et al., 2018). Foundational STEM courses have the highest breadth of incoming student backgrounds and expertise. and typically require a larger proportional investment in terms of instructional spaces, resources, learning materials, and preparation of instructors and teaching assistants, all of which are reflected in the financial data and are a key factor in student-centered financial planning.4 These are just a sampling of the issues where combining various forms of data, including financial, can help inform institutional decision making about instruction and the extent to which equitable and effective teaching is prioritized. Institutional leaders often have to make difficult decisions about competing priorities and finite resources. Various forms of data, including financial data, can help administrators, leaders, and instructors to build a culture and infrastructure that supports equitable and effective teaching with an eye toward sustainable, long-term programmatic and instructional changes.
Infrastructure, policies, and practices also relate directly to data in terms of what data are available to whom and for what purposes. Data are often siloed within institutions, which inhibits evidence-informed decisions related to instruction, students, and instructional staff (Swing et al., 2016). Instructors and administrators are often unaware of the challenges that students face and the complexity of the classroom situations that can arise in STEM courses, as well as the bottlenecks created in STEM curriculum (as discussed in Chapters 6 and 7). Data, made usable in transparent and multi-dimensional representations, coupled with professional development around its interpretation and use, can provide substantial support for the creation of equitable and effective instructional environments. When
___________________
4 More information about the benefits of student-centered financial planning is available at https://www.nacubo.org/Press-Releases/2024/NACUBO-Student-Success-Hub-Highlights-Financial-Links-to-Equitable-Student-Outcomes
financial data are also included, there is an opportunity for leadership, instructors, and business officers to work together to develop approaches that are feasible and prioritize equitable and effective teaching in a sustainable manner that benefits the institution as a whole.
Student-Focused Policies
One important function of institutional leaders in supporting equitable and effective teaching is an examination of policies that impact teaching and learning. One example of a policy that directly affects student success is transfer articulation (see Chapter 7). Credit hour limits, procedures for transfer requests, and agreements among local or state institutions must be decided by institutional administrators; academic advisors and program directors are responsible for developing course-to-course equivalencies; faculty are most likely to be the ones to look at course learning objectives and syllabi to ensure that the content meets the needs of the students within a given program (see Chapter 6). Institutional alignment across these efforts will ensure that decisions at each level are guided by the same values, promoting equitable and effective teaching for all students, as articulated in the seven Principles of this report. Lacking such alignment can lead to the exclusion of some groups of students. For example, when students transferring from a two-year college are not given equivalent credit, such as for a first-year biology course, only because a research institution does not believe the two-year college version of the course has sufficient “rigor,” then populations of students who are likely to begin their higher education career through a two-year college route (a larger percentage of underrepresented, low-income, and/or first-generation students) will be systematically and unfairly penalized by the receiving institution requiring them to take a course a second time in their college career.
Other policies that affect students’ equitable access to an excellent education are those related to the timing of course withdrawals and the impact of withdrawals on prior grades, policies connecting a student’s permission to register for courses based on their account balance, practices around grades in a student’s first term in college, academic integrity policies and how they are reported, monitored, and enforced, practices around admission to limited-enrollment degree programs, and rules and procedures related to changing a major. Students who must work in order to pay for school may be at a significant disadvantage when their ability to register for the following term’s courses is held up. Each of these policies, when examined with an equity lens, can derail the educational plans of some students more than others based on their race, income level, or other identity characteristics. An equity lens, or “equity-mindedness,” has been described by scholars (Bensimon, 2007, 2018) as a way of thinking that centers—brings
attention to—and seeks to resolve persistent patterns of inequity. The underlying message is that the reasons behind any given policy or practice should be consistent with the educational mission of the institutions and not inadvertently put barriers in front of some groups of students.
Instructor-Focused Policies
Creating a student-ready institution that provides and supports equitable and effective teaching involves not only policies that directly affect students, but also those that have a direct impact on instructors and an indirect impact on students. Among these are the extent to which they have access to data about their students, support to utilize those data, expectations for teaching, such as numbers of courses/sections/students, modality of teaching, and time to prepare and support for improving teaching such as that provided by centers for teaching and learning or other related structures (Molinaro et al., 2020). It also matters how teaching assignments are allocated (to different categories of instructors) and resourced: support staff, types of teaching spaces, availability of needed equipment or materials, course scheduling, and so on. Some of these decisions rest solely within the purview of the academic unit (Chapter 6) but all of it can be influenced by policies and cultural expectations created by the institutional leadership. These policies and actions are embedded in the system depicted in Figure 9-1. In addition, realistic access to professional learning requires that institutional leaders provide realistic compensation for the time required for everyone who teaches to do that learning and subsequently apply it to their courses.
A practice that spans an entire institution—both across disciplines and vertically through levels of leadership—is creating true access to professional learning and development (PLD) and an expectation that faculty will participate (Chapter 8 discusses various approaches to PLD). While faculty have expertise in their academic discipline, formal preparation in methods of educating others varies. Furthermore, there is plentiful research demonstrating that some approaches to teaching are more effective and equitable than others in higher education (Freeman et al., 2014; Sathy & Moore, 2020; Theobald et al., 2020) It follows that professional learning is needed if institutions expect their instructors to educate in equitable and effective ways, and for all students to learn and graduate successfully. Access to professional learning means that all categories of faculty, instructors, and teaching assistants can develop the necessary skills with evidence-based pedagogies and inclusive teaching practices through ongoing programming and resources available to them through their institution or professional societies. As mentioned in Chapter 8’s discussion of professional learning, a single workshop or isolated experience is not likely to be effective unless
it instigates further engagement by instructors. In most situations, research has shown that long-term professional learning experiences are needed in order to increase the likelihood of success (Bifulco & Drue, 2023; Castillo-Montoya et al., 2023; Hakkola et al., 2021; Miller et al., 2022).
Compensation and hiring policies may also have an impact on student success at the institution. Hiring practices and how they influence the diversity of faculty across departments and faculty categories are important for developing an inclusive community for all students. Simply hiring diverse faculty is not enough. Work at all levels is needed to ensure that effective faculty are retained and promoted. Valuing teaching from all types of instructors is one step to cultivating an equitable environment.
These faculty-focused policies are an example of actions that require an alignment of efforts between unit practices and institutional policies. For example, one important component of faculty retention specifically related to teaching is how teaching evaluation, recognition, and reward practices are carried out. This is clearly a complex subject that interacts with many other components of the larger system, and one that does not always get prioritized by the academic unit or the institution.
Reflect on the Role of Grades
Letter grades are considered a staple of the educational system, but the history of the system is more complex than usually recognized today (Durm, 1993; Schneider & Hutt, 2013). In higher education, there have been numerous approaches to grading over the past several hundred years; a 100-point scale, common in the 19th century, was phased out in the 1900s such that by the 1940s an A–F approach was fairly ubiquitous, potentially still tied to a 100-point scale but considered more reliable due to potential inconsistencies with how points were assigned (Schinske & Tanner, 2014). A detailed description of both the history of grading and what grading is meant to achieve for students and the institution can be found in the paper by Schinske and Tanner (2014), including a discussion of the emergence of grading which is commonly associated with foundational STEM courses:
As research on intellectual ability appeared to show that, like other continuous biological traits, levels of aptitude in a population conformed to a normal curve, some experts felt grades should similarly be distributed according to a curve in a classroom (Finkelstein, 1913). Distributing grades according to a normal curve was therefore considered as a solution to the subjective nature of grading and a way to minimize interrater differences in grading (Guskey, 1994). Others worried that measuring aptitude was different from measuring levels of classroom performance, which might not be normally distributed (Schneider & Hutt, 2013). (Schinske & Tanner, 2014, p. 158).
More and more, the use of a normal distribution within the sample of an individual course has led to concerns about the validity of grades representing learning (Bowen & Cooper, 2021; Eyler, 2024). Some schools have experimented with the removal of grades in favor of assessment models like student portfolios or narrative grading, while others have looked at using “badges” or “microcredentials” to signify mastery of specific skills rather than approaches that assign letters or numbers without context and therefore do not connect grades to specific skills or abilities (Ferguson & Whitelock, 2024; Reed, 2023). Some institutions have chosen to not calculate grade point averages (GPAs) and/or not record failing grades (e.g., Burke, 2022), as repeating a course is considered penalty enough and recording failing grades creates a GPA burden that is very difficult to overcome. More recently, there have been suggestions of alternative grading approaches usable in classrooms (Townsley & Schmid, 2020) that attempt to make grading more relevant and equitable, including approaches such as specifications grading. Specifications grading is a term for a combination of multiple types of grading approaches. It includes mastery grading in which students demonstrate proficiency before continuing to the next topic as well as competency-based grading where students can show mastery through a choice of assessments. A variation is contract grading in which the instructor offers multiple options in advance and the students have choices about which components to complete, and the anticipated grade or requirement for credit is known in advance based on the choices selected by the student. The instructor selects the options for assignments in a way designed to ensure students meet the learning objectives for the course (Bonner, 2016; Howitz et al., 2021; Larson, 2023; Nilson, 2015; Townsley & Schmid, 2020).
Some research suggests that the most commonly used grading systems have the most negative impact on students with the lowest number of systemic advantages and that small differences in points awarded that lead to different letter grades can impact a student’s choice of major (Castle et al., 2024; Dika & D’Amico, 2016; Koester et al., 2016; Li & Xia, 2024; Malespina & Singh, 2023; Matz et al., 2017; Seymour & Hunter, 2019; Whitcomb et al., 2021; Wright et al., 2016; Xie et al., 2015). Therefore, these approaches to grading create barriers to inclusive STEM outcomes at the course and program levels for students belonging to one or more of the commonly underrepresented groups: low socio-economic status; first-generation college attendance; female; lesbian, gay, bisexual, transgender, queer/questioning, intersex, asexual/aromantic/agender, plus other related identities (LGBTQIA+); and/or racial/ethnic minoritized status. Furthermore, even with all the variations in grading approaches—which arise for a range of factors, from different instructors teaching the same course to inconsistencies between courses in an academic unit, and from differences
between academic units to the even more varied approaches that are possible between different institutions—it can be easy to make high-stakes decisions about who is qualified to continue in STEM, receive a STEM degree, and how worthy they are as graduates—all based on the imprecise measure of grades.
Campus leadership also has a role in shaping the use of grades; one face of this is to organize an analysis of the courses where inequities across groups are more prevalent, with particular emphasis on large, introductory courses. Such an audit of grading practices and outcomes requires follow-up with support for implementing improvements in grading practices that are supported by the literature to avoid introducing additional bias into the grading approaches (Bowen & Cooper, 2021; Ekstrom et al., 1994; Felder & Brent, 2024; Feldman, 2023; van Dusen et al., 2021).
In the end, administrators need to consider that historically, grades have been assigned a tremendous amount of value, but they are actually an approach that takes limited input and assigns those generalizations an outsized impact. Additional research on more holistic approaches to evaluating the capabilities of STEM students is needed, with special attention to emphasis on approaches that value the ability to learn and overcome struggle.
ANALYZE INSTITUTIONAL DATA
This section focuses on how data can be leveraged to define goals, measure progress, and guide movement toward equitable and effective STEM teaching at the institutional level. We purposefully choose to consider data as they pertain to the range of interrelated dimensions that together make up the institution, from the individual instructor in the classroom to the college or university administrator who makes and implements institution-wide policies. Data are crucial at every level as stakeholders think about goals, measure progress, and guide improvement. The committee acknowledges that structures and resources may vary widely among institutions, so we discuss issues from a general perspectives and guidance that can be applied in a variety of ways.
As reflected in Principle 5: Multiple forms of data, data and self-reflection are important tools that provide evidence to inform continuous improvement. Data alone, however, do not make change happen. Data are an input to decision making and one component of raising awareness of potential challenges in achieving equitable and effective teaching—an awareness that can be used to increase buy-in and participation across the institution. Without data, understanding what is and is not working in undergraduate education, and why, becomes extremely difficult, leaving stakeholders to make decisions based on anecdotal experiences and individual perspectives. It is crucial, then, to use data to determine what the potential challenges are and to discern what further data points might be needed to craft a solution.
Data are part of a complex system that can help reveal insights into intent, support, and culture and can reveal information about these various factors and their interrelationships. Thus, it is important to remain aware of the power dynamics at play when data are gathered, interpreted, and used to inform action. As Taylor (2020) described, leaders of student success initiatives are increasingly pressured to use the language of data-driven decision making to justify programs and their corresponding results. There may also be significant challenges for everyone involved in student success initiatives to access the data used to drive decision-making processes, and data-driven decision making can pit student success initiatives against each other by setting up competition for resources, and silence institutional personnel who push back against data-driven reform efforts (Taylor, 2020). Taylor’s analysis shows that quantitative data are often inappropriately treated as superior to qualitative data points when both are valuable. The perspectives and life experiences of the people designing studies and those collecting and analyzing qualitative and quantitative data influence the categories that are chosen and many other aspects of data use (Castillo & Gillborn, 2022; Delamont, 2012; Gillborn et al., 2023; Hatch, 2023).
Ways of Using Data
There are multiple types of data and multiple ways to use data. Both quantitative and qualitative data can be used for strategic planning, program evaluation, and instructors’ annual reviews. Quantitative data may comprise the student outcomes that programs often use to gauge students’ progress, including course pass/fail/withdrawal rates, Fall-to-Fall retention, enrollment numbers, and student responses to class evaluations, among others. Qualitative data may be in-class assessments given by the instructor asking students how they feel about their learning (e.g., exit tickets students submit before leaving class, Plus/Delta charts where students submit quick answers to what added value (plus) and what should be changed (delta), exit interviews when students leave or graduate from programs; focus groups and individual interviews with students; and content analyses of artifacts (e.g., syllabi, student assignments). Different data points can be used to guide planning and action at the course, program, department, college, and institutional levels. McNair et al.’s (2020) book From Equity Talk to Equity Walk: Expanding Practitioner Knowledge for Racial Justice in Higher Education provides a thorough discussion of how equity-informed data can guide reform at multiple levels. The book is centered around themes of great relevance to equitable and effective education and how to promote equity through alignment of mission and priorities, and strategies. Institutions can invest in or fine-tune their data structure to support such missions and priorities and to inform strategies. For example, Meraz (2022)
described an effort in the California community college system wherein faculty were given access to an equity dashboard that allowed faculty to view the academic performance of their students based on gender, race, and ethnicity. These data, along with targeted professional development resources and communities of practice, allowed faculty to diagnose issues in their teaching and make improvements (Carlson, 2022; Gonzales & Culpepper, 2024; Meraz, 2022).
Thoughtful data collection and usage involves three important areas: intent, guidance, and cultural context. Each of these areas can be considered at the class, academic unit, or institutional level. Here we provide examples at various levels as all are relevant to the thinking of institutional leaders using data to drive change.
The intent behind collecting and using data can span from supportive to supportive and critical (called here as “critical friend”) to purely judgmental. Take, for instance, the range of responses to the instructor of a course with a large gap in D, F, and withdraw (DFW) rates by gender or race. A purely supportive approach may ask questions to better understand the situation and why inequities exist and what is within the instructor’s control to alter; a slightly more critical friend might highlight teaching practices that have been shown to reduce DFW rates with the hope that the instructor sees the value in engaging in professional development in these areas. A critical friend approach could employ the supportive approach, acknowledging that a change is needed to improve the situation, but also work with the instructor to identify and strongly suggest potential interventions while supporting ongoing data exploration. A judgmental approach would label the instructor as not able to work with different groups of students and take some punitive action based on the judgment.
At the department level, a department chair may see data that one or more of their introductory courses have an unacceptable DFW rate of 30% to 60%. A supportive approach in this case might consist of acknowledging the poor outcomes with colleagues and blaming the admissions unit, the preparation of the students, and/or their willingness to work hard. If adopting a critical friend perspective, the chair could acknowledge the challenge of working with students from varied backgrounds, acknowledge the rates are unacceptable, and create a plan supporting instructor development and course structure and pedagogical experimentation to work toward improvement. A judgmental perspective may have the chair decide that the current instructors are incapable and replace them with a new group to try again and hope for a better result. An important point is that the intent can be aimed toward the students, the instructors, and/or the administrators. What appears to be supportive, critical, or judgmental can dramatically change based on who the actions are aimed at. Box 9-1 gives a brief example of
BOX 9-1
Moving Toward Equity via Course Redesign
Using the disaggregated data for formative and summative course redesigns has led to marked improvements in student success. For instance, the data on DFW rates for students in math gateway courses at the Department of Mathematics at The University of Texas at San Antonio helped to drive initiatives for change.
The institution made decisions that decreased class sizes, increased the salaries of the faculty teaching these courses, and brought in more teaching assistants to provide students with personalized support.
Increased performance in mathematics was seen, with the D, F, and withdraw rate on these courses dropping from 35% in Fall 2019 to 25% in Fall 2021. Furthermore, these gains were sustained over time and through the challenges of the pandemic.
Academic innovation is fundamentally rooted in data-informed strategies, underscoring the importance of consistent data analysis to shape effective educational practices (Principle 5: Multiple forms of data). This precise approach enables specific interventions tailored to the diverse needs of student groups (Principle 2: Leveraging diverse interests, goals, knowledge, and experiences).
SOURCES: Vito et al. (2024); Gutiérrez (n.d.).
how data have been used by an institution to inform redesign of courses in ways that promote higher student success.
In connection with intent comes the approach toward guidance. Some instructors will receive significant guidance on how to use data and others will be left to figure it out on their own. When considering the main ways that instructional data are shared with instructors and administrators, it is often via a report or dashboard. Those modalities by themselves do not dictate the level of guidance that accompanies them, but often they end up as stand-alone tools that are intended to inspire the “necessary” actions in their users but do not clearly identify or enumerate those actions.
Some reports and dashboards come with informational material to provide guidance both for interpretation and potential actions. This guidance can be written into the report and/or part of a compilation of interlinked dashboards, each with its own individual guidance, or can come in the form of formalized instruction. The lessons can be short term, allowing full access upon completion, and/or can take the form of a more extended learning community where faculty and administrators are exposed to the assumptions, limitations, and capabilities of the data along with ideas for structuring student-centered asset-based conversations and actions at the level of instructors and department administrators.
A few examples of this work include (a) the Foundational Course Initiative (FCI)5 at the University of Michigan where large introductory courses are revised with departmental teams working with the Center for Research on Learning & Teaching and supported by extensive disaggregated data, (b) the Indiana University faculty Learning Analytics Learning Community,6 (c) the California State University Equity Dashboards and Community College dashboards,7 and (d) the Sloan Equity & Inclusion in STEM Introductory Courses Stem Equity Learning Community project8 and resulting course equity reports and communication approach (Carlson, 2022; Meraz, 2022, Rehrey et al., 2019).
Often, these more elaborate approaches are facilitated by teaching and learning center staff, an institutional researcher/learning analytics expert, and/or an administrator focused on educational effectiveness. Since guidance in data interpretation always comes with a point of view, the perspective of the author/deliverer of the guidance needs to be explicit to avoid concerns of hidden messages. In the learning community situation, the group facilitator helps to guide, but not completely dictate, the direction of conversation allowing this approach to often be more palatable and accessible to multiple viewpoints and interpretations of the data (Margalef & Roblin, 2018; Ortquist-Ahrens & Torosyan, 2009).
Lastly, we turn to the importance of cultural context. Culture includes behaviors, norms, and expectations defined at the disciplinary, institutional, and departmental levels with disciplinary and departmental norms often playing the key role in STEM fields. The setting in which data are interpreted can influence how they are perceived and interpreted, and potentially which actions are taken based on them. Potential questions to ask to help reflect on the setting, culture, and context that might be influencing data use include the following: Does the discipline pride itself on the complexity of their area? Is interdisciplinarity valued? Is there a belief that few are able to succeed in the area due to exceptional intellect needed to succeed? Is there greater value placed on abstract/theoretical components than applied? Does the department believe they are one of the few upholding rigor? Does the institution espouse and reward an equity-minded student-centered perspective toward undergraduate education? The answers to these questions and more could affect what data will be collected, who they will be shared with,
___________________
5 More information about the FCI is available at https://crlt.umich.edu/fci
6 More information about the Learning Community is available at https://ctl.indianapolis.iu.edu/Programs/teaching-with-learning-analytics
7 More information about the Course Equity Portal is available at https://cep-info.dashboards.calstate.edu/
8 More information about the SEISMIC Collaboration is available at https://www.seismicproject.org/about/overview/
how they will be perceived and, ultimately, what impact they may have on STEM education.
Even after the data and approach to their use are clear in the form of intent, guidance, and cultural context, it is still important to consider the individual, or individuals, using the data and their own individual experiences with data and with teaching. Often STEM faculty and administrators have dealt primarily with quantitative data. While they will often have some experience with survey data, data from sources such as observations, focus groups, and interviews can be less familiar. In addition, some STEM focused researchers may be uncomfortable incorporating qualitative data that apply to feelings, perceptions, and opinions, and may benefit from guidance on how this type of data can be interpreted.
Measuring Progress in Multiple Ways
The measurement of progress implies the use of one or more forms of data to make informed decisions about the current state as well as suggest metrics for a desired state. Two forms of data, and an intermediate form, are important to consider: quantitative, qualitative, and categorical (a mixed form containing features/elements of both). Quantitative data are numerical (e.g., age, test outcomes, number of credits attempted, percent of total points, course enrollment, etc.), tend to answer questions such as how many, how much, and how often, and are usually collected as part of the institution’s standard processes (admissions, Registrar) and through various assessments. Qualitative data describe perceptions, characteristics, behaviors, and experiences (e.g., course experience/evaluation of teaching, sense of belonging, perceptions of classroom and campus climate, etc.), tend to answer questions such as what, why, and how, and are collected through surveys, focus groups, interviews, and/or observation.
The intermediate form of categorical data (e.g., gender, Pell eligibility,9 race/ethnicity, major, letter grade, enrollment type, prior schooling, veteran, etc.) is based on generally constructed definitions and is usually collected through questionnaires or derived based on quantitative data (Pell status). Categorical data are generally considered qualitative, but they are separated here as they are based less on experience or opinion and more on shared definitions which tend to be mostly stable over time but are subject to the understanding of those that define it. For example, the dominant racial/ethnic group defines who is in the underrepresented group, but the
___________________
9 Pell-eligible students are those from families with low household income. The precise income levels vary with family size and several other factors. Students access Pell grants via the Free Application for Federal Student Aid. Pell grants are frequently used as a proxy for social and economic status in discussion of higher education.
definition is subject to interpretation and can change over time. This can be seen in how Asian/Asian-American workers are not considered underrepresented in science and engineering fields by the National Center for Science and Engineering, National Science Foundation (NCSES, 2023a), while the Department of Education classifies Asian American and Native American Pacific Islander Serving Institutions as Minority Serving Institutions based on their percentage of Asian/Asian-American students. Neither of the definitions used make the actual people involved immune to experiences similar to other minoritized groups. Categorical data also tend to raise a small number of questions of objectivity, though maybe it should raise more. Classification is complex but useful for data analysis. First-generation attendance, Pell eligibility, gender, and other categorical variables can and often do change definition over time so some students may fit one or more of these categories in any given academic year but not in the next.
In general, quantitative data hold a privileged position in STEM conversations likely due to the similarity in numerical analysis approaches applied in STEM research. Qualitative data often are the least appreciated in STEM as they are often relegated to the realm of “opinion” because they relates to perception and experience, realms most often associated with social sciences (which many research funding agencies in the United States recognize as part of STEM but many natural and physical scientists may not). Even with these challenges in what type of data may be favored, to cultivate equitable and effective teaching, data can be used to depict the lived experience of the students. This need will most often require all three forms of data to be available for analysis and interpretation.
Data Dashboards and Decision Making
Understanding what types of data are useful to measure progress does not mean they are readily available to those who can use them to support and enhance equitable and effective teaching. In most higher education settings, whether community college, four-year liberal arts, or comprehensive research institutions, the data are often siloed in a combination of settings that can include numerous categories:
- Admissions: past student records and demographics
- Registrar: course and program registrations and outcomes
- Financial aid: Pell grants and scholarships
- Personnel: instructor information
- Individual classrooms: course elements, student participation, and assessments
- Departments: evaluations and teaching assignments
- Special programs: participation
- Advising: participation and reasons
- Supports: tutoring and other support participation
- Budget and finance: income and expenses
- Facilities: classroom space affordances
- Health: physical and mental
- Extracurricular participation: clubs, sports
- Alumni: post-graduation outcomes
Depending on the size and resources of an institution, some of the data-collecting and interpreting functions may be combined. An institutional research entity (from one individual to many individuals) may have the express duty of combining data from multiple data silos and reporting this synthesis to leadership, with a subset of data used in reporting to a broader campus community. Externally administered undergraduate experience surveys (e.g., University of California Undergraduate Experience Survey,10 National Survey of Student Engagement,11 Community College Survey of Student Engagement,12 and student surveys by the Higher Education Research Institute13) can also inform improvement efforts. Data from surveys can provide a window into student experiences at the department, college, and institution levels over time, highlighting positive and negative trends. These tools can be expensive for institutions, however, and it may be difficult to get students to spend the time needed to provide the necessary information, which can result in low response rates and less helpful data.
Data governance and security, from completely absent to highly evolved, also determines the process and availability of data for informing equitable and effective teaching. Some institutions have handed over the process of data ingestion, clean-up, analysis, and visualization to private entities, often at substantial cost. This can lead to limited access for individual instructors. Local efforts focused on equity in introductory STEM courses have developed in a variety of institutions including those associated with the Association of American Universities (AAU) STEM initiative,14 NSF Improving Undergraduate STEM Education grants, HHMI Inclusive Excellence
___________________
10 More information about the University of California Undergraduate Experience Survey is available at https://www.ucop.edu/institutional-research-academic-planning/services/survey-services/UCUES.html
11 More information about the National Survey of Student Engagement is available at https://nsse.indiana.edu/
12 More information about the Community College Survey of Student Engagement is available at https://www.ccsse.org/
13 More information about the surveys conducted by the Higher Education Research Institute is available at https://heri.ucla.edu/overview-of-surveys/
14 More information about AAU’s STEM initiative is available at https://www.aau.edu/education-community-impact/undergraduate-education/undergraduate-stem-education-initiative
projects,15 and the Sloan Foundation-funded Sloan Equity & Inclusion in STEM Introductory Courses (SEISMIC) Collaboration.16 Some institutional efforts turn into national-level consulting approaches such as Georgia State University’s National Institute for Student Success.17 In general, external funding has enabled a variety of tools and approaches to be developed and disseminated to a broader number of schools. While attempts at understanding how equity-focused visualizations can impact faculty thinking (Reinholz et al., 2023), more complex collections of visualizations and suggestions are available in the form of course equity reports.18
Here we present several examples of data dashboards or other tools for collecting, collating, analyzing, and visualizing data. These examples were chosen to illustrate a variety of approaches and a variety of data types. Most of the tools have other features we do not explore here as well as new features that may have been recently added or are currently under development. Because tools of this complexity are relatively new, the impact that such approaches could have on efforts to achieve equitable and effective teaching has not been methodically studied. We share them here for their potential promise and for the kinds of questions they can raise for decision makers. An approach for making your own dashboard19 to visualize inequities has also been described (Tatapudy et al., 2024).
Box 9-2 shows an example of an equity dashboard that provides data on academic units that provide courses for students who are majoring in another field. This example was chosen to illustrate how thinking of donor and acceptor departments can inform the pathways students take through their undergraduate education. The student flows as they leave one major and declare another can be considered in the context of the number of effectors students hold. An effector is term that signifies a student as belonging to an underserved group and serves as a mechanism for better understanding inequities. The University of California, Santa Barbara, has used this tool along with a suite of complimentary tools to improve understanding of student experiences and has coupled the tools with professional development on their use.
Box 9-3 shows a different type of equity dashboard that is designed for use by an instructor and is easily accessible from course software they are
___________________
15 More information about the HHMI Inclusive Excellence projects is available at https://www.hhmi.org/programs/inclusive-excellence-3
16 More information about the SEISMIC Collaboration is available at https://www.seismicproject.org/
17 More information about NISS is available at https://niss.gsu.edu/
18 More information about course equity reports is available at https://www.seismicproject.org/seismic-central/the_selc_grant/
BOX 9-2
Equity Dashboards at the University of California, Santa Barbara
The University of California, Santa Barbara (UCSB) has created a guided pathway system for instructors and administrators to engage with data. For these visualizations, the campus developed the terminology “effectors of opportunity”—e.g., systemic factors shown to have a relationship to educational access—to disaggregate the data. Students might hold 0, 1, 2, or 3 effectors: low-income, first-generation attendee, and/or member of a minoritized group (1, 2, 3), or none of these (0).
Via questions surrounding specific “pages” of extensive data dashboards, the pathway provides disaggregated visualizations for users to begin by examining quarter-by-quarter program-level patterns, such as a flow diagram showing graduation rates in the major for students who “start in” (see Figure 9-2-1) or “switch into” (see Figure 9-2-2) majors (for entering first-year or transfer students; note that UCSB is on the quarter system).
Then, depending on what users of the system observe, they can move to other visualizations to look at more specific points in the selected group’s trajectories to learn more, e.g., at what point in the program different effector groups switch in/out via change of major; pre-major to major trajectories; etc. Next, users can look at disaggregated data focused on course performance at multiple levels (e.g., division, major, lower or upper division). Finally, instructors can receive course-specific reports and equity-focused questions relevant to teaching: Data-Enhanced Teaching and Learning (DETAiL), and the institution’s Course Equity Reports (originally developed as part of the SEISMIC collaborative).
SOURCE: Data Dashboards, https://otl.ucsb.edu/data-dashboards (University of California, Santa Barbara, n.d.).
BOX 9-3
Equity Dashboards at University of Nebraska-Lincoln
At the University of Nebraska-Lincoln, every instructor can access course-specific dashboards through a customized link embedded in the course software. Instructors can view average assignment grades disaggregated by a wide variety of demographics, including first generation or rural/urban background. The assignments can be summed into assignment groups (see Figure 9-3-1).
Instructors can link out to another view that shows them their course DFW rates across semesters disaggregated by the same set of demographics (see Figure 9-3-2). For small enrollment courses, instructors can sum across semesters in order to reduce noise. These views are complemented by campus-wide initiatives to engage and support instructors in self-reflection as they trial pedagogical interventions.
SOURCE: Course Insights, https://analytics.unl.edu/ (University of Nebraska-Lincoln, n.d.).
already using. It illustrates how a set of tools can be created to visualize disaggregated data for course and department outcomes as well as disaggregated outcomes from the learning management system that is in use by the entire university. It is notable that this tool was made broadly available with minimal oversight at the University of Nebraska-Lincoln, and that the institution is slowly growing its community of users.
Box 9-4 shows a third example of equity dashboards that demonstrates how a tool can provide customized results for each instructor within a 24-institution system, the California State University system. It shows instructors disaggregated course outcomes along with any equity indicators observed in their data. Notably the data are coupled with guidance and self-reflection opportunities to help instructors consider the implications of what they see and determine how they can use it to decide where to focus their efforts at improving teaching.
BOX 9-4
Equity Dashboards at California State University
The California State University Course Equity Portal provides each faculty member across the whole system of 24 institutions with historical course grade records and identifies students with notable equity gaps in the rates at which they received low or non-passing grades. Criteria for selecting which courses to show faculty include effect size, overall size of the difference (e.g., must be greater than 10 percentage points difference), and recency. The portal intentionally biases presentation toward more recent terms rather than showing gaps in a course that may have occurred many years ago (see Figure 9-4-1).
They include several pop-up FAQs explaining (a) what is shown and why it was selected/criteria used; (b) why it matters—some faculty may not see a clear connection between their in-class teaching and student contact and the long-term outcomes of their students (e.g., retention and graduation rates) so these connections are discussed; and (c) self-reflection questions (see Figure 9-4-2). This last is critical in part because it could be natural for instructors viewing data like these to slip into a defensive and/or deficit mindset; the self-reflection questions can help faculty move themselves toward more constructive/productive questions.
SOURCE: Course Equity Portal, https://www.studentsuccessanalytics.com/courseEquityPortal.html (California State University, n.d.).
Box 9-5 presents one of the earliest tools (2016) that provided detailed aggregated student information to each instructor about the classes they have taught in the past as well as upcoming classes. Additionally, it provided instructors information on how different student characteristics, as well as intersectional identities, performed in the courses they taught, giving them a first-even view of inequities in outcomes. This tool provided a form of inspiration for the other three tools mentioned. Even though this is an old tool, it provides a clear way to see disparities in access by instructor and is notable in part because many across higher education still do not have access to such information on their own students.
Critically, users of data, whether instructors, department heads, or institutional administrators, need support and guidance to disaggregate, interpret, and use the data in ways that minimize bias (see, e.g., McNair et al., 2020). Simply providing data without context and appropriate discussion can lead to unintended consequences or encourage a deficit mindset. For example, predictive analytics could be used to identify students needing additional support or to label or exclude students. Equity gaps in student performance could be used to identify instructors needing additional support or professional learning opportunities, or to judge and punish.
BOX 9-5
Know Your Students at the University of California, Davis
Know Your Students (KYS) at the University of California, Davis (UC Davis) is a web application developed to help instructors plan and execute their courses. It provides information on learning and grades for small and large classes. KYS presents aggregate student demographics and academic data at the course level as well as student testimonials about their background and experience at UC Davis (see Figure 9-5-1). It also serves as a tool that instructors can use to survey students in a course, examine inequities at the assignment level, and reflect on the language used in recorded lectures.
The tool was designed with the intention of providing information that might inspire empathy and motivate course change efforts that could improve overall student learning and equity.
All users only have access to the courses they have taught. Any instructor who would like access to the full set of charts and visualizations showing demographic information about the students they teach as well as historical grade gaps must go through training and/or consultation with the teaching center. In the example, the full dashboard is shown as a strip at left with two highlighted sections showing general class demographic variables for current class (top) and specific equity gaps selectable by group over the time the course is taught (below).
SOURCE: Know Your Students, https://cee.ucdavis.edu/know-your-students (University of California, Davis, n.d.).
Tools and Approaches Across the Instructional System
While equitable and effective teaching might at first appear to be solely the domain of an individual instructor impacting their classroom, the reality is that moving toward teaching that is equitable can have a much broader impact when such change is approached systemically. Many introductory STEM courses are taught by groups of instructors that, when coordinated, can impact a much larger group of students than any individual instructor. In addition, courses connect with each other within and between departments, and none of them exist independently of academic unit and institutional policies (see Figure 9-2). Therefore, institutional leaders might have to consider these interconnections and the data about the courses in their decision-making process. Institution-level change can benefit from tools and approaches that measure and support progress across various levels of users from the individual to the instructor group to the academic unit, collection of academic units, and the whole campus. Specific ideas and approaches to data at the various levels have been illustrated in Chapters 6 on academic units, courses, and curriculum and Chapter 7 on pathways. Here we illustrate some of the interconnections across academic units that help explain why course-level data might be of interest to institutional leaders (Figure 9-2).
Similarly to how the data suggested by Figure 9-2 can be of use at multiple levels of the system, other course-level data can inform systemic change efforts. Buy-in to those systemic change efforts can be enhanced by sharing of data to STEM instructors when it is done in a thoughtful way and via useful informative tools. The majority of STEM instructors have been trained to look for and appreciate data. Unfortunately, in most higher education classrooms they are limited to the data that they generate by their students’ level of learning through homework, quizzes, and tests. While these forms of data provide insights into students’ performance, they tend to reflect the outcomes of students’ prior opportunities rather than offering insights into potential inequities. As a result, these assessments offer limited guidance on how instructors or instructional teams can address and mitigate these inequities. Implementing alternative grading methods that assess specific learning outcomes can provide instructors with additional and more detailed data (see Chapter 5) and help them see the inequities that they might be able to address.
Administrative leaders can encourage and support instructors’ access to valuable information such as who is enrolled, which can be shared via readily accessible dashboards or course reports. Real-time information about student engagement with course materials via the learning management system can help gauge who may need additional attention and system-level early warning systems can guide instructor-student interventions where they
NOTE: Data can be collected and analyzed at the level of individual instructors, at a course level, as connected between courses in a sequence, within an academic program, how courses connect between programs, as well as aggregated at the level of related academic units and/or the institution.
are most needed. Additionally, affective measures (qualitative or quantitative) such as sense of belonging and departmental climate can be part of a campus-wide data collection effort that is distributed back to departments and instructors and teaching and learning centers can help gather and interpret qualitative data that can help an instructor make instructional corrections when they still have the chance to impact student outcomes. In all these instances, departmental and institutional leaders can create the expectations and infrastructure for these forms of data to be normalized throughout academic units and the institution.
When considering course-level innovations to foster effective and equitable teaching, there are multiple approaches to data for instructors and instructional communities that are being employed. Course teams and learning communities rely heavily on administrative support both for data access as well as for resources to make a well-informed coordinated community possible. Groups of instructors could work together, informed by detailed disaggregated data of prior student outcomes, in order to create courses that prioritize equitable and effective teaching. Additionally, providing structure at the introductory levels via course coordinators, whose primary job is to create a team teaching a highly structured course that is meeting in multiple sections with multiple instructors throughout one term, can provide a consistent student experience that has continuous improvement at its core. Lastly, supporting departmental action teams (see Chapter 6) and interdepartmental learning communities can ensure that valuable instructional expertise gained in one discipline has the opportunity to impact other disciplines within related academic units. None of these activities can be optimized without data resources, compensation for time spent on this, and a clear message of the value of equitable and effective teaching communicated throughout the academic unit and institution.
When considering data at the level of the academic unit there are multiple opportunities that allow STEM units to gauge their progress toward equitable and effective teaching—teaching communicated throughout the academic unit and institution.
When considering data at the level of the academic unit there are multiple opportunities that allow STEM units to gauge their progress toward equitable and effective teaching, including
- Student flow
- Elements of program structure
- Course taking patterns
- Course outcomes
- Bottlenecks in the curriculum
- Climate
- Sense of belonging
Analysis can consider the data in relation to policies and potential for differential impact on different student groups. Student flow can be studied with tools that make clear the paths that students are taking into and out of programs. Such tools can help leaders of academic units determine where a deeper look is warranted. Program structure can help a program look at how their pre- and co-requisite structures enable or block the potential for timely graduation and whether current requirements reflect actual student needs. Bottlenecks in the curriculum can then be identified, and potential course and/or structural changes encouraged to smooth student progress in a program (see Chapter 6). Aside from logistical and course outcome improvements, one also needs to be aware of the climate, student perceptions, and their sense of belonging. Examining policies within a course, academic unit, and beyond can help instructors and administrators to understand the impact the accreted structures created have on equitable and effective student outcomes. For all this work, academic unit leadership and above need to ensure that data are available for all forms of improvement work and encourage data collection that is both qualitative and quantitative and that gives information of various types (e.g., student learning, student engagement, instructor engagement, teaching practices, instructor participation in PLD, etc.) as well as providing time, resources, and guidance on how to use these forms of data.
While some of the examples mentioned in this section focus at the academic unit level (e.g., the department), many of the same approaches mentioned will also work when reviewing these elements at the institutional level—whether that is within a college or school within a university, a collection of academic units, student services units, or the whole institution. In addition, as mentioned earlier, this level of data can be of value to institutional leaders working to make systemic change and to drive continuous improvement toward equitable and effective teaching.
VALUE AND REWARD TEACHING
Recognizing and rewarding faculty for implementing equitable teaching practices is essential to achieving equitable and effective undergraduate STEM education. The institutional reward system can support and encourage such teaching practices at the classroom and departmental levels by valuing teaching activities and rewarding equitable and effective teaching. This may be done through revised promotion and tenure criteria that value innovative and inclusive teaching, professional development opportunities, and appropriate compensation for the additional efforts required to transform teaching practices. Revised teaching evaluation processes are necessary to support these revised criteria. Aligning rewards with the goals
of equitable and effective teaching can motivate educators to adopt and maintain these practices.
Different Approaches to Teaching Evaluation
Many higher education institutions nationwide have been using student end-of-term surveys as a form of teaching evaluation. In many cases, these surveys serve as the sole basis for judging the quality of teaching. While many of the surveys used include options for students to contribute written comments, the quantitative portion of the survey data tends to be the focus of annual faculty evaluations, such as considerations for merit raises or promotion. This is largely due to the ease with which the quantitative data are collected, processed, reported, and compared.
Such heavy reliance on this approach to evaluating teaching is a concern, particularly within the context of our larger conversation about inclusion and equity. Numerous studies have documented biases that exist in these student teaching surveys. While most studies on this subject have examined the effects of faculty gender on survey scores (Boring, 2017; Flaherty, 2022; MacNell et al., 2015; Mitchell & Martin, 2018), bias has also been seen to result from the instructor’s race/ethnicity, accent, attractiveness, age, and LGBTQIA+ identity (Basow & Martin, 2012; Heffernan, 2021; Kreitzer & Sweet-Cushman, 2021). Furthermore, there is little evidence that the scores on these surveys correlate with measures of student learning or teaching quality, and some evidence to suggest that, in fact, they do not (Esarey & Valdes, 2020; Spooren et al., 2013). This is not to say that the student voice is not important in understanding what takes place in a course. Instead, the information provided by student end-of-term surveys must be understood as feedback and a reflection of the students’ own experience, not as a measure of teaching quality, and used appropriately within that interpretation (D’Agostino & Kosegarten, 2015). One important way to do this is to include multiple sources of evidence about an instructor’s teaching, as described in Chapter 8.
Holistic approaches to teaching evaluation (see Chapter 6) have the advantage that they can take into account the aspects of teaching that take place outside of a classroom setting—whether that be the laboratory, the field, or an online classroom—and would thus be invisible to students. For example, there is a significant amount of work that takes place before a course begins, such as planning a syllabus to include selected source materials, creating assessments and learning progressions, preparing course materials, and engaging in professional development. While students will see the finished products of these efforts, the reasoning behind them may not be evident to a student; yet the approach to these planning steps may indeed distinguish higher and lower quality teaching. Other components of
teaching include advising students on such topics as their educational journeys, majors, career aspirations, specific courses, undergraduate research opportunities, or internships as well as engaging in community outreach, or carrying out scholarly studies on teaching. These and other aspects of teaching can be represented and properly valued in a holistic approach to teaching evaluation, balancing the limited information from the in-class feedback that has long been provided through student end-of-term surveys.
Reform of the Reward and Recognition System
Because the current approach of evaluating faculty teaching using primarily (or solely) student surveys is so deeply flawed, it is difficult to properly recognize and reward teaching with this approach to measurement. Therefore, we consider a discussion of the methods used for the evaluation of teaching a core component of a larger call to reform teaching and the educational experience for all students. Similar arguments have been made by others, such as the Boyer 2030 Commission report, which states, “Aligning the faculty rewards structure with the stated educational mission of the university is the most important reform we can make to ensure sustained, authentic institutional change in the quality of undergraduate education” (2022, p. 33). Similarly, others have said that when an institution uses a well-rounded process to measure and reward teaching it is both fairer to faculty members and better for students (McMurtrie & Supiano, 2021).
If considering the example of a college or university in which faculty have both a teaching and research requirement in their contract, every faculty member knows that they can achieve recognition for their innovations, productivity, and creativity in the arena of research. They are able to demonstrate their contributions through publications, grants, presentations, and other metrics of research productivity, which are occurring on a regular basis and hold legitimacy for others in the department or institution. This varied list of metrics underscores the reasons that a multi-dimensional approach to examining teaching quality is also needed. In fact, for faculty in disciplines where the connections between their discipline-based work and equity issues are less clear, attention to equity-minded teaching can be viewed as at best a distraction and, at worst, a waste of time (McGee, 2020; Perez et al., 2023). Creating a system of evaluating teaching that everyone at the institution can trust to be meaningful is a first step in raising the value of teaching in the reward and recognition system and ensuring that all teaching can be both equitable and excellent.
While the actions related to implementing a holistic teaching evaluation approach will take place at the department level of an institution, the work that will be required to formulate and use these practices will need support from institutional leaders. In fact, reform of faculty evaluation in ways that
will support equity-minded teaching is a third-order organizational change (Marbach-Ad et al., 2016; Miller & Fairweather, 2016). However, without changes to campus-level faculty evaluation policies, including workload, annual and merit review, and promotion policies, it will be difficult to move equity-minded teaching into the core of an institution.
SUMMARY
The higher education space has many types of institutions that vary in their specifics. However, all institutions are responsible for implementing equitable and effective teaching, and all face challenges in this work as well. The committee acknowledges that the resources, structures, and titles of the actors may be very different in different types of institutions, but the need to provide courses for students resonates across disciplinary and institutional types. Data can be useful at all institution types and at all levels within institutions. Multiple types of data can be used to provide various perspectives on the components of the system and their interconnections and that can in turn inform decision making to improve equitable and effective teaching.
Conclusion 9.1: Policies and procedures at the institutional level can either impede or promote implementation of the Principles for Equitable and Effective Teaching. Change toward equitable and effective teaching will require coordinated effort from multiple levels of institutional leadership and a culture of growth that is responsive to the needs of students and instructors. Upper-level administrators (e.g., deans and provosts) can analyze and reform policies and practices so that the institutional reward system for faculty, instructors, and academic unit leaders is aligned with the goal of equitable and effective teaching and all stakeholders are supported in change efforts.
Conclusion 9.2: Institutional change is an ongoing process of continuous improvement that can include (a) opportunities to become familiar with goals and principles of change, (b) recognition of an academic unit’s culture, (c) attention to power dynamics in the institution, and (d) communication among key stakeholders.
Conclusion 9.3: Data, both aggregated and disaggregated, are a key tool to understand, enact, and monitor change. Both quantitative and qualitative data are needed to fully understand what is happening in a system and to provide information to guide change efforts. Reflective analysis of data best guides policy and practice decisions and informs ongoing efforts at improvement. Grades, and the approaches to assigning them, do not convey the full complexity of information about student learning.
This page intentionally left blank.
