3
Considerations in Identifying Engineering Impacts on Society

This chapter outlines the many ways in which engineering research and education impact society, the role that the National Science Foundation (NSF) has played in bringing these impacts about, and the means that it employs to evaluate societal impacts. It begins with an overview of engineering’s influences on our everyday lives and then briefly addresses the role of research funding, focusing on NSF. The chapter concludes with a description of the agency’s “broader impacts” criterion, a required component of funding applications that addresses a proposed undertaking’s “potential to benefit society and contribute to the achievement of specific, desired societal outcomes” (NSF, 2022; p. 3). These considerations informed the development of the committee’s approach to identifying engineering impacts made possible by NSF investments, which is addressed in the next chapter.

OVERVIEW – IMPACTS OF ENGINEERING ON SOCIETY

The engineering profession and the engineers it is comprised of have radically influenced our world for the better. Over the past century, engineering innovations such as spacecraft, centralized water treatment, and modern conveniences such as air conditioning have brought ideas to reality and provided for societal needs. For example—as highlighted in the National Academy of Engineering (NAE) publication A Century of Innovation: Twenty Engineering Achievements that Transformed our Lives (Constable and Somerville, 2003)—electrification is a foundational engineering improvement. Not only did it change the course of economic development of the United States, but many of the other achievements recognized in the book and cited in this report require electricity and a reliable electrical power grid as prerequisites. Table 3-1 presents a complete list of the innovations cited in the publication.

TABLE 3-1 “The Greatest Engineering Achievements of the 20th Century” Identified by the National Academy of Engineering and a Consortium of Professional Engineering Societies

SOURCE: Constable and Somerville (2003).

Just as engineering solutions advanced society in the past, they will also play an important role in meeting the needs of a future under the pressure of population growth and emerging threats to the public. In 2008 the NAE Committee on Engineering’s Grand Challenges identified 14 grand challenges and opportunities for engineering during the world’s next few generations which fell into four “broad realms of human concern—sustainability, health, vulnerability, and joy of living” (NAE, 2017; p. 1).¹⁶ The challenges, listed in Table 3-2, have since been taken up by the Grand Challenge Scholars Program, an effort being pursued by engineering schools throughout the United States that is designed to prepare students to be the generation that solves the grand challenges facing society in this century.¹⁷

TABLE 3-2 The “Grand Challenges for Engineering” Identified in 2008

SOURCE: NAE (2008).

New challenges such as those highlighted above require new ways of thinking, and the engineer of the future will need to be trained to meet interdisciplinary challenges that promote the sustainable development of our world (Tabas et al., 2019). They will need to respond to emerging needs of people across all cultures of our globalizing world, which will require insights and knowledge currently held in the social and political sciences. Preparing this type of holistic engineer will require schools to become more appealing to those who have not traditionally entered the engineering profession, including changing the messaging from engineering being not “for everyone” to engineering being a creative avenue that students of all kinds can pursue to improve the world (NAE, 2008).

___________________

¹⁶ Note that the NAE Grand Challenges for Engineering report was originally published in 2008 and updated in 2017.

¹⁷ The NAE Grand Challenges for Engineering website is https://www.engineeringchallenges.org/.

ENGINEERING INNOVATION AND FEDERAL FUNDING

Research underpins the important engineering breakthroughs mentioned in A Century of Innovation and NAE’s Grand Challenges, and research requires sources of support. While commercial interests fund and conduct most research and development work in the United States, the federal government still plays an important role. In 2019, federal sources accounted for 21 percent of basic and applied research funding (NSF, 2022). The Department of Defense and Department of Health and Human Services (primarily, via the National Institutes of Health) were the largest funders, followed by the Department of Energy, National Science Foundation, NASA, and the Department of Agriculture.

These agencies’ research support programs do not operate in isolation, however. Indeed, complementary federal agency support has been instrumental in driving engineering innovation in the United States. This complementary support may take different forms. In some circumstances, differences in mission drive when a particular agency is likely to be the primary support mechanism. The “ARPA’s”—the Defense Advanced Research Project Agency, Advanced Research Project Agency–Energy, and Advanced Research Project Agency for Health—are tasked with funding cutting-edge, high-risk research efforts that may, if they show promise, be fostered through their next stage of development by other agencies like NSF. In other cases, particular agencies maintain infrastructure like laboratories and testing facilities that other entities can utilize, saving costs and allowing better utilization of specialized operations and support staff. And research in areas that overlap agency responsibilities may be supported by multiple agencies. Studies that examine health and the built environment, for example, might be cofunded by such diverse federal entities as the U.S. Environmental Protection Agency, National Institutes of Health, Department of Energy, National Institute of Standards and Technology and the Department of Housing and Urban Development. Overall, the complementary support from federal agencies has been a cornerstone of engineering innovation in the U.S., enabling groundbreaking research, fostering cross-sector collaborations, and providing essential resources that promote continuous advancement in engineering.

Three examples of federal funding are cited here to illustrate both the catalyzing role of the NSF and the range of activities it has supported.¹⁸ First, a series of National Academies of Sciences, Engineering, and Medicine (National Academies) reports starting in the mid-1990s illustrated the complex nature of information technology (IT) research and the interdependencies among subfields of computing and communications research (most recently,¹⁹ NASEM, 2020a). This work showed that the IT sector is not self-sufficient and that research support, sometimes taking place over decades, has been crucial to the sector’s commercial success. Key projects with agency funding include the creation of NSFNET, a predecessor to the Internet that promoted advanced research and education networking across the United States.

NSF funding has also had an impact in the field of nanotechnology. Generally speaking, the rapid growth of innovation in this field was found to be directly correlated to funding (Huang et al., 2005). Huang and colleagues (2006) found that research funded by NSF and patents authored by NSF-funded researchers demonstrate notably greater influence, as indicated by patent citation metrics over the 2001–2004 timeframe, compared with other groups used for

___________________

¹⁸ Chapter 2 provides a history of NSF’s support of engineering-related research and education and information on the mechanisms that the agency uses to provide that support.

¹⁹ A complete list of the reports in this series is contained in Table 1-1.

comparison. The impact of patents authored by the NSF increasingly expanded over the lifespan of the patent, underscoring the enduring significance of fundamental research in the long term.

As a final example, additive manufacturing—an emerging technology that has revolutionized manufacturing—has greatly benefited from NSF support (IDA Science and Technology Policy Institute, 2013). NSF funded predecessor technologies and supported early advanced manufacturing patents to become proof-of-concepts and prototypes in three major commercial technology areas—binder jetting, powder bed fusion, and sheet lamination (Peña et al., 2014). As advanced manufacturing technologies have developed, NSF has supported research on new processes, novel applications for existing processes, and initiatives related to benchmarking and road mapping.

IMPACTS OF NSF PROGRAMS AND ERCS

As noted in this report, NSF’s Engineering Research Centers (ERCs) have had enormous impact, creating strong partnerships with the private sector and universities which foster multifaceted interdisciplinary research and generating thousands of engineering graduates and millions of dollars of economic benefits (NSF, 2015; Roessner et al., 2010). ERCs have helped launch new fields and entirely new systems approaches in areas such as bioengineering and nanosystems (Preston and Lewis, 2020). They have also created disruptive technologies in neurotechnology and biorenewable chemicals by facilitating cross-disciplinary collaboration and industry partnerships. In 2007 the return on NSF investment in ERCs was estimated to be 50:1, and this has since continued to grow (Preston and Lewis, 2020).

NSF-led programs have also provided immense economic benefits to the nation. Having generated more than $9 billion in private investment from 2014 to 2020, NSF’s Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs have helped small business turn their ideas into marketable products and services (U.S. Congress, 2020). The SBIR program was found to commercialize research at a considerable rate, generate substantial knowledge-based outputs such as patents, and create companies that otherwise would not exist (NASEM, 2015). Led by NSF, the National Nanotechnology initiative (NNI), a highly successful cross-disciplinary and interagency coordination effort, was found to have provided key advancements in research and critical support of responsible development of nanotechnology (NASEM, 2020b). NNI also developed important opportunities for nanotechnology workforce training to strengthen national competitiveness.

The NSF’s Designing Materials to Revolutionize and Engineer our Future program produced groundbreaking research that was essential to advancing the White House’s Material Genome Initiative (MGI), launched in 2011 (NASEM, 2022). This accelerated the discovery and fast and efficient deployment of advanced materials. PARADIM (Platform for the Accelerated Realization, Analysis, and Discovery of Interface Materials), one of four material innovation platforms associated with MGI, hosted users from 41 universities and national labs and resulted in 140 journal publications from 2016 to 2021 (Nutt, 2021). The platform’s most notable discoveries include a new type of topological insulator²⁰ and a form of galfenol (an alloy of iron and gallium) that is the world’s highest-performance magnetostrictive material. These impacts

___________________

²⁰ A topological insulator is a substance that has an interior that exhibits the properties of an electrical insulator and a surface that acts as a conductor.

are just a few examples among the multitude of impacts of NSF programs that spur engineering innovations that benefit the economy.

THE NSF BROADER IMPACTS REVIEW CRITERION

Overview

A core component of proposals submitted to NSF is the broader impacts criterion, defined as the potential of the research to benefit society and contribute to the achievement of specific desired societal outcomes (NSF, 2019). Currently NSF is the only federal agency that has such a requirement for its proposals. From 1981 to 1997, NSF proposals were evaluated on (1) researcher performance competence, (2) intrinsic merit of research, (3) utility of research, and (4) effect of research on infrastructure of science and engineering. However, in 1997 a reassessment by the National Science Board distilled the four criteria into two evaluation categories: intellectual merit and broader impacts (NSF, 1997). The change came from a confluence of factors including recommendations from the Committee on Equal Opportunities in Science and Engineering, the passing of the Government Performance and Results Act, and the “NSF in a Changing World” strategic plan (NSF, 1995) which included a long-term goal of promoting knowledge in service of society.

While NSF avoids dictating the exact societal outcomes a project should target, it aims to ensure that public funding supports research with tangible societal benefits beyond fundamental knowledge expansion. The NSF website²¹ offers various examples of Broader Impacts Criteria (BIC), which are illustrative but not exhaustive, including:

• inclusion;
• science, technology, engineering, and mathematics (STEM) education;
• public engagement;
• societal well-being;
• STEM workforce development;
• partnerships between academia, industry, and others;
• national security;
• economic competitiveness; and
• infrastructure for research and education.

Prior iterations have categorized broader impacts as: (1) infrastructure for science, (2) broadening participation, (3) training and education, (4) academic collaboration, (5) K–12 outreach, (6) potential society benefits, (7) outreach/broad dissemination, and (8) partnerships with potential users of research results (Roberts, 2009, based on a July 2007 NSF broader impacts guidance memorandum cited in the paper); other impacts have included (1) increased public scientific literacy, (2) increased public engagement with science and technology, (3) broadened participation, (4) development of a diverse STEM workforce, (5) development of a globally competitive STEM workforce, and (6) increased economic competitiveness of the United States (Verdín, 2017; based on NSF’s 2016 Proposal & Award Policies and Procedures

___________________

²¹ https://new.nsf.gov/funding/learn/broader-impacts.

Guide). NSF’s initiatives are widespread in both subject matter and programmatic nature, encompassing broader impacts that range from those inherent to the research itself to broadening participation benefits from STEM outreach programs.

A variety of efforts have been undertaken to evaluate and address NSF’s broader impact criterion. These include a study conducted by the National Academy of Public Administration in coordination with NSF that identified the need for a more consistent application of broader impacts as a criterion during proposal reviews (NAPA, 2001). In 2010, Congressional approval of the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (COMPETES) Reauthorization Act²² mandated broader impacts and ordered higher education institutions to support principal investigators achieving broader impacts in their work.

In 2013 the National Alliance for Broader Impacts (NABI) was formed through funding from an NSF Research Coordination Network grant. Made up of approximately 200 institutions from across the United States, NABI was created to develop institutional capacity in STEM fields to respond to the broader impacts criterion. The organization developed a guiding document in 2015 for reviewers that was the first nation-wide attempt to standardize how review panels assess NSF proposals’ broader impact plans. A follow up NABI report in 2018 assembled data from years of its annual summits and two NSF reports on broader impacts implementation and application across directorates. The findings of this report highlighted seven issues common to all relevant stakeholder groups to better inform how NSF’s broader impacts criterion evaluation should be carried out. These issues were largely tied to the lack of clarity and consistency of broader impact evaluation as well as a lack of resources to support broader impacts at the individual, institutional, and national level. The document, The Current State of Broader Impacts: Advancing Science and Benefiting Society, is listed as one of their key additional resources to stakeholders today (NABI, 2018).

Evaluations of the Effects of Broader Impacts Reviews

Over recent years, scholars have assessed how broader impacts have been distributed among the categories described in the previous section (Kamenetzky, 2013). The most frequent categories included were teaching and training, broad dissemination, and infrastructure enhancement. Verdín (2017) found that broader impact statements were most likely to include (1) increased public scientific literacy, (2) public engagement in science and engineering, and (3) developing a diverse STEM workforce. Watts et al. (2015), on the other hand, found that activities seeking to broaden participation of underrepresented groups in STEM fields were reported less frequently. Cultural differences between scientific fields have been found to play a large role in how these fields view the broader impacts criterion, causing them to propose certain types of impacts in their submissions (Kamenetzky, 2013). Political considerations have also been proposed as influencing the types of broader impacts that researchers mention or omit in their award proposals (Roberts, 2009).

Researchers who submitted proposals that mentioned benefits to society were found to be no more likely to propose dissemination of their results to those who could use them than researchers who only spoke to the broader impacts of their work for science (Roberts, 2009).

___________________

²² Public Law 111-358.

This finding suggests that the potential societal benefits identified when describing broader impacts do not necessarily lead to actual societal benefits and that results relevant to the public are often not publicized outside of the scientific community. Actions are thus needed to communicate broader impacts along with dissemination activities to both researchers and the general public to build a better knowledge base regarding societal impacts, especially among underrepresented groups.

The creation of NSF’s broader impacts criterion along with agency and university efforts to create institutional capacity around them have led to important progress in linking research projects and societal impacts. However, it was found that the current broader impacts framework could be improved by assessing who receives the benefits created by NSF-funded research (Bozeman, 2020) and by establishing metrics for broader impact assessment (Verdín, 2017). Without such an analysis, NSF-funded research could sustain or even exacerbate inequalities that exist today (Woodson et al., 2021). The Inclusion-Immediacy criterion (IIC) created by Woodson and Boutilier (2022) seeks to fill this gap and shed light on the impacts of funded NSF projects on marginalized communities. IIC evaluates NSF grants based on the people who will benefit from the research and characterizes the grant based on the alignment of the research and the broader impacts. Using this framework has the potential to benefit not only marginalized communities but the entire innovation ecosystem, as inequality limits NSF’s mission to advance national health, prosperity, welfare, and security (NSF, n.d.).

REFERENCES

NAE (National Academy of Engineering). 2008. Changing the conversation: Messages for improving public understanding of engineering. Washington, DC: The National Academies Press. https://doi.org/10.17226/12187 (accessed February 22, 2024).

NAE. 2017. NAE grand challenges for engineering. Washington, DC: The National Academies Press. https://nae.edu/187212/NAE-Grand-Challenges-for-Engineering. (Note that this report was originally published in 2008 and updated in 2017.)

NAPA (National Academy for Public Administration). 2001. A study of the National Science Foundation’s criteria for project selection. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=dbeb6cd123afda1f77369490fd27ffcb3e2df6d9 (accessed February 22, 2024).

NASEM (National Academies of Sciences, Engineering, and Medicine). 2015. SBIR at the National Science Foundation. Washington, DC: The National Academies Press. https://doi.org/10.17226/18944 (accessed February 22, 2024).

NASEM. 2020a. Information technology innovation: Resurgence, confluence, and continuing impact. Washington, DC: The National Academies Press. https://doi.org/10.17226/25961 (accessed February 22, 2024).

NASEM. 2020b. A quadrennial review of the National Nanotechnology Initiative: Nanoscience, applications, and commercialization. Washington, DC: The National Academies Press. https://doi.org/10.17226/25729 (accessed February 22, 2024).

NASEM. 2022. NSF efforts to achieve the nation’s vision for the Materials Genome Initiative: Designing Materials to Revolutionize and Engineer Our Future (DMREF). Washington, DC: The National Academies Press. https://doi.org/10.17226/26723 (accessed February 22, 2024).

NSF (National Science Foundation). n.d. About NSF. https://beta.nsf.gov/about (accessed February 8, 2023).

NSF. 1995. NSF in a changing world. https://www.nsf.gov/pubs/1995/nsf9524/contents.htm.

NSF. 1997. NSF to adopt new merit review criteria. https://www.nsf.gov/news/news_summ.jsp?cntn_id=102789 (accessed February 22, 2024).

NSF. 2015. NSF engineering research centers: Creating new knowledge, innovations, and technologies for over 30 years (No. 15–810). Arlington, VA: National Science Foundation.

NSF. 2019. Proposal and award policies and procedures guide. https://www.nsf.gov/pubs/policydocs/pappg19_1/index.jsp (accessed February 22, 2024).

NSF. 2022. Perspectives on Broader Impacts. https://nsf-gov-resources.nsf.gov/2022-09/Broader_Impacts_0.pdf (accessed February 22, 2024).

Nutt, D. 2021. $22.5M NSF grant accelerates materials discovery. Cornell Chronicle, May 25. https://news.cornell.edu/stories/2021/05/225m-nsf-grant-accelerates-materials-discovery (accessed February 22, 2024).

Peña, V., B. Lal, and M. Micali. 2014. U.S. federal investment in the origin and evolution of additive manufacturing: A case study of the National Science Foundation. 3D Printing and Additive Manufacturing 1(4):185–193. https://doi.org/10.1089/3dp.2014.0019 (accessed February 22, 2024).

Preston, L., and C. Lewis. 2020. Agents of change: NSF’s engineering research centers—A history. Lynn Preston Associates LLC. https://erc-history.erc-assoc.org/.

Roberts, M. R. 2009. Realizing societal benefit from academic research: Analysis of the National Science Foundation’s broader impacts criterion. Social Epistemology 23(3–4):199–219. https://doi.org/10.1080/02691720903364035 (accessed February 22, 2024).

Roessner, J., L. Manrique, and J. Park. 2010. The economic impact of engineering research centers: Preliminary results of a pilot study. Journal of Technology Transfer 35:475–493. https://doi.org/10.1007/s10961-010-9163-x (accessed February 22, 2024).

Tabas, B., U. Beagon, and K. Kövesi. 2019. Report on the future role of engineers in society and the skills and competences engineering will require. https://arrow.tudublin.ie/engschcivrep/16 (accessed February 22, 2024).

U.S. Congress. 2020. America’s seed fund: A Review of SBIR and STTR. Hearing before the Subcommittee on Research and Technology, Committee on Science, Space, and Technology, House of Representatives, One Hundred Sixteenth Congress, second session, February 5, 2020. https://www.congress.gov/event/116th-congress/house-event/LC65772/text (accessed March 20, 2024).

Verdín, D. 2017. Quantifying and assessing trends on the National Science Foundation’s broader impact criterion. 2017 ASEE Annual Conference & Exposition Proceedings: 28778. https://doi.org/10.18260/1-2--28778.

Watts, S. M., M. D. George, and D. J. Levey. 2015. Achieving broader impacts in the National Science Foundation, Division of Environmental Biology. BioScience 65(4):397–407. https://doi.org/10.1093/biosci/biv006 (accessed February 22, 2024).

Woodson, T., and S. Boutilier. 2022. Impacts for whom? Assessing inequalities in NSF-funded broader impacts using the inclusion–immediacy criterion. Science and Public Policy 49(2):168–178. https://doi.org/10.1093/scipol/scab072 (accessed February 22, 2024).

Woodson, T. S., E. Hoffmann, and S. Boutilier S. 2021. Evaluating the NSF broader impacts with the inclusion–immediacy criterion: A retrospective analysis of nanotechnology grants. Technovation 101(C) https://ideas.repec.org//a/eee/techno/v101y2021ics0166497220300821.html (accessed February 22, 2024).