Emerging Hazards in Commercial Aviation—Report 2: Ensuring Safety During Transformative Changes (2024)
Chapter: Executive Summary
Executive Summary
In compliance with a directive in Section 132 of the Aircraft Certification, Safety, and Accountability Act of 2020,1 the Transportation Research Board of the National Academies of Sciences, Engineering, and Medicine tasked a study committee with identifying, categorizing, and analyzing emerging safety trends in air transportation, including offering advice to Congress, the Federal Aviation Administration (FAA), industry, and others on options for improving means for identifying, monitoring, understanding, and addressing emerging aviation safety risks. Throughout its activities, the committee relies on a definition of “emerging” that refers to something “becoming apparent or prominent.” Thus, the committee interprets emerging trends in safety to include both new hazards emerging via proposals for new technologies or operations, as well as current concerns that may be evolving to become prominent. This report marks the second installment of a series of six reports to be issued within a span of 10 years.
This report focuses on safety management for new entrants and for transformative changes to technologies and operations that cannot simply just apply current methods. “Transformative changes” are defined here as changes that are sufficiently novel that their impact on safety and risk cannot be extrapolated from current data and analysis methods. Such changes are highlighted in the potential systemic stressors noted in the first report with “new entrants” and “new technologies and operations.”
___________________
1 H.R.8408, 116th Cong. (2019–2020). Aircraft Certification Reform and Accountability Act. https://www.congress.gov/bill/116th-congress/house-bill/8408.
This report frames its analysis using the established principles applied in aviation safety management. Safety management occurs across the entire lifecycle of technologies and concepts of operation, from their design, through their implementation via production, operations, and maintenance. Such safety management spans multiple processes, including (1) safety risk management processes before new technologies and operations are implemented, to identify hazards, assess their risk, and implement appropriate safeguards and mitigations, largely focusing on certification and regulatory approval before implementation; (2) safety assurance processes once new technologies and operations are implemented via ongoing data collection and analysis to support continuous improvement; (3) safety policy and objectives determining the high-level properties of the organization integrating a commitment to safety; and (4) safety promotion across all levels of the organization.
First, the committee examined how to identify and address emerging safety concerns via safety risk management with transformative changes to technology and operations. Second, examining safety assurance, the committee noted that, while extensive data gathering activities support monitoring of current concerns, gaps exist in data and analysis methods for predicting new hazards that may emerge with transformative technologies and operations; similarly, methods for predicting and then continuing to monitor for safety concerns are vital to the safe implementation of transformative changes to technology and operations. Third, examining safety policies and promotion across the organization, the committee noted in its first report how FAA’s assessment of its internal safety culture was then in a formative state requiring further development and review. In this report the committee follows up on the present status of FAA’s safety culture assessment and extends this discussion to the question of how to foster and monitor the safety cultures of organizations across the industry.
SUMMARY OF FINDINGS AND RECOMMENDATIONS
Safety Risk Management
The starting point for managing aviation safety must be during design of new technologies and operations and continue through their testing and evaluation before implementation. Known as safety risk management, this discipline includes appropriate processes for identifying hazards, assessing their risk, and implementing appropriate safeguards and mitigations commensurate with risk; these are critical contributions to identifying, categorizing, and analyzing emerging safety trends in aviation. FAA plays three vital roles in safety risk management:
- Certification of individual products, personnel and practices of designers, producers, maintainers, and aircraft operators (managed by the Office of Aviation Safety [AVS]);
- Establishing the collective Operating and Flight Rules and procedures by which aircraft are operated within the National Airspace System (NAS) (managed by AVS and Air Traffic Organization [ATO]);
- Deployment, operation, and maintenance of the air traffic management system (new equipment, operations, and procedures) that provide aircraft separation, air traffic control and air traffic management (managed by the ATO).
Proposals for certifying transformative changes poses a unique challenge when current methods of demonstrating compliance do not sensibly describe their characteristics. Literal adherence to the existing regulations may unnecessarily complicate any attempt at certification or suggest workarounds that do not necessarily support safety. FAA’s role should be viewed as ensuring an acceptable level of risk while at the same time providing a reasonable path for innovation within the broad intent of safety risk management. (F2-1)2
Recommendation 2-1: The Federal Aviation Administration Office of Aviation Safety should evaluate current regulations, guidance, and standards to determine if they limit or preclude certification of transformative changes in technologies and commensurate changes in personnel, production, and operating certificates, or appear to disincentivize new features that may enhance safety in different ways than historically assumed. Where such a situation is found, certification processes and standards should be updated to promote effective safety risk management within these new developments, including collaboration with industry where appropriate.
Similarly, the committee found that, for most types of operations, the FAA AVS does not currently have mechanisms for comprehensive safety risk management that spans the different FAA organizations certifying technologies, personnel, and operating certificates, and governing the general operating and flight rules for the NAS. Since the regulatory offices are organized around how the NAS operates today, transformative changes can be difficult, if not impossible, to address. (F2-2)
___________________
2 Findings as they appear in the chapters are denoted herein at the end of each paragraph as F(chapter number-finding number), or 2-1 in this case.
Recommendation 2-2: The Federal Aviation Administration should establish cross-cutting positions, staffed by officials responsible for examining proposed transformative changes that simultaneously seek to manage safety and risk by purposeful, integrated changes to technologies, personnel, and operating certificates, and authorized to identify suitable means of compliance for, and certify, such transformative changes based on their collective mitigation of risk.
Current standards for certification and operational approval for many functions necessary for safe flight prescribe technologies, roles, and skills for personnel, and operating rules that assume historic concepts of operations and architectures. These assumptions do not fit with, and can conflict with, proposed transformative changes to aviation technology and operations. Instead, performance-based standards can direct assessments of how well a proposed technology or operation meets safety standards and do so in a manner directly supporting certification and operational approval. (F2-3)
Recommendation 2-3: The Federal Aviation Administration Office of Aviation Safety should lead the definition of performance-based standards for flight-critical functions that allow for different equipage, different methods of performing the functions, and a different distribution of who or what is performing the functions’ constituent activities, including changes in distribution of activity between human or machine, on the aircraft or on the ground, by the operator (e.g., airline), air traffic or a third party. These performance-based standards should be developed to the point that they can serve as a basis for certification and for approving new operations within the National Airspace System.
The committee also found challenges beyond certification of individuals’ products and operating procedures: radical transformations to general operating and flight rules are being proposed, or will be necessary to safely operate proposed vehicles. However, there are no systematic, rigorous, broadly documented methods for identifying, categorizing, and analyzing the risks that may emerge with novel operating and flight rules, especially where the changes are sufficiently transformative that they cannot be predicted via extrapolations from the current day. (F2-4)
Recommendation 2-4: Congress should ensure that fundamental and generalizable research is chartered to establish a systematic and repeatable framework for analyzing and designing civil aviation general operating and flight rules that enable novel technologies and operations to share the airspace with current operations. This requires fundamental and generalizable research beyond the Federal Aviation Administration’s
current research expertise and portfolio, and may need the support of other agencies conducting fundamental research into operational system-wide safety. These methods must calculate the appropriate performance requirements for all the functions noted earlier, be able to identify and characterize safety concerns created or mitigated by the operating and flight rules, and be detailed enough to articulate the specific activities, and required information needs and control authority, of all actors and stakeholders and how these activities combine to create safe operations.
Likewise, different operators cannot use conflicting general operating and flight rules in the same airspace, highlighting that these flight rules cannot be developed independently by individual applicants, but must have some guiding structure and constraints that are fair to all in terms of access to the airspace and safety for all in the airspace while minimizing undue constraints on innovation. (F2-5)
Recommendation 2-5: The Federal Aviation Administration (FAA) and the aviation industry should develop a process for defining the general operating and flight rules that enable new, innovative operations to coexist in the airspace with other innovative operations as well as legacy operations. FAA should launch a single-purpose group (e.g., task force, aviation rule making committee) including representatives of all relevant stakeholders from legacy and new operations for the purpose of identifying the fundamental structure of and constraints on operating and flight rules for broad classes of current and proposed new operators in the National Airspace System. This should include consensus-based recommendations for development of technical standards that characterize all relevant aspects of the proposed operations relevant to assessing their safety and that identify the essential constraints on and performance measures of new operating rules needed to address potential risks emerging from changes in the operations.
Safety Assurance
Even after the most rigorous safety risk management process, safety concerns may continue to emerge as a system is implemented and operated. Some concerns may only become apparent in real operations. Other concerns may only be very subtle and require more experience and data to identify and characterize. Furthermore, new concerns may arise over years or decades as a technology or operation that had been “safe” becomes stressed by changes in the operating environment and other external factors.
Thus, the second component of safety management, safety assurance, depends on the ongoing collection and analysis of data, both to understand possible concerns and to constantly monitor for the unexpected. Furthermore, safety assurance is only truly effective if it not only monitors safety performance but also uses its insights to both direct and manage change within an organization, and to reflect upon, and continuously improve, safety management overall.
The committee found that proposed transformative changes in technology and operations provide both challenges and opportunities to rethink the appropriate data to collect to support safety assurance. Given that many new technologies can record a wide range of new measures, the opportunity exists to systematically determine what potential data streams best monitor for potential safety concerns that may emerge with innovation. When addressing this opportunity, the safety assurance process should particularly consider the unique safety concerns with changing roles of human and machine, and with the unique concerns with monitoring software-intensive functions. This should also capitalize on criteria used in certification and approval of transformative changes, including monitoring of criteria applied earlier in certification and other safety risk management processes based on performance-based standards and on special criteria and conditions. (F3-1)
Voluntary safety reporting programs (VSRPs) are a vital data source in safety assurance, given the ability of personnel throughout the NAS to detect and describe anomalies. Changing roles between human and machines, particularly with increasingly automated functions and remotely piloted aircraft, may impact human observability of safety concerns. In such cases, the personnel expected to provide VSRP reports, and the questions they are asked within the report, may need to change. Likewise, further digital data may be required to make up for gaps in, and to make sense of, VSRP reports. (F3-2)
Recommendation 3-1: The Federal Aviation Administration Office of Aviation Safety should determine the process and criteria by which an applicant can demonstrate that their proposed data set is appropriate for safety assurance when implementing transformative changes in technology and operation. This determination should be sufficiently proactive to identify where new sensors and recording mechanisms need to be built into systems during their design and certification to then enable safety assurance during their operation. These data sets can leverage the criteria used in safety risk management (including certification and operational approval) to demonstrate safety of their new attributes through performance-based standards and through the special conditions and criteria.
Given the many forms of data analysis enabled by fields such as data mining and machine learning, more and better is possible in both characterizing and predicting known (or hypothesized) safety concerns, and monitoring for the unknown. Unfortunately, little research has been conducted and documented in the public domain on how such methods of data analysis can be applied to the full range of analysis that aviation safety assurance requires. The committee knows of no recent research by federal agencies. Likewise, the sensitive nature of aviation data, and the absence of publicly available data sets, has limited open research into the extension of general data analysis methods to aviation.
Extensive data analysis methods have been established in other domains and industries suitable for expanding the capabilities of aviation safety assurance. These methods particularly span the needs of transformative changes to technologies and operations, where simple definitions of exceedances cannot span the open questions in possible safety concerns that need to be monitored for; instead, transformative changes require a range of methods, from those suitable for monitoring for, characterizing and predicting potential concerns to detecting the unknown. (F3-3)
Recommendation 3-2: The Federal Aviation Administration Office of Aviation Safety should determine the process and criteria by which an applicant can demonstrate that their proposed data analysis methods are appropriate for safety assurance when implementing transformative changes in technology and operation, and that they are appropriate for the data set being collected. This determination should specifically support both (1) characterizing phenomena that are only hypothesized or poorly understood as a result of transformative changes; and (2) monitoring for situations and conditions that are unknown and statistically abnormal, and thus should be flagged for further evaluation.
Safety Culture
Organizational processes and culture are critical aspects of effective safety management. The committee examined several aspects of safety culture. They first examined the annual safety culture assessment by the FAA AVS. Second, because other safety-critical industries, particularly nuclear power, have begun to explore the safety culture of regulators and its effect on operators, the committee examined how the constructs of regulatory culture may apply to AVS. Finally, the committee examined strategies FAA could use to ensure that industry organizations—especially “new entrants”—develop and actively maintain and shape mature safety cultures.
Examining the FAA AVS safety culture survey and assessment, the committee supports the AVS shift in 2023 to a survey based on the 10
International Atomic Energy Agency (IAEA) safety culture traits and its efforts to improve the response rate. The appointment by AVS of a manager with experience in the safety culture assessment process is appropriate. These steps represent good progress, although the committee observes that the congressional request for the annual assessment of the AVS safety culture was enacted in December 2020, while the second survey had been completed at the time of the committee’s briefing in March 2024, the overall second annual assessment was incomplete. (F4-1)
A safety culture survey represents but the initial step in an ongoing dialogue and discovery process between senior management and employees about the safety culture of an organization and how it can be strengthened. For the committee to carry out its charge of reviewing AVS’s annual safety culture assessment requires greater insight into the steps being taken by AVS to assess its culture beyond the survey and how it is using what it is learning to strengthen the AVS safety culture. (F4-2)
Success in strengthening the AVS safety culture depends heavily on the direct, visible, and frequent engagement of senior AVS management with front-line employees in enabling, enacting, and elaborating the AVS safety culture. These are not responsibilities that can be delegated to the manager responsible for the safety culture survey. At the time of this writing, the committee lacks evidence about such a level of engagement by the highest levels of AVS management in learning from safety culture assessment and implementing actions aimed at strengthening AVS’s safety culture. (F4-3)
The emerging concept of the safety culture appropriate for a regulator reflects the view of the IAEA that, if regulators expect the organizations that they regulate to have good safety cultures, they must understand and model safety culture principles and behaviors themselves. To date there is insufficient research in defining, and distinguishing between, the traits and behaviors of the safety cultures of regulators and the safety culture of those they oversee. The emerging concept of regulatory safety culture is one that AVS can monitor and learn from. Moreover, FAA could support the development of this concept and its assessment through its research budget. (F4-4)
The safety culture of any large organization does not change quickly. An annual survey is too frequent to pick up shifts in employee perspectives about the organization’s safety culture. (The Nuclear Regulatory Commission conducts a survey every 3 to 5 years.) The ongoing assessment process of the AVS safety culture can also employ appropriate cycles of other processes, such as focus groups, ongoing dialogue with front-line employees, and input from the employee voluntary reporting system to help AVS articulate and mature its safety culture. (F4-5)
Recommendation 4-1: Congress should continue requiring that the Federal Aviation Administration Office of Aviation Safety (AVS) assess its safety culture, but allow AVS the flexibility to reduce the frequency of the safety culture survey, and in alternate years allow AVS to focus more of its annual assessment efforts on formal and informal communication by leadership, conduct of focus groups and other forms of dialogue with employees about their perceptions of AVS’s safety culture, and feedback to employees about what leadership is learning through the assessment process and the changes it is making in response. Within this process, AVS should identify two or three major goals the organization has in strengthening its safety culture and a short list of actions it will be taking in the next assessment. Likewise, AVS should identify the safety culture traits and behaviors it should model as a regulator to the industry organizations it oversees.
Recommendation 4-2: The safety culture assessment manager that Federal Aviation Administration (FAA) Office of Aviation Safety (AVS) has added to its staff should report regularly to the AVS Associate Administrator and the FAA Administrator, both of whom should be responsible for appropriate actions to enable a strong, and continuously improving, AVS safety culture.
The committee also examined how FAA can monitor and respond to industry organizations’ safety cultures. The components of safety management, the data they generate, and their implementation in a safety management system (SMS), can provide FAA with a primary mechanism for monitoring organizations’ safety culture maturity, how safety management will be integrated into operations and management by new entrants, and how safety management can be applied throughout the organization for continual improvement. (F4-6)
Promising safety indicators to support both industry organizational structures and safety culture and FAA monitoring would include testing of employee competence in understanding and executing risk controls; feedback between front-line employees and system developers regarding identification and management of hazards; identification and response to unanticipated emerging hazards within the organization; assessments of employee voluntary reports, SMS audits, audit deficiencies, and corrective action plans and timeliness of response to them; and other indicators of the organizations’ preoccupation with what might go wrong and continual improvement. (F4-7)
These findings on safety culture contribute to recommendations next for safety management that is integrated across disciplines, organizations, and the lifecycle of new products and operations.
Integrated Safety Management
The tasks of identifying, monitoring, understanding, and addressing emerging aviation safety risks cannot be fully achieved by only looking at separate aspects of the NAS and by applying sequential processes. Instead, safety is only effectively managed when all aspects of safety management are purposefully coordinated, and when they integrated across the many organizations involved.
Looking specifically at managing safety across organizations, safety cannot be regulated by examining only the safety management processes of a prime organization that involves third-party suppliers or service providers in support of their design, production or aircraft operation; likewise, it is not sufficient for the regulator to oversee each organization separately. Instead, the prime organization’s decision to involve others, including purchasing their products and services, requires a deliberate, layered approach to safety management that ensures all contributions together comply with their safety risk management and safety assurance processes, and are based on appropriate organizational processes and culture. The regulator has the role in overseeing that this layered safety management is implemented and continuously monitored and used to manage safety across all constituent organizations. (F5-1)
Recommendation 5-1: The Federal Aviation Administration Office of Aviation Safety should establish the personnel, mechanisms, and policies that enable oversight of effective layered safety management of an organization applying for certification, ensuring that this safety management also spans the contributions of those third parties whose products and services contribute to safety. This oversight must ensure that this layered safety management is not only implemented correctly at the time of initial certification but also continuously applied.
Often, the entities that seek to certify products, to certify personnel, and to operate and maintain aircraft are separate, uncoordinated organizations, and they may be active at very different points in time. Safety is bolstered when each entity, at the time of its activity, is expected to capture the data and knowledge that informs safety management by those reasonably expected to use the same technologies or operations after them. (F5-2)
Recommendation 5-2: The Federal Aviation Administration Office of Aviation Safety should identify and characterize the data and knowledge associated at each stage in designing, testing, maintaining, and operating aircraft that then can be useful to safety management in the life of the product or operation. This data and knowledge should be required to
be captured at the time, and later integrated into subsequent activities in support of both safety risk management and safety assurance.
Recent events suggest that the FAA AVS may already be challenged to regulate all aspects of the NAS. Transformative changes will further pose challenges with managing new risks. To address both current and likely future technologies and operations will require adequate funding and staffing, and requires this staff has the requisite technical expertise across the full spectrum of technologies. Furthermore, AVS staff must have the training and vision to oversee broad safety management processes spanning the life of a product (design, production, and operation), spanning multiple organizations and considering the organization structures and culture needed in these organizations that they oversee. (F5-3)
Recommendation 5-3: The Federal Aviation Administration Office of Aviation Safety should evaluate its personnel requirements in light of the demands placed on the workforce in identifying and addressing both existing and emerging risks. Emphasis should be placed on expertise required to oversee and evaluate new and emerging technologies and operations, to oversee the transition from safety risk management to safety assurance as new technologies and operations are implemented, to support the maturation of safety culture within the industry organizations it oversees, and to ensure rigorous safety management processes within all the contributing organizations that impact aviation safety.
Even when all is properly regulated and evaluated in best faith, any first-time implementation represents a new frontier in knowledge in which the unexpected can manifest subtly or suddenly and violently. In aviation, even reasonably small changes to otherwise-solid systems have a history of the unexpected occurring. Transformative changes in technology and operations reflect an even larger step-function change in the knowledge required to design systems and operations safely, and even to know what tests and monitoring to run. Each transformative change represents a step-change in knowledge—and each of these step changes may have gaps that are undetectable and inscrutable until later, with experience. Thus, it is important to constantly understand that, even after the best safety risk management before implementation, something unexpected may happen once something new takes flight.
Thus, the aviation industry, and FAA in all its roles, should remain vigilant for emerging safety risks as new technologies and operations are implemented—to detect the precursor before it manifests as an accident, to investigate unexpected behaviors and effects to characterize their safety and risk, and to be open-minded and prepared to seek new mitigations to newly identified risks. (F5-4)
This page intentionally left blank.
