Appendix C: Technical Memorandum on Methodology (Task 7 Deliverable)

NCHRP 23-20 GUIDEBOOK FOR IMPLEMENTATION OF UAS OPERATIONAL CAPABILITIES

Technical Memorandum on Methodology to Achieve Research Objectives

Prepared for
National Cooperative Highway Research Program
Transportation Research Board
of
The National Academies of Sciences, Engineering, and Medicine

WSP USA
1250 23rd Street NW, Suite 300
Washington, DC 20037

September 2023

Table of Contents

INTRODUCTION

Research Topics and Objectives

METHODOLOGY

Focus Group Participants

Focus Group Organization

Conduct Focus Group Meetings

Gap Analysis

Analysis

Conclusion

REFERENCES

Table of Tables

Table 1. Agendas for Focus Group Meetings

Table of Figures

Figure 1. Methodology Timeline

Figure 2. Focus Group Meeting 1 Representation

Figure 3. Focus Group Meeting 1 Attendee UAS Experience

Figure 4. Focus Group Meeting 1 Attendee AAM Experience

Figure 5. Focus Group Meeting 2 Representation

Figure 6. Focus Group Meeting 2 Attendee UAS Experience

Figure 7. Focus Group Meeting 2 Attendee AAM Experience

Figure 8. Coded Transcript Example

Figure 9. Excel Example of Referenced Codes

National Cooperative Highway Research Program Project 23-20: Guidebook for Implementation of Uncrewed Aircraft Systems (UAS) and Advanced Air Mobility (AAM) Operational Capabilities

List of Acronyms

Introduction

UAS technologies have a long-standing history and have been widely accepted by State DOTs over the last decade, yet, in many regards, UAS is still an emerging sector. While UAS adoption continues to offer many benefits and opportunities to State DOTs, it also poses many challenges. UAS and AAM technologies rapidly evolve and mature, making it difficult for stakeholders to find their bearings in the planning process to mature operational capabilities. Developing a comprehensive Guide to address the current UAS environment and the nascent nature of AAM requires the right methodology. This document outlines the execution of the project panel-approved methodology.

The project panel approved the proposed methodology—a mixed-method approach that includes focus groups and supplemental interviews, as needed—to enable the development of a Guide. The use of focus groups and interviews are both long-established and proven qualitative research methods (Morgan and Spanish, 1984; Mashuri et al., 2022); however, as with any research methodology, there are strengths and limitations. Focus groups and interviews are time-efficient and cost-efficient ways to collect data. Focus group discussions can create a group synergy that can enable the group to work together and share profound insights into complicated topics (Advantages of Focus Groups, 2021). When properly designed, focus groups and interviews can produce rich qualitative data sets for analysis. Focus groups can also serve as a valuable platform to understand how something should change or progress (Gibbs, 1997). In the context of this project, focus groups and supplemental interviews were the best methodologies to collect the necessary data to develop the Guide for maturing UAS and AAM capabilities.

Potential limitations to these methods include a lack of anonymity among participants that could influence their responses. A similar challenge is that the moderator can have a strong impact on the overall feeling of the focus group that could also influence participants’ responses. The third potential limitation is assembling enough people to establish a representative sample of the larger population group.

The research team developed an overall strategy to mitigate the first two noted limitations. The research team and focus group moderators established an inclusive and open environment to foster the sharing of diverse thoughts and opinions by developing semi-structured, non-biased questions, and using poll questions, the meeting chat function, and whiteboards. These various platforms allowed participants to share their thoughts in ways in which they were most comfortable.

The moderators are subject matter experts and knowledgeable on each of the topics discussed within the focus group, and they approached the focus group meetings with the recognition that each member had valuable contributions to be make. The moderators strived to remain unbiased while leading the discussion of the predetermined questions to draw out the views of each participant, and they continued to encourage participation through the various platforms throughout the focus group meetings.

For the third noted potential constraint, the research team mitigated this limitation by expanding the invitation list for the focus group. This list was developed to account for people with varying experience across each stakeholder group (e.g., participation from well-established State DOT UAS programs and from newer, less-established State DOT UAS programs). The list of invitees was developed with the goal that at least four participants would be able to represent each stakeholder group. Because the main

audience of the Guide is expected to be State DOTs, this stakeholder group was purposely designed to have a larger representation.

Research Topics and Objectives

As part of the preparation for executing the methodology that is thoroughly described in the following section, each research topic was evaluated. The research team framed each research topic as problem statements with associated research objectives. This exercise served as a guiding and foundational point for the development and execution of the methodology. The research topic problem statements and objectives are listed below.

• Future UAS integration through UAS traffic management (UTM), AAM, and UAM

UAS integration continues to mature as State DOTs adopt the technologies into traditional workflows. UAS package deliveries have started in a variety of locations throughout the United States, and additional opportunities and future use cases using UAS and AAM aircraft are being explored. There are still many unknowns regarding the integration of these future use cases, but this project aims to understand the systems needed (e.g., ground infrastructure, UTM) to enable integration of UAS and AAM into the NAS and into existing transportation and cargo ecosystems.

• Identification of the roles and responsibilities among State DOTs, federal and local agencies, including areas of overlap and gaps in responsibilities

The FAA has clearly defined its role in UAS and AAM integration and has organized its work in five key areas: aircraft, airspace, operations, infrastructure, and community (FAA, 2022a). The overarching role and responsibility of the FAA across these areas is to ensure UAS and AAM integration maintains the highest levels of safety that have been established and solidified in modern-day aviation (FAA, 2022b). Although the roles and responsibilities of the FAA are clearly defined, many State DOTs and local agencies may be unsure of their role in this new era of transportation. An objective of this project is to identify the roles and responsibilities at the federal, state, and local levels and to identify the potential overlap and gaps that exist between these various stakeholder groups.

• Identification of specific use cases that may become applicable to State DOTs (e.g., Air Cargo, passenger air mobility, Emergency Services)

Numerous potential use cases are being explored for smaller UAS and larger AAM aircraft. Although circumstances will vary from state to state, many of these use cases could become relevant to State DOTs. State DOTs may take on responsibilities at various levels for the oversight or management of UAS and AAM infrastructure. This project will identify specific use cases that may become applicable to State DOTs.

• Framework for coordinating resources from State DOTs or local agencies with UAS integration programs

As UAS technology continues to advance and transportation applications continue to mature, there is a growing need to establish a framework for coordinating resources between State DOTs or local agencies and UAS programs. This framework should ensure that integrating UAS into transportation infrastructure is safe, effective, and efficient, and that resources are used appropriately to support this integration. Although State DOTs have opportunities to coordinate through industry associations and the FHWA Every Day Counts (EDC) UAS efforts, a potential gap is a framework for consistent

coordination of resources. This project aims to confirm if this is indeed a gap, and to then develop a framework that could be instituted to enable ongoing State DOT collaboration and coordination of resources.

• Outreach strategies for coordinating UAS integration between multiple jurisdictions and engaging with private stakeholders

UAS and AAM integration will not succeed if done in a silo. For actual integration to occur, the coordinated efforts of countless agencies and organizations across local, state, and federal jurisdictions are required. This project aims to identify historically successful strategies, best practices, and lessons learned from the surface transportation and legacy aviation industries regarding stakeholder engagement and cross-jurisdictional coordination.

• Funding resources available for State DOTs or local agencies for the implementation of UAS (including AAM/UAM) operational capabilities

Securing initial and, perhaps more importantly, ongoing funding is often a leading concern for State DOTs starting or maturing a UAS program. Understanding potential funding resources available to State DOTs and how to acquire a dedicated UAS/AAM budget is essential to a program’s long-term success. This research aims to identify funding resources

• Requirements of workforce qualifications and training programs

Recruiting, training, and retaining a qualified workforce is a challenge faced by many State DOTs and throughout the UAS and AAM industries. Understanding what qualifications should be required for emerging use cases and any potential responsibilities State DOTs may take on is largely unknown. Best practices regarding UAS training programs continue to evolve, and necessary components of future AAM-related training programs are yet to be determined.

• The requirements for implementing the UAS operational capabilities by State DOTs, including IT infrastructure and security, data retention policy, organization structure and workflow, and training

Several State DOTs have successfully implemented UAS technologies into their operations and have established robust programs with internal training programs, policies and procedures, organizational structures, UAS integrated workflows, and UAS data management plans. Many other State DOTs are working toward maturing their UAS programs but are encountering challenges. These challenges range from simply not having quality guidance to a lack of workforce to cover current responsibilities. The objective of this research is to work closely with the State DOTs that have mature and established UAS programs to understand and identify best practices and necessary requirements for maturing the implementation of UAS operational capabilities.

Methodology

The methodology and associated research activities were executed over the course of 16 weeks as depicted in Figure 1.

Focus Group Participants

The first research activity was to select the focus group participants. Participants for the focus group were chosen primarily from the established survey participant list used in Task 2. This list for invitees was expanded in some of the categories to ensure adequate representation of the stakeholder group. The identified stakeholder groups developed for the survey distribution were also used as the foundation for the focus group. These stakeholder groups include:

• State DOTs and local transportation agencies.
• State Divisions of Aeronautics.
• Multimodal transportation planning organizations.
• UAS OEMs.
• Passenger air mobility OEMs.
• UAS/AAM service providers.

Adequate representation of at least four participants from each stakeholder group was the goal when seeking representation for participation in the focus group. When using focus groups as a research methodology, it is best practice to invite more participants than the established goal because participants may need to withdraw for various reasons. The research team invited more people per stakeholder group and contacted potential participants individually to explain the scope and goals of the focus group to secure commitments for the anticipated two meetings.

Focus Group Organization

Simultaneously with the first research activity, the date for the first focus group meeting was selected and scheduled using a scheduling tool to ensure the best date and time was selected for the maximum number of participants. The first focus group meeting was scheduled for July 5, 2023, and the second focus group meeting was scheduled for August 4, 2023. Both meetings were scheduled as virtual meetings on Zoom. Once dates were selected, focus group invitations and details were sent to participants. The meeting agendas were also developed and finalized concurrently with scheduling the meetings. The research topics were divided between the two meetings and times assigned to each topic.

Each meeting lasted 2 hours; the meeting agendas for both focus group meetings are included in Table 1.

Table 8. Agendas for Focus Group Meetings

Another key to the success of the focus group meetings was the development of the semi-structured questions to be used in the discussions of the research topics. The previously identified gaps from the Task 3 Gap Analysis served as the basis for these questions. Semi-structured questions are written in an open-ended format and designed to encourage participants to freely share their thoughts. Semi-structured questions allow for focused questioning, while still giving the focus group moderator the flexibility to dive deeper on relevant topics that emerge during the discussion (Adeoye-Olatunde and Olenik, 2021). The semi-structured method is ideal for exploring participants’ thoughts on complex open-ended topics and yields a rich data set.

The research team designed the focus group questions in an unbiased way, avoiding the use of leading questions. The neutral questions were developed to target the identified research gaps and extract data from the focus group participants to address these gaps. Focus group best practice is to use engagement, exploration, and exit questions to maximize the participation throughout the course of the focus group meeting (Then et al., 2014). Engagement questions are used to establish the baseline topic and warm up the participants. Exploration questions are designed to be the core questions for the focus groups and areas expected to take the most time. Exit questions are used to check if anything was

missed and to provide an opportunity for participants to share last-minute thoughts. The research team designed the flow of questions for each research topic to follow this sequence.

The research team also wanted to provide multiple opportunities to participate or various platforms in which participants could answer questions. The integrated tools on the Zoom platform provide an excellent opportunity to accomplish this. In addition to the open dialogue, each focus group meeting had a series of poll questions where participants could anonymously answer. Virtual whiteboards were used to allow participants to write down their thoughts on a sticky note and stick it to the whiteboard in a collaborative environment.

Conduct Focus Group Meetings

The focus group meetings were held virtually using Zoom. Both focus group meetings were recorded to help with notetaking and to provide recorded transcripts for analysis. Although the focus group moderators were taking notes throughout the course of the meetings, the recordings served as the primary data capture tool for later in-depth analysis.

Focus Group Meeting #1

The first focus group meeting was held July 5, 2023, and had 54 attendees. Forty-three of these attendees completed the poll question that asked them to anonymously report their affiliations; the results of this poll are provided in Figure 2. This information offers insight into how well each stakeholder group was represented in the meeting. Representation from the Multimodal Transportation Planning group was one short of the goal. Although representation was low for the federal agencies, this group was not a formal stakeholder group.

In addition to having each stakeholder group represented by at least four people, the research team had the goal of having a range of experience in UAS and AAM. The range of participant experience can be seen in Figures 3 and 4.

Focus Group Meeting #2

The second focus group meeting was held August 4, 2023, with 49 total attendees. The second meeting also included opening poll questions to understand the representation and experience level of those in attendance. Unfortunately, many representatives had technical difficulties accessing these initial poll questions resulting in only 20 attendees being able to respond to the questions. Figure 5 shows the affiliation of the participants, while Figures 6 and 7 demonstrate the mixed level of experience across UAS and AAM that was represented. In general, stakeholder representation and experience levels in the second meeting were similar to the first focus group meeting.

Gap Analysis

Task 3 of the project identified outstanding knowledge gaps following the Literature Review and Survey. One of the goals of the focus groups is to collect data to address these gaps. It often takes more than one focus group meeting to produce valid data and results, which is why the research team held the two focus group meetings. Following the first focus group meeting, members of research team compared notes, reviewed the recordings, and processed the data through a coding process which is outlined below the “Analysis” section. After analyzing the data from the first focus group meeting, the research team developed questions in preparation for the second meeting. This gap analysis process was repeated following the second meeting, and the research team determined that enough data had been secured from the focus group meetings and supplemental interviews were not needed at this time. If additional information is necessary later in the development of the Guide, the supplemental interviews can be scheduled.

Analysis

Following the completion of each focus group meeting, the data were thoroughly analyzed using the following process. The recordings from both focus group meetings were transcribed using transcription software. Thematic analysis was the approach used to analyze the meeting transcripts and the data from the meeting chats and whiteboards. This approach included properly formatting the transcripts, meeting chats, and whiteboards for the coding process. Once the data were formatting, each entry was thoroughly read and coded.

Qualitative data coding of transcripts can occur in a variety of ways. Coding can refer to numbers being assigned to various words or phrases, or a system of abbreviations or words and short phrases to describe bits of data. In qualitative coding, there are two categories of code: a priori codes and empirical codes. A priori codes are codes that are defined prior to conducting any data collection, typically related to categories that the researcher expects to have confirmed. Empirical codes are codes created after data collection while analyzing the data set (Gibson and Brown, 2009).

Using empirical codes with an inductive approach ensures that themes can be derived from the participants and the collected data. A deductive (a priori) approach approaches the data with a predetermined framework and seeks confirmation within the data. It is important to note that because the themes are data-driven, they may not necessarily always directly relate to the original focus group meeting questions. For example, if the participants expound on an answer to one of the previously developed focus group questions, this could lead to additional discussion on new topics. (Kiger and Varpio, 2020).

To allow new ideas or themes to emerge from the data set, the research team employed an empirical coding system by assigning short phrases to various segments of data from the transcripts and using these brief phrases as the codes. Each transcript was formatted for coding by placing the transcript in a left-hand column and noting the codes in the right-hand column. Each transcript was carefully read, and the ideas and concepts were highlighted in the transcript on the left and then a corresponding code was assigned and noted on the right. Figure 8 is an excerpt from one of the coded transcripts that illustrates this process. The arrow points to an assigned code that can be seen in the data shown in Figure 9. All 134 formatted pages from the two focus group meetings, associated chats, and the whiteboard data underwent the same coding process. The data set produced 390 total codes that proceeded to additional analysis.

The 390 codes from the coding process were exported into a Microsoft Excel workbook for further analysis. Each code was assigned a code reference number, and the source for each code was noted. After these reference numbers were allocated, the codes were sorted by alphabetical order. Then using a formula, the primary code was separated from the code description. The filter function was applied to the primary code column to allow each code category to be searched. Figure 9 shows an excerpt from the Excel document to show an example of how the codes were organized and analyzed. This example correlates to Figure 8 where it shows what was said in the focus group and the code that was assigned. That code appeared in the organized data on line 179 with the code reference number of 17, which can be seen in Figure 9.

Conclusion

The execution of the approved methodology for developing the comprehensive Guide to address the maturing use of UAS in transportation agencies and the nascent nature of AAM has been successful. Focus groups proved effective in producing a rich data set to address the research gaps and achieve the objectives of the project.

Using focus groups as the qualitative research method was efficient in terms of both cost and time throughout the data collection period. In addition, the thorough preparation prior to the focus groups, and active facilitation of the group synergy and insights generated a comprehensive and quality data set for analysis. To mitigate potential limitations such as lack of anonymity, moderator influence, and representative sample size, the research team implemented strategies to foster an inclusive and open environment.

The data collected from the focus groups will serve as a valuable resource for informing the Guide and providing guidance to State DOTs and other stakeholders involved in UAS integration and coordination efforts. The in-depth analysis of the data set produced results that will be presented, analyzed and expanded, and used to develop the Draft Guide, which will be reviewed by the panel.

The methodology employed in this project has demonstrated its effectiveness in gathering qualitative data and generating insights to inform decision-making and guide the development of resources for the UAS and AAM industry. The results of this project will contribute to the continual advancement and maturity of UAS and AAM capabilities, enabling their safe and efficient integration into the NAS and existing transportation ecosystems.

References