CHAPTER 3

Data Collection and Analysis

This chapter presents the methods used to collect and analyze data from the selected sources.

3.1 Stakeholder Lists and Relational Analysis

Because this evaluation relied heavily on qualitative data and analysis, the evaluation team first sought to identify strong candidates for interviews and focus groups. Program stakeholders were individuals affiliated with organizations that participate in some or all program activities, from defining the program’s overall goals and directions, to selecting initial ideas for EDC Innovations during the call of ideas, to the implementation stage of the EDC Innovations. Internal stakeholders were those affiliated with FHWA, such as FHWA leadership, EDC Program staff, and representatives of other FHWA programs, departments, and offices. External stakeholders were representatives of organizations that are not FHWA and include state DOTs, local government agencies, national associations, industry, as well as consultants hired by the EDC implementation teams to support the dissemination of innovations by the FHWA to support EDC Innovations. The evaluation team generated lists of potential participants from the document review (described below) and from individuals recommended by the EDC staff as well-informed on the EDC Program’s goals and activities (see Table 2).

A goal of the evaluation design was to ensure that the full spectrum of perspectives was captured in the data collection and analysis, and that the evidence for evaluation findings was not biased in favor of a particular group. The data collection also was sensitive to the different positions of groups within the program’s structure. Because the EDC Program is a state-led effort by design, state-level stakeholders are overrepresented. The availability of the contact data and the degree to which the contacts listed were representative of their associated groups affected decisions on data collection, as described in the next subsection.

3.2 Interviews

Initially, the evaluation team conducted seven background interviews to obtain historical information relevant to process evaluation and contextual information to inform the design of subsequent data collection and analysis techniques. The background interviews provided the research team with situational awareness about the different types of stakeholders involved in EDC and the specific roles staff perform to select, promote, and implement Innovations. With this awareness, the research team determined that individual interviews and focus groups would be appropriate and efficient data collection methods for eliciting comprehensive input from the range of EDC stakeholders were developed.

Table 2. Sample stakeholder groups.

Separate interview guides were developed for FHWA staff, state DOT staff, and representatives of member associations representing the surface transportation community. Appendix D contains the interview guide developed for state DOT staff with fewer than 3 years of experience with the EDC Program. The interview guides contain semi-structured questions tailored to the background and roles of these individuals. Each interview lasted 30 to 60 minutes. All interviews were conducted using web video conferencing (e.g., Zoom) or by phone. After obtaining consent from the subject, interviews were recorded for use in validating notes taken for later analysis. In some instances, the research team conducted follow-up interviews with individuals who agreed to speak with us more than once. Those were not counted as separate interviews in the interview tracking.

Table 3 provides a summary of the number and type of interviews conducted. As noted above, internal stakeholders were those who are part of the FHWA and charged with supporting the implementation of the EDC Program, including work by the FHWA to support EDC-designated Innovations. External stakeholders were primarily representatives of state DOTs, but included representatives from associations, universities, and others who work with state and local authorities on developing and selecting Innovations. Additional interviews were conducted for the integrative case studies, as discussed in Section 3.4 below.

Table 3. Interviews completed.

Table 4. Overview of documents analyzed.

^* The FHWA website for the Every Day Counts Program and related activities contains numerous pages of content that cannot be counted as conventional documents. The evaluation team reviewed all relevant content available through the website.

3.3 Document Review

This section describes document review undertaken to answer the key evaluation questions and inform other components of the project. Table 4 lists the types of documents reviewed.

3.4 Implementation Plans

EDC implementation plans are developed by the IDTs to help guide states in understanding, assessing, and implementing innovations not yet widely used and known in the transportation industry. The IDTs receive training organized by the EDC Program to assist them in formulating and organizing these plans. Although not completely standardized, each plan presents similar information, including but not limited to a background description of the EDC Innovation, the challenges associated with it, and the various parties involved in implementation. The plans provide guidance on assisting state DOTs and other parties in adopting and institutionalizing each innovation and define obstacles to overcome. EDC Innovations for which the implementation plans have been analyzed are as follows:

• Collaborative Hydraulics
• Advanced Geotechnical Methods in Exploration
• Data Driven Safety Analysis
• e-Construction
• e-Ticketing and Digital As-Builts
• Smarter Work Zones
• Stakeholder Partnering
• Targeted Overlay Pavements Solutions
• Value Capture
• Weather-Savvy Roads

Although implementation plans are written by and for the IDTs and not for public distribution, these documents provide information relevant to the process evaluation questions and to designing metrics for capturing the Program’s outputs and outcomes. A description of these documents is included in Appendix C.

3.5 FHWA Website Content

The FHWA maintains a website with sections for the EDC Program, the STIC Network, the AID Demonstration Program, the Accelerating Market Readiness (AMR) Program, and other resources. Below is the content from the FHWA website reviewed:

• EDC Program site
• STIC Network site

• AID Demonstration Program site
• AMR Program site

These websites provided the basis for an initial database of EDC Innovations developed for this evaluation. The database includes key information on each innovation, a list of contacts for the EDC Program and contacts for each implementation team, and a taxonomy to classify EDC innovations along various parameters. The website also provided evidence of how the EDC Program engages key stakeholders through its various activities, including EDC summits and newsletters.

3.6 EDC Summit Summary and Baseline Reports

The EDC Summit Summary and Baseline Reports provide an overview of each of the EDC Summits and show the estimated level of adoption (implementation stage) for each state for each Initiative. These reports were used to analyze the development of specific innovation ideas, as well as the change in overall innovative topics from year to year. Six Summit Summaries and Baseline Reports are available for analysis.

3.7 EDC Progress and Final Reports

For each Round, the EDC Program staff publishes progress reports and a final report. The final reports exist for every Round except EDC Round 1. One progress report was published in EDC Round 2, and each subsequent Round has three progress reports, published every 6 months. These documents aim to summarize the status of deployment for the Innovations from their respective Rounds. All the reports include a description of each innovation; maps and charts for each innovation that show the implementation stage for each state; and an EDC Innovation “Spotlight” that focuses on the impacts of the innovation in a single state. The implementation stages show the status of deployment at the time of the report for the innovations and have been standardized since EDC Round 3. The Innovation Spotlight included a small section in EDC Rounds 2 and 3 but grew to a full page in EDC Rounds 4, 5, and 6. EDC Rounds 2, 3, and 4 include an additional section called Project Highlights, which summarizes the impacts and status of the innovation across several select states. Last, EDC Rounds 5 and 6 include an appendix in the form of a table that includes the implementation stage for each state and each innovation.

The final reports were used to determine how the status of deployment, or implementation stage, has changed as time passed for each innovation. The authors compared the status at the time of the final report to statuses reported in the baseline and progress reports. Progress reports were useful in getting a sense of activities, challenges, and outstanding needs throughout the process.

3.8 Newsletters and Legacy Reports

The EDC Program offers two newsletters: EDC News, a biweekly publication, and Innovator, released bimonthly. These newsletters serve as sources of information and inspiration in the transportation sector. EDC News, launched in 2013, keeps professionals updated with the latest developments, success stories, and valuable resources related to the EDC Program. In contrast, Innovator, dating back to 2007, takes a broader perspective, spotlighting innovative practices, technologies, and Innovations spanning the entire transportation sector. Beyond the EDC Program Innovations, Innovator covers a wide array of innovations, including those within the STIC Innovations and state-led projects. Together, these publications shed light on the transformative potential of innovation in shaping the future of transportation and highway construction and showcase the achievements of the EDC Program and Innovations. To date, there have been

98 issues of Innovator and 271 issues of EDC News. As of March 2023, EDC News had 34,154 subscribers and Innovator had 20,303 subscribers. The “open rate” (share of emailed publications opened by the recipient) for both newsletters averages 25% to 30% per issue, putting them in the top 20% of federal newsletters based on that metric.

3.9 Integrative Analysis and Case Studies

Interviews and document analysis were merged into a single process for quantitative analysis. Excerpts from transcripts were copied into a qualitative data management system and reviewed by team members to develop a content coding system to track both common interview themes and unique perspectives conveyed in the interviews. The coding scheme was developed from the evaluation questions and during the analysis phase, the evaluation team reviewed the coded excerpts and synthesized them into more general findings for each evaluation question.

To bring together the disparate array of data sources and items, the evaluation team researched structured case studies, each focusing on a key thematic area that encompasses multiple EDC Innovations. Evaluations contain case studies to investigate how complex processes unfold in a program; how programs deal with situations with a high degree of variability in their interventions and contexts; and how a program adapts to a changing environment. The EDC Program involves a diverse set of Innovations implemented in organizations that range widely in scale, scope, and stakeholder management. By using a series of case studies, the evaluation can provide a limited degree of generalization in its findings (Yin 1992). The case studies used in this evaluation helped the team to investigate the complex path through which program activities contribute to long-term outcomes and fulfill both programmatic and FHWA strategic goals.

The case studies for this evaluation encompassed developments through successive Rounds of the EDC Program. The EDC database developed by the evaluation team classified Innovations based on a standardized set of characteristics and terms, looking at the nature of the innovation, the intended outcomes, and the transportation topics related to that innovation. The team then selected clusters of innovations to investigate, comparing similarities among innovations within a thematic area and contrasting the management of EDC Innovations between thematic areas. Each case study examined the following aspects of the given innovation:

1. Background on the innovation covered, including the initial development and how the feasibility of the innovations was proven;
2. The specific problem or need in the surface transportation community addressed by the innovations, and how they relate to EDC and FHWA goals;
3. The criteria and conditions examined by the EDC Program in selecting each EDC Innovation;
4. EDC Program activities undertaken to promote adoption of the innovation;
5. Initial response from EDC stakeholders, including the benefits highlighted and barriers encountered that factored into the decision to adopt each innovation covered;
6. Comparative study of the diffusion of the innovations across diverse state DOTs;
7. Patterns and scale of adoption of the innovations after the end of the EDC Program support; and
8. Ultimate outcomes of the innovations covered, the approaches used by organizations to adopt Innovations after involvement with the EDC Program, and the innovation culture.

The evaluation team conducted case study research on the following EDC Innovations:

• Geosynthetic Reinforced Soil-Integrated Bridge System (GRS-IBS)
• 3D Engineered Models
• Intelligent Compaction
• Data-Driven Safety Analysis

• e-Construction
• Pavement Preservation
• Using Data to Improve Traffic Incident Management
• Weather-Responsive Management Strategies
• e-Ticketing and Digital As-Builts

The evidence collected through these case studies is integrated into the findings presented in Chapter 4 and Chapter 5. Using the case study method enabled the evaluation team to understand how support for EDC Innovations evolved within each Round and across multiple Rounds as the EDC Program matured.