Page 35

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Appendix A. Strengths and Weaknesses of Current Evaluation Methods

Knowledge, Attitudes and Perceptions, Beliefs, Norms, and Intentions

Knowledge
Objective	To measure knowledge of traffic safety-related campaigns or initiatives and knowledge surrounding specific safety initiative (e.g., knowledge around seat belt usage and enforcement)
Data Collection Methods	Quantitative data: surveys and questionnaires (e.g., phone, internet, intercept) Qualitative data: interviews, focus groups, and open-ended survey items
Resources	Survey and interview guide design, data collection staff, and database management.
Strengths	Weaknesses
Measured objectively through surveys, tests, questionnaires Can be quantified, making it easier to track over time Provide a baseline understanding of prior knowledge, allowing for comparison	Self-reported data is subject to response bias. Respondents might provide response based on what is perceived as socially desirable Does not always correlate with behavior change The quality of assessment tools can impact the reliability of the data collected

Attitudes and Perceptions
Objective	To assess how the target audience thinks and feels about road safety topics, campaign messages, and behaviors.
Data Collection Methods	Same as Knowledge – see above
Resources	Same as Knowledge – see above
Strengths	Weaknesses
Insights into attitudes and perceptions have the potential to help inform targeted interventions Inform customized campaign messaging to address specific concerns	Response bias can be influenced by the characteristics of the participant group. For example, if the study participants are already predisposed to safety consciousness, there may be a limitation to how much their consciousness of safety will improve When respondents are asked to complete a survey more than once (e.g., pre-and-post), it introduces the potential for response bias. Individuals willing to participate in multiple survey instances are more likely to exhibit more interest or concern regarding the topic

Page 36

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Beliefs (includes behavioral beliefs, control beliefs, and normative beliefs)
Objective	To gain insight into the beliefs of the target audience and how these beliefs influence their attitudes and behaviors
Data Collection Methods	Same as Knowledge – see above
Resources	Same as Knowledge – see above
Strengths	Weaknesses
Provides insights into beliefs have the potential to help inform targeted interventions Inform customized campaign messaging to address specific concerns	Self-reported data is subject to response bias. Respondents might provide response based on what is perceived as socially desirable

Norms
Objective	To measure intent or desire to engage or change a specific behavior
Data Collection Methods	Same as Knowledge – see above
Resources	Same as Knowledge – see above
Strengths	Weaknesses
Measuring norms can allow for more targeted messaging to align with audiences’ attitudes and beliefs Alignment with perceived norms can encourage behavior change	Self-reported data is subject to response bias. Respondents might provide response based on what is perceived as socially desirable Norms can be subjective and difficult to quantify There is culture variation in norms for different population groups Limited control on external factors that influence norms (e.g., social influence, media)

Intentions
Objective	To measure intent or desire to engage or change a specific behavior
Data Collection Methods	Same as Knowledge – see above
Resources	Same as Knowledge – see above
Strengths	Weaknesses
Can be used as a proxy for behavior when behavior cannot be observed or measured Understanding audience intentions allows for more targeted messaging Can be cost-effective (e.g., surveys and questionnaires) relative to observations or naturalistic driving studies Can be quantified if using Likert-type questions	Self-reported data is subject to response bias. Respondents might provide response based on what is perceived as socially desirable Used as a proxy for behavior, therefore may not correlate with behavior change.

Page 37

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Self-reported data on intentions is stronger when coupled with other data, such as naturalistic observation

Campaign Awareness and Recognition

Campaign Awareness
Objective	To assess how successful the campaign was in reaching the intended audience and raising awareness about key messages and objectives
Data Collection Methods	Quantitative data: surveys and questionnaires (e.g., phone, internet, intercept) Qualitative data: interviews, focus groups, and open-ended survey items for qualitative data that do not present campaign stimuli/materials to prompt a response (unaided recall)
Resources	Survey and interview guide design Data collection staff Database management
Strengths	Weaknesses
Serves as a moderating variable, influencing how the independent variable (campaign presence or absence) impacts the dependent variables (measured campaign effects) Can serve as an indicator for interpreting outcome evaluation data Provides insights into whether the campaign is reaching target audience	If used as the only measurement variable, it may present challenges because memory is not a true measure of the effect of the campaign. Memory is a limited predictor of attitudinal and behavioral change.

Campaign Recognition
Objective	To assess whether the campaign has left a lasting impression on the target audience
Data Collection Methods	Same methods as Campaign Awareness above but presenting campaign stimuli/materials to prompt a response (aided recall)
Resources	Same as Campaign Awareness – see above
Strengths	Weaknesses
Can compare recognition rates across different campaign elements Provides insight into whether the campaign is reaching and resonating with the target audience	Same as Campaign Awareness – see above

Page 38

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Media Metrics

Units of Messaging/Frequency
Objective	To provide insight into the volume and frequency of campaign media being published
Data Collection Methods	Tracking of posts, usually facilitated through the online platform (e.g., Facebook, Twitter) or tracking ad buys in traditional media
Resources	Social media management tools (e.g., Hootsuite, CrowdTangle) and others may track posts that come from different accounts if they use the same hashtag If working with traditional media (e.g., Radio, TV, newspaper), the source should track and provide these numbers
Strengths	Weaknesses
Demonstrates the volume of campaign-related media disseminated Easy to track and calculate from the start As accurate as the tracking method Allows for confirmation of dissemination activities and adjustments as necessary	Does not provide insight into the number of people who saw the media, how long they looked at it, or how much they engaged with it Only as accurate as your tracking method For social media tracking, use of Facebook Ads Manager, Twitter/X Ads Manager, or other social platforms

Earned Media
Objective	Shows how much free publicity the campaign earned
Data Collection Methods	Same media tracking tools as Units of Messaging/Frequency—see above
Resources	Traditional media: generally manually tracked; the source will likely contact the campaign managers to ask about the campaign and one can track from there Digital/social media: Social media management tools (e.g., Hootsuite and CrowdTangle) can track the use of campaign hashtags, but manual tracking and searching may be done to collect posts that do not use the campaign hashtags
Strengths	Weaknesses
Shows the appeal of the campaign messaging with media outlets and influencers Shows the reach of the campaign outside of paid efforts	More difficult to track than directly paid-for media The amount and type of coverage can vary widely

Page 39

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Counting mentions without context can lead to incorrect assumptions about campaign effectiveness, given that mentions may be both positive and negative

Website Metrics
Objective	Shows how many people used the campaign-related website, if and how they interacted with it; how effective campaign media was in getting folks to visit the website.
Data Collection Methods	Metrics can be tracked through Google Analytics To track where website visitors come from, you can create unique links through UTM codes and services like Bitly
Resources	Google Analytics Bitly UTM codes
Strengths	Weakness
Can show how effective the website is in attracting and maintaining visitors’ attention Simple to track if it is set up well at the start of the campaign	Only provides quantitative data on the volume of people looking at the website, their interactions with it, and their time spent on it Conclusions are made based on inferences from these without the additional context that would be gained in surveys/focus groups

Social Media Engagement
Objective	Assess the level of interaction users have with content and evaluate a campaign’s reach, awareness, and resonance with the target audience
Data Collection Methods	Metrics can be tracked through Google Analytics Built-in analytics tools that allow tracking engagement such as likes, shares, and clicks Third-party social media tools that gather and analyze data across multiple platforms
Resources	Google Analytics Built-in media analytic tools Social media management tools (e.g., Hootsuite, Buffer)
Strengths	Weaknesses
High engagement can indicate if a campaign content resonates with an audience	Can be reliant on the type of platform rather than the message content

Page 40

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Real-time feedback on content, which allows for quick adjustment and responsiveness to feedback

Engagement rates can fluctuate due to external factors unrelated to the road safety campaign (e.g., social events, trending topics)
Platform algorithms can change, affecting the visibility and reach of content

Social Media Followers
Objective	Measure the rate/number of followers; assess the ability to expand audience and reach of a campaign
Data Collection Methods	Same as Social Media Engagement – see above Built-in analytics tools that provide real-time follower count Manual counts
Resources	Same as Social Media Engagement – see above
Strengths	Weaknesses
Quantifiable metric that provides a clear numerical representation of an online audience Use data to benchmark campaign performance against past campaigns Segmentation by demographics and interest allows tailor messaging to better resonate with specific audience groups	Follower demographic characteristics are typically unknown Does not measure influence or impact at the individual level Counts can fluctuate due to external factors unrelated to the road safety campaign (e.g., changes to platform algorithm, people unfollowing platforms)

Reach
Objective	Assess the number of different people who have seen or heard a campaign message or content
Data Collection Methods	Same as Social Media Engagement – see above Built-in analytics tools that provide metrics on number of viewers/listeners
Resources	Same as Social Media Engagement – see above
Strengths	Weaknesses
Quantifiable measurement of how many people have been exposed to the campaign Use data to benchmark campaign performance against past campaigns Track progress over time Assess whether the target audience has been reached	Often only an estimate Can be reliant on the type of platform and algorithm High reach on one platform does not equate the same level of impact as reach on another platform High reach does not equate to impact or influence on the audience
Impressions

Page 41

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Objective	Assess the number of times the campaign has been presented to audience
Data Collection Methods	Same as Social Media Engagement – see above
Resources	Same as Social Media Engagement – see above
Strengths	Weaknesses
Quantifiable metric that provides data on campaign visibility Use data to benchmark campaign performance against past campaigns	Can be reliant on the type of platform and algorithm High exposure or impressions does not equate to influence or impact on the audience Does not differentiate between repeated exposure to the same individual and unique individuals

Campaign Cost-Effectiveness and Efficiency

Cost-Effectiveness Analysis
Objective	To evaluate the cost of the campaign relative to the outcomes achieved in general
Data Collection Methods	Review of campaign budgets Completion of a cost worksheet
Resources	Campaign expenditure data (i.e., all costs associated with the campaign) Outcome data (e.g., traffic enforcement data) Data analytic tools and data analyst
Strengths	Weaknesses
It is possible to compare multiple programs evaluated in different contexts and different years Show the user what can be achieved at what cost Useful when comparing methods of achieving a specific objective based on the lowest cost or greatest effectiveness for a given campaign cost	Requires additional data collection on media metrics for comparison Relying solely on CEA, without incorporating CBA can lead to bias results. When benefits are excluded from the evaluation of a campaign, it becomes challenging to determine which project is more socially desirable

Cost-Benefit Analysis
Objective	To evaluate the cost of the campaign relative to improved safety outcomes
Data Collection Methods	Same as CEA—see above
Resources	Same as CEA—see above

Page 42

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Strengths	Weaknesses
Able to evaluate programs with multiple outcomes Allows user to decide whether the expected benefits are worth the cost Helps determine how to maximize the net benefits (economic efficiency) of a program	Requires additional data collection on media metrics for comparison Requires several assumptions about the monetary value of benefits Does not include the value of difficult-to-monetize benefits; however, they may be socially desirable

Secondary Quantitative Metrics

Speed Metrics
Objective	To evaluate average speed/speed over the posted speed limit to measure the impact of anti-speeding messaging on drivers’ behavior
Data Collection Methods	Speed detection devices on roadways that cover areas served by the campaign Law enforcement speeding citation records Surveys and questionnaires (i.e., Likert-style questions - quantitative methods)
Resources	Surveillance technology (e.g., speed radar)
Strengths	Weaknesses
Exact enumeration of speeding behaviors in specific locations	Difficult to ascertain whether drivers have seen the campaign messaging Difficult/impossible to establish a control group Weather may affect speeds over time (e.g., rainy months vs dryer months), and the amount of light in daytime Self-reported data is subject to response bias

Incidences of Alcohol-Impaired Driving
Objective	Assess the instances where alcohol-impaired driving is occurring in a specific population
Data Collection Methods	Crash reports Law enforcement DUI records Breathalyzer and blood test (e.g., BAC test) Surveys and questionnaires Checkpoints and sobriety check data
Resources	Surveillance technology (e.g., BAC testing kits, video surveillance) Trained staff to collect and store data (e.g., law enforcement officers, medical professionals) Statistical analysis software (i.e., Joint Point regression) and data analyst

Page 43

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Strengths

Weaknesses

Measures the impact of road safety campaigns on reducing alcohol-impaired driving incidents
Objective and quantifiable metric
Baseline data on incidence

Incident rates can be influenced by external factors, not related to the road safety campaign
Resource- and time- intensive
Self-reported data is subject to response bias
Impaired driving incidents can go unnoticed or unreported, leading to an underestimation of true incidences
Law enforcement needs to be trained in the proper collection and management of the data
Data collection activities may fluctuate based on fluctuation in law enforcement staffing (with less staffing, data collection may not be a priority)

Number of Crashes
Objective	Assess the occurrences of traffic crashes and collisions.
Data Collection Methods	Same as Traffic Fatalities—see above
Resources	NHTSA’s Model Minimum Uniform Crash Criteria provides a model set of data elements that should be collected to enable safety professionals to conduct data-driven analyses Statistical analysis software (i.e., Joint Point regression) and data analyst
Strengths	Weaknesses
Objective and quantifiable metric Baseline data on incidence Generally, it provides basic information on the date, location, injury severity, types of vehicles, and characteristics of persons involved	It is important to consider confounding variables (e.g., new laws, events, societal trends) that may impact crash statistics during/after the campaign period It is also important not to measure campaign effects by accident data only because: Accidents are rare events, so tracking them makes it difficult to detect changes year by year Accidents are influenced by multiple factors, and often cannot be limited to just the target problem behavior; and Official data may not be detailed enough to test specific hypotheses. Correlating crashes with alcohol involvement takes additional analysis Inconsistent reporting -– different states and jurisdictions have different regulations for

Page 44

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

what is reported in a crash report, and in the threshold of what is required for a crash to be reported
Depending on agency capacity, there can be a long gap between when the crash occurs and when the data is reported
Law enforcement needs to be trained in the proper collection and management of the data
Reduction in data collection with fluctuations in law enforcement staffing

Traffic Fatalities
Objective	Assess the number of lives lost due to traffic crashes or collisions
Data Collection Methods	Secondary data from public records, like official accident statistics, statistics from insurance companies, and hospital statistics
Resources	Statistical analysis software (i.e., Joint Point regression) and data analyst
Strengths	Weaknesses
Objective and quantifiable metric Baseline data on incidence	Official statistics were not compiled for the purpose of campaign evaluation, so important details and context may be missing (date and time of particular events, causes of accidents, linkages to one or more behaviors)

Nonfatal Road Traffic Injuries
Objective	Assess the number of injuries due to traffic accidents or collisions
Data Collection Methods	Data obtained from trained operators who reviewed hospital discharge reports and hospital admission records
Resources	Train individuals to review secondary data and extract specific data points
Strengths	Weaknesses
Objective and quantifiable metric Baseline data on incidence	May not capture all injuries that occur (if there is no hospital record) Data variables being reported only by law enforcement with no verification by healthcare sources is a potential weakness. Issues related to failure to provide sufficient detail in the accident reports were the cause of concern for this weakness.

Behaviors

Page 45

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Risky Behaviors (including distracted driving, risky cycling behavior, speeding, drinking and driving, and fatigued driving)
Objective	Often used as a substitute for crash data, as an outcome measure. In the absence of crash data, measuring behavior that has some known relationship to risk of death or injury can provide some insight into the real-life effects of the campaign
Data Collection Methods	Field observation Intercept surveys (pre-and post- intervention) State and local agencies data on citations for risky behaviors Recommend measuring by comparing a control group with an experimental group, using objective measures (e.g., offences, random breath testing to measure BAC, seatbelt wearing), or subjective measures (e.g., self-reported behaviors)
Resources	Data collection staff (e.g., multiple field observers, intercept interviewers) Training data collection staff
Strengths	Weaknesses
Intercept surveys have a higher response rate than mail surveys and can be more cost-effective than conducting on-site observations State-level reporting shows actual and verified data from police officers Large-scale surveys collecting a high number of observations will provide the most accurate sample of the road user population.	Data collection is time-consuming Often, no observational data is available for the period before the intervention was implemented Intercept surveys are more expensive and time-consuming to conduct than mail, telephone, or web surveys There is potential for response bias Pre-and post- surveys are subject to selection history threat (selection differences between the participants in the intervention and the comparison groups lead to differences in exposure or impact of historical events) and may introduce testing or sensitization threat (restricted generality due to participants responding to / or evaluating treatment differently because testing instruments or procedures sensitize them prior to the treatment) Drawing information from state-level reporting on citations does not allow one to control the criteria for such violations, the level of detail provided, or the timeline in which it is reported

Page 46

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Protective Behavior
Objective	To assess behaviors that promote road safety
Data Collection Methods	Intercept questionnaire survey (pre-and post-intervention) Observation of seat belt use and car seat use at checkpoints
Resources	Data collection staff (e.g., intercept interviewers) Training data collection staff IHS observation seat belt survey protocol Statistical analysis software (i.e., Joint Point regression) and data analyst
Strengths	Weaknesses
Intercept surveys have a higher response rate than mail surveys and can be more cost-effective than conducting on-site observations	Data collection is time-consuming There is potential for response bias Like Risky Behaviors, pre-and post- surveys are subject to selection history threat and may introduce testing or sensitization threat Law enforcement conducting seat belt use/car seat check points need to be trained data collection and management Reduction in data collection with fluctuations in law enforcement staffing

Other

Unconscious Feelings
Objective	To assess the target audience’s subconscious emotional response to a road safety campaign
Data Collection Methods	Facial recognition software (e.g., GfK-EMO scan) Surveys and questionnaires
Resources	Facial recognition software and staff who know how to operate the software Skilled staff who can help interpret meaning and implication in unconscious feelings
Strengths	Weaknesses
Captures emotional response that may not always be apparent through traditional data collection methods Can help complement self-reported attitudes and behaviors	Complex and resource-intensive, requiring specialized equipment (e.g., facial recognition software) and expertise Confirmation of readings can rely on self-reporting, making it subjective and influenced by outside factors. This can also make it challenging to attribute results directly to a road safety campaign

Page 47

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Data Collection Methods

Individual Interview (Semi-structured, structured, unstructured)
Objective	Hold discussions with individuals to understand in depth what they believe were the effects of the campaign
Resources	• Data inputting • Data analysis programs
Strengths	Weaknesses
Can see the process and outcome of evaluation Can ask sensitive questions Assures confidentiality Can be conducted virtually to reduce costs Provides direct quotes in participants’ own words Allows flexibility with question structure (e.g. follow-up questions, clarifications) Potential for larger range of views compared with focus groups	Data analysis may be difficult Interviewers can bias questions and responses Time-consuming May require assistance from others to help develop protocol and database programs Subject to response bias Requires a good facilitator

Surveys
Objective	Obtain information from individuals about changes in knowledge, attitudes, and/or behavior
Resources	• Database to manage and analyze large amounts of data • May require participant incentives • Requires a larger sample size than focus groups
Strengths	Weaknesses
Pre/post use to evaluate outcomes Can be confidential and/or anonymous Can obtain information from a lot of people Can be conducted online to eliminate data entry Does not require experienced personnel	More effective when categorical responses are used Impersonal as questions are not typically in depth Unable to follow-up if any responses are unclear Subject to response bias Pretest and Post testing may be difficult to complete

Page 48

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Focus Groups
Objective	Hold discussions with small groups of people to understand in depth what they believe were the effects of the campaign
Resources	• Requires participant incentives • Experienced personnel required • Higher cost than surveys
Strengths	Weaknesses
Good for outcome evaluation Provides more detailed/in-depth data Can be conducted virtually to reduce costs Provides direct quotes in participants’ own words	In-person focus groups require additional facilities Requires a good facilitator Analysis and interpretation of data is time-consuming Subject to response bias

Meta-Analysis
Objective	A systematic study used to assess previous research to understand the campaigns effectiveness
Resources	• Database to manage and analyze large amounts of data • Multiple researchers (usually) needed • Access to online article databases • Data analysis software
Strengths	Weaknesses
Ability to utilize a large set of data Structure of search maintains consistency and ensures breadth of searches Allows for defined scope and variables Defines number of articles found, number that met inclusion criteria, and number that were used Does not require participant incentives Increases generalizability of individual studies	Cannot control for all aspects of study quality Search criteria may limit results New articles may be released after search is conducted Full text articles may not be available Exclusion criteria may accidentally exclude relevant articles Time-consuming Susceptible to publication bias Heterogeneity of researchers

Literature Review (Cochrane Reviews, Systematic Reviews, and Narrative Reviews)
Objective	A systematic study used to assess previous research to understand the campaigns effectiveness
Resources	• Limited software needed • Access to online article databases • Can be done by staff members
Strengths	Weaknesses
Ability to utilize a large set of data	Heterogeneity of researchers

Page 49

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Structure of search maintains consistency and ensures breadth of searches
Allows for defined scope and variables
Defines number of articles found, number that met inclusion criteria, and number that were used
Does not require participant incentives
Increases generalizability of individual studies
Scope can be adjusted based on criteria and resources
Uses already available information
Most common evaluation method

Some articles didn’t articulate all needed data types
Cannot control for all aspects of study quality
Results may not reflect behavior targets but may be due to campaign quality
Full text articles may not be available
Exclusion criteria may accidentally exclude relevant articles
Susceptible to publication bias
Difficult to be comprehensive
May have too narrow of scope
Time-consuming

Case Study
Objective	Case studies are used to describe programs or experiences in depth and can encompass one or multiple evaluation methods
Resources	Evaluation resources over time May need extensive software
Strengths	Weaknesses
Can combine evaluation of the process and outcome Powerful means for program descriptions Can identify personal experience, quotes, and unique processes	Time-consuming May be difficult to organize May be difficult to describe program May be resource heavy

Observation
Objective	Observations gather program information such as program operations or visual changes
Resources	• Can be done by staff members • supplies needed for recording and observing
Strengths	Weaknesses
Can give immediate understanding of evaluation progression and results Can observe where intervention or campaign occurs Can provide data that shows behavioral trends over time Potentially removes bias from experimenter presence Could reveal unintended consequences that may not have been considered Could provide additional context regarding behavior of interest	May be difficult to interpret information Interpretations may be biased by researchers Observed events may be time dependent Observers may influence actions and reactions of those being observed Time-consuming Could be difficult to determine demographic characteristics Details could be impacted by restricted viewing angles

Page 50

Suggested Citation: "Appendix A: Strengths and Weaknesses of Current Evaluation Methods." National Academies of Sciences, Engineering, and Medicine. 2025. Objectives, Components, and Measures of Effective Traffic Safety Public Awareness and Education Efforts. Washington, DC: The National Academies Press. doi: 10.17226/29155.

Cannot follow-up with observed participants regarding campaign knowledge and awareness

Existing Statistics
Objective	Existing statistics are used before, during and/or after the campaign to determine campaign effectiveness based on recorded results
Resources	Relatively low resource requirement Flexible data collection timing
Strengths	Weaknesses
Unobtrusive source of data Flexible data collection timing Baseline data for comparison can be found relatively easily Relatively straightforward analysis	Large amounts of data typically required for good results No control over data collection methodology making verification of accuracy and reliability difficult or impossible