Decision makers need to know what works and what doesn’t in order to make informed choices about transportation investments. Hard evidence, rather than opinions, should be the source of this information. Making use of hard evidence, though, can be challenging when transportation proposals come from policy makers who have already made a public commitment to an idea, such as building a high-speed rail system. Presenting new information on such vision-driven projects may mean confronting, and perhaps contradicting, established political positions. Nonetheless, evidence on what works is vital to making wise transportation decisions. Experiments are the most effective way to show what works—or might work—in a given setting, especially when the experimental conditions are close to the decision situation at hand. Experiments vary in validity and transferability, but opportunities to experiment with transportation are all around us.
Four Types of Experiments
Planned and unplanned changes in transportation systems occur all the time. If we study them as experiments, there are opportunities to answer causal questions and to learn what works and what doesn’t. We can learn from four kinds of experiments: laboratory experiments, analytic models, field experiments, and natural experiments.
Laboratory Experiments
The lab allows us to test the feasibility and performance of new technologies and innovations such as engines, fuels, sensors, or crash protection systems. We can use simulations or focus groups to explore factors affecting human behavior, including the effects of cell phone distractions on driver performance, or reactions to new service concepts for public transit. Laboratory experiments can only answer narrowly-defined questions, however, and cannot address large scale feasibility issues, such as market and political responses, implementation issues, or longer term economic and environmental consequences. Additionally, transportation decision makers may find lab results to be too distant from reality, particularly when results conflict with their prior opinions.
Analytic Models
Models allow planners to experiment with designs and service concepts without the expense or safety constraints of field experiments. Modeling has long been a core activity of transportation planning and is most commonly used to analyze demand and network equilibrium. Model-based experiments are imperfect, however, and subject to biases, most notably when forecasting travel demand and system costs. Models extrapolate behaviors of systems and the people who interact with them, so they have limited ability to test substantially new concepts. Because of these limitations, analytic models may not be the best tool for decisions about large investments or radical service changes. Decision makers may discount the credibility of models, as they do for lab experiments.
Field Experiments
Planners can conduct field experiments when they implement new systems or services and observe the outcomes. The results of real-world experiments are more complex and nuanced than the outcomes of laboratory or model-based experiments. The real world is richer and more credible, and thus outcomes are more convincing and harder to dismiss. During the planning process, transportation decision makers commonly look for concrete examples and are influenced more by experiences and observations than by forecasts and reports.
People often look to other places for guidance, asking questions like, “Why can’t we have high-speed rail like they have in Europe?” Planners and decision makers should certainly learn from what others have experienced, but they should also exercise caution because results of field experiments are not always transferable. What works in Paris may not work in Los Angeles. If we look only for what we want to find, the risk of confirmation bias can outweigh the benefits of learning from others’ field experiments.
Natural Experiments
Natural experiments often arise in the form of uncontrollable events, such as hurricanes, floods, storm surges, heavy snowfalls, and drought. These extreme weather events probe the vulnerabilities of infrastructure and service systems and test our defense and response mechanisms. Mother Nature has ways of finding out where we are weak by conducting stress tests we would never voluntarily undertake. Nature’s experiments sometimes uncover vulnerabilities that might lead to serious problems in the future if we experience significant rise in sea level, more frequent and severe flooding, prolonged droughts, or heat waves. Recent unexpected outcomes have included:
- Commuter rail lines washed out when a privately-owned dam failed in hurricane-driven rains (Metro-North Port Jervis Line, 2011)
- Reduced pavement life resulted from prolonged submersion in flood waters (Nebraska, Missouri, and Platte Rivers, 2011)
- Train derailments and infrastructure damaged by an extended heat wave (Northbrook, Illinois, 2012)
The lessons learned from these natural experiments can guide preparations for the future, inform and motivate adaptation strategies, and reduce both risks and costs in the long run. Importantly, they may build support for actions that address extreme and uncertain natural events.
Creating a Culture of Experimentation
We can learn more about the vulnerability of our transportation systems and what projects and policies work by establishing a culture of experimentation that both initiates field experiments and capitalizes on selected natural learning opportunities. In particular, there is a special incentive to turn major natural disasters into learning experiments that help us prepare for future extremes.
Learning from Nature
Opportunistic learning from nature is a particularly important way to address the potential consequences of climate change and natural disasters such as earthquakes and tsunamis. For example, after Hurricane Katrina devastated the New Orleans area, several studies were conducted to understand the vulnerabilities in southern Louisiana. At-risk organizations elsewhere also learned from this natural experiment. The New York Metropolitan Transportation Authority (MTA) developed its own hurricane-storm surge response policies, adding lessons from Katrina to its experience with the heavy snowstorm that stranded buses and trains in New York in December 2010. These natural experiments informed MTA’s policies that called for shutting down mass transit services in the face of Hurricane Irene in 2011 and the much stronger Hurricane Sandy in 2012, saving countless lives and dollars.
We do not choose the time and place for such events, but we can learn from them. We must have mechanisms in place both to capture data and to provide flexibility to direct resources to analyses and interpretations. Agencies affected by extreme weather events sometimes conduct post mortem evaluations to prepare for future events—exactly what happened in New Orleans after Katrina and New York after Sandy. While these have high value for local use, national learning may call for more broadly focused and systematic assessments. Figure 1 presents a checklist that could be a starting point for planning and learning from experiments in anticipation of extreme weather events. The most important steps will assure that the necessary data are collected and preserved to support maximum post hoc learning.
Turning Events into Experiments
The national interest in experimenting grows when projects have the potential to guide major decisions. Investments in high-speed rail (HSR) are an important example.
HSR proposals have generated both broad interest and substantial controversy. Uncertainties associated with HSR investments include costs, performance, market response, and finance mechanisms. California’s program may offer the best opportunity to gather critical, US-based evidence to inform future HSR choices. Because HSR projects are planned interventions, they can be structured as experiments from the outset to increase their value as learning tools.
To view California’s HSR program as an experiment, it makes sense to focus federal dollars in one place to conduct a meaningful test, rather than follow the political tradition of spreading federal funding over multiple small or partial projects across the nation. Furthermore, HSR projects are typically built in stages, which can expedite the first opportunities for learning from them. The resulting knowledge benefit will be greater, and will come sooner, if the initial stage can target major topics of uncertainty, such as market response and financing. If the HSR system is built in corridors where there is substantial demand potential, we have the opportunity to learn about the market response, the resulting revenue stream, and the interests in investing future public and private funds in HSR. This strategy would generate lessons more quickly and contribute to the reduction of long-term national investment risk.
Sharing the Data
Collecting, archiving, maintaining, and sharing data form the foundation for experimentalizing projects and events. Cities archive data to assess long-term outcomes, such as shifts in land use, environmental consequences, and individual and organizational travel behaviors. Archived data also helps to answer questions that may arise in the future. The volume and scope of data now collected are rapidly expanding because of innovations in Intelligent Transportation Systems and social media. The increasing breadth of available data is balanced by the steady decline in storage costs.
Data sharing opens the door for many researchers to analyze experiments and, through replication, to test the findings of others. Such replication can provide more robust, and sometimes different, interpretations of experiments. Data sharing can be contentious because of privacy concerns as well as proprietary interests in publication primacy. These issues have become prominent in behavioral research as the ability to collect highly personalized, microscale data has exploded.
When there is broader interest in the outcome of an experiment, it may be appropriate to assemble resources at the national scale to invest in extra data collection, analysis, and interpretation. Support for experiments of broader significance might come from federal or pooled state funds. For example, the American Association of State Highway and Transportation Officials sometimes provides funds to address problems of interest to multiple states. Providing national support for experimentalizing projects requires not only a source of funding, but also an objective mechanism for selecting those projects.
Conclusion
Academic programs and professional organizations may be best positioned to conduct experiments because they are already equipped to evaluate them. They also are able to define standards and templates for data collection, establish principles for data stewardship and sharing, develop practical methods for experimental design, conduct meta-analyses of findings from multiple studies, and promote peer review to enhance the quality of experimental studies.
Federal and local institutions should work to support an experimental culture by promoting the value of learning from experience. Local learning, in particular, brings high value to local transportation decision making because it reduces the “not invented here” reaction—“owned” results are harder to ignore. Government investment in experimentation should include not only incremental funding within established action programs, but also a policy commitment to planning experiments, collecting, archiving and sharing data, and using the results in decision processes.
Creating an experimental culture will help us learn more, faster, through systematic studies of changes from within and threats from without. Ultimately, this will help to create a better transportation future for everyone.
This article is adapted from the Melvin Webber Lecture given at UC Davis in 2012. Special thanks to William L. Garrison, University of California, Berkeley, whose ideas about experimentation helped motivate this work.
Further Readings
Nancy Cartwright and Jeremy Hardie. 2012. Evidence-Based Policy: A Practical Guide to Doing It Better, Oxford, UK: Oxford University Press.
William T. Gormley Jr. 2011. “From Science to Policy in Early Childhood Education,” Science, 333: 978–981.
Joseph L. Schofer, Tim Lomax, Thomas Palmerlee, and Johanna Zmud. 2006. “Transportation Information Assets and Impacts: An Assessment of Needs,” Transportation Research Circular E-C 109, Transportation Research Board.
United Nations and the World Bank. 2010. Natural Hazards, UnNatural Disasters: The Economics of Effective Prevention, Washington, DC.
