Plan quality evaluation researchers typically evaluate plans in relation to whether they contain certain desirable features. Best practice dictates that plans be evaluated by at least two readers and that researchers report a measure of the extent to which the readers agree on whether the plans contain the desirable features. Established practice for assessing this agreement has been subject to criticism. We summarize this criticism, discuss an alternative approach to assessing agreement, and provide recommendations for plan quality evaluation researchers to follow to improve the quality of their data and the manner in which they assess and report that quality.

State mitigation plans play a critical role in supporting disaster loss reduction and long-term resiliency of human communities. The Disaster Mitigation Act of 2000 requires all states to prepare mitigation plans. Based on six principles of plan quality, we content analyzed a sample of 30 coastal state plans to determine how well they support mitigation. Findings indicate that although plans scored moderate to low for all plan quality principles, plan quality has modestly improved over the past decade. In addition, some states scored low for one principle, which can undermine implementation of otherwise high-scoring plans for the remaining principles. Recommendations are offered on how plan quality evaluation can be used to guide and monitor state development of hazard mitigation plans.

Attention to if and how plans are implemented is increasing. Among the conceptions of plan implementation used by scholars, plan conformance and plan performance are most prominent, though rarely have both been assessed in a single study. We assess plan conformance and a modified version of performance, which we call influence, for a sample of more than 100 local hazard mitigation plans developed in response to a national planning requirement in the United States. Our findings indicate that progress on implementation of the policies included in the plans (plan conformance) is moderate, but that the rate of progress varies widely by the type of policy and the state in which the local jurisdiction is located. The findings also indicate that coordination of hazard mitigation plans (plan influence) is strongest with other emergency management agency-led planning initiatives and weaker with planning initiatives led by other agencies. National officials interested in advancing local implementation of national goals need to consider the degree to which state and local governments are given autonomy to select and prioritize local policies included in plans, as well as constraints on coordination across planning initiatives.

No abstract

During the last twenty years, more than forty-five publications have sought to measure and evaluate the quality of plans using content analysis methods. We examine reasons for this growth in the literature and its contributions and limitations. We also examine whether the research methods described in these publications conform to recommended practices in the methodological literature on content analysis to determine whether plan quality researchers are likely to be generating reliable and reproducible plan quality data. We provide seven recommendations plan quality researchers can follow to address these weaknesses and improve the reliability and reproducibility of their data.