How Well Do Structural Demand Models Work? Counterfactual Predictions in School Choice

Abstract

Discrete choice demand models are widely used for counterfactual policy simulations, yet their out-of-sample performance is rarely assessed. This paper uses a large-scale policy change in Boston to investigate the performance of discrete choice models of school demand. In 2013, Boston Public Schools considered several new choice plans that differ in where applicants can apply. At the request of the mayor and district, we forecast the alternatives' effects by estimating discrete choice models. This work led to the adoption of a plan which significantly altered choice sets for thousands of applicants. Pathak and Shi (2014) update forecasts prior to the policy change and describe prediction targets involving access, travel, and unassigned students. Here, we assess how well these ex ante counterfactual predictions compare to actual outcome under the new choice sets. We find that a simple ad hoc model performs as well as the more complicated structural choice models for one of the two grades we examine. However, the structural models' inconsistent performance is largely due to prediction errors in applicant characteristics, which are auxiliary inputs. Once we condition on the actual applicant characteristics, the structural choice models outperform the ad hoc alternative in predicting both choice patterns and policy relevant outcomes. Moreover, refitting the models using the new choice data does not significantly improve their prediction accuracy, suggesting that the choice models are indeed "structural." Our findings show that structural demand models can effectively predict counterfactual outcomes, as long there are accurate forecasts about auxiliary input variables.

Immigration Lottery Design: Engineered and Coincidental Consequences of H-1B Reforms

“…For example, Walk-Open with One Lottery is similar to Exemptions-First and Walk-Open with Two Lotteries policy is similar to the Reserve-Initiated. Boston ultimately eliminated the reserve system, motivated partially by a concern that these issues were too complex and partially by potential interactions with other contemporaneous reforms (Shi, 2014;Pathak and Shi, 2017). It is interesting that the USCIS independently considered and deployed closely related solutions to their similar problems and faced a similar period of uncertainty regarding these policies functioning.…”

School Choice with Asymmetric Information: Priority Design and the Curse of Acceptance

“…Indeed, school districts often do have some control over the design of their priority structures and some have attempted to (re-)design their priorities to achieve particular objectives. For example, Dur et al (2018) and Pathak and Shi (2017) explore consequences of different design decisions relating to the development of walk zones in Boston, a city which has taken an active role in the design of its priorities. Roughly, schools are divided into quality tiers, and a student's menu consists of the two closest tier 1 schools, the four closest tier 2 schools, etc.…”

On rank dominance of tie‐breaking rules

“…33 Such preferences were also used in Kojima and Pathak (2009) and also referred to as popularity‐based preferences, for example, in Gimbert, Mathieu, and Mauras (2019). Moreover MNL preferences are often used to estimate demand in school choice problems (e.g., Shi (2015), Agarwal and Somaini (2018), Pathak and Shi (2013)). …”