Testing the Validity of a Demand Model: An Operations Perspective

Abstract

The fields of statistics and econometrics have developed powerful methods for testing the validity (specification) of a model based on its fit to underlying data. Unlike statisticians, managers are typically more interested in the performance of a decision rather than the statistical validity of the underlying model. We propose a framework and a statistical test that incorporate decision performance into a measure of statistical validity. Under general conditions on the objective function, asymptotic behavior of our test admits a sharp and simple characterization. We develop our approach in a revenue management setting and apply the test to a data set used to optimize prices for consumer loans. We show that traditional model-based goodness-of-fit tests may consistently reject simple parametric models of consumer response (e.g., the ubiquitous logit model), while at the same time these models may "pass" the proposed performance-based test. Such situations arise when decisions derived from a postulated (and possibly incorrect) model generate results that cannot be distinguished statistically from the best achievable performance-i.e., when demand relationships are fully known. We develop our approach in a revenue management setting and apply the test to a data set used to optimize prices for consumer loans. We show that traditional model-based goodness-of-fit tests may consistently reject simple parametric models of consumer response (e.g., the ubiquitous logit model), while at the same time these models may "pass" the proposed performance-based test.Such situations arise when decisions derived from a postulated (and possibly incorrect) model, generate results that cannot be distinguished statistically from the best achievable performance -i.e., when demand relationships are fully known.

Markdown Pricing with Unknown Fraction of Strategic Customers

“…A broader take-away from our work is that while studying RM with full information can lead to valuable insights, ultimately RM optimization should not be considered independently of estimation issues. We are encouraged by the recent work of Besbes et al (2010), who propose estimation procedures optimized for a specific pricing optimization problem, and of Levin et al (2009b), who actively learn an aggregate demand process that allows for multiple sources of underlying uncertainty. We also applaud recent streams of research into robust decision making and active learning in RM.…”

Learning and Pricing with Models That Do Not Explicitly Incorporate Competition

“…Cooper and Li (2012) study a setup similar to Cooper et al (2006), but consider protection levels generated by a "buy-up" model. Besbes et al (2010) consider a single seller who wants to choose a price to maximize expected revenue, but who does not know the true demand function. The seller restricts attention to a parameterized family of demand functions, calculates a parameter value that gives the best fit to observed data, and then chooses a price according to certainty equivalent control.…”

Monotone Approximation of Decision Problems

“…The criteria used to measure the approximation error (such as mean square error or maximum absolute error) are typically disconnected from the underlying decision problem. Besbes et al (2010), motivated in part by this deficiency, develop a hypothesis test for the significance of a parameterization based on sample data of objective-function values when comparing the parameter-induced optimum to a nonparametric baseline estimate of the objective function. In sample-sparse environments, however, parametric methods are of limited use only, because the data does not induce any reasonable assumptions on the model class.…”

“…Second, the approach combines the problems of model estimation and optimization by evaluating the approximation error using any user-defined robustness criterion, such as average performance, worst-case performance, expected gain, or competitive ratio. The approach is therefore in the spirit of recent advances in operational statistics (Shanthikumar and Liyanage 2005, Lim et al 2006, and Besbes et al 2010, which emphasize the importance of using a metric for the evaluation of an approximation error that can be directly related to the cost of mistakes in the underlying decision. An optimal robust decision is obtained together with an optimal approximation error by solving a pair of nested optimization problems.…”