Data vs. Methods: Quasi-Experimental Evaluation of Alternative Sample Selection Corrections for Missing College Entrance Exam Score Data

Abstract

In 2007, Michigan began requiring all high school students to take the ACT college entrance exam. This natural experiment allows us to evaluate the performance of several parametric and semiparametric sample selection correction models. We apply each model to the censored, prepolicy test score data and compare the predicted values to the uncensored, post-policy distribution. We vary the set of model predictors to imitate the varying levels of data detail to which a researcher may have access. We find that predictive performance is sensitive to predictor choice but not correction model choice. All models perform poorly using student demographics and school- and district-level characteristics as predictors. However, all models perform well when including students’ prior and contemporaneous scores on other tests. Similarly, correction models using group-level data perform better with more finely disaggregated groups, but produce similar predictions under different functional form assumptions. Our findings are not explained by an absence of selection, the assumptions of the parametric models holding, or the data lacking sufficient variation to permit useful semiparametric estimation. We conclude that “data beat methods” in this setting: gains from using less restrictive econometric methods are small relative to gains from seeking richer or more disaggregated data.