Assessing Decision-Making Confidence in the Presence of Data Uncertainty

Abstract

Assumptions are made liberally in statistical analyses, but violations to them are rarely tested. When they are, little attention is paid to whether or not the downstream decision is changed and by how much. We show that once an analyst specifies a notion of making an actionable decision based on what they observe from their modeling pipeline, there is a duality with having confidence in that decision given the assumptions made in said modeling pipeline. In particular, we illustrate how analysts can specify a "confidence in decision'' (CID) metric by which they can assess how violations to an assumption can result in loss in correctness, and therefore in confidence. We show that a variety of these CID metrics are possible, and propose a visualization framework that allows data analysts to cohesively and easily test how deviations from modeling assumptions could impact not just estimates of particular quantities, but also how they may or may not result in lost confidence in decision making, depending on what the analyst prioritizes. Two assumptions of interest that are commonly violated, through which we illustrate the CID procedure, are measurement error and missing not at random data. Two common settings in which decisions need to be made that are illustrated are political polling and public health interventions.