Assessing Interchangeability among Raters with Continuous Outcomes in Agreement Studies

Abstract

In various medical settings, new raters are available to take measurements for evaluations of medical conditions. One may want to use new and existing raters simultaneously or replace the existing raters with the new one due to low-cost or portability. For both situations, the raters should be interchangeable, such that it makes no clinical difference in which raters measure the subject in the population of interest. This is a problem of claiming sufficient agreement among the raters. Existing literature on assessing agreement is limited to two raters, and those for more than two raters have various issues of interpretability or unrealistic assumptions. This dissertation proposes new overall agreement indexes for multiple raters by extending the preferred pairwise agreement indexes, coverage probability, total deviation index, and the relative area under the coverage probability. The new indexes have intuitive interpretations regarding the clinical judgment of interchangeability. A unified generalized estimating equation (GEE) approach is developed for inference. Simulation studies are conducted to assess the performance and theoretical properties of the proposed approach. A blood pressure dataset is used for illustration. Due to limited literature on sample size calculation in the agreement study, this dissertation investigates sample size formulas with pre-specified power if one of the proposed overall agreement indices is used for claiming satisfactory interchangeability. While the sample size formulas based on the inference of the GEE framework is somewhat complicated, simplified formulas giving conservative sample size estimation is also proposed for easy implementation. Our simulation studies indicated that the sample size formulas work well if the resulting number of subjects is at least 30 where each rater takes about 3 replicates. We demonstrate how to design an agreement study based on a pilot blood pressure data set. The U.S. Food and Drug Administration recommends using a regression-based approach in case of replacing a new device with a commercially marketed device. In the third part, we discuss the potential pitfalls of this approach and compare it with the coverage probability approach, the currently preferred approach for assessing agreement. A respiratory rate dataset is used to illustrate the issues of the regression-based approach.