Design, Measurement, & Analysis

Digital experiments and randomized control trials are crucial for understanding the potential effectiveness and implications of decisions and policies. Experiment design research explores statistical and machine learning strategies and methods for sampling, effect sizes, and parameter estimation. More recently, adapting such strategies to digital experimentation settings has become important for understanding heterogenous treatment effects, bias in estimation, as well as statistical and practical significance.

• Maxwell, S. E., Delandey, H. D., Kelley, K., (2027). Designing Experiments and Analyzing Data: A Model Comparison Perspective (4th edition). Routledge.

• Stamey, W., Kelley, K., Chattopadhyay, B., & Bandyopadhyay, T. (2026). Fast Selection from Multiple Treatments: A Sequential Method for Principled Digital Experimentation. Information Systems Research, forthcoming

• Kelley, K. & Yu, X. (2026). Power Enhancement in High-Dimensional
  Heterogeneous Mediation Analysis, Journal of the American Statistical Association, forthcoming.

• Kelley, K. (2026). DMAR: Design, Measurement, and Analysis in R. Comprehensive R Archival Network.

• Abbasi, A., Somanchi, S., & Kelley, K., (2025). The Critical Challenge of Using Large-scale Digital Experiment Platforms for Scientific Discovery. MIS Quarterly 1 March 2025; 49 (1): 1–28.

• Somanchi, S., Abbasi, A., & Kelley, K., Dobolyi, D., & Yuan, T. T. (2023). Examining User Heterogeneity in Digital Experiments. ACM Transactions on Information Systems, 41(4), 1-34.

• Anderson, S. F., & Kelley, K. (2022). Sample Size Planning for Replication Studies: The Devil is in the Design. Psychological Methods, 2024 Oct; 29(5), 844-867.

• Somanchi, S., Abbasi, A., Dobolyi, D., Kelley, K., & Yuan, T. T. (2021). User and Session Heterogeneity in Digital Experiments: A Framework for Analysis and Understanding. MIT Conference on Digital Experimentation, November 4-5, 2021

• Kelley, K., Darku, F. B., & Chattopadhyay, B. (2019). Sequential Accuracy in Parameter Estimation for Population Correlation Coefficients. Psychological Methods, 24(4), 492-515

• Kelley, K., Darku, F. B., & Chattopadhyay, B. (2018). Accuracy in Parameter Estimation for a General Class of Effect Sizes: A Sequential Approach. Psychological Methods, 23(2), 226-243

• McFowland III, E.Somanchi, S., & Neill, D. B. (2018). Efficient Discovery of Heterogeneous Treatment Effects in Randomized Experiments Via Anomalous Pattern Detection. arXiv, forthcoming, https://arxiv.org/pdf/1803.09159.pdf

• Kelley, K., & Preacher, K. J. (2012). On Effect Size. Psychological Methods, 17(2), 137-152

• Preacher, K. J. & Kelley, K., (2011). Effect Size Measures for Mediation Models: Quantitative Strategies for Communicating Indirect Effects. Psychological Methods, 16(2), 93-115