Designing for Digital Experimentation

Digital experiments and randomized control trials are crucial for understanding the potential effectiveness and implications of decisions and policies. Experiment design research explores statistical and machine learning strategies and methods for sampling, effect sizes, and parameter estimation. More recently, adapting such strategies to digital experimentation settings has become important for understanding heterogenous treatment effects, bias in estimation, as well as statistical and practical significance.

• Abbasi, A., Somanchi, S., & Kelley, K., (2024). The Critical Challenge of Using Large-scale Digital Experiment Platforms for Scientific Discovery. MIS Quarterly, forthcoming.

• Somanchi, S., Abbasi, A., & Kelley, K., Dobolyi, D., & Yuan, T. T. (2023). Examining User Heterogeneity in Digital Experiments. ACM Transactions on Information Systems, 41(4), 1-34.

• Anderson, S. F., & Kelley, K. (2022). Sample Size Planning for Replication Studies: The Devil is in the Design. Psychological Methods, forthcoming.

• Somanchi, S., Abbasi, A., Dobolyi, D., Kelley, K., & Yuan, T. T. (2021). User and Session Heterogeneity in Digital Experiments: A Framework for Analysis and Understanding. MIT Conference on Digital Experimentation, November 4-5, 2021

• Kelley, K., Darku, F. B., & Chattopadhyay, B. (2019). Sequential Accuracy in Parameter Estimation for Population Correlation Coefficients. Psychological Methods, 24(4), 492-515

• Kelley, K., Darku, F. B., & Chattopadhyay, B. (2018). Accuracy in Parameter Estimation for a General Class of Effect Sizes: A Sequential Approach. Psychological Methods, 23(2), 226-243

• McFowland III, E.Somanchi, S., & Neill, D. B. (2018). Efficient Discovery of Heterogeneous Treatment Effects in Randomized Experiments Via Anomalous Pattern Detection. arXiv, forthcoming, https://arxiv.org/pdf/1803.09159.pdf

• Kelley, K., & Preacher, K. J. (2012). On Effect Size. Psychological Methods, 17(2), 137-152

• Preacher, K. J. & Kelley, K., (2011). Effect Size Measures for Mediation Models: Quantitative Strategies for Communicating Indirect Effects. Psychological Methods, 16(2), 93-115