[gov3009-l] GOV 3009 (Applied Stats Workshop), 5/1 -- Kosuke Imai

Dear Applied Statistics Workshop Community, Our last meeting of this semester will be on Wednesday, May 1 (12:00 EST). Kosuke Imai presents "Does AI help humans make better decisions? A methodological framework for experimental evaluation" (joint work with Eli Ben-Michael, D. James Greiner, Melody Huang, Zhichao Jiang, Sooahn Shin). <When> May 1, 12:00 to 1:30 PM, EST Lunch will be available for pick-up inside CGIS K354. <Where> In-person: CGIS K354 Zoom: https://harvard.zoom.us/j/93217566507?pwd=elBwYjRJcWhlVE5teE1VNDZoUXdjQT09 <Abstract> The use of Artificial Intelligence (AI) based on data-driven algorithms has become ubiquitous in today's society. Yet, in many cases and especially when stakes are high, humans still make final decisions. The critical question, therefore, is whether AI helps humans make better decisions as compared to a human-alone or AI-alone system. We introduce a new methodological framework that can be used to answer experimentally this question with no additional assumptions. We measure a decision maker's ability to make correct decisions using standard classification metrics based on the baseline potential outcome. We consider a single-blinded experimental design, in which the provision of AI-generated recommendations is randomized across cases with a human making final decisions. Under this experimental design, we show how to compare the performance of three alternative decision-making systems--human-alone, human-with-AI, and AI-alone. We apply the proposed methodology to the data from our own randomized controlled trial of a pretrial risk assessment instrument. We find that AI recommendations do not improve the classification accuracy of a judge's decision to impose cash bail. Our analysis also shows that AI-alone decisions generally perform worse than human decisions with or without AI assistance. Finally, AI recommendations tend to impose cash bail on non-white arrestees more often than necessary when compared to white arrestees. The paper is available on arXiv: https://arxiv.org/pdf/2403.12108.pdf <2023-2024 Schedule> GOV 3009 Website: https://projects.iq.harvard.edu/applied.stats.workshop-gov3009 Calendar: https://calendar.google.com/calendar/u/0?cid=Y18zdjkzcGF2OWZqa2tsZHJidTlzbm… Best, Jialu -- Jialu Li Department of Government Harvard University *https://jialul.github.io/ <https://jialul.github.io/>*