Dear Applied Statistics Workshop Community,
Just a reminder, our last meeting of this semester will be on Wednesday,
May 1 (12:00 EST). Kosuke Imai presents "Does AI help humans make better
decisions? A methodological framework for experimental evaluation" (joint
work with Eli Ben-Michael, D. James Greiner, Melody Huang, Zhichao Jiang,
Sooahn Shin).
<When>
May 1, 12:00 to 1:30 PM, EST
Lunch will be available for pick-up inside CGIS K354.
<Where>
In-person: CGIS K354
Zoom:
https://harvard.zoom.us/j/93217566507?pwd=elBwYjRJcWhlVE5teE1VNDZoUXdjQT09
<Abstract>
The use of Artificial Intelligence (AI) based on data-driven algorithms has
become ubiquitous in today's society. Yet, in many cases and especially
when stakes are high, humans still make final decisions. The critical
question, therefore, is whether AI helps humans make better decisions as
compared to a human-alone or AI-alone system. We introduce a new
methodological framework that can be used to answer experimentally this
question with no additional assumptions. We measure a decision maker's
ability to make correct decisions using standard classification metrics
based on the baseline potential outcome. We consider a single-blinded
experimental design, in which the provision of AI-generated recommendations
is randomized across cases with a human making final decisions. Under this
experimental design, we show how to compare the performance of three
alternative decision-making systems--human-alone, human-with-AI, and
AI-alone. We apply the proposed methodology to the data from our own
randomized controlled trial of a pretrial risk assessment instrument. We
find that AI recommendations do not improve the classification accuracy of
a judge's decision to impose cash bail. Our analysis also shows that
AI-alone decisions generally perform worse than human decisions with or
without AI assistance. Finally, AI recommendations tend to impose cash bail
on non-white arrestees more often than necessary when compared to white
arrestees.
The paper is available on arXiv: https://arxiv.org/pdf/2403.12108.pdf
<2023-2024 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/u/0?cid=Y18zdjkzcGF2OWZqa2tsZHJidTlzbm…
Best,
Jialu
--
Jialu Li
Department of Government
Harvard University
*https://jialul.github.io/ <https://jialul.github.io/>*
Dear Applied Statistics Workshop Community,
Our last meeting of this semester will be on Wednesday, May 1 (12:00 EST).
Kosuke Imai presents "Does AI help humans make better decisions? A
methodological framework for experimental evaluation" (joint work with Eli
Ben-Michael, D. James Greiner, Melody Huang, Zhichao Jiang, Sooahn Shin).
<When>
May 1, 12:00 to 1:30 PM, EST
Lunch will be available for pick-up inside CGIS K354.
<Where>
In-person: CGIS K354
Zoom:
https://harvard.zoom.us/j/93217566507?pwd=elBwYjRJcWhlVE5teE1VNDZoUXdjQT09
<Abstract>
The use of Artificial Intelligence (AI) based on data-driven algorithms has
become ubiquitous in today's society. Yet, in many cases and especially
when stakes are high, humans still make final decisions. The critical
question, therefore, is whether AI helps humans make better decisions as
compared to a human-alone or AI-alone system. We introduce a new
methodological framework that can be used to answer experimentally this
question with no additional assumptions. We measure a decision maker's
ability to make correct decisions using standard classification metrics
based on the baseline potential outcome. We consider a single-blinded
experimental design, in which the provision of AI-generated recommendations
is randomized across cases with a human making final decisions. Under this
experimental design, we show how to compare the performance of three
alternative decision-making systems--human-alone, human-with-AI, and
AI-alone. We apply the proposed methodology to the data from our own
randomized controlled trial of a pretrial risk assessment instrument. We
find that AI recommendations do not improve the classification accuracy of
a judge's decision to impose cash bail. Our analysis also shows that
AI-alone decisions generally perform worse than human decisions with or
without AI assistance. Finally, AI recommendations tend to impose cash bail
on non-white arrestees more often than necessary when compared to white
arrestees.
The paper is available on arXiv: https://arxiv.org/pdf/2403.12108.pdf
<2023-2024 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/u/0?cid=Y18zdjkzcGF2OWZqa2tsZHJidTlzbm…
Best,
Jialu
--
Jialu Li
Department of Government
Harvard University
*https://jialul.github.io/ <https://jialul.github.io/>*
Dear Applied Statistics Workshop Community,
Our next meeting will be on Wednesday, April 24 (12:00 EST). Siyu Heng
presents "Design-Based Causal Inference with Missing Outcomes: Missingness
Mechanisms, Imputation-Assisted Randomization Tests, and Covariate
Adjustment."
<When>
April 24, 12:00 to 1:30 PM, EST
Lunch will be available for pick-up inside CGIS K354.
<Where>
In-person: CGIS K354
Zoom:
https://harvard.zoom.us/j/93217566507?pwd=elBwYjRJcWhlVE5teE1VNDZoUXdjQT09
<Abstract>
Design-based causal inference, also known as randomization-based or
finite-population causal inference, is one of the most widely used causal
inference frameworks, largely due to the merit that its statistical
validity can be guaranteed by the study design (e.g., randomized
experiments) and does not require assuming specific outcome-generating
distributions or super-population models. Despite its advantages,
design-based causal inference can still suffer from other data-related
issues, among which outcome missingness is a prevalent and significant
challenge. This work systematically studies the outcome missingness problem
in design-based causal inference. First, we propose a general and flexible
outcome missingness mechanism that can facilitate finite-population-exact
randomization tests for the null effect. Second, under this flexible
missingness mechanism, we propose a general framework called "imputation
and re-imputation" for conducting finite-population-exact randomization
tests in design-based causal inference with missing outcomes. This
framework can incorporate any imputation algorithms (from linear models to
advanced machine learning-based imputation algorithms) while ensuring
finite-population-exact type-I error rate control. Third, we extend our
framework to conduct covariate adjustment in randomization tests and
construct finite-population-valid confidence sets with missing outcomes.
Our framework is evaluated via extensive simulation studies and applied to
a cluster randomized experiment called the Work, Family, and Health Study.
Open-source Python and R packages "iArt" (*i*mputation-*A*ssisted *r*
andomization *t*est) are developed for implementation of our framework.
This talk is based on joint work with Yang Feng and Jiawei Zhang. The
working paper is available on arXiv: https://arxiv.org/abs/2310.18556
<2023-2024 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/u/0?cid=Y18zdjkzcGF2OWZqa2tsZHJidTlzbm…
Best,
Jialu
--
Jialu Li
Department of Government
Harvard University
*https://jialul.github.io/ <https://jialul.github.io/>*
Dear Applied Statistics Workshop Community,
Just a quick reminder, our next meeting is Wednesday, April 17 (12:00 EST).
Connor Jerzak presents "Selecting Optimal Candidate Profiles in Adversarial
Environments Using Conjoint Analysis" (Joint with Kosuke Imai).
<When>
April 17, 12:00 to 1:30 PM, EST
Lunch will be available for pick-up inside CGIS K354.
<Where>
In-person: CGIS K354
Zoom:
https://harvard.zoom.us/j/93217566507?pwd=elBwYjRJcWhlVE5teE1VNDZoUXdjQT09
<Abstract>
Conjoint analysis, an application of factorial experimental design, is a
popular tool in social science research for studying multidimensional
preferences. In such experiments in the political analysis context,
respondents are asked to choose between two hypothetical political
candidates with randomly selected features, which can include partisanship,
policy positions, gender and race. We consider the problem of identifying
optimal candidate profiles. Because the number of unique feature
combinations far exceeds the total number of observations in a typical
conjoint experiment, it is impossible to determine the optimal profile
exactly. To address this identification challenge, we derive an optimal
stochastic intervention that represents a probability distribution of
various attributes aimed at achieving the most favorable average outcome.
We first consider an environment where one political party optimizes their
candidate selection. We then move to the more realistic case where two
political parties optimize their own candidate selection simultaneously and
in opposition to each other. We apply the proposed methodology to an
existing candidate choice conjoint experiment concerning vote choice for US
president. We find that, in contrast to the non-adversarial approach,
expected outcomes in the adversarial regime fall within range of historical
electoral outcomes, with optimal strategies suggested by the method more
likely to match the actual observed candidates compared to strategies
derived from a non-adversarial approach. These findings indicate that
incorporating adversarial dynamics into conjoint analysis may yield unique
insight into social science data from experiments.
<2023-2024 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/u/0?cid=Y18zdjkzcGF2OWZqa2tsZHJidTlzbm…
Best,
Jialu
--
Jialu Li
Department of Government
Harvard University
*https://jialul.github.io/ <https://jialul.github.io/>*
Dear Applied Statistics Workshop Community,
Just a quick reminder, our next meeting is Wednesday, April 10 (12:00 EST).
Melissa Dell presents “Efficient OCR for Building a Diverse Digital
History.”
<When>
April 10, 12:00 to 1:30 PM, EST
Lunch will be available for pick-up inside CGIS K354.
<Where>
In-person: CGIS K354
Zoom:
https://harvard.zoom.us/j/93217566507?pwd=elBwYjRJcWhlVE5teE1VNDZoUXdjQT09
<Abstract>
Thousands of users consult digital archives daily, but the information they
can access is unrepresentative of the diversity of documentary history. The
sequence-to-sequence architecture typically used for optical character
recognition (OCR) – which jointly learns a vision and language model - is
poorly extensible to low-resource document collections, as learning a
language-vision model requires extensive labeled sequences and compute.
This study models OCR as a character level image retrieval problem, using a
contrastively trained vision encoder. Because the model only learns
characters’ visual features, it is more sample efficient and extensible
than existing architectures, enabling accurate OCR in settings where
existing solutions fail. Crucially, the model opens new avenues for
community engagement in making digital history more representative of
documentary history. Beyond OCR, the presentation will also discuss how
large differences in sample efficiency across different neural network
architectures influence the types of learning that are often most suited
towards academic applications, particular for low resource settings.
<2023-2024 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/u/0?cid=Y18zdjkzcGF2OWZqa2tsZHJidTlzbm…
Best,
Jialu
--
Jialu Li
Department of Government
Harvard University
*https://jialul.github.io/ <https://jialul.github.io/>*
