Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on April 19 (12:00 EST). Michela
Carlana will present "Revealing Stereotypes: Evidence from Immigrants in
Schools."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:45 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
We study how people change their behavior after learning they are biased.
Teachers in Italian schools give lower grades to immigrant students
relative to natives with comparable ability. In two experiments, we reveal
to teachers their own bias, measured by an Implicit Association Test (IAT).
Randomizing the timing of disclosure, we find that learning one’s IAT
before deciding end-of-term grades reduces the native-immigrant gap in
grades. IAT disclosure and generic debiasing have similar average effects,
but there is heterogeneity: teachers with more negative stereotypes do not
respond to generic debiasing but change their behavior when informed about
their own IAT.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on April 12 (12:00 EST). Naoki
Egami will present "Empirical Strategies Toward External Validity:
Framework and External Robustness."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:45 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
Over the last few decades, social scientists have developed and applied a
host of statistical methods to make valid causal inferences, known as the
credibility revolution. This trend has primarily focused on internal
validity — researchers aim to unbiasedly estimate causal effects within a
study. However, one of the most important long-standing methodological
debates is about external validity — how scientists can generalize causal
findings beyond a specific study. This question of external validity has a
long history in the social sciences, going back to at least the 1960s, and
it has recently become even more essential, given that huge opportunities
and challenges of accumulating causal knowledge have become evident.
In this talk, I will discuss a set of empirical strategies to improve
external validity in practice. I briefly introduce a formal framework of
external validity (Egami and Hartman, 2022; APSR) that synthesizes diverse
external validity concerns. Then, I will propose a new simple approach to
quantify the robustness of experimental results to external validity bias
(Devaux and Egami, 2022; Egami and Rothenhäusler, 2023+). In particular, I
introduce a measure of external robustness, which ranges from 0 to 1 and
represents how well causal effects estimated in one’s study can be
generalized to other populations and contexts. Researchers can estimate
this quantity using only experimental data (i.e., no additional data
collection), and users can also account for unmeasured confounders. I
discuss a debiased estimator, which is consistent and asymptotically normal
under mild rate conditions that allow for the use of machine learning
estimators. Finally, I provide default benchmarks and discuss practical
guides about how to report external robustness in practice using the R
package “exr” (https://github.com/naoki-egami/exr).
Papers: (1) https://naokiegami.com/paper/external_full.pdf (2)
https://naokiegami.com/paper/external_robust.pdf
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on April 5 (12:00 EST). Fredrik
Sävje will present "A Design-Based Riesz Representation Framework for
Randomized Experiments."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:45 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
We describe a new design-based framework for drawing causal inference in
randomized experiments. Causal effects in the framework are defined as
linear functionals evaluated at potential outcome functions. Knowledge and
assumptions about the potential outcome functions are encoded as function
spaces. This makes the framework expressive, allowing experimenters to
formulate and investigate a wide range of causal questions. We describe a
class of estimators for estimands defined using the framework and
investigate their properties. The construction of the estimators is based
on the Riesz representation theorem. We provide necessary and sufficient
conditions for unbiasedness and consistency. Finally, we provide conditions
under which the estimators are asymptotically normal, and describe a
conservative variance estimator to facilitate the construction of
confidence intervals for the estimands.
Paper: https://arxiv.org/abs/2210.08698
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on March 29 (12:00 EST). James M.
Robins will present "Target Trials and Structural Nested Models: Emulating
RCTs using Observational Longitudinal Data."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
Target trials are RCTs one would like to conduct but cannot for ethical,
financial, and/or logistical reasons. Consequently, we must emulate such
trials from observational data. A novel aspect of target trial methodology
is that, for purposes of data analysis, each subject in the observational
study is ‘enrolled’ in all target trials for which the subject is eligible,
instead of a single trial. I will describe recent theoretical results
connecting target trial methodology and structural nested models. I will
discuss a novel inferential conundrum that arises from this connection.
Finally I will discuss the question: How do we validate causal estimates
from observational data.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on March 22 (12:00 EST). Carlos
Velasco Rivera will present "On-Platform Experimental Research on Facebook
and Instagram in the 2020 Election."
<Where>
Hybrid: CGIS K354 or Zoom (the presenter will join us via Zoom)
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
We will discuss a groundbreaking collaboration among over two dozen
independent (that is, not paid by Meta) academics and a team of Meta
researchers. Since early 2020, this group has worked together to evaluate
the role of Facebook and Instagram in the 2020 U.S. presidential election.
The collaboration has, as of now, resulted in over a dozen pre-registered
(observational and experimental) designs for academic research papers. In
this presentation, we will focus on the experimental interventions that
were designed to test the causal impact of Facebook and Instagram on all of
the project’s key variables of inquiry: political participation; political
polarization; knowledge and misperceptions; and beliefs about democratic
norms. The project included multiple experiments, including full
deactivation of platform use as well as various changes to the way in which
participants encountered information on the platform.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on March 8 (12:00 EST). Cory
McCartan will present "Estimating Racial Disparities when Race is Not
Observed."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
The estimation of racial disparities in health care, financial services,
voting, and other contexts is often hampered by the lack of
individual-level racial information in administrative records. In many
cases, the law prohibits the collection of such information to prevent
direct racial discrimination. As a result, many analysts have adopted
Bayesian Improved Surname Geocoding (BISG), which combines individual names
and addresses with the Census data to predict race. Although BISG tends to
produce well-calibrated racial predictions, its residuals are often
correlated with the outcomes of interest, yielding biased estimates of
racial disparities. We propose an alternative identification strategy that
corrects this bias. The proposed strategy is applicable whenever one’s
surname is conditionally independent of the outcome given their
(unobserved) race, residence location, and other observed characteristics.
Leveraging this identification strategy, we introduce a new class of
models, Bayesian Instrumental Regression for Disparity Estimation (BIRDiE),
that estimate racial disparities by using surnames as a high-dimensional
instrumental variable for race. Our estimation method is scalable, making
it possible to analyze large-scale administrative data. A validation study
based on the North Carolina voter file shows that BIRDiE reduces error by
up to 84% in comparison to the standard approaches for estimating racial
differences in party identification.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on March 1 (12:00 EST). Laura
Hatfield will present "Adaptive metrics for an evolving pandemic."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
Throughout the COVID-19 pandemic, policymakers have proposed risk metrics,
such as the CDC Community Levels, to guide local and state decision-making.
However, risk metrics have not reliably predicted key outcomes and often
lack transparency in terms of prioritization of false positive versus false
negative signals. They have also struggled to maintain relevance over time
due to slow and infrequent updates. In this talk, I will highlight recent
work that makes two key contributions to address these weaknesses of risk
metrics. I first present a framework to evaluate predictive accuracy based
on policy targets related to severe disease and mortality, allowing for
explicit preferences toward false negative versus false positive signals.
This approach allows policymakers to optimize metrics for specific
preferences and interventions. Second, I will present a novel method to
update risk thresholds in real-time. Our proposed adaptive metrics have a
unique advantage in a rapidly evolving pandemic context. In this talk, I
also connect these ideas to new causal identification strategies in
difference-in-differences.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on February 22 (12:00 EST). Molly
Offer-Westort will present "Adaptive experimental designs for policy
learning and evaluation, with applications to Facebook Messenger studies."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
When running multi-arm trials, experimenters may wish to both learn and
evaluate data-driven policies; for example, learning which version of
treatment is most effective and evaluating the effect of that treatment in
comparison to a control condition. Response adaptive algorithms, which
dynamically update treatment assignment mechanisms based on observed
response, facilitate experimental designs where the most data is collected
about the most effective interventions, and can improve policy learning
over conventional randomized trials. I discuss design decisions when
running adaptive experiments, and considerations for inference when using
adaptively collected data. I review applications to Facebook Messenger
studies using different adaptive algorithms.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on February 15 (12:00 EST). David
Ham will present "Design-Based Confidence Sequences for Anytime-valid
Causal Inference."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
Many organizations run thousands of randomized experiments, or A/B tests,
to statistically quantify and detect the impact of product changes.
Analysts take these results to augment decision-making around deployment
and investment opportunities, making the time it takes to detect an effect
a key priority. Often, these experiments are conducted on customers
arriving sequentially; however, the analysis is only performed at the end
of the study. This is undesirable because strong effects can be detected
before the end of the study, which is especially relevant for risk
mitigation when the treatment effect is negative. Alternatively, analysts
could perform hypotheses tests more frequently and stop the experiment when
the estimated causal effect is statistically significant; this practice is
often called "peeking." Unfortunately, peeking invalidates the statistical
guarantees and quickly leads to a substantial uncontrolled type-1 error.
Our paper provides valid confidence sequences from the design-based
perspective, where we condition on the full set of potential outcomes and
perform inference on the obtained sample. Our design-based confidence
sequence accommodates a wide variety of sequential experiments in an
assumption-light manner. In particular, we build confidence sequences for
1) the average treatment effect for different individuals arriving
sequentially, 2) the reward mean difference in multi-arm bandit settings
with adaptive treatment assignments, 3) the contemporaneous treatment
effect for single time series experiment with potential carryover effects
in the potential outcome, and 4) the average contemporaneous treatment
effect in panel experiments. We further provide a variance reduction
technique that incorporates modeling assumptions and covariates to reduce
the confidence sequence width proportional to how well the analyst can
predict the next outcome.
Paper: https://arxiv.org/abs/2210.08639
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on February 8 (12:00 EST). Yi
Zhang will present "Safe Policy Learning under Regression Discontinuity
Designs."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
The regression discontinuity (RD) design is widely used for program
evaluation with observational data. The RD design enables the
identification of the local average treatment effect (LATE) at the
treatment cutoff by exploiting known deterministic treatment assignment
mechanisms. The primary focus of the existing literature has been the
development of rigorous estimation methods for the LATE. In contrast, we
consider policy learning under the RD design. We develop a robust
optimization approach to finding an optimal treatment cutoff that improves
upon the existing one. Under the RD design, policy learning requires
extrapolation. We address this problem by partially identifying the
conditional expectation function of counterfactual outcome under a
smoothness assumption commonly used for the estimation of LATE. We then
minimize the worst case regret relative to the status quo policy. The
resulting new treatment cutoffs have a safety guarantee, enabling policy
makers to limit the probability that they yield a worse outcome than the
existing cutoff. Going beyond the standard single-cutoff case, we
generalize the proposed methodology to the multi-cutoff RD design by
developing a doubly robust estimator. We establish the asymptotic regret
bounds for the learned policy using the semi-parametric efficiency theory.
Finally, we apply the proposed methodology to empirical and simulated data
sets.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
