Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on March 1 (12:00 EST). Laura
Hatfield will present "Adaptive metrics for an evolving pandemic."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
Throughout the COVID-19 pandemic, policymakers have proposed risk metrics,
such as the CDC Community Levels, to guide local and state decision-making.
However, risk metrics have not reliably predicted key outcomes and often
lack transparency in terms of prioritization of false positive versus false
negative signals. They have also struggled to maintain relevance over time
due to slow and infrequent updates. In this talk, I will highlight recent
work that makes two key contributions to address these weaknesses of risk
metrics. I first present a framework to evaluate predictive accuracy based
on policy targets related to severe disease and mortality, allowing for
explicit preferences toward false negative versus false positive signals.
This approach allows policymakers to optimize metrics for specific
preferences and interventions. Second, I will present a novel method to
update risk thresholds in real-time. Our proposed adaptive metrics have a
unique advantage in a rapidly evolving pandemic context. In this talk, I
also connect these ideas to new causal identification strategies in
difference-in-differences.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on February 22 (12:00 EST). Molly
Offer-Westort will present "Adaptive experimental designs for policy
learning and evaluation, with applications to Facebook Messenger studies."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
When running multi-arm trials, experimenters may wish to both learn and
evaluate data-driven policies; for example, learning which version of
treatment is most effective and evaluating the effect of that treatment in
comparison to a control condition. Response adaptive algorithms, which
dynamically update treatment assignment mechanisms based on observed
response, facilitate experimental designs where the most data is collected
about the most effective interventions, and can improve policy learning
over conventional randomized trials. I discuss design decisions when
running adaptive experiments, and considerations for inference when using
adaptively collected data. I review applications to Facebook Messenger
studies using different adaptive algorithms.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on February 15 (12:00 EST). David
Ham will present "Design-Based Confidence Sequences for Anytime-valid
Causal Inference."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
Many organizations run thousands of randomized experiments, or A/B tests,
to statistically quantify and detect the impact of product changes.
Analysts take these results to augment decision-making around deployment
and investment opportunities, making the time it takes to detect an effect
a key priority. Often, these experiments are conducted on customers
arriving sequentially; however, the analysis is only performed at the end
of the study. This is undesirable because strong effects can be detected
before the end of the study, which is especially relevant for risk
mitigation when the treatment effect is negative. Alternatively, analysts
could perform hypotheses tests more frequently and stop the experiment when
the estimated causal effect is statistically significant; this practice is
often called "peeking." Unfortunately, peeking invalidates the statistical
guarantees and quickly leads to a substantial uncontrolled type-1 error.
Our paper provides valid confidence sequences from the design-based
perspective, where we condition on the full set of potential outcomes and
perform inference on the obtained sample. Our design-based confidence
sequence accommodates a wide variety of sequential experiments in an
assumption-light manner. In particular, we build confidence sequences for
1) the average treatment effect for different individuals arriving
sequentially, 2) the reward mean difference in multi-arm bandit settings
with adaptive treatment assignments, 3) the contemporaneous treatment
effect for single time series experiment with potential carryover effects
in the potential outcome, and 4) the average contemporaneous treatment
effect in panel experiments. We further provide a variance reduction
technique that incorporates modeling assumptions and covariates to reduce
the confidence sequence width proportional to how well the analyst can
predict the next outcome.
Paper: https://arxiv.org/abs/2210.08639
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on February 8 (12:00 EST). Yi
Zhang will present "Safe Policy Learning under Regression Discontinuity
Designs."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
The regression discontinuity (RD) design is widely used for program
evaluation with observational data. The RD design enables the
identification of the local average treatment effect (LATE) at the
treatment cutoff by exploiting known deterministic treatment assignment
mechanisms. The primary focus of the existing literature has been the
development of rigorous estimation methods for the LATE. In contrast, we
consider policy learning under the RD design. We develop a robust
optimization approach to finding an optimal treatment cutoff that improves
upon the existing one. Under the RD design, policy learning requires
extrapolation. We address this problem by partially identifying the
conditional expectation function of counterfactual outcome under a
smoothness assumption commonly used for the estimation of LATE. We then
minimize the worst case regret relative to the status quo policy. The
resulting new treatment cutoffs have a safety guarantee, enabling policy
makers to limit the probability that they yield a worse outcome than the
existing cutoff. Going beyond the standard single-cutoff case, we
generalize the proposed methodology to the multi-cutoff RD design by
developing a doubly robust estimator. We establish the asymptotic regret
bounds for the learned policy using the semi-parametric efficiency theory.
Finally, we apply the proposed methodology to empirical and simulated data
sets.
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei
