Dear Applied Statistics Workshop Community,
Our next meeting of the semester will be on February 15 (12:00 EST). David
Ham will present "Design-Based Confidence Sequences for Anytime-valid
Causal Inference."
<Where>
CGIS K354
Bagged lunches are available for pick-up at 11:40 (CGIS K354).
Zoom:
https://harvard.zoom.us/j/99181972207?pwd=Ykd3ZzVZRnZCSDZqNVpCSURCNnVvQT09
<Abstract>
Many organizations run thousands of randomized experiments, or A/B tests,
to statistically quantify and detect the impact of product changes.
Analysts take these results to augment decision-making around deployment
and investment opportunities, making the time it takes to detect an effect
a key priority. Often, these experiments are conducted on customers
arriving sequentially; however, the analysis is only performed at the end
of the study. This is undesirable because strong effects can be detected
before the end of the study, which is especially relevant for risk
mitigation when the treatment effect is negative. Alternatively, analysts
could perform hypotheses tests more frequently and stop the experiment when
the estimated causal effect is statistically significant; this practice is
often called "peeking." Unfortunately, peeking invalidates the statistical
guarantees and quickly leads to a substantial uncontrolled type-1 error.
Our paper provides valid confidence sequences from the design-based
perspective, where we condition on the full set of potential outcomes and
perform inference on the obtained sample. Our design-based confidence
sequence accommodates a wide variety of sequential experiments in an
assumption-light manner. In particular, we build confidence sequences for
1) the average treatment effect for different individuals arriving
sequentially, 2) the reward mean difference in multi-arm bandit settings
with adaptive treatment assignments, 3) the contemporaneous treatment
effect for single time series experiment with potential carryover effects
in the potential outcome, and 4) the average contemporaneous treatment
effect in panel experiments. We further provide a variance reduction
technique that incorporates modeling assumptions and covariates to reduce
the confidence sequence width proportional to how well the analyst can
predict the next outcome.
Paper:
https://arxiv.org/abs/2210.08639
<2022-2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/embed?src=c_3v93pav9fjkkldrbu9snbhned8…
Best,
Shusei