Dear Applied Statistics Workshop Community,
Our next meeting will be on October 18 (12:00 EST). Dae Woong Ham presents
"Design-Based Confidence Sequences: A General Approach to Risk Mitigation
in Online Experimentation."
<When>
October 18, 12:00 to 1:30 PM, EST
Lunch will be available for pick-up inside CGIS K354.
<Where>
In-person: CGIS K354
Zoom:
https://harvard.zoom.us/j/93217566507?pwd=elBwYjRJcWhlVE5teE1VNDZoUXdjQT09
<Abstract>
Randomized experiments have become the standard method for companies to
evaluate the performance of new products or services. In addition to
augmenting managers' decision-making, experimentation mitigates risk by
limiting the proportion of customers exposed to innovation. Since many
experiments are on customers arriving sequentially, a potential solution is
to allow managers to ``peek'' at the results when new data becomes
available and stop the test if the results are statistically significant.
Unfortunately, peeking invalidates the statistical guarantees for standard
statistical analysis and leads to uncontrolled type-1 error. Our paper
provides valid design-based confidence sequences, sequences of confidence
intervals with uniform type-1 error guarantees over time for various
sequential experiments in an assumption-light manner. In particular, we
focus on finite-sample estimands defined on the study participants as a
direct measure of the incurred risks by companies. Our proposed confidence
sequences are valid for a large class of experiments, including multi-arm
bandits, time series, and panel experiments. We further provide a variance
reduction technique incorporating modeling assumptions and covariates.
Finally, we demonstrate the effectiveness of our proposed approach through
a simulation study and three real-world applications from Netflix. Our
results show that by using our confidence sequence, harmful experiments
could be stopped after only observing a handful of units; for instance, an
experiment that Netflix ran on its sign-up page on 30,000 potential
customers would have been stopped by our method on the first day before 100
observations.
<2023 Schedule>
GOV 3009 Website:
https://projects.iq.harvard.edu/applied.stats.workshop-gov3009
Calendar:
https://calendar.google.com/calendar/u/0?cid=Y18zdjkzcGF2OWZqa2tsZHJidTlzbm…
Best,
Jialu
--
Jialu Li
Department of Government
Harvard University
jialu_li(a)g.harvard.edu