• I n 2021, we published a blog post titled " Increasing experimentation accuracy and speed by using control variates ," describing how we reduce the variance of metrics using CUPED in our experimentation platform. • This is a follow-up on how CUPED has evolved at Etsy since then. • Spoiler - It’s changed a lot, decreasing our average experiment duration by 3 days! • Etsy’s mission is to Keep Commerce Human. • To achieve this, we need to understand the impact each change to our platform has on our buyers’ and sellers’ experience. • Whether that involves changing the color of the “Buy Now” button on the Etsy app or updating elements of how our algorithms rank search results, we leverage large-scale online experimentation to iterate on and improve the things we build.

Article Summaries:

  • Etsy has updated its experimentation platform to further shorten the time needed to run A/B tests. Building on a 2021 blog post that introduced CUPED (Controlled‑Experiment Using Pre‑Experiment Data), the company now reports a 3‑day average reduction in experiment duration. CUPED uses pre‑experiment visitor data as covariates in a linear regression to cut metric variance, boosting statistical power and allowing smaller sample sizes. Early deployments cut variance by 7 % on average-up to 30 % in some cases-leading to earlier decisions and faster product iterations. The update underscores Etsy’s focus on rapid, data‑driven improvements for buyers and sellers.
  • Etsy has updated its use of CUPED (Controlled‑Experiment Using Pre‑Experiment Data) to cut experiment duration by an average of three days. The variance‑reduction technique, first deployed in 2021, leverages pre‑experiment visitor data to adjust key metrics such as conversion rate, thereby lowering statistical noise and reducing required sample sizes. The latest iteration delivers up to 30 % variance reduction in some tests, enabling decisions to be made roughly a day earlier than before. By shortening the experimentation cycle, Etsy aims to accelerate product iterations and improve the experience for buyers and sellers while maintaining rigorous statistical confidence.
  • Etsy’s experimentation platform has refined its use of CUPED (Controlled‑Experiment Using Pre‑Experiment Data) to cut average experiment duration by three days. The variance‑reduction technique leverages pre‑experiment visitor data-such as prior week purchase counts-to adjust key metrics like conversion rate, lowering metric variance and thus the sample size needed for statistical significance. Since its 2021 rollout, CUPED has delivered a 7 % average variance reduction (up to 30 % in some cases) and enabled decisions to be made roughly a day earlier. The update underscores Etsy’s focus on faster, data‑driven improvements to buyer and seller experiences.
  • Etsy has updated its CUPED (Controlled‑Experiment Using Pre‑Experiment Data) technique to cut average experiment duration by three days. The variance‑reduction method, first deployed in 2021, uses pre‑experiment visitor data-such as prior week purchases-to adjust key metrics like conversion rate, thereby lowering statistical noise. The latest iteration delivers up to 30 % variance reduction in some tests, enabling faster, more accurate decisions. By shrinking the required sample size, experiments reach statistical significance sooner, accelerating the product development cycle while maintaining Etsy’s commitment to a human‑centered commerce experience.
  • Etsy has updated its CUPED (Controlled‑Experiment Using Pre‑Experiment Data) technique to cut experiment duration by an average of three days. The variance‑reduction method, first deployed in 2021, uses pre‑experiment visitor data as covariates in a linear regression to lower metric noise, thereby boosting statistical power and allowing smaller sample sizes. The latest iteration delivers up to 30 % variance reduction in some tests, enabling decisions to be made roughly a day earlier on average. Etsy’s goal is to accelerate learning in its large‑scale online experimentation pipeline, improving buyer and seller experiences faster while maintaining rigorous statistical confidence.

Sources: