• Imagine you have a fabulous idea to drive more sales on Etsy by giving out free ice cream with every purchase. • How would you know if it will actually work? • One way to test this out is to run an experiment ! • An experiment is a learning tool that can help us see if a new change really affects user behavior and key metrics we care about. • At Etsy, we run nearly 2000 experiments a year for testing new ideas to improve the user experience for our buyers and sellers. • For our ice cream idea, we could set up our experiment as an A/B test on the listing page, where one group of users (the “control” group) sees the usual “Buy it now” button, while the other (the “treatment” group) sees a shiny new version that says, “Buy it now and get a FREE ICE CREAM!”.

Article Summaries:

  • Etsy’s experimentation platform, Catapult, historically offered only 13 hard‑coded segmentations, limiting teams to pre‑defined slices of experiment data. To accelerate learning from the roughly 2,000 A/B tests run annually, Etsy engineers redesigned Catapult so experimenters can now define custom segmentations directly within the platform. This change eliminates the need for engineering code changes for each new segmentation, tripling the number of available slices and enabling deeper, faster insights into how different user groups respond to product changes. The update empowers teams to uncover hidden patterns-such as varying responses to promotions-without waiting for backend updates.
  • Etsy’s in‑house experimentation platform, Catapult, historically offered only 13 hard‑coded segmentations, limiting teams to pre‑defined user slices. Adding new segmentations required engineering code changes, a slow, non‑scalable process that hindered rapid experimentation. The recent update now lets experimenters define their own segmentations directly within Catapult, tripling the available segmentation options. This empowers teams to analyze experiments by custom user groups-such as ice‑cream lovers versus non‑lovers-without waiting for engineering support. The change accelerates insight generation across Etsy’s 2,000 annual experiments, enabling deeper, faster analysis of how different user segments respond to new features.
  • Etsy’s in‑house experimentation platform, Catapult, has been upgraded to let experimenters create their own segmentations directly within the system. Previously, only 13 hard‑coded segmentations were available, and adding new ones required engineering changes to the data pipeline, a slow, non‑scalable process. The new feature enables teams to define custom slices of data-such as user interests or purchase history-without code changes, tripling the number of usable segmentations. This empowers analysts to uncover deeper, faster insights from experiments, improving the ability to understand how different user groups respond to new features.
  • Etsy’s experimentation platform, Catapult, previously offered only 13 hard‑coded segmentations for A/B test analysis, requiring engineering changes to add new ones. The company has now enabled experimenters to define custom segmentations directly within the platform. This change has tripled the number of available segmentations and accelerated insight generation across teams. By allowing analysts to slice data by user traits such as ice‑cream preference or price sensitivity, Etsy can uncover nuanced effects that were previously masked in aggregate results, improving the speed and depth of experimentation outcomes.
  • Etsy has upgraded its in‑house experimentation platform, Catapult, to let teams create custom segmentations without engineering intervention. Previously, Catapult offered only 13 hard‑coded segmentations that refreshed daily, and adding new ones required code changes to the data pipeline-a slow, non‑scalable process. The new system now supports user‑defined segmentations, tripling the number of available slices and enabling faster, deeper analysis of A/B test results. This change is expected to accelerate insights across Etsy’s 2,000 annual experiments, such as evaluating how different user groups respond to new features like a “free ice‑cream” promotion.

Sources: