​​False Positive Risk in A/B Testing

There is a much greater probability than generally expected that a statistically significant test outcome is in fact a false positive. In industry jargon: that a variant has been identified as a “winner” when it is not.

https://georgi-georgiev.medium.com/false-positive-risk-in-a-b-testing-ba2c76e258c4

via @ABtesting

​​The good Lukas Vermeer’s A/B testing game simulator is here. It’s an interactive way to understand hypothesis testing better.

You control the backlog and running experiments for a small development team. Each day you make decisions that impact product development and overall sales.

Your objective is to maximise total sales.

via @ABtesting

​​It takes a Flywheel to Fly: Kickstarting and Keeping the A/B testing Momentum

Practitioners from hundreds of companies and several different industries reached out with thoughts, concerns and tips on how to kickstart and scale an A/B testing program from “Crawl” or “Walk” milestones, to the “Run” or “Fly” stages.

via @ABtesting

​​Four Ways to Improve Statistical Power in A/B Testing (without increasing test duration, duh)

Exploration simple and potent techniques for enhancing power without prolonging test durations. By grasping the significance of key parameters such as allocation, MDE, and chosen KPIs, data analysts can implement straightforward strategies to elevate the effectiveness of their testing endeavors. This, in turn, enables increased data collection and provides deeper insights into their product.

via @ABtesting