Comparative Probability Metrics: Using Posterior Probabilities to Account for Practical Equivalence in A/B tests

DOI10.1080/00031305.2021.2000495OpenAlexW3209485542MaRDI QIDQ5050831

Nathaniel T. Stevens, Luke Hagar

Publication date: 18 November 2022

Published in: The American Statistician (Search for Journal in Brave)

Full work available at URL: https://doi.org/10.1080/00031305.2021.2000495

zbMATH Keywords

Markov chain Monte CarloBayesian estimationpractical significanceA/B testingdesign and analysis of experimentstwo-group comparisons

Mathematics Subject Classification ID

Statistics (62-XX)

Uses Software

• rjags
• JAGS
• BayesDA

Cites Work

• Testing Statistical Hypotheses of Equivalence and Noninferiority
• Bayes Factors
• Moving to a World Beyond “p < 0.05”
• ValidP-Values Behave Exactly as They Should: Some Misleading Criticisms ofP-Values and Their Resolution WithS-Values
• The p-Value Requires Context, Not a Threshold
• p-Values, Bayes Factors, and Sufficiency
• An Introduction to Second-Generation p-Values
• A Proposed Hybrid Effect Size Plus p-Value Criterion: Empirical Evidence Supporting its Use
• Three Recommendations for Improving the Use of p-Values
• The False Positive Risk: A Proposal Concerning What to Do About p-Values
• Moving Towards the Post p < 0.05 Era via the Analysis of Credibility
• Blending Bayesian and Classical Tools to Define Optimal Sample-Size-Dependent Significance Levels
• Abandon Statistical Significance
• How Large Are Your G-Values? Try Gosset’s Guinnessometrics When a Little “p” Is Not Enough
• Coup de Grâce for a Tough Old Bull: “Statistically Significant” Expires
• The ASA Statement on p-Values: Context, Process, and Purpose

This page was built for publication: Comparative Probability Metrics: Using Posterior Probabilities to Account for Practical Equivalence in A/B tests