Hi all!
I am a newbie in the topic of Bayesian AB testing and tried to follow the example for revenue-like metrics (Beta+Gamma distributions) from here.
I encountered a problem that AA test simulations (100 sims, 500k users in group per simulation) on the data I have show significant differences too often. From 30% to 50% of simulations shows 0 out of HDI (depends on selected prior)
I think it’s because of the data structure - very right skewed.
Conversion to target action - 1%, so data contains 99% of zeroes.
For users with target action metric’s histogram looks like this:
What steps should I take to reduce False Positives?