How to read it
A lower p-value means the observed conversion-rate gap is less likely under a no-difference assumption.
Before calling a winner
Avoid stopping early just because the result crosses a threshold. Run through complete business cycles and use the decision alongside product judgment.
Practical guardrails
- Use final assignment counts, not partial analytics snapshots.
- Check that traffic quality stayed stable across variants.
- Treat very small samples as directional, even with a good-looking lift.