r.robert.

Free tool · Robert × Dor

Is the lift real
or noise?

Compare conversion before and after a change, or between two stores. Get the difference, a confidence interval, a p-value and a plain-English verdict, plus how much traffic you need to detect the lift you care about.

Conversion rate significance calculator

What are you comparing?
Before
After
Confidence level

Use periods of equal length that cover the same weekdays, e.g. four weeks before and four weeks after a change.

Is the difference real?

Likely real

After converts higher (+2.50 pts), and a gap this size would show up by chance only 0.6% of the time if nothing had changed.

Conversion
20.00% → 22.50%
Before → After
Difference
+2.50 pts
+12.5% relative
95% confidence interval
+0.73 pts to +4.27 pts
excludes zero
p-value
0.006
z = 2.77

How much traffic do you need?

Relative, e.g. 10% means 20% → 22%.
Per store or period.

To reliably spot a 10% change from 20.0% conversion (95% confidence, 80% power) you need about 6,510 visitors in each period, or 44 days each at 150 a day.

Show your work

  • Conversion = transactions ÷ visitors Before: 840 ÷ 4,200 = 20.00% · After: 900 ÷ 4,000 = 22.50%
  • Pooled rate p = (transactions A + B) ÷ (visitors A + B) 1,740 ÷ 8,200 = 0.2122
  • z = difference ÷ √(p(1 − p)(1/visitors A + 1/visitors B)) 0.0250 ÷ 0.0090 = 2.77; two-sided p-value = 0.006
  • 95% interval = difference ± 1.96 × √(pA(1 − pA)/visitors A + pB(1 − pB)/visitors B) 0.0250 ± 1.96 × 0.0090
  • Visitors per period = (zα√(2p̄(1−p̄)) + zβ√(p1(1−p1) + p2(1−p2)))² ÷ (p2 − p1)² = 6,510

Assumes each visit is an independent chance to buy. Groups shopping together, repeat visitors, weather and promotions add variation the formula can’t see, so treat borderline results (p near your threshold) as “not proven yet”.

Measure real visitors with Dor (opens dorhq.com in a new tab)

Dor counts visitors at the door and pairs them with your POS sales. $150 per location per month plus $300 hardware.

Worked example.

A store rearranged its front table. In the four weeks before, 4,200 people came in and 840 bought (20.00%). In the four weeks after, 4,000 came in and 900 bought (22.50%).

  • Difference: 22.50% − 20.00% = +2.50 points, a 12.5% relative lift.
  • Pooled rate: 1,740 ÷ 8,200 = 0.2122. Standard error: √(0.2122 × 0.7878 × (1/4,200 + 1/4,000)) = 0.0090.
  • z = 0.025 ÷ 0.0090 = 2.77, so the two-sided p-value is about 0.006.
  • 95% interval: +0.73 to +4.27 points. It excludes zero, so the lift is likely real.

To reliably detect a smaller 10% lift from 20% (to 22%), you’d need about 6,500 visitors in each period, around 44 days each at 150 visitors a day.

“Likely real” means the change probably isn’t chance. It doesn’t prove the new table caused it; a holiday, weather or a competitor closing could do the same. Comparing with a store that didn’t change helps.

Reading the result.

  • p-value: how often you’d see a gap at least this big if conversion hadn’t really changed. Below 0.05 is the usual bar for 95% confidence.
  • Confidence interval: the range of true differences consistent with your data. If it straddles zero, you can’t rule out no change. If it’s wide, collect more traffic.
  • Don’t peek and stop. Decide the test length up front. Checking every day and stopping the moment it looks significant inflates false positives.
  • Use visitors, not transactions alone. Without a door count you can’t tell a conversion change from a traffic change.

Questions retailers ask.

How do I know if a change in store conversion is significant?

Compare the two conversion rates with a two-proportion z-test. Enter visitors and transactions for both periods; if the p-value is below 0.05, the difference is unlikely to be chance at 95% confidence.

What p-value should I use?

0.05 (95% confidence) is the common default. Use 0.01 when acting on a false positive would be expensive, such as a chain-wide refit, and 0.10 for cheap, easily reversed changes.

Why isn’t my lift significant when it looks big?

Small stores have few visitors, so conversion naturally swings day to day. A 3-point jump on 300 visitors is often noise; the same jump on 5,000 visitors usually isn’t. The traffic estimate shows how long to wait.

Does significant mean my change caused the lift?

No. It means the difference is unlikely to be random. Seasonality, weather, promotions or local events can also move conversion. Compare against a control store or the same weeks last year.

How many visitors do I need to test a store change?

It depends on your baseline conversion and the lift you want to detect. From 20% conversion, spotting a 10% relative lift at 95% confidence and 80% power takes roughly 6,500 visitors per period.

Test changes on real visitor counts.

Significance needs visitors, not just sales. Dor counts everyone who walks in and pairs it with your POS so every store test has a denominator.

Book a Dor demo (opens dorhq.com in a new tab)