So welcome to another exciting episode of ‘fun with stats’ with your host cliff harris.
Lets examine some stats for changes that are under analysis (using random sampling) with the buy page of a fictional pc political strategy game. Lets assume it sells for $20 to make things simple, and to take into account occasional discounts.
Lets also imagine (theoretically) that the game sells on portals which take a 30% cut and that it has a direct sell option which (theoretically) takes a 10% cut. So a portal sale earns $14 and a direct one earns $18.
The raw before and after the change stats are heavily skewed due to traffic variations, but basically we have this:
3rd October onwards (old buy page)
Direct buy button clicks 427
Buy page hits 3203.
Portal buy button clicks 152
direct sales share = 13.3%
8th October onwards (changed buy page)
Direct buy button clicks 194
Buy page hits 451.
Portal buy button clicks 23
direct sales share = 43%
So far, all this does is say that the percentage of buyers who choose the direct option has gone up. Because we get email details + a higher percentage of the sale, then we can assume this is a boost for us, but it’s not ‘free money’ We are not getting new sales, just converting portal sales to direct ones. So the difference is really only to be found in that 20% difference. To work out the actual difference we need to guess total portal and direct theoretical (assuming all clicks are sales, they really aren’t…) income, and compensate for traffic variation…
3rd October onwards (old buy page)
Portal income $2,128
Direct Income $7,686
Income per visitor from portals: $0.66
Income per visitor from direct: $2.39
Income per visitor overall: $3.06
8th October onwards (changed buy page)
Portal income $308
Direct Income $3,492
Income per visitor from portals: $0.68
Income per visitor from direct: $7.74
Income per visitor overall: $8.42
Holy crap. Have I done this right? And this is with A/B testing so actually only HALF the visitors are getting the new buy page, so the effect is actually double this. Assuming my maths is correct, the percentage of people who visit my site and THEN buy on a portal is actually fairly low, meaning that encouraging them to buy direct (but still having multiple portal options displayed) seems to have very little downside. The income from portals actually even rose very slightly $0.66 to $0.68, which is a statistical irrelevance. That change from $3.06 to $8 is not though. It’s real.
So obviously I need a lot MORE data to prove I’m right, so I’m going to leave my experiment running a few more days. I have some D3 ads running now which will drive in a bit more traffic which will help. You can’t really extrapolate anything from under 1,000 clicks on anything. But it looks promising.
BUT THE FUNNEL…
What this means is that getting someone to my buy page now earns $8 not $3. Now as it happens, the actual; abandonment rate is fairly high, because many people see the price (only on the buy page..) and then don’t buy, but there is value there in the stored intent and later discount-purchases, or second-thoughts and return buyers. Lets assume an abandonment rate of 80%. That makes a visitor now worth $1.68 rather than $0.61. The problem is not everyone gets to the buy page, home page to buy page hits happen at 35% so real values are old system-> $0.21 per visitor, new system -> $0.59.
The difference between the viability of an ad campaign or PR campaign targeting $0.21 and $0.59 per visitor is huge. It’s hard getting $0.20 CPC. It’s easy to get $0.50 CPC.
This is why I care about this stuff. Plus I’m a stats head and I enjoy it :D