User:Cook Me Plox/ads testing

This is a scratch space for some of the data and decision-making we are doing while testing display ads.

Baseline traffic

I wanted to get some unbiased traffic stats, since I knew we did about 5 million Cloudflare worker requests a day (but not all of them resulted in real pageviews)

• 167,549 total worker requests served in about 40 minutes a bit before peak on Saturday - we get about ~5m a day, so multiply these numbers by about 30 to get daily estimates.
  • 20,069 (12.0%) are 302 redirects (exclude - don't really count as pages for our purposes)
  • 13,498 Special:Search redirects
  • 2,386 Special:Lookup redirects
  • 3,820 Special:RecentChanges atom feed - should disappear
  • 183 Special:Random redirects
  • 182 other 302 redirects (mostly submits). Exclude.
• 949 (0.6%) 301 redirects (mostly people using address bar to get around). Exclude.
• 63 (0.04%) 404s (exclude)
• 13,600 (8.1%) are from search engine crawlers. Mostly our old friends Google and Bingbot, but also some weird ones we maybe don't want. They don't count as eyeballs or people for this purpose.
• Remainder (79.3%) are served content pages. of those:
• 1,159 searches results pages (0.7%), 131,709 articles (99.3%).
• 107,102 (80.6%) logged out, desktop
• 21,839 (16.4%) logged out, mobile
• 3,873 (2.9%) logged in, desktop
• 54 (0.04%) logged in, mobile

Big picture: we do about 3.2m logged-out desktop pageviews per day, 700k logged-out mobile pageviews per day. That's a slightly higher desktop percentage than I was expecting.

Footer ad test

We have been testing footer ad on a random (but sticky) 5% subset of logged-out desktop users, starting on February 12th. This test was done with NitroPay, our current ad tech provider (to put everyone at ease, we will almost-certainly never enter into any sort of long-term binding agreement that ties us to a particular ad company - it's important that the wiki community can pull the plug if that's what needs to happen).

With the caveats that we are extrapolating based on early data (and some of these findings may be particular to NitroPay rather than advertising in general), these were the main takeaways:

Nobody seemed to notice.

5% isn't everyone, of course, but I was quite surprised that we went a whole week testing an ad without a single complaint (or even a comment) from anyone who wasn't already involved in the testing. Nothing on Reddit or Discord or anywhere else - it seemed like nobody cared.

We don't think any "bad ads" got served.

This was something we were super worried about, but related to the previous point, we got no reports or commentary at all. We were expecting to have to chase up with the NitroPay guys regularly, but after a week, the only "bad ad" reports were from Jayden and me testing stuff. I think some of this is due to the NitroPay guys doing a pretty thorough job with a domain block list (which I was skeptical about the effectiveness of, but so far hasn't let anything through that we're aware of).

About 70% of desktop visitors are using some sort of AdBlock.

We serve about 3.2 million logged-out desktop pageviews a day to what appear to be real users. However, the reporting script was only reporting about 50 thousand pageviews per day (remember this experiment was running for 5% of users). Simple math says that about 70% of the relevant pageviews blocked the reporting.

70% is insanely high, TBH - it's higher than I've seen anywhere else (about twice what I was expecting ahead of time), and it's a tough blow for the effort to raise 2 full-time salaries with ads. I think it worth trying to do some testing of our own to try to confirm that number.

90% of the revenue is from the United States.

• 51% of our traffic is from the US
• US users are about 1.7 times more likely to NOT have AdBlock (64% of our total monetizable pageviews are from the US)
• Ads served to US users are worth about 5 times more than the rest of our users. This part is fucking crazy, especially because the rest of our users are largely in other highly-developed countries (Western Europe, Canada, AU/NZ). With the footer, we're getting about $0.43 per thousand impressions from the US, nowhere else is even a third of that. The UK is ten times lower, at like 5 cents per thousand. I knew that the US was king for advertising, but I hadn't realized how much the landscape had shifted in the last few years after GDPR and COVID.

Put it all together, and the Land of the Free is responsible for about 10 times as much wiki ad revenue as the rest of the world combined.

The main takeaway

If current trends hold, rolling out the footer ad to all logged-out desktop users would make approximately $90k-$110k per year.

There are reasons to think this number may be an underestimate:

• February is usually a fairly slow month for advertisers - usually things peak in Q4 for Christmas, and everyone scales back at the beginning of the new year. There's just not that much competition for the ad space this time of year, so all else equal, we might expect an annualized rate that's about 30% higher than what we're looking at in February.
• We only very recently started advertising, and having no advertising history makes advertisers less willing to bid on our space. We're already seeing a small uptick in the CPM numbers this week.

On the other hand, RuneScape and OSRS are somewhat more popular right now than they have been over most of the last year. A 25% downturn in player count (near the low end of what it was at this year) would probably reduce our revenue by the same amount.

The bad news is that it's looking like a footer ad will not fully fund what we need. It's close (especially adding on a projected $20k/year from Patreon, and whatever's left over from the Jagex fee), but it's just not enough. If the footer revenue was 30% more, we would probably be having a very interesting conversation about trying to tighten the rest of the budget. It may still get there eventually, but IMO we need to at least try something else.

The main options are sidebar or sitenotice. People have told me that they would probably prefer sidebar over sitenotice, although I'm interested in getting more opinions on that. So with that in mind, we will probably trial a sidebar ad for the next few days (or some mix of either a sidebar or footer depending on page size - not both). I strongly suspect this will be enough to cover our funding requirements.

More nebulous questions

Just some out-of-the-box thoughts based on the recent testing. These are not things I'm currently proposing.

• The US is 90% of our revenue. Is there any world where it makes sense to only advertise to Americans? Advertising in Europe has so many GDPR-ish things that can go wrong. It's where nearly all the liability is, but for such a small fraction of the money.
• Mobile is only like 16% of our traffic (even less than I expected), but they (mostly) don't use AdBlock. It's possible that 40% of our monetizable traffic is on mobile. Does this swing the pendulum at all towards advertising on mobile (maybe instead of desktop)? I still think the answer is no, but it's not as straightforward as I had expected.