Switch Off Your Lightbox

Mar 16, 2018

Why Overlays Are Bad For Your Site And Can Hurt Your Revenue

During pledge week, public media organizations across the country take to the airwaves and the Internet, raising money in support of their journalism and programming. Inside stations, it’s a time of hustle and bustle, of camaraderie and anxiety. But underneath it all lies a tension -- a balance between fundraising efforts and a duty to deliver quality programming. To be good stewards of our public funding, we should ask ourselves what’s the right balance of campaign messaging versus clear delivery of our content.

Fortunately, for those of us working in the digital space, there are good ways of defining what that balance looks like. In 2012, NPR partnered with Michigan Public Radio to do some important work in this area. They ran a test to evaluate just how effective different forms online display ads were at raising money. Do bigger, more intrusive ad units drive more revenue? The answer then was no. The 2012 test measured four different display ad units, including push-downs and lightboxes, and discovered that none was more effective than a standard 300x250 ad.

Despite Michigan Public Radio’s good research, lightbox and interstitial ads had become even more common on public media sites in recent years. Had stations started finding them effective? We did an informal poll of stations’ confidence in the lightbox ad format and got a tepid response from a handful of digital leaders. It seemed as if many were using them out of convenience, if nothing else.

It’s risky, however, to use lightboxes, interstitials, popups or modals as a default marketing tool. Advertising techniques that interrupt access to content have been viewed negatively by users as far back as 2004. It seems user reception to the format, hasn’t really changed today. A 2017 study showed that modals were still one of the most hated ad formats. And given Google’s statement that use of standalone interstitials could negatively affect sites’ placement in search results, we figured it was really time to do another test.  

In June 2017, wamu.org ran a digital-only “silent” fundraising campaign. We measured not only how many donations were made by users who clicked on our lightbox ad, we also measured the ambient effect of the lightbox’s presence: Were users who were exposed to the lightbox at some period during the campaign more likely to donate than those who saw no lightbox at all?

The findings were interesting.

Sessions that included a lightbox had a 0.32% conversion rate compared to sessions with no lightbox which had a 0.25% conversion rate. At first glance, this seemed like a win for the lightbox.

However, it’s important to note that if a segment has a higher percentage of sessions, then the conversion rate will likely drop for that segment. Conversion rate, here, is calculated as transactions per session.  It can be hard to wrap your head around this fact, but in particular instances such as this, a lower conversion rate is not necessarily a bad thing. It might just mean that the segment you are measuring is visiting the site more often. They’re donating, but returning to do other stuff too. This was the case with our non-lightbox segment which saw a much higher session/user rate. If we had blindly looked at this conversion rate, we might have been led astray.

Rather than using a session conversion rate, we decided to explore exposure to the lightbox via user segments. Looking at users who were exposed to the lightbox during the campaign and the revenue potential of that segment painted a different picture. We saw that users who were never exposed to a lightbox promo were much more likely to donate than users who were shown the lightbox promo. We also saw that the average revenue per user (ARPU) was much higher when users were not exposed to the lightbox. In this particular test on wamu.org, we saw a spread of 49% in favor of not having a lightbox -- $0.55/user vs. $0.82/user..

During pledge week in September 2017 we ran another study. This time we spruced up our ad creative to include sharp imagery. We also used DFP (Doubleclick for Publishers) to performance-optimize the delivery of the creative, thinking this might make the effort more effective.

While the lightbox ads had some of the highest click-through rates we’ve seen on any ad creative, once again the presence of the lightbox seemed to discourage donations. We broke this study into two tests. Test A ran at the beginning of the campaign, and Test B ran at the end of the campaign. During the first test, average revenue per user was 15% higher for those who didn’t see the lightbox. In the second test, as the campaign neared its end, non-lightbox sessions still accounted for a 9% higher ARPU. We ran a final test in December 2017 and saw the biggest spread in lightbox and non-lightbox ARPU yet -- 67% in favor of running no lightbox at all.

Many of our stakeholders were surprised by the results. Why would the simple presence of an ad unit discourage donations? Although the answer to this question is invariably complicated, we do know that many users find lightboxes intrusive and annoying. This, after all, is one of the reasons Google has threatened to penalize publishers for using them.

Others at our station asked, if $X dollars of donations were directly attributable to the lightbox, wouldn’t that mean we would have lost that amount had the lightbox not run. On the contrary, because our non-lightbox segments performed a percentage better, it suggests that we might have made that percentage more without the lightbox.

Putting aside discussion of lost revenue, we should still ask if the approach takes time and energy away from our staff in terms of ad production and management, if it interferes with the overall user experience on the site, and if it may potentially harm our Google search rank, should we really be doing this?

Interestingly, the final nail in our lightbox coffin came as a result of Google’s announcement that the Chrome browser would block ads from sites that delivered intrusive ad units. Although there has been some debate as to whether publishers could skirt Chrome’s ad blocker by limiting the amount of time intrusive ad formats, like lightboxes, are present, our teams were not willing to jeopardize losing so much display ad inventory. You can read more on how Chrome’s ad filtering works on the Chromium Blog.

To recap, there are at least four very good reasons that you should stay away from lightboxes:

1. Users don’t like them. Studies from 2004 have shown this and the same is true today.

2. They can hurt your site’s PageRank.

3. They may not boost fundraising revenue, and as our tests have shown, may actually decrease your fundraising potential.

4. They put your display advertising revenue in jeopardy by increasing the chances that all ads on your site may be blocked by browsers such as Chrome.

In spirit of our public media mission, we should be looking for ways to deliver content experiences in the most unobtrusive way, with the least amount of commercial interruptions. It’s also clear from our tests that other ways of promoting fundraising campaigns perform just as well, or even better, than lightbox overlays. With the additional push from Google to discourage publishers from using intrusive ad formats, there should be no question that it is finally time to turn off your lightboxes.