Paywall AB testing is a necessary strategy for any app publisher wanting to optimize revenue. In this article we’ll provide an introduction to paywall testing by covering the following topics:
The paywall screen is how apps make money through IAPs and subscriptions. In fact, most app publishers spend very little time updating their paywall let alone optimizing it.
The paywall can be one of the essential elements contributing to your success in growth and retention this year. Let’s take look:
Focusing on app growth usually means a combination of efforts including:
It’s important that new users see your paywall during their first session. In fact, 70-80% of subscription starts occur on the day of install (D0) according to data from AppsFlyer and Redbox Mobile.
Considering this, it’s important to think of your paywall as not one monolithic object that all users flow through regardless of origin. That would be like having a single landing page on your web site regardless of traffic source.
Conceptually, you want different paywall instances aligned to each acquisition source:
To optimize in-app revenue, your paywall needs to be operated like a growth marketing asset, not a static coded screen.
Most app publishers use push notifications for app engagement. Push can be a helpful way to improve D7 retention, but you should also consider what happens once the user is in the app.
It may not be obvious, but the paywall isn’t just important for growth. It’s important for retention as well especially for subscription apps. Consider that most app subscriptions are auto-renewable. The desired state for app publishers is for a user to be a paying subscriber on an annual plan with auto-renew turned on.
However, subscription customers can end up in other states as well. The paywall can help you message users in context of those other states to nudge them to that desired place.
Here are a few example of retention paywall instances tied to user context:
The key to effective retention paywalls is to power your experience with individual-level subscriber data. This means your external marketing channels, the in-app experience, and everything in-between should all have the same context about a user’s subscription journey. This way, the experience including marketing assets will be personalized and relevant.
Yes, you should be conducting paywall AB testing. The exception to the rule is if your app doesn’t have much usage. If this is you, focus on top-of-the-funnel activities such as ASO keyword tuning.
If you do have steady app traffic, you almost certainly will be able to improve in-app revenue via one or more of the following:
If your app has only one paywall, you can benefit from paywall AB testing. Most apps convert at low enough rates that much upside exists. Here are a few guidelines:
Sadly, it’s not uncommon to see freemium app with free-to-paid conversion rates of less than 1%. With testing, you can boost conversion to 5-15% or more.
For apps with a hard paywall, conversion rates depend on whether you offer a free trial or not. It’s not uncommon to see trial conversions of 5% or more. With testing, you can boost free trial conversion rates to 30-50% or more.
No matter what your conversion rate is, your paywall can be improved. The only question is by how much.
Just realize that the one-size-fits-all paywall won’t be perfectly aligned with each and every user. For fully optimized revenue, you’ll need to move to multiple, segmented paywalls.
As we’ve explained, to optimize in-app revenue it’s important to have different paywalls focused on different acquisition channels and user context. To say this another way, you should have a paywall segmentation strategy.
With paywall segmentation, you and an opportunity to conduct paywall AB testing that is focused on improving the specific metric that a certain paywall is responsible for.
For instance, you could run a test of the paywall presented to users acquired via Apple Search Ads to tightly focus on improvement to that channel. For instance, you could focus on improving your ROAS (Return on Advertiser Spend - defined as the total revenue generated divided by total ad spend).
Similarly, you can test variants of your Annual Offer to see which does a better job of converting existing monthly subs to your annual plan.
To prioritize your testing strategy, focus on the key metrics you’re trying to improve first. Also look for the opportunities that have a lot of room for improvement so you can get some quick wins.
If you’re convinced your app is a good candidate for paywall AB testing, let’s take a look at how you actually run a test. We’ll cover the three key elements to help you get started with your first test:
There are different tools and techniques for running an AB test. For the purpose of this article, which pertains to mobile app paywall AB testing, you have the following options:
There are pros and cons to each of these approaches. If you’re already using a general purpose tool, something like Optimizely might make sense. The downside, is it’s not particularly mobile-first and certainly wasn’t built to focus on testing paywalls.
A solution like Firebase is closer to mobile-first, but is not as friendly for the marketing folks on your team that are used to great campaign management systems.
In our humble opinion, Nami is the easiest way to ship a mobile paywall and yes, conduct paywall AB tests. In fact, no code is needed to design the paywall or run the test. Our powerful campaign engine helps you with paywall segmentation so can focus the right test on the right outcome.
Now that you’ve chosen a tool, it’s time to figure out what you want to test. As we’ve discussed you may have multiple paywalls each playing a different role in your growth and retention strategy.
This means the elements you will want to test really depends on the paywall and it’s purpose. Here are just a few things you might test based upon common mobile app paywall designs:
Remember, the element you want to change needs to align with the metric you want to influence. For example, if you want to drive more users into choosing the annual plan the possible elements to test may narrow to:
Your test can include multiple changes to the paywall or a single change. There are pros and cons to each. Multiple changes may help you make progress faster while a single change may help you gain better intuition from each test.
Now that you’ve chosen what you want to test, it’s time to create your variant. If you're using a no code tool like Nami, it’s easy to create the variant for paywall ab testing.
As a best practice, variant A should be the paywall that is already active in your app so you have a baseline to start with. Variant B contains the changes you want to test.
Next, you need to decide how to allocate traffic to to your variants. It could be as simple as a 50%/50% split or 80%/20% split. The specific granularity of traffic allocation will depend on the tool you’re using. Nami allows any whole number so long as the total equals 100%.
Some considerations when deciding how to allocate traffic:
Your test is running and results are coming in. You’re probably eager to find out the winner. However, just because Paywall B is converting better, doesn’t mean it’s the winner. It might be, but to be sure you need to reach statistical significance.
A statistical significant result is one that has not occurred by random chance. This is why it’s important not to look at those early results which may appear like a conclusion when in fact there’s not yet enough yet to reach such a conclusion.
You might be wondering how long the test will take to reach statistical significance. The answer is somewhat frustrating: it depends. It depends how much traffic each variant is receiving as well as how many conversions you’ve received against that traffic.
Amongst all this statistics talk, there is good news! Most AB testing tools will do the math for you and crown a winner once statistical significance is reached with high confidence.
In this article we’ve talked all about paywall AB testing. The AB naturally means we’re testing one thing against another. This limits the test to just two paywalls.
Marketers who have performed testing in other contexts will likely be familiar with multi-variant testing (MVT). While it may sound appealing to test more things at once (in this case, more paywall variations), it’s not for everyone.
MVT requires much more traffic to deliver a statistically significant result. Given how anemic most mobile app conversion rates are, it’s our recommendation that you stick to AB testing even with a lot of traffic.
There’s so much opportunity to improve without the complexity of MVT. Once you’re chasing single percentage points of optimization, you can consider introducing MVT into your testing arsenal.
Paywall AB testing is an essential for any app publisher wanting to optimize in-app revenue. In fact, paywall AB testing can boost conversion rates by 2-3X or more through straightforward changes.
At Nami, we’re focused on helping app publishers thrive in the app economy. We’re doing this by turning the paywall into a full-fledged marketing asset. Paywall AB testing is just one of the capabilities available as part of our solution. If you’re interested in getting a demo or learning more, we’re happy to help.
Discover the insights from paywall A/B testing for mobile apps to maximize your revenue and user engagement. Learn how to find the right pricing structures, subscription models, and paywall designs to improve your content monetization strategy.
Discover the benefits of using native UI elements for paywall screens in mobile apps. From faster loading times to improved security, learn how leveraging these elements can maximize your revenue and user satisfaction.