Guides
Start typing to search…

A/B Testing Your Subscription Manager

You can A/B test different versions of your Subscription Manager—including themes and Cancel Flows—by integrating your own A/B testing platform like LaunchDarkly, AB Tasty, or Optimizely. This approach gives you full control over variant bucketing and experiment configuration while leveraging Ordergroove's theme system. This flexibility can also be leveraged to show different Subscription Manager themes for different business use cases, even in absence of running a test.

This guide walks you through setting up themes, routing users to the correct variant, and analyzing your results.

Create the themes you’d like to test

Open up Ordergroove and go to the Subscription Manager page. Create an “A” Theme and a “B” Theme you’d like to test. Typically, the "A" theme (control) will be your live theme, and your "B" theme (variant) will be a draft theme. You can choose an existing draft theme, duplicate one of your existing themes, or create a new one.

Once you’ve determined which draft theme will be your variant, click on it and note the ID that appears in the URL. For example, if your theme editor has the URL https://rc3.ordergroove.com/subscriptions/manager/68ed0b19937ea0e734b27e34/, your theme ID will be 68ed0b19937ea0e734b27e34. The live theme will not have an ID (“live” appears in the URL instead).

We recommend updating your draft theme's name to make it clear that it's currently being used in an A/B test.

Link Users to the Correct Subscription Manager Variant

Once you have your variant theme ID, find where you are currently linking users to the Subscription Manager. The location and URL will depend on your e-commerce platform. On Shopify, most merchants have the theme file ordergroove_subscription_interface_link.liquid, which links to /apps/subscriptions/manage/. If you have the “Ordergroove SMI” block instead, reach out to us for instructions.

Next, you need some way to bucket your users into an "A" variant and a "B" variant. This may involve using a third-party A/B testing platform or writing your own bucketing logic. Ideally, you should ensure that the same user gets assigned the same variant for the duration of their session, either by using a cookie or by making the variant bucketing deterministic based on some user attribute (e.g., user ID).

Once the user has been assigned to either the control or variant, update the Subscription Manager URL to link the user to the corresponding Subscription Manager theme. The URL will stay the same for users who were assigned theme "A" (the currently live theme). For users who were assigned theme "B", add a main_theme query parameter to the Subscription Manager URL. This parameter will be the same as the variant theme ID you recorded in step 1.

For instance, let's say users assigned to the "B" variant should get the draft theme with ID 68ed0b19937ea0e734b27e34. Assuming /apps/subscriptions/manage/ is the URL of your Subscription Manager, then your link to that page should be updated as follows:

  • “A” users: /apps/subscriptions/manage/
  • “B” users: /apps/subscriptions/manage/?main_theme=68ed0b19937ea0e734b27e34

Here's what that could look like in Shopify, though note that the variant bucketing in this example is non-deterministic and will be different on every page load:

{% if customer %}
  {% assign random_seed = 'now' | date: '%N' %}
  {% assign bucket_check = random_seed | modulo: 2 %}
  
  {% if bucket_check == 0 %}
    {% assign subscription_url = '/apps/subscriptions/manage/' %}
  {% else %}
    {% assign subscription_url = '/apps/subscriptions/manage/?main_theme=688a5aa2d793723e6b8b27e9' %}
  {% endif %}
  
  <p>
    <a href="{{ subscription_url }}" 
       class="btn btn--small">
      Manage Your Subscriptions
    </a>
  </p>
{% endif %}

Finish Your A/B Test

Once you are finished running the A/B test, restore your Subscription Manager link to the way it was before, and it will start loading the live theme again. For help analyzing the results of your A/B test, reach out to your Ordergroove representative with the themes you tested and when the test was live.

Updated 9 days ago