Control Groups – Mobile App Testing Experiments

Have you ever wondered how you can accurately attribute user activity or even increased revenue to a Pulsate campaign? Maybe you sent a campaign and saw an increase in user activity or transactions but were wondering if that would have happened anyway without the campaign being sent. If any of that sounds familiar then control groups are the perfect tool for you.

WHAT IS A CONTROL GROUP?

A control group is typically a small group of users that you do not send a campaign to so that you can compare the attribution rate of the campaign between the control group and the larger group of users who did receive the campaign.

One of the problems that mobile marketers often face is that it is difficult to accurately attribute a user action to that user receiving a campaign. A user may have just received a campaign for one of your products and purchased that product but how do you know that they wouldn’t have purchased that product anyway?

If you use a control group, you can compare the behavior of users who received the campaign with those in the control group who did not receive it and accurately determine the level of attribution for the campaign.

Control groups work best when they are used on campaigns that are sent to a large group of users. In Pulsate, we recommend that campaigns are sent to at least 10,000 users to ensure that the control group results remain statistically relevant.

HOW DO CONTROL GROUPS WORK?

For most campaigns, we do not know how many users will receive the campaign when it is sent because new users are being added to the segment for that campaign or if it is a beacon or geofence campaign, new eligible users will enter the specific locations where the beacon or geofence are located during the period of the campaign.

For that reason, when a control group is enabled for a campaign, Pulsate will choose a random sample of users to be added to that control group and not receive the campaign. If a control group is set to 10% of users that are eligible to receive a campaign, this means that every campaign recipient has a one in ten chance of being added to the control group. This also means that the control group may not be exactly 10% of the recipients because we did not know how many users would receive the campaign until it was finished.

USING CONTROL GROUPS

When you send a Pulsate campaign you can select whether or not to add a control group in the Targeting section of the Campaign Builder. To enable a control group, tick the checkbox beside Enable Control Group.

Enabling a Control Group

Once the control group is selected, you can decide what percentage of campaign recipients should be added to the control group. For most use cases, a control group of 10% is a good choice.

Specifying the size of a Control Group

When you have enabled and specified your control group, a block will be added to the Targeting simulator displaying the percentage of users that will be added to the control group.

The control group

Before you send the campaign, you will have an opportunity to check that the control group has been accurately set on the Campaign Review page.

Campaign review screen

When a campaign is finished, you can export a campaign report to find out what the difference in behavior was between the users in the control group and the users who did receive the campaign. To export a campaign report, go to the campaign list screen and click on the dropdown to the right of the campaign you sent.

An email will be sent to the email address of the Admin that is currently signed into Pulsate. The email will contain links to CSV files of both the users who received the campaign and those who did not so that you can accurately compare the attribution rate between both groups.