A/B Tests

Watch Video play_circle

ByteBrew comes equipped with the most advanced A/B Testing platforms for games and apps where you can put any hypothesis to the test by splitting your users into vairant groups to analyze how your players perform across different groups.

reviewsPopular Use Case Use ByteBrew A/B Tests to measure where opportunities are to increase player engagement by being able to track not only how different buckets of users retain in your game, but also how they reach your game's vital conversion events.

Required to Start using A/B Tests

tag

To start using our A/B Testing dashboard on ByteBrew, you must first have initialized the ByteBrew SDK in your game or app and implemented remote configs in your game code where ever you'd want to set up tests in your game. To learn how to integrate Remote Configs in your games, go to the remote configs section in your platform's SDK integration page.

notification_importantImportantA/B Tests utilize remote configs to operate. If you plan on using A/B Tests in your games, then make sure to implement remote configs during the SDK integration.

infoNoteBest practice for implementing Remote Configs is to implement them in your game code where ever you think you might want to send update throughout your game's lifespan (Ex. Enemy Spawn Counts, Bonus Reward Amounts, Daily Items Sales, Ad Placements, etc.). It's always better to implement more Configs in your game code and not need to use them then to realize later that you should have implemented them at the stat.

Create A/B Tests

tag

To setup an A/B Test dashboard on the ByteBrew dashboard, follow the steps below:

Select "Create A/B Test" in the top right corner of the dashboard to open the A/B Test creation panel.

Follow the rest of the staged prompts below under this section for setting up your A/B test's Details, Filters, Groups and Goals.

Setup A/B Test Details

The first stage of setting up an A/B Test is inputting your A/B Test Details. Follow the steps below to setup your test's details:

Test Name: Choose a name for the test.

Test ID: Add the Unique ID of the test. This will only be used in your dashboard to identify the test in your filters panel.

Start Date: Set your desired Start Date for the test to begin.

End Date (Optional): Optionally, you can also select a determined end date for the test.

Select "Next" to move on to setting pu your A/B Test Filters.

Setup A/B Test Filters

The second stage of setting up an A/B Test is inputting your A/B Test Filters. Follow the steps below to setup your test's filters:

Select User Filters (Optional): Choose any user filters you'd like the A/B Test use. You have the option to choose:

infoNoteYou will not be able to edit the filter settings of your A/B Test once it is created. To update the filter settings of a test, you must create a new test.

Country Filter	Choose users from certain countries to be used in the A/B Test.
Device Filter	Select which devices you'd like to include in the A/B Test.
Language Filter	Select languages on user's devices you'd like to be apart of the A/B Test.
OS Filter	Choose OS versions you'd like the test to filter through.
Build Version Filter	Choose Build Versions you'd like the test to use.

New User Percentage: Use the dial to choose what percentage of new user's you'd like to be eligable for the A/B Test. Default value is set to 25%.

infoNoteIt is recommended to set this value at default to allow you to get a healthy set of users flowing into the test, while being able to keep room for any coinciding A/B Tests you might be running or choose to run in the future.

Setup A/B Test Groups

The third stage of setting up an A/B Test is inputting your A/B Test Groups. Follow the steps below to setup your test's groups:

notification_importantImportant If you are using a config key in a running remote config, you can not use that same config key to run an A/B test. If you need start running an A/B Test with a config key in a live remote config, then just remove the running remote config to use it in an A/B Test.

Add Variant Test Groups: Add user groups for your test equal to the number of configs you want to distribute in the split test experiment. By default, ByteBrew pre-creates (1) Control Group and (1) Group A. To add another group, click the "Add Group" button. Each time that you add a test it will split your user's share distribution evently. (Ex. 3 Groups: Control Group, Group A, and Group B will each have 33% user distribution)

Input Config Keys: After creating the groups for the A/B Test, input the Key Value pairs into each group you've created. A/B Tests will be distributed to users like a remote config based on the group the user falls under. Follow the rest of the staged prompts below under this section.

notification_importantImportant Remember your config Keys. These Keys will be used when retrieving the remote configs in the SDK for the A/B Test to operate effectively.

Additional Advanced Option: One advanced feature on ByteBrew A/B Tests is the ability to set multiple sub-configs under a single Group. This lets you can have a list of different features tied to Key Value pairs that are being tested in one group. To illustrate, if you wanted to test the engagement of different reward screen end cards in your game you could set up two groups with multiple sub-configs that change several attributes of the end card in each group like: banner color, reward amount, button color, positive re-inforcement phrasing, and more.

Setup A/B Test Goals

The fourth stage of setting up an A/B Test is inputting your A/B Test Goals. Follow the steps below to setup your test's goals:

Goal Metrics: Choose your Goal Metrics for your A/B Test.

Track Extra Metrics: Add extra metrics you'd like to track during the test. You can select as many metrics as you'd like to track. If you forget to add an extra metric during the A/B Test creation process, ByteBrew allows you to add that metric to the test even while the test is live.

lightbulbTip ByteBrew gives you the ability to track an entire host of vital gameplay metrics. Develop a great Goal Metric strategy in your A/B Tests by selecting multiple metrics to track in your test.

See the list below for the full list of available metrics to track during the A/B Test:

fiber_manual_record

Retention Day 1

fiber_manual_record

Retention Day 7

fiber_manual_record

Retention Day 30

fiber_manual_record

Playtime

fiber_manual_record

Session

fiber_manual_record

Session Length

fiber_manual_record

ARPPU

fiber_manual_record

ARPDAU

fiber_manual_record

Ad Events

fiber_manual_record

Purchases

fiber_manual_record

Custom Events

Choosing Custom Events as a goal will allow you to use any Custom Events and subparameter that you track in your game as metric conversion event to analyze and compare how each group achieves that specific event.

infoNoteYou can have up to 2 different conversion events being tracked in any individual A/B Test.

Manage A/B Tests

tag

After you've created an A/B Test you have the ability to manage or edit settings under the test, even while a test is running. To find the editable settings in an A/B Test, go into the test you want to edit and locate the row of buttons at the top of the panel such as: Edit Details, Edit Configs, Stage Rollouts, and Full Rollouts. Learn how to use each of these setting panels in the sections below.

Edit Details

To edit your Test's details click the "Edit Details" button inside the test to see the re-configurable items in the panel. See the list of editable fields in the list below:

fiber_manual_record

Name of the Test: You are able to edit the name of the test on your dashboard under the Edit Details panel.

fiber_manual_record

Goal Metric: The goal metric you've chosen when creating your test is able to be swapped with another metric you want to feature as the goal for your test. To change the goal metric, select another metric under the "Goal Metric" dropdown and save the panel.

fiber_manual_record

Track Extra Metrics: All extra tracked metrics you've selected during your test's creation process can be edited or expanded. To remove an extra metric under the "Edit Details" panel, click the remove icon on the tracked metric name. To add an additional metric to the list your test is tracking click the Trackc Extra Metrics dropdown, choose the metric you want to add and save the panel changes.

Edit Configs

To edit your Test's configs click the "Edit Configs" button inside the test to see the re-configurable items in the panel. See the list of editable fields in the list below:

fiber_manual_record

Edit Group Key/Values: Using the Edit Configs panel, you can edit your Test's existing key/value pairs setup in your variant groups.

fiber_manual_record

Adding Key/Values: If there are additional variables you'd like to add to an existing test group, you can add additional Sub-Configs to an existing group in the Edit Configs panel.

infoNoteUpdating the configs in a running test will automatically be applied to all users participating in the test.

Staged Rollouts

Staged Rollouts is a powerful feature exclusive to the ByteBrew platform that lets you validate your split tests results enabling you to depricate all under performing variant groups in your test and promote a single winning variant group to run against the control group. To stage rollout, go into your running test and select the "Stage Rollouts" button to open the Staged Rollouts panel and follow the steps below:

Click the group dropdown and locate the group you'd like to promote

Optional: Choose a new, new user percentage for the test. This is not required, but the option is listed in the event that you want to introduce a larger user population to your running test.

Select the Stage button to apply your changes.

infoNoteIf you only have two (2) groups created in your game, then you will not be able to stage rollout a group.

Full Rollouts

Full Rollouts give the ability to send a global update to all your players with a chosen winning variant. To rollout a completed test to your players follow the steps below:

Open the Full Rollout Panel by selecting the "Full Rollout" button in your test.

Once inside the Full Rollouts panel, select the group you'd like to rollout under the Variant dropdown.

Select Save in the panel to apply the global rollout.

notification_importantImportantLaunching a Full Rollout will send the updated group to all yours in your game. This will end the test and convert the winning group's config key/value pair into a remote config on your Remote Configs dashboard.

A/B Test Statuses

tag

There are four different tagged statuses for A/B Tests. See table below for their descriptions:

Active	Active A/B Tests are running and distributing the test to users.
Pause	Paused A/B Tests will stop distributing the split test to new users. However, users who currently are apart of the test will still get delivered the group configs. Any paused test can be resumed and made active again
Archive	Archived A/B Tests will stop distributing the test to all users, including to users who are currently apart of the test. In an Archived state, you will still be able to see the user data from the test in the dashboard and filters. Archived test cannot be resumed.
Delete	Deleted A/B Tests will be fully removed from the entire game and also removed from dashboard filters. Deleting a test cannot be undone.

FAQs

What is required to start using A/B Tests?

Do you tell me who the winner of the A/B Test is?

We only display all the metrics you want to track in your A/B Test; giving you the power to determine who the winner of your test is.

Can I use Custom Events in A/B Tests?

Yes, you can use any Custom Events from your game as a tracked metric when setting up your A/B Test.

What is a Staged Rollout?

Staged Rollouts are a great way to verify your tests results by rolling out 1 of your variant groups against your control group and pausing any other groups in your test.

Can I re-stage rollout a different variant group if I've already stage rolled out a different one?

Yes, you can choose another group to re-rollout by heading back to the "Staged Rollouts" settings panel and choosing the new group you'd like to rollout. However, you cannot restore the A/B test to it's original format prior to staging.

Can I resume an A/B Test?

A/B Tests can be resumed if you had paused the test previously. Archived and Deleted A/B Test can not be resumed.

Can I create an A/B Test using the same Key being utilized in a live remote config?

You will not be able to set up a test using a config key that is already being utilized in a running remote config. To use a Key already in-use, end the remote config with the key.

help How long does it take for my Configs in my A/B Test to deliver to my players? A/B Tests deliver configs using your Remote Configs integration. Remote Configs delivered in milliseconds to your players after the ByteBrew SDK is initialized.