A/B Tests

Watch Video play_circle

ByteBrew comes equipped with the most advanced A/B Testing platforms for games and apps where you can put any hypothesis to the test by splitting your users into vairant groups to analyze how your players perform across different groups.

Required to Start using A/B Tests

tag

To start using our A/B Testing dashboard on ByteBrew, you must first have initialized the ByteBrew SDK in your game or app and implemented remote configs in your game code where ever you'd want to set up tests in your game. To learn how to integrate Remote Configs in your games, go to the remote configs section in your platform's SDK integration page.

Create A/B Tests

tag

To setup an A/B Test dashboard on the ByteBrew dashboard, follow the steps below:

1

Select "Create A/B Test" in the top right corner of the dashboard to open the A/B Test creation panel.

2

Follow the rest of the staged prompts below under this section for setting up your A/B test's Details, Filters, Groups and Goals.

Setup A/B Test Details

The first stage of setting up an A/B Test is inputting your A/B Test Details. Follow the steps below to setup your test's details:

1

Test Name: Choose a name for the test.

2

Test ID: Add the Unique ID of the test. This will only be used in your dashboard to identify the test in your filters panel.

3

Start Date: Set your desired Start Date for the test to begin.

4

End Date (Optional): Optionally, you can also select a determined end date for the test.

4

Select "Next" to move on to setting pu your A/B Test Filters.

Setup A/B Test Filters

The second stage of setting up an A/B Test is inputting your A/B Test Filters. Follow the steps below to setup your test's filters:

1

Select User Filters (Optional): Choose any user filters you'd like the A/B Test use. You have the option to choose:

Country Filter Choose users from certain countries to be used in the A/B Test.
Device Filter Select which devices you'd like to include in the A/B Test.
Language Filter Select languages on user's devices you'd like to be apart of the A/B Test.
OS Filter Choose OS versions you'd like the test to filter through.
2

New User Percentage: Use the dial to choose what percentage of new user's you'd like to be eligable for the A/B Test. Default value is set to 25%.

Setup A/B Test Groups

The third stage of setting up an A/B Test is inputting your A/B Test Groups. Follow the steps below to setup your test's groups:

1

Add Variant Test Groups: Add user groups for your test equal to the number of configs you want to distribute in the split test experiment. By default, ByteBrew pre-creates (1) Control Group and (1) Group A. To add another group, click the "Add Group" button. Each time that you add a test it will split your user's share distribution evently. (Ex. 3 Groups: Control Group, Group A, and Group B will each have 33% user distribution)

2

Input Config Keys: After creating the groups for the A/B Test, input the Key Value pairs into each group you've created. A/B Tests will be distributed to users like a remote config based on the group the user falls under. Follow the rest of the staged prompts below under this section.

Additional Advanced Option: One advanced feature on ByteBrew A/B Tests is the ability to set multiple sub-configs under a single Group. This lets you can have a list of different features tied to Key Value pairs that are being tested in one group. To illustrate, if you wanted to test the engagement of different reward screen end cards in your game you could set up two groups with multiple sub-configs that change several attributes of the end card in each group like: banner color, reward amount, button color, positive re-inforcement phrasing, and more.

Setup A/B Test Goals

The fourth stage of setting up an A/B Test is inputting your A/B Test Goals. Follow the steps below to setup your test's goals:

1

Goal Metrics: Choose your Goal Metrics for your A/B Test.

2

Track Extra Metrics: Add extra metrics you'd like to track during the test. You can select as many metrics as you'd like to track. If you forget to add an extra metric during the A/B Test creation process, ByteBrew allows you to add that metric to the test even while the test is live.

See the list below for the full list of available metrics to track during the A/B Test:

fiber_manual_record
Retention Day 1
fiber_manual_record
Retention Day 7
fiber_manual_record
Retention Day 30
fiber_manual_record
Playtime
fiber_manual_record
Session
fiber_manual_record
Session Length
fiber_manual_record
ARPPU
fiber_manual_record
ARPDAU
fiber_manual_record
Ad Events
fiber_manual_record
Purchases
fiber_manual_record
Custom Events

Choosing Custom Events as a goal will allow you to use any Custom Events and subparameter that you track in your game as metric conversion event to analyze and compare how each group achieves that specific event.

Manage A/B Tests

tag

After you've created an A/B Test you have the ability to manage or edit settings under the test, even while a test is running. To find the editable settings in an A/B Test, go into the test you want to edit and locate the row of buttons at the top of the panel such as: Edit Details, Edit Configs, Stage Rollouts, and Full Rollouts. Learn how to use each of these setting panels in the sections below.

Edit Details

To edit your Test's details click the "Edit Details" button inside the test to see the re-configurable items in the panel. See the list of editable fields in the list below:

fiber_manual_record
Name of the Test: You are able to edit the name of the test on your dashboard under the Edit Details panel.
fiber_manual_record
Goal Metric: The goal metric you've chosen when creating your test is able to be swapped with another metric you want to feature as the goal for your test. To change the goal metric, select another metric under the "Goal Metric" dropdown and save the panel.
fiber_manual_record
Track Extra Metrics: All extra tracked metrics you've selected during your test's creation process can be edited or expanded. To remove an extra metric under the "Edit Details" panel, click the remove icon on the tracked metric name. To add an additional metric to the list your test is tracking click the Trackc Extra Metrics dropdown, choose the metric you want to add and save the panel changes.

Edit Configs

To edit your Test's configs click the "Edit Configs" button inside the test to see the re-configurable items in the panel. See the list of editable fields in the list below:

fiber_manual_record
Edit Group Key/Values: Using the Edit Configs panel, you can edit your Test's existing key/value pairs setup in your variant groups.
fiber_manual_record
Adding Key/Values: If there are additional variables you'd like to add to an existing test group, you can add additional Sub-Configs to an existing group in the Edit Configs panel.

Staged Rollouts

Staged Rollouts is a powerful feature exclusive to the ByteBrew platform that lets you validate your split tests results enabling you to depricate all under performing variant groups in your test and promote a single winning variant group to run against the control group. To stage rollout, go into your running test and select the "Stage Rollouts" button to open the Staged Rollouts panel and follow the steps below:

1

Click the group dropdown and locate the group you'd like to promote

2

Optional: Choose a new, new user percentage for the test. This is not required, but the option is listed in the event that you want to introduce a larger user population to your running test.

3

Select the Stage button to apply your changes.

Full Rollouts

Full Rollouts give the ability to send a global update to all your players with a chosen winning variant. To rollout a completed test to your players follow the steps below:

1

Open the Full Rollout Panel by selecting the "Full Rollout" button in your test.

2

Once inside the Full Rollouts panel, select the group you'd like to rollout under the Variant dropdown.

3

Select Save in the panel to apply the global rollout.

A/B Test Statuses

tag

There are four different tagged statuses for A/B Tests. See table below for their descriptions:

Active Active A/B Tests are running and distributing the test to users.
Pause Paused A/B Tests will stop distributing the split test to new users. However, users who currently are apart of the test will still get delivered the group configs. Any paused test can be resumed and made active again
Archive Archived A/B Tests will stop distributing the test to all users, including to users who are currently apart of the test. In an Archived state, you will still be able to see the user data from the test in the dashboard and filters. Archived test cannot be resumed.
Delete Deleted A/B Tests will be fully removed from the entire game and also removed from dashboard filters. Deleting a test cannot be undone.

FAQs

To start using our A/B Testing dashboard on ByteBrew, you must first have initialized the ByteBrew SDK in your game or app and implemented remote configs in your game code where ever you'd want to set up tests in your game.

We only display all the metrics you want to track in your A/B Test; giving you the power to determine who the winner of your test is.

Yes, you can use any Custom Events from your game as a tracked metric when setting up your A/B Test.

Staged Rollouts are a great way to verify your tests results by rolling out 1 of your variant groups against your control group and pausing any other groups in your test.

Yes, you can choose another group to re-rollout by heading back to the "Staged Rollouts" settings panel and choosing the new group you'd like to rollout. However, you cannot restore the A/B test to it's original format prior to staging.

A/B Tests can be resumed if you had paused the test previously. Archived and Deleted A/B Test can not be resumed.

You will not be able to set up a test using a config key that is already being utilized in a running remote config. To use a Key already in-use, end the remote config with the key.

Need some help?

Join the ByteBrew Discord community to chat with our team and other developers on the platform.