Handling A/B testing

It is well established that A/B testing is a core strategy in mobile development.

However, Waldo’s primary mission is to reliably replayreplay - A replay is the evaluation of a test in a particular context: A specific device configuration (for example, “iPhone X on iOS 13.5 in French”), a specific application build, or a specific point in time. and validate the behavior of your app by validating assertions as it executes your user flows. This requires your app to behave deterministically. This simply means that your app must perform identically every time a test is runrun - A run is a set of replays performed concurrently. Given a set of one or more tests and a set of one or more device configurations, each test is evaluated against each device configuration using a specific application build..

An active A/B test in your app complicates that mission by providing for more than one “acceptable” behavior. In some cases, it may even render a testtest - A test is a flow combined with a set of assertions that define the criteria for acceptance. impossible to replay. For example, if your app has two variations:

  • Variation A: ask the user to register first, and then ask them about their preferences
  • Variation B: ask the user about their preferences first, and then ask them to register

Assuming that A/B testing is active, it is easy to see that the behavior of your app is no longer deterministic.

Therefore, we strongly encourage you to disable A/B testing on all builds that you upload to Waldo.

If you want to test your variations, it is straightforward to do so by creating an explicit test for each variation: for example, “Signup – Variation A” and “Signup – Variation B”. We recommend an approach where you set a unique template variable for each email address you sign up with in your tests. For instance, you can set variationA{HASH}@yourdomain.com and variationB{HASH}@yourdomain.com.

See Variables for more details on setting string patterns in your tests.