Understanding your Results

The “Runs” page is a key area of the Waldo dashboard and where you'll spend most of your time. This is where you can see all the past runs for each of the different versions of your app, as well as where you can dive into each.

Replay status

During a run, your test replays on multiple device configurations. Each replay completes with a specific status.

No matter the status of a replay, Waldo provides you with a couple of tools to help you understand what occurred during that replay:

  • Watch recording This tool displays a video recording of your test replaying on the device at actual speed.
  • Download logs This tool fetches the system log generated on the device as your test replayed. It includes only entries directly generated by your app, beginning at app launch. This means that if your test is dependent on other tests, the log includes entries for those dependent tests as well.

For more visit Run status

App crashed

If your test fails to replay on a device configuration because your app crashed, Waldo indicates exactly where in the flow the crash occurred and what interaction caused it.

What can you do?

  • In addition to downloading the system log as previously discussed, you can also download the crash report generated on the device. (If you are targeting Android, the crash report directly included in the system log.) Together, these can help you diagnose what caused your app to crash.
  • You may also want to reproduce the issue locally on one of your own devices. In this case, you should click on the Builds tab and download the corresponding version.

Cannot replay

If your test fails to replay on a device configuration because of the current context, you should examine the replay to help you understand the failure. Typically, the failure is for one the following reasons:

  • Your app’s behavior has changed.
  • There is a bug in your app.
  • It is the first time Waldo has replayed on that device configuration, and Waldo is not able to automatically determine how to correctly replay the flow there.

What can you do?

  • If your app is simply not behaving properly, you can use the same approach as discussed previously under “App crashed” to diagnose the root cause of the problem.
  • If your app has changed, or Waldo cannot determine how to proceed on one step of the flow, or both, you can simply select Update test or directly fix the interaction in the details of the step (See Fixing Interactions for more details). This provides Waldo with another reference recording of your flow under the new conditions. Waldo is smart about which recording to use for each condition.
  • In some cases, the replay happens too quickly (the infamous “wait problem”). Specifically, an element is visible on screen and can be interacted with, but its action depends on your app being in a particular state. In such a case, Waldo may try to tap too soon. For example, let us imagine that your app allows the user to upload a photo, displays the photo as it is uploading, and, once the upload completes, allows the user to tap on the photo to navigate to another screen. During recording, everything works correctly. Upon replay, if the timing is off, the tap occurs too soon on the photo and nothing happens because the upload is not finished. To force Waldo to wait longer on that particular screen, you can simply add an assertion on any element that will only appear after uploading completes (perhaps a checked checkbox, or some informative text). For that, click on the name of the test at the top, and add an assertion there.
    See Assertions to learn more.

Assertions failed

Even if your test replays successfully, Waldo may indicate one or more assertions that do not pass validation.

What can you do?

As in the case with “Cannot replay” discussed previously, you should first determine whether the failed assertion indicates an actual problem (the app is not behaving properly), or that the assertion needs to be updated (or removed entirely).

If you determine that the assertion is outdated, you can click on the Fix assertions button and either:

  • Fix assertion Make sure it now uses a new value. For example, if a button now says “Sign up” instead of “Register”, and that is the expected behavior, you can simply update the value of the text assertion attached to it.
  • (delete) Sometimes an assertion is no longer useful. If you delete it by mistake, you can simply go back to the test definition page and add it again.

Note: when you fix an assertion, it is evaluated in the future against the new state of the flow (the screen on the right-hand side). In particular, in the case of a similarity assertion, the new threshold that you set is compared in the future against that right-hand screen.

For a step-by-step guide visit Fixing Assertions

Dependency error

This status means that one of the dependencies could not be replayed properly. Thus Waldo was not even able to reach the point of attempting the replay of that particular test.

What can you do?

There is not much to do on the test itself. Instead, you should see how to fix the dependency test directly.

In some very rare cases, you might see a test in a dependency error status even if all other tests are passing. This can happen if one of the early tests is flaky. Consider this situation:

  • Sign Up -> Create Message
  • Sign Up -> Go to Settings

To play Create Message and Go to Settings, Waldo has to play the test Sign Up twice. Now imagine if Sign Up is flaky and passes 50% of the time: you might end up with Sign Up as a success (it passed when replaying Create Message), and Create Message as a success (it passed after Sign Up), but Go to Settings has a dependency error because when we attempted to replay Sign Up the second time, there was an issue.

Recording a strong test suite early on is essential to ensuring your dependencies succeed in future runs.

App failure

If we could not replay the test in Waldo for some network reasons (on our side), you will get an App failure error. It can usually be resolved by retrying the replay.