medium.marathonlabs.io Open in urlscan Pro
162.159.153.4  Public Scan

Form analysis
0 forms found in the DOM

Text Content

Open in app

Sign up

Sign In

Write


Sign up

Sign In




I WANT TO RUN ANY NUMBER OF ANDROID UI TESTS ON EACH PR. EXISTING SOLUTIONS.
PART V

Eugene Matsyuk

·

Follow

Published in

MarathonLabs

·
9 min read
·
1 day ago

397



Listen

Share

Series of articles:

 1. I want to run any number of Android UI tests on each PR. Your actions? Part
    I
 2. I want to run any number of Android UI tests on each PR. Cost. Part II
 3. I want to run any number of Android UI tests on each PR. Existing solutions
    (BrowserStack, Firebase Test Lab). Part III
 4. I want to run any number of Android UI tests on each PR. Existing solutions
    (SauceLabs, AWS Device Farm, LambdaTest, Perfecto Mobile). Part IV
 5. I want to run any number of Android UI tests on each PR. Existing solutions
    (emulator.wtf, Marathon Cloud). Part V

Hello everyone!

This is the last article in a series focused on finding efficient solutions for
running UI tests on each PR with ease. I will explore the two most promising and
attractive options: emulator.wtf and Marathon Cloud. Finally, I will draw
conclusions based on the findings of this series.


Photo by ZQ Lee on Unsplash


EMULATOR.WTF


SUPPORTED PLATFORMS

emulator.wtf is focused on only Android native UI tests. No Browser, no iOS, no
Appium. Only Android.


INTERFACE

The solution solely offers a CLI tool, with a setup that is clearly explained in
detail within the documentation. Configuration is only delivered through CLI
parameters and not through files. The commands are blocking, which simplifies
the usage of the CLI in CI/CD. However, it would be helpful if emulator.wtf
provided a runtime of test execution to view the overall status of test
execution.




STABILITY

It efficiently utilizes Orchestrator and clears app states before every test if
you want, among other things.

emulator.wtf provides a useful feature for retrying flaky tests using the “ —
num-flaky-test-attempts” parameter. It’s important to read about this parameter
carefully. As a user, if I set “ — num-flaky-test-attempts 3,” I expect the
failed test to be repeated a maximum of three times until success. However, the
current solution repeats the test an additional three times always, even if the
first retry was successful.



Here’s some good news: if a test that previously failed turns green after a
retry, the entire test run is considered a success:



But some things are still unclear to me. I have developed an application with a
unique feature — 15% of its tests are flaky, which means a random function
simulates some backend problems or other issues. I found emulator.wtf to be the
first solution that allows retries, unlike previous options. After running the
APK on emulator.wtf with the parameter “- num-flaky-test-attempts 2,” I obtained
the following results:



Have a look at “test 7”. I’m having trouble understanding why the solution
starts new attempts when the first test passed. Next, even after the second
attempt failed, the third attempt was successful. In the end, the test was
marked as a failure and the entire run becomes failed. It looks weird.

Although the solution is functional, its stability is not optimal when compared
to other options previously considered. During tests that involved 10 parallels,
almost all attempts encountered at least one failure. Nevertheless, this issue
can be resolved by rerunning flaky tests, with only minimal impact on overall
time.


TIME AND SCALABILITY

One of the major benefits of using emulator.wtf is the time it saves. The
platform offers a variety of sharding strategies, starting with a random
distribution of tests and moving toward smart distribution using historical
data. I have not come across any information regarding the maximum limit of
shards allowed. So, I guess it could be a very big number.

Let’s have a look at the numbers produced by emulator.wtf:

10 parallels:

 * Uploading app apk: 00:00:03.75
 * Uploading test apk: 00:00:00.35
 * Total time: 00:02:01

5 parallels:

 * Uploading app apk: 00:00:03.75
 * Uploading test apk: 00:00:00.35
 * Total time: 00:03:57

Amazing! I was able to experiment with various test sets and models thanks to a
generous 100 hours of free time. During my tests, I noticed that the total time
for the same test set sometimes varied greatly depending on the model used. For
instance, I ran 300 tests on both the NexusLowRes (version 29) and Pixel2
(version 29) models using 13 parallel emulators. Surprisingly, the total time
for the NexusLowRes version was only 8 minutes and 59 seconds, while the Pixel2
version took 17 minutes and 31 seconds. However, most of the time, the total
time was close to the minimum. It's worth noting that we may have a large number
of simultaneous runs on each pull request, so some runs may take longer than
usual.


REPORTS

The dashboard has a minimalistic design but includes all the necessary
information for analysis, such as logs, videos, and stack traces.



Furthermore, emulator.wtf offers the opportunity to analyze the stacktrace using
chatGPT technology behind the scenes.




COST

At emulator.wtf, developers pay for spent hours with Unlimited concurrent
emulators:



All price details are available here and here.


MARATHON CLOUD

Marathon Cloud is built on the widely-used OpenSource UI test runner, Marathon
Runner. All the functionalities offered by Marathon Runner are automatically
supported by Marathon Cloud. To know more about the features of Marathon Runner,
you can refer to the comprehensive documentation.


SUPPORTED PLATFORMS

Marathon Cloud specializes in mobile testing for Android and iOS using only
native tools. At the moment, Appium is not included in Marathon's services.


INTERFACE

As a user, you have the choice between running your tests through the UI
Dashboard or the CLI tool.

Let's discuss how to utilize the UI. The initial element that catches the user's
attention is the Dashboard:



Starting a new run is a straightforward process. Simply click on the appropriate
button “New Run” located in the top right corner and select the type of test you
are searching for:



The CLI tool operates in a blocking mode and provides runtime support. A
screenshot demonstrating its appearance is shown below



The documentation provides clear instructions on how to interact with Marathon
Cloud using CLI methods.


STABILITY

Marathon Cloud prioritizes overall stability. The solution conducts tests
exclusively on emulators and turns on Marathon Runner settings like "Clear an
app's state between test executions" to ensure consistent performance.

I would like to delve into the topic of retries in greater detail. As previously
mentioned, flaky tests can be a persistent issue in UI testing across various
platforms. Unfortunately, the cause of these tests failing could be anything,
even something as unpredictable as the phase of the moon. Therefore, it is
crucial to have a straightforward method for retrying failed tests.

There is only one tool that fully supports the retry mechanism without any
unclear things, and that's Marathon Cloud. With Marathon, you can set the number
of retries for failed tests (not shards or devices, just tests!), as well as the
maximum number of failed tests allowed. Marathon also collects statistics on
test runs, including their success rate and duration, and uses this information
to optimize future runs. For instance, if a particular test has a low success
rate (let's say below 75%), Marathon will include two runs of the same test in
the test suite to save time on reruns.



Read more in the Retries and Flakiness chapters of the Marathon documentation.

Regarding the stability of Marathon Cloud, I am unable to definitively state
that it is a flawless solution. Sometimes, a user may encounter unreliable tests
caused by infrastructure problems, as with previous solutions. Fortunately, this
issue is being addressed through automatic retries. Even runs where 15% of tests
experience random runtime exceptions ultimately yield successful outcomes:



I want to bring to your attention that Marathon Cloud is the only platform
capable of managing a test suite consisting of 15% flake tests in our
experiments.


TIME AND SCALABILITY

I need to notice that only Marathon Cloud provides a “15-minute” promise about
the time execution of any number of tests (50, 1000, 4000, doesn’t matter) and
frees a user of necessity to predict the appropriate number of parallels to fit
in some particular time frame and other params which are set up by default.
That’s why the number of parallels may vary from one run to another.

Have a look at the numbers produced by Marathon Cloud (50_0 suite):

 * Run Time: 00:04:03
 * Report Time: 00:00:18

The Dashboard doesn't display the time taken for uploading apks.


REPORTS

Marathon Cloud utilizes Allure reports, which offer a highly convenient method
for showcasing test reports.




There are various tools available to analyze failures such as stacktraces, logs,
and videos for each test.


COST

The prices for Marathon Cloud are presented in this link and in the picture
below:



Just a quick reminder that the "Cloud" option provides the promise of executing
an unlimited number of tests for 15 minutes.


FINAL SUMMARY

I'm almost finished with the fifth article comparing different cloud solutions
for running UI tests. I'm now ready to share my final conclusions.

When searching for a cloud solution to run UI tests on each PR, it's important
to establish specific requirements. These criteria will help evaluate and
compare various options. The first article provides a comprehensive overview of
this subject. However, it's worth noting that the set of requirements may differ
for only night or prerelease runs for example.

In this series of articles, I considered the next solutions: Marathon Cloud,
Firebase Test Lab, BrowserStack, emulator.wtf, SauceLabs, AWS Device Farm,
Perfecto Mobile, LambdaTest.

When comparing options, the initial factor to consider is cost. Typically, there
are two pricing models to choose from: "Pay for a device per month" or "Pay for
spent hours/minutes". For PR runs, opting for the “Pay for spent hours/minutes”
policy tends to be the most cost-effective choice. Detail of comparing is
available in the second article. The “Pay for spent hours/minutes” policy is
suggested by only 5 solutions: Marathon Cloud, emulator.wtf, SauceLabs, Firebase
Test Lab, and AWS Device Farm.

Further, it's important to take into account factors such as Time, Scalability,
and Stability. However, AWS Device Farm falls short in these areas due to its
lack of emulators and parallelization, as well as its high cost of $10.2 per
hour. While SauceLabs is a decent solution, it may not be the best choice in
terms of time and price (at $4.8 per hour). Both solutions are described in this
article. As a result, only three options remain: Marathon Cloud, emulator.wtf,
and Firebase Test Lab.

To run UI tests successfully, managing flaky tests is essential. This may
involve implementing basic retry mechanisms, scheduling runs that account for
flaky tests, and other things. Only Marathon Cloud offers these features
seamlessly, without requiring developers to put in extra effort. Also, Marathon
Cloud offers the same functionalities for iOS, allowing it to run on iOS
Simulators at the same price as Android. It means you can use a single solution
with consistent settings for both Android and iOS at the same time.


Android
Ui Testing
Infrastructure
Espresso
Firebase Test Lab


397

397



Follow



WRITTEN BY EUGENE MATSYUK

1.3K Followers
·Editor for

MarathonLabs

Co-Founder at marathonlabs.io | Google Developer Expert

Follow




MORE FROM EUGENE MATSYUK AND MARATHONLABS

Eugene Matsyuk

in

MarathonLabs


I WANT TO RUN ANY NUMBER OF ANDROID UI TESTS ON EACH PR. EXISTING SOLUTIONS.
PART IV


HAVE A LOOK AT SAUCELABS, AWS DEVICE FARM, LAMBDATEST, AND PERFECTO MOBILE

8 min read·May 31

153





Eugene Matsyuk

in

MarathonLabs


I WANT TO RUN ANY NUMBER OF ANDROID UI TESTS ON EACH PR. EXISTING SOLUTIONS.
PART III


IN THIS ARTICLE, I AM GOING TO TELL ABOUT THE MOST POPULAR SOLUTIONS TO RUN UI
TESTS ON ANDROID — BROWSERSTACK, AND FIREBASE TEST LAB.

10 min read·May 18

225





Anton Malinskiy

in

MarathonLabs


MARATHON 0.8.0 RELEASE NOTES


A LOT HAS CHANGED SINCE THE LAST RELEASE IN SEP 2022. MARATHONLABS HAS SINCE
GROWN INTO SOMETHING BIGGER

5 min read·Mar 14

257





Eugene Matsyuk

in

ProAndroidDev


I WANT TO RUN ANY NUMBER OF ANDROID UI TESTS ON EACH PR. COST. PART II


WE WILL EXPLORE AND EVALUATE THE VARIOUS SOLUTIONS TO RUN UI TESTS ON PRS WHICH
ARE AVAILABLE IN THE MARKET.

5 min read·May 6

243

1



See all from Eugene Matsyuk
See all from MarathonLabs



RECOMMENDED FROM MEDIUM

Shubham Garg

in

ProAndroidDev


EFFECTIVE TESTING WITH ANDROID TEST ONLY MODULES


WE HAVE ALL HEARD ABOUT HOW MODULARIZING YOUR ANDROID APP INTO DIFFERENT MODULES
IS BENEFICIAL FOR CODE MAINTAINABILITY AS WELL AS BUILD…

5 min read·Jan 14

60

2




Badea Mihai Bogdan


GUIDE TO E2E TESTING WITH MAESTRO ON ANDROID


AUTOMATED END TO END (E2E) TESTING ON MOBILE APPLICATIONS IS A POWERFUL TOOL
THAT CAN HELP YOU ENSURE THE QUALITY AND RELIABILITY OF YOUR…

3 min read·Jan 14

2






LISTS


NOW IN AI: HANDPICKED BY BETTER PROGRAMMING

248 stories·11 saves


STAFF PICKS

357 stories·119 saves


Robert Martin


MOBILE APP PERFORMANCE TESTING WITH APPIUM


HARDLY ANY MODERN-DAY ORGANIZATION WOULD THINK OF FUNCTIONING WITHOUT A MOBILE
APPLICATION. OVER 60% OF BUSINESSES HAVE AN APP OR ARE IN…

4 min read·Jun 10

32





Lucky Senjaya Darmawan

in

julotech


IMPLEMENTING DYNAMIC FEATURE MODULES IN OUR ANDROID APP


SHARING TIPS ON CODE ARCHITECTURE, UX, AND TESTING BASED ON OUR EXPERIENCE

6 min read·Mar 27

115





Fran Soto

in

BestSecret Tech


CREATING A SPLIT FLAP DISPLAY IN COMPOSE


WHAT IS A SPLIT FLAP

8 min read·Jun 12

23

1




Abhishek Srivastava


ANDROID SCREENSHOT TESTING FOR UI


(AUTOMATED TESTING THAT HELPS PREVENT VISUAL REGRESSIONS IN ANDROID APPS)

9 min read·Feb 26

7




See more recommendations

Help

Status

Writers

Blog

Careers

Privacy

Terms

About

Text to speech

Teams

To make Medium work, we log user data. By using Medium, you agree to our Privacy
Policy, including cookie policy.