Dave Hunt, Automationeer

State of Performance Test Engineering (H2/2019)

2020-01-06T15:50:42+00:00

It’s a new year, and time for me to post an update on the state of Firefox performance test engineering. The last update was in July 2019 and covered the first half of the year. This update covers the second half of 2019.

Team

The team consists of 9 engineers based in California, Toronto, Montreal, and Romania.

Tests

We currently support 4 frameworks:

Are We Slim Yet (AWSY) - memory consumption by the browser
build_metrics - build times, installer size, and other compiler-specific insights
Raptor - page load and browser benchmarks
Talos - various time-based performance KPIs on the browser

At the time of writing, there are 511 test suites (query) for the above frameworks. This is up from 263 in the H1/2019 report.

The following are some highlights of tests introduced in H2/2019:

Dashboards

In addition to updating health.graphics and related dashboards based on test changes, we have also contributed summary dashboards for media playback performance and power usage.

We have also contributed several improvements to the Perfherder tool used by the performance sheriffs for monitoring and investigating regression and improvement alerts. Some example are listed below:

Ability for sheriffs to assign alerts
Allow users to retrigger from the compare view
New view showing active performance tests
Highlight prioritised alerts for investigation
Show measurement unit in graphs

Hardware

Our tests are running on the following hardware in continuous integration:

38 Moto G5 devices (with two of these dedicated to power testing)
23 Pixel 2 devices (with two of these dedicated to power testing)
16 Acer Aspire 15 laptops (2017 reference hardware)
35 Lenovo Yoga C630 laptops (Windows ARM64)
2 Apple MacBook Pro laptops (used for power testing)

See this wiki page for more details on the hardware used.

Sheriffs

We have 3 performance sheriffs 🤠 dedicating up to 50% of their time to this role. Outside of this, they assist with improving our tools and test harnesses.

Q3/2019

During the third quarter of 2019 our perfomance tools generated 1110 alert summaries. This is an average of 12 alert summaries every day.

Of the four test frameworks, build_metrics caused the most alert summaries, accounting for 42% of the total. The raptor framework was the second biggest contributor, with 30%.

Of the alert summaries generated, 13% were improvements. Of the alert summaries showing regressions, 51% were determined to be invalid, 5% were backed out, 23% were accepted, and 18% were fixed.

Our performance sheriffs triaged 63% of alerts within a day, and an additional 22% within 3 days.

Bugs were filed within 3 days for 53% of confirmed regressions, with a further 16% within 5 days.

Here are some highlights for some of our sheriffed frameworks:

Are We Slim Yet (AWSY)

On July 22nd we detected up to 18.9% improvement in AWSY for macOS. This was caused by Paul Bone’s patch on bug 1567366, which switched to using MADV_FREE_REUSABLE.
The largest fixed regression detected from AWSY was noticed on August 2nd, and had an impact of up to 21.57% regression to images. This was attributed to bug 1570745 and was fixed by switching AWSY to the common scenario for new tab page rather than the new user experience.

Raptor

The improvement alert with the highest magnitude for Raptor was created on July 16th, which saw up to a massive 46.06% improvement to page load time on desktop. The gains were attributed to bug 1541229, which tweaked idle detection during page load.
The largest regression alert that was ultimately fixed for Raptor was generated on August 13th. This showed up to 21.86% regression to the cold page load tests on Android, and was caused by bug 1557282. This was also noticed in Telemetry and was fixed via bug 1575794.

Talos

For Talos, the alert showing the largest improvement was created on August 13th, which showed up to 13.09% improvement to ts_paint. It was attributed to bug 1572646, which optimised picture cache tiles that are solid colours.
The largest fixed regression alert for Talos was spotted on August 8th, and includes a 4.24% hit to ts_paint on Windows. It was caused by bug 1539651, and was fixed by Brian Grinstead in bug 1573158.

Q4/2019

In the final quarter, our perfomance tools generated 996 alert summaries. This is an average of 11 alert summaries every day.

Of the four test frameworks, build_metrics caused the most alert summaries, accounting for 49% of the total. The raptor framework was the second biggest contributor, with 25%.

Of the alert summaries generated, 14% were improvements. Of the alert summaries showing regressions, 50% were determined to be invalid, 8% were backed out, 17% were accepted, and 10% were fixed.

Our performance sheriffs triaged 61% of alerts within a day, and an additional 26% within 3 days.

Bugs were filed within 3 days for 67% of confirmed regressions, with a further 11% within 5 days.

Here are some highlights for some of our sheriffed frameworks:

Are We Slim Yet (AWSY)

The largest improvement noticed for AWSY was on October 11th, where we saw up to a 3.88% decrease in Base Content JS. This was attributed to the work by André Bargull bug 1570370 to move language tag parsing and Intl.Locale to C++.
On September 30th, a 6.15% regression was noticed in JS, caused by bug 1345830. This was fixed by Gijs in bug 1586220.
The open regression with the highest impact for AWSY was detected on June 12th. Whilst there were a large number of improvements in this alert, the 12.67% regression to images was also significant. The regression was caused by bug 1558763, which changed the value of a preference within Marionette. It looks like this may have been fixed, but our performance sheriffs have yet to verify this.

Raptor

The most significant improvement detected by Raptor was on December 2nd. Up to 35.59% boost to many page load and benchmark tests on macOS. This was due to Nathan Froyd’s patch in bug 1599133 to enable constructing Sequence from moved nsTArrays.
On November 4th, a regression alert of 9.53% was reported against the Wikipedia page load test for bing.com on desktop. It turned out that bug 1591717 caused the unexpected regression. It was fixed by a patch by Emilio Cobos Álvarez to turn layout.css.notify-of-unvisited off for now.
Due to many page recordings being recreated, there are a lot of open alerts that require closer examination.

Talos

On December 18th we detected an improvement of up to 32.21% to several tests on macOS. This was thanks to Chris Manchester’s work on enabling PGO (Profile-Guided Optimisations) for the platform in bug 1604578.
Talos detected a 35.21% regression to perf_reftest_singletons on November 14th, which was caused by bug 1588431 and fixed by Emilio Cobos Álvarez in bug 1596712.
The open regression alert with the highest magnitude was opened on November 26th, and reports up to 196.47% regression to tp5o. It was caused by bug 1512011, which replaced mozhttpd with wptserve in Talos.

State of Performance Test Engineering (H1/2019)

2019-07-02T12:04:42+01:00

Late in 2018 I stepped out of the familiar position of automation engineer, and into the unknown as an engineering manager. A new team was formed for me to manage, focusing on performance test engineering. Now here we are, just over six months in, and I’m excited to share some updates!

Team

The team consists of 10 engineers based in California, Toronto, Montreal, and Romania.

The following team members joined us in H1/2019:

Alexandru Ionescu 🇷🇴
Marian Raicof 🇷🇴
Alexandru Irimovici 🇷🇴
Arnold Iakab 🇷🇴
Ken Rubin 🇺🇸

We also welcomed Greg Mierzwinski, who was converted from a contractor to full time employee.

Despite some early signs of interest, there were no community contributions in this period.

Tests

We currently support 4 frameworks:

Are We Slim Yet (AWSY) - memory consumption by the browser
build_metrics - build times, installer size, and other compiler-specific insights
Raptor - page load and browser benchmarks
Talos - various time-based performance KPIs on the browser

At the time of writing, there are 263 test suites (query) for the above frameworks.

The following are some highlights of tests introduced in H1/2019:

Raptor power tests for Android:
- Idle foreground
- Idle background
Raptor page load tests for Android:
- Expanded from 4 sites to 26
Raptor cold page load tests for Android and desktop
Raptor page load tests for Reference Browser and Firefox Preview

Hardware

Our tests are running on the following hardware in continuous integration:

35 Moto G5 devices (with two of these dedicated to power testing) 📱
38 Pixel 2 devices (with two of these dedicated to power testing) 📱
16 Windows UX laptops (2017 reference hardware) 💻
35 Windows ARM64 laptops (Lenovo Yoga C630) 💻

Sheriffs

We have 2 performance sheriffs 🤠 dedicating up to 50% of their time to this role. Outside of this, they assist with improving our tools and test harnesses. We also have 2 sheriffs in training to meet the needs of the additional tests we’re adding, and to provide support when needed.

Q1/2019

During the first quarter of 2019 our perfomance tools generated 863 alert summaries. This is an average of 9.7 alert summaries every day. Of the four test frameworks, build_metrics caused the most alert summaries, accounting for 35.1% of the total. The raptor framework was the second biggest contributor, with 29.9%.

Of the alert summaries generated, 106 (12.3%) were improvements. 13 alerts associated with regressions were fixed, and 9 were backed out. The remaining alerts were either reassigned, marked as downstream, marked as invalid, are still under investigation, or were accepted.

Here are some highlights for some of our sheriffed frameworks:

Are We Slim Yet (AWSY)

On February 15th we detected up to 20.66% improvement in AWSY. This was caused by Paul Bone’s patch on bug 1433007, which allowed the nursery to use less than a single chunk.
The largest fixed regression detected from AWSY was noticed on March 20th, and had an impact of up to 9.53% regression to the heap unclassified. This was attributed to bug 1429796 and was fixed a week later by Myk Melez.
There are no open regressions for AWSY for this period!

Raptor

The improvement alert with the highest magnitude for Raptor was created on March 25th, which saw up to a massive 41.04% improvement to page load time on Android. The gains were attributed to bug 1537964, which raised the importance of services launched from the main process.
The largest regression alert that was ultimately fixed for Raptor was generated early on in the quarter on January 3rd. This showed up to 120.65% regression to the assorted-dom benchmark, caused by bug 1467124. A fix was quickly identified by the author, Jan de Mooij and pushed within 24 hours of the notification from our sheriff.
Our biggest regression that remains unresolved was created on January 23rd, and includes 20.38% page load regression for Reddit on desktop, and appears to have been caused by bug 1485216. Due to excessive noise, we have since disabled the Time to First Interactive (TTFI) measurement, which is the metric that caused this regression. This will make confirming any fix here more difficult.

Talos

For Talos, the alert showing the largest improvement was created on January 9th, which showed up to 38.14% improvement to tsvg_static, tsvgr_opacity, and tsvgx. It was attributed to bug 1518773, which was a WebRender update.
The largest fixed regression alert for Talos was spotted on January 23rd, and includes a 26.73% hit to tscrollx and tp5o_scroll. It was caused by bug 1522028, and was fixed by Glenn Watson via 1523210 within a couple of days of being notified by our sheriffs.
The worst open regression alert for Talos was created on February 5th, and contains a 28.66% slow down to the about-preferences test. It looks like this has stalled with a patch to enable ASAP mode for the test, which caused failures and was subsequently backed out. According to Mike Conley in this comment, “ASAP mode causes us to paint ASAP after the DOM has been dirtied, without waiting for vsync”.

Q2/2019

In the second quarter, our perfomance tools generated 831 alert summaries. This is an average of 9.23 alert summaries every day. Of the four test frameworks, raptor caused the most alert summaries, accounting for 42.2% of the total. The build_metrics framework was the second biggest contributor, with 32.7%. There have been additional tests landed for Raptor, new page recordings, and a focus on improving page load performance, which are all likely to be reasons for the increase in Raptor alerts in this period.

Of the alert summaries generated, 98 (11.79%) were improvements. 19 alerts associated with regressions were fixed, and 6 were backed out. The remaining alerts were either reassigned, marked as downstream, marked as invalid, are still under investigation, or were accepted.

In Q1 we introduced a first_triaged timestamp in Perfherder (bug 1521025), which allows us to track how much time passes between the alert generation and the initial triage from a performance sheriff. Using this, we can see that our perf sheriffs triaged 56.1% of alerts within a day, and an additional 24.7% within three days.

In Q2 we introduced a bug_updated timestamp in Perfherder (bug 1536040) to more closely track the time between the alert and the correct bug being identified, which is where we handover to the regression author and test owners. The data doesn’t cover all of Q2, but so far we see that 63.3% of alerts complete this period within 3 days, and 87.4% complete within 1 week.

Here are some highlights for some of our sheriffed frameworks:

Are We Slim Yet (AWSY)

The largest improvement noticed for AWSY was on June 9th, where we saw up to a 5.98% decrease in heap unclassified and base content js. This was attributed to a patch from Emilio Cobos Álvarez in bug .
On May 30th, a 9.54% regression was noticed across several tests, caused by bug 1493420. Fortunately, this was promptly followed by a fix from Emilio Cobos Álvarez.
The open regression with the highest impact for AWSY was detected on June 12th. Whilst there were a large number of improvements in this alert, the 12.67% regression to images was also significant. The regression was caused by bug 1558763, which changed the value of a preference within Marionette. It looks like this may have been fixed, but our performance sheriffs have yet to verify this.

Raptor

The most significant improvement detected by Raptor was on May 22nd. Up to 20.42% boost to page load of several sites was seen, which was thanks to Andrew Creskey’s investigation and Michal Novotny’s patch in bug 1553166 to make disk cache sizing less restrictive on mobile.
On April 30th, a regression alert of 52.62% was reported against the Raptor page load test for bing.com on desktop. It turned out that bug 1514655 caused the unexpected regression, and Emilio Cobos Álvarez was able to post a fix within a day of the regression report!
Due to many page recordings being recreated, there are a lot of open alerts that require closer examination.

Talos

On April 13th we detected an improvement of up to 99.24% to the tp5n nonmain_startup_fileio test. This was thanks to Doug Thayer’s work on optimising prefetching of Windows DLLs in bug 1538279.
Talos detected a 3.78% regression to tsvgr_opacity on March 31st, which was caused by bug 1539026. After an initial fix was backed out, Lee Salzman was able to fix the regression.
The open regression alert with the highest magnitude was opened on April 29th, and reports a 38.25% decrease in the average frame interval while animating a simple WebGL scene. It was discovered by the glterrain test, and attributed to 1547775. It does look like this test returned to the pre-regression values, but we have yet to confirm if this was expected or if the regression remains.

Survey

I’d really like to find out how our team is doing, and hear if you have any feedback or suggestions for any way we can improve. Please take a couple of minutes to complete the very simple satisfaction survey I have created. Thanks!

EuroPython 2018

2018-09-14T11:52:42+01:00

In July I took the train up to beautiful Edinburgh to attend the EuroPython 2018 conference. Despite using Python professionally for almost 8 years, this was my first experience of a Python conference. The schedule was packed, and it was challenging deciding what talks to attend, but I had a great time and enjoyed the strong community feeling of the event. We even went for a group run around Holyrood Park and Arthur’s Seat, which I hope is included in the schedule for future years.

Now that the videos of the talks have all been published, I wanted to share my personal highlights, and list the talks I saw during and since the conference. I still haven’t caught up on everything I wanted to see, so I’ve also included my watch list. First, here’s the full playlist of talks from the conference

Here are my top picks from the talks I either attended or have watched since:

I also wanted to highlight the following lightning talks:

Here is a list of the other talks I either attended at the conference or have watched since:

Here’s my list of talks I have yet to watch:

Were you at EuroPython 2018? Let me know if you have any favourite talks that aren’t already on my list! I’m keen to attend again next year, if my travel schedule allows for it.

Python unit tests now running with Python 3 at Mozilla

2018-06-29T15:48:42+01:00

I’m excited to announce that you can now run the Python unit tests for packages in the Firefox source code against Python 3! This will allow us to gradually build support for Python 3, whilst ensuring that we don’t later regress. Any tests not currently passing in Python 3 are skipped with the condition skip-if = python == 3 in the manifest files, so if you’d like to see how they fail (and maybe provide a patch to fix some!) then you will need to remove that condition locally. Once you’ve done this, use the mach python-test command with the new optional argument --python. This will accept a version number of Python or a path to the binary. You will need to make sure you have the appropriate version of Python installed.

Once you’re ready to enable tests to run in TaskCluster, you can simply update the python-version value in taskcluster/ci/source-test/python.yml to include the major version numbers of Python to execute the tests against. At the current time our build machines have Python 2.7 and Python 3.5 available.

To summarise:

Remove skip-if = python == 3 from manifest files. These are typically named manifest.ini or python.ini, and are usually found in the tests directory for the package.
Run mach python-test --python=3 with your target path or subsuite.
Fix the package(s) to support Python 3 and ensure the tests are passing
Add Python 3 to the python-version for the appropriate job in taskcluster/ci/source-test/python.yml.

At the time of writing, the pythonclock.org tells me that we have just over 18 months before Python 2.7 will be retired. What this actually means is still somewhat unknown, but it would be a good idea to check if your code is compatible with Python 3, and if it’s not, to do something about it. The Firefox build system at Mozilla uses Python, and it’s still some way from supporting Python 3. We have a lot of code, it’s going to be a long journey, and we could do with a bit of help!

Whilst we do plan to support Python 3 in the Firefox build system (see bug 1388447), my initial concern and focus has been the Python packages we distribute on the Python Package Index (PyPI). These are available to use outside of Mozilla’s build system, and therefore a lack of Python 3 support will prevent any users from adopting Python 3 in their projects. One such example is Treeherder, which uses mozlog for parsing log files. Treeherder is a django project, which recently dropped support for Python 2 (unless you’re using their long term support release, which will support Python 2 until 2020).

Updating these packages to support Python 3 isn’t necessary that hard to do, especially with tools such as six, which provides utilities for handling the differences between Python 2 and Python 3. The problem has been that we had no way to run the tests against Python 3 in TaskCluster. This is no longer the case, and Python unit tests can now be run against Python 3!

So far I have enabled Python 3 jobs for our mozbase unit tests (this includes the aforementioned mozlog), and our mozterm unit tests. There are still many tests in mozbase that are not passing in Python 3, so as mentioned above, these have been conditionally skipped in the manifest files. This will allow us to enable these tests as support is added, and this condition could even be used in the future if we have a package that doesn’t have full compatibility with Python 2.

Now that running the tests against multiple versions of Python is relatively easy, it’s a great time for me to encourage our community to help us with supporting Python 3. If you’d like to help, we have a tracking bug for all of our mozbase packages. Find a package you’d like to work on, read the comments to understand what you need and how to get set up, and let me know if you get stuck!

Prototype multi-device Firefox tests

2018-03-20T14:26:25+00:00

With Firefox Accounts, you can access your tabs, history, and bookmarks from any device. You can even send tabs from one device to another, which is great when I find myself on a page that’s not optimised for mobile, or if I get distracted at the weekend and find something I want to pick up when I get to work on Monday morning. While these features are awesome, I’ve had issues when the sync isn’t triggered, or things don’t go as expected. Some of these issues are known (and are being addressed), but currently it’s too easy for regressions to be introduced.

Let’s take the simple use case of saving a bookmark using Firefox on your phone, and later opening the bookmark on Firefox on desktop. In this scenario we have the mobile client, the Firefox Accounts service, the Sync service, and the desktop client. We could be using Firefox on Android or iOS as our mobile client, and we could be using Firefox on macOS, Linux, or Windows. Other scenarios could involve multiple different mobile clients, such as syncing between a tablet and phone. There’s a lot of configuration necessary, and many variations. Whilst each of the components have their own automated tests, there’s currently no automation to take care of the basic end-to-end scenarios.

Part of the issue is that there are many individual components, and many ways they can be combined. Integration testing is currently carried out manually, which is time-consuming, and doesn’t allow us to cover as many scenarios and device combinations as we’d like. Introducing automation to cover the basic scenarios will allow the testers more time to focus on exploration and more edge cases.

Over the last few weeks I’ve built a proof-of-concept test harness to automate end-to-end testing of multiple clients and server components. Initially I have focused on the previously mentioned scenario of syncing a bookmark from mobile to desktop, and limited to Firefox on iOS and macOS for now. Rather than create something entirely from scratch, I’ve brought together existing solutions for this initial prototype. This allowed me to pull something together relatively quickly, but does also bring some limitations and questions along.

The Scenario

Let’s remind ourselves of our initial scenario:

Firefox on iOS:
- Open a website
- Save a bookmark
- Sign into Firefox Accounts
- Perform initial sync
Firefox on macOS:
- Sign into Firefox Accounts
- Perform initial sync
- Verify new bookmark exists

Finally, we’ll need to present these results in a way the user can interpret, and can investigate in the event of a failure. For this we’ll need a framework that can pull everything together, which is where I started.

The Test Framework

My language of choice is Python, and my preferred test framework is pytest. Being able to use something that I’m already familiar with for the framework allowed me to focus on the more challenging areas. By using pytest I’m also able to take advantage of several plugins I have built and maintain for other projects. Finally, if we decide in the future to land any part of this into mozilla-central, it shouldn’t require too many changes as Python and pytest are already in use there.

Generating Firefox Accounts

As a prerequisite, we need credentials for Firefox Accounts. Fortunately, we already have the PyFxA package. This allowed me to create a pytest fixture to create accounts as needed and destroy them when they’re done with. The following is a slightly simplified version of the fixture, which creates an account, verifies it, and ultimately destroys it once it’s no longer needed:

@pytest.fixture
def fx_account():
    account = TestEmailAccount()
    client = Client('https://api-accounts.stage.mozaws.net/v1')
    password = ''.join([random.choice(string.ascii_letters) for i in range(8)])
    session = client.create_account(account.email, password)
    account.fetch()
    message = account.wait_for_email(lambda m: 'x-verify-code' in m['headers'])
    session.verify_email_code(message['headers']['x-verify-code'])
    yield {'email': account.email, 'password': password}
    account.clear()
    client.destroy_account(account.email, password)

Whilst building out the prototype it was useful to have a Firefox Account that wasn’t immediately destroyed, which is why I built a command line tool for creating and destroying accounts.

Automating Firefox on iOS

Fortunately there is already a suite of automated UI tests for Firefox on iOS, so I was able to build on the existing code. For our scenario I was able to get away with only making a few changes:

Created an IntegrationTests.swift file for the new script. Note that although this is technically a test itself, I’m referring to it as a script. This is because it only forms part of the overall integration test, and is essentially executed as a script. Of course, any failure encountered while running it will result in a test failure.
Added LaunchArguments.StageServer as a launch argument in BaseTestCase.swift so the staging environment would be used for Firefox Accounts and Sync.
Switched from using type to typeText in FxScreenGraph.swift for the Firefox Accounts login screen. This allowed entry of characters not displayed on the initial keyboard layout. If we want to use the on screen keyboard, then we’ll need to enhance the screen graph to support switching layouts. Alternatively, we could avoid using such characters.
Added the ability to set the timeout for waitforExistence as loading the Firefox Accounts login screen and performing the initial sync were occasionally taking longer than the default of 5 seconds.
Modified the existing Fennec_Enterprise_XCUITests scheme to include environment variables for the Firefox Account email and password so they can be used from the script.

With these changes, I was able to create the script to open a website and save it as a bookmark:

func testFxASyncBookmark () {
    // Go to a webpage, and add to bookmarks
    navigator.createNewTab()
    loadWebPage("www.example.com")
    navigator.nowAt(BrowserTab)
    bookmark()

    // Sign into Firefox Accounts
    navigator.goto(FxASigninScreen)
    waitforExistence(app.webViews.staticTexts["Sign in"], timeout: 10)
    userState.fxaUsername = ProcessInfo.processInfo.environment["FXA_EMAIL"]!
    userState.fxaPassword = ProcessInfo.processInfo.environment["FXA_PASSWORD"]!
    navigator.performAction(Action.FxATypeEmail)
    navigator.performAction(Action.FxATypePassword)
    navigator.performAction(Action.FxATapOnSignInButton)
    allowNotifications()

    // Wait for initial sync to complete
    waitforExistence(app.tables.staticTexts["Sync Now"], timeout: 10)
}

To run XCUITests from outside of Xcode, you need to use the xcodebuild command line tool. So, using FxACLI to create a test account, I can run my new script using the following commands:

$ fxacli create
Account created!
 - 🌐  https://api-accounts.stage.mozaws.net/v1
 - 📧  test-a478e06856@restmail.net
 - 🔑  CokFkuRA
Account verified! 🎉
$ export FXA_EMAIL=test-a478e06856@restmail.net FXA_PASSWORD=CokFkuRA
$ xcodebuild test -scheme Fennec_Enterprise_XCUITests -destination "platform=iOS Simulator,name=iPhone X" -only-testing:XCUITests/IntegrationTests/testFxASyncBookmark
--- snip ---
** TEST SUCCEEDED **
$ fxcli destroy
Account destroyed! 💥

In order to run the script from my pytest framework, I created a fixture named xcodebuild. This fixture patches the environment with the Firefox Account variables, and yields an XCodeBuild object:

@pytest.fixture
def xcodebuild_log(pytestconfig, tmpdir):
    xcodebuild_log = str(tmpdir.join('xcodebuild.log'))
    pytestconfig._xcodebuild_log = xcodebuild_log
    yield xcodebuild_log

@pytest.fixture
def xcodebuild(fx_account, monkeypatch, xcodebuild_log):
    monkeypatch.setenv('FXA_EMAIL', fx_account.email)
    monkeypatch.setenv('FXA_PASSWORD', fx_account.password)
    yield XCodeBuild(xcodebuild_log)

The XCodeBuild object has a test method, which requires the identifier of the test to run. When the test method is called, the xcodebuild binary is started in a new process, and the output is redirected to a log file to later attach to the HTML report. The following test, although incomplete at this point, demonstrates running the XCUITest from pytest:

def test_sync_bookmark_from_device(xcodebuild):
    xcodebuild.test('XCUITests/IntegrationTests/testFxASyncBookmark')

I noticed early on that once a test has signed into Firefox Accounts, the next test to run will remember the email address used and only ask for a password when attempting to sign in. There’s likely a less expensive solution, but for now I’ve resolved this by running xcrun simctl shutdown all to shutdown all simulators, followed by xcrun simctl erase all to wipe them.

Automating Firefox on Desktop

So far we’ve added a bookmark in Firefox on iOS and performed an initial sync. We now need to sign into Firefox Accounts on desktop Firefox, perform a sync, and verify the new bookmark is added. Like our UI tests for Firefox on iOS, we already have a solution for performing integration tests for Sync. It’s called TPS, and I with the following tweaks I was able to get it to work for my prototype:

Added “mobile” bookmark folder to bookmarks.jsm, which is necessary to verify bookmarks in this location.
Removed attempt to load and validate the ping schema. The _tryLoadPingSchema method attempts to read a schema file from disk, which isn’t present for my prototype so I’ve removed that, and another related code path.

The source code for TPS is stored in mozilla-central, and I don’t want the entire repository to be a requirement for running my prototype. If we decide that TPS is the best approach for these tests, then we’d probably need to find a better way to share the code than my current approach of simply copying the extension source code and making local changes.

TPS works by launching Firefox with the extension installed and passing an argument to the test to run. This means it’s necessary for us to write a TPS test in JavaScript for our scenario:

EnableEngines(["bookmarks"]);

var phases = { "phase1": "profile1" };

// expected bookmark state
var bookmarksExpected = {
"mobile": [{
  uri: "http://www.example.com/",
  title: "Example Domain"}]
};

// sync and verify bookmarks
Phase("phase1", [
  [Sync],
  [Bookmarks.verify, bookmarksExpected],
]);

The phases allow Firefox to be restarted and for syncing to be performed across multiple profiles. Whilst this could be useful, I’ve currently enforced a single phase/profile for my prototype.

There’s already a TPS test runner written in Python, so I was able to selectively pick what I needed for my prototype. I created several pytest fixtures that work together to package the TPS extension, configure a Firefox profile, and in a similar to our xcodebuild fixture, provide an interface for executing a test:

@pytest.fixture(scope='session')
def tps_addon(tmpdir_factory):
    name = str(tmpdir_factory.mktemp('addon').join('tps'))
    shutil.make_archive(name, 'zip', os.path.join(here, 'tps'))
    os.rename('{}.zip'.format(name), '{}.xpi'.format(name))
    yield '{}.xpi'.format(name)

@pytest.fixture
def tps_config(fx_account):
    yield {'fx_account': {
        'username': fx_account.email,
        'password': fx_account.password}}

@pytest.fixture
def tps_log(pytestconfig, tmpdir):
    tps_log = str(tmpdir.join('tps.log'))
    pytestconfig._tps_log = tps_log
    yield tps_log

@pytest.fixture
def tps_profile(tps_addon, tps_config, tps_log):
    preferences = {
        'extensions.autoDisableScopes': 10,
        'extensions.legacy.enabled': True,
        'identity.fxaccounts.autoconfig.uri': urls['content'],
        'tps.config': json.dumps(tps_config),
        'tps.logfile': tps_log,
        'tps.seconds_since_epoch': int(time.time()),
        'xpinstall.signatures.required': False
    }
    yield Profile(addons=[tps_addon], preferences=preferences)

@pytest.fixture
def tps(pytestconfig, tps_log, tps_profile):
    yield TPS(pytestconfig.getoption('firefox'), tps_log, tps_profile)

I also added a required command line option for the path to Firefox. The TPS test runner actually downloads the latest Firefox nightly, which could be an option in the future.

The TPS object provided by the tps fixture provides a run method, which takes a test file as an argument. After adding this to our Python test we have something that looks like this:

def test_sync_bookmark_from_device(tps, xcodebuild):
    xcodebuild.test('XCUITests/IntegrationTests/testFxASyncBookmark')
    tps.run('test_bookmark.js')

Now we have a working prototype that satisfies our scenario. Note that there’s not much to see while the TPS test is running, however if you open the settings menu in Firefox you can watch the state transition from not signed in, to signed in, and the initial sync being performed. If you’re really quick you can also see the mobile bookmark appearing in the menu.

Running the Tests

To run the tests you will need to first follow the instructions for building Firefox on iOS. You will also need to ensure you have legacy Python (2.7) installed (there are dependencies that have not yet been updated to support modern Python). Finally, install pipenv, which will take care of the remaining Python dependencies.

You can then run the tests as follows, making sure you set the correct path to your Firefox binary:

$ cd python
$ pipenv install
$ pipenv run pytest --firefox=/path/to/Firefox.app/Contents/MacOS/firefox-bin

The tests will build and install the application to the simulator, which can cause a delay where there will be no feedback to the user. Also, note that each XCUITest that is executed will shutdown and erase data from all available iOS simulators. This assures that each execution starts from a known clean state.

Reviewing Results

Hopefully you’ll see something like the following in your console:

$ pipenv run pytest --firefox=/path/to/Firefox.app/Contents/MacOS/firefox-bin
============================= test session starts =============================
platform darwin -- Python 2.7.13, pytest-3.4.2, py-1.5.2, pluggy-0.6.0 -- /path/to/python2.7
cachedir: .pytest_cache
rootdir: /path/to/firefox-ios/python, inifile: pytest.ini
plugins: metadata-1.6.0, html-1.16.1, mozlog-3.7
collected 1 item

test_integration.py::test_sync_bookmark_from_device PASSED               [100%]

-------------- generated html file: /path/to/results/index.html ---------------
========================= 1 passed in 273.26 seconds ==========================

Even if the test fails, you should see the ‘generated html file’ somewhere in your console. Open this file in Firefox to review the results. If there was a failure, the report will include the details as shown in the console. You’ll also find environment details including the version of desktop Firefox being used.

For each test there’s a link to logs for TPS and xcodebuild. At the moment these are included regardless of the outcome, however they’re mostly valuable for investigating failures. In the future we may decide to exclude them when tests pass.

What’s Next?

Well, the next thing is to gather feedback on this prototype. Does it make sense to use XCUITests, TPS, and pytest, or have I missed something that would improve the integration testing between multiple devices? If you’ve read this far, I suspect you have some opinions. Please get in touch by leaving a comment or finding me on IRC, Slack, Twitter, email, etc!

After I’ve given some time for feedback to reach me, we’ll need to device how to rollout the prototype so that it can start to provide value. Initially we might start with a dedicated Mac Mini available via remote access to trigger the tests.

Then we’ll need to review the test cases that we’d want to write. Ideally we’d keep the suite relatively small, as the tests will take some time to run. The idea is to cover the basic functionality where risk is high, and the more obscure scenarios will be covered by manual testing.

Other ideas for future plans include:

Pre-populate Firefox with history instead of creating them through the user interface. Our prototype test loads a website and adds it to the bookmarks as a user would. Doing it this way is slow and prone to failure, so if we can pre-populate Firefox with this it would be an improvement.
Experiment with going back-and-forth between devices. This could be achieved by saving and restoring state between sessions, or by using an alternative tools that would allow for switching between multiple concurrent sessions.
Create a prototype for Firefox on Android. Perhaps we can reuse parts of this prototype, although we’d probably look into using some combination of UIAutomator, Espresso, and Appium for the Android automation parts.
Experiment with Appium as an alternative to XCUITest. I went with XCUITest because we already had something in place, but perhaps there’s some value in switching these tests to using Appium. It’s at least worth investigating.
Experiment with introducing WebDriver to the prototype. If we have a scenario that requires us to interact with either Firefox or web content (such as Firefox Accounts) then we may need to introduce WebDriver (Selenium).
Look into setting up Continuous Integration for these tests. If the tests prove to be valuable and stable, then it would be ideal to run them whenever a new version of Firefox is avaiable. This could be a simple cron schedule, or something like a Jenkins agent running the tests when triggered.
Create a prototype for Firefox on other devices. In the future we may have Firefox Accounts integration in Firefox for FireTV, so it would be great if we could include coverage here, too.

I’ve been pushing my code to a branch on my fork of Firefox on iOS. You can see a comparison between my branch and the upstream master here.

Demo

If you’re unable to run the tests locally, here’s a recording of the test running along with reviewing the HTML report and logs:

Command line tool for Firefox Accounts

2018-03-15T10:27:39+00:00

When testing services that depend on Firefox Accounts, it’s useful to be able to create disposable test accounts. Fortunately we’ve had this ability from the very early days of the service, and our automated tests make heavy use of PyFxA to create, verify, and ultimately destroying accounts. As useful as this is, it hasn’t been particularly easy to create accounts for the purposes of manual testing. For the rare occasion that I’ve needed an account, I’ve either created them manually via main user interface with a disposable email account, or I’ve created a simple one-off script to create a batch of accounts. As I had this need again recently, I decided to write a simple command line tool for creating verified accounts and subsequently destroying them.

To use the tool, you’ll need Python (despite a dependency not claiming to have support for Python 3, it’s working fine for me using 3.6) and pip. It can then be installed by simply running pip install fxacli

Any account created by the tool will be stored in a JSON file named .accounts in the working directory. This allows for one or more accounts to be created, and then destroyed without the need to specify the target account.

To create an account:

$ fxacli create
Account created!
 - 🌐  https://api-accounts.stage.mozaws.net/v1
 - 📧  test-72a888a3f6@restmail.net
 - 🔑  IvOhSLzI
Account verified! 🎉

To destroy the most recently created account:

$ fxacli destroy
Account destroyed! 💥
 - 🌐  https://api-accounts.stage.mozaws.net/v1
 - 📧  test-72a888a3f6@restmail.net
 - 🔑  IvOhSLzI

To destroy all accounts created using the tool, pass the --all flag.

If you want to destroy an account not created by the tool, or if your .accounts file is for any reason unreadable, you can pass --email and --password options.

By default all accounts will be created/destroyed from the staging instance of Firefox Accounts. This is highly recommended, but if you know what you’re doing, the target environment can be specified using the --env command line option.

Visit the FxACLI package on PyPI for links to the source code, issue tracker, and more.

Effective CI for Firefox projects developed in GitHub

2017-06-28T00:00:00+01:00

Whilst the canonical repository for the Firefox source code uses Mercurial, it’s becoming increasingly popular for Firefox projects to use GitHub for development. When it’s time to ship, many of these projects will land their code inside the canonical repository for inclusion in the upcoming Firefox release. There are a few challenges that come with this approach.

Primarily, we won’t know when a change in the project introduces a regression in Firefox until we attempt to land the project alongside the Firefox source code. Similarly, we won’t know when a change to Firefox might cause a regression in the project.

This quarter I spent some time discussing these issues, learning how different teams are addressing the problems, and helping to define a more generic approach for the future.

Current solutions

Here’s a summary of some the solutions I discovered teams are using:

ActivityStream are adding their project to a local copy of the Firefox source code and pushing to an alternate branch. This then runs all the tests and reports results to Treeherder. This relies on access to the alternate branch, and is currently a manual process.
Normandy download an archive from gecko-dev (a read-only Git mirror of the Firefox repository), combine this with the project repository, build Firefox, and run project specific tests. Every two weeks or so, they bundle the changes up and push to the Firefox repository.
debugger.html use a Docker image with the Firefox source code, and run project specific tests in Circle CI.

Proposed solution

Whilst I wasn’t able to pull together a prototype this quarter, I did draft a proposal based on my investigations:

Each project would create an entry script mcmerge.sh, which would be responsible for integrating the project source code with the Firefox source code. In some instances this might simply copy files, but it could also apply patches as needed.
Whenever a pull request is opened against the GitHub project, or a change is pushed:
- Check if the author of the pull/push is authorised to trigger continuous integration. This might be that they’re a repository collaborator, or have authority to push to the main Firefox repository. If they are not authorised, we stop here.
Prepare a new patch using the vcssync tool currently in development, which will take the entire GitHub project repository at the current revision and apply it in the patch to a vacant subdirectory in the Firefox repository.
Push the prepared patch to our Try Server, which will trigger tests according to the commit message. This syntax should be configurable in the project repository, but would likely default to all platforms/tests.
If triggered by a pull request, publish a comment indicating that the tests have been triggered and linking to the results in Treeherder.
During the regular Firefox build, we’d scan for these projects and execute any mcmerge.sh scripts we encountered. This would integrate the project with the Firefox source code.
Build Firefox and execute the tests as usual.
Reviewers of the pull request would be expected to check the results of the Try push as part of their review.

I also identified a few possible directions this may evolve in the future:

We could use the same mechanism to automatically land changes that are pushed to the GitHub repository into the Firefox repository. This would mean that the Firefox repository would effectively hold a read-only mirror of the GitHub project. This would remove the need for manually pushing the project to the Firefox repository, and would allow us to determine when changes to the Firefox source code causes regressions in the GitHub projects. This may not be desirable for all projects, and would have implications for the sheriffs, who may need to back out changes that cause bustage.
Rather than perform full builds of Firefox, we may be able to optimise these in a similar fashion to the artefact builds. This isn’t something we can do now, but it is a desirable build optimisation for other reasons.
Due to the intermittent nature of our tests, it’s not currently possible to give a pass/fail indication in the GitHub pull request. If the scope of the tests is narrow enough and they’re stable, it’s possible we could enhance this to poll for the result and indicate the outcome in the pull.

Next steps

As mentioned, I was unfortunately not able to bring together a prototype based on my proposal. I discovered early on that I wasn’t the only one looking into this, and I even feel that the other work in this area is already in better hands. The work in bug 1364561 and bug 1357597 are particularly relevant to this project. It’s possible that I’ll return to this project in the future, but for now I’ll be following the progress and contributing to the discussions without actively working on the implementation.

Reporting test results to Treeherder

2017-04-10T00:00:00+01:00

Many of the web and services automated tests at Mozilla run in Jenkins, and until recently our instance was public. This meant it was easy for both paid and volunteer contributors to discover test failures, file issues, and provide fixes either for the tests or the projects they serve. Unfortunately, just like any software, Jenkins has had some security vulnerabilities. Last year, one of these prompted us to remove public access to our instance.

As Jenkins is much more than just a dashboard of results, a compromised instance could cause loss of data, or worse, remote execution of code. Since removing access to Jenkins, we’ve been looking into alternative ways to share our test results publicly.

Last month I wrote about how we’re making the test results accessible via ActiveData. This allows anyone to query our results and perform data analysis and visualisation on them. Whilst this is going to prove to be a valuable tool, it doesn’t provide a convenient dashboard for the results and is unlikely to attract much interest from the community.

The next step is submitting our results to Treeherder - a public dashboard for commits to Mozilla projects, which displays results of tasks such as builds, linting, and automated tests. Treeherder can be used to monitor the health of projects, and as a tool for investigating failures and raising defects, making it the perfect home for our test results.

The remainder of this post details the steps necessary to submit our test results to Treeherder. Note that whilst working on this I had local instances of most of the services and applications involved. This is a good practice as it reduces latency, and means you’re not filling up an live instances with experimental test data.

Creating a Pulse user

Treeherder ingests information about jobs from Pulse exchanges. Pulse is a RabbitMQ cluster, and if you’re unfamiliar (as I was), I can highly recommend the tutorials to learn more. The first step was to sign into PulseGuardian and create a user for publishing our messages.

Adding project repositories

To have our project repositories shown in Treeherder, it’s necessary to tell Treeherder about them. Then, to get result sets showing up you’ll need to add TaskCluster integration for the repositories. For each revision, TaskCluster will send a message to Pulse, which Treeherder uses to build a collection of result sets for each repository. Note that you will need a GitHub organisation owner or repository administrator to enable TaskCluster integration. It’s also worth noting that historic revisions will not be available in Treeherder, so only new commits will show up.

Generating a message

Treeherder provides a schema that messages are expected to validate against. As we’re using Jenkins, and have recently migrated to declarative pipelines with a shared library, I wrote a submitToTreeherder step to generate the payload using json-schema-validator to ensure we’re conforming to the schema. Whilst developing locally I used a simple Python script based on the RabbitMQ tutorials, as this made it much easier to iterate on the payload. Interestingly, this validation actually highlighted a couple of issues with the schema (1352402, 1352403). I also encountered issues with the Jenkins pipelines (JENKINS-43195, JENKINS-43246).

Submitting the message

Once the message has been generated and validated against the schema, we send it as our Pulse user to an exchange including the username with a routing key that contains the username and project. Our Pulse username is fxtesteng, so our exchange would be exchange/fxtesteng/jobs, and for the FxAPOM project our routing key would be fxtesteng.fxapom.

Using Pulse Inspector

At this point Treeherder isn’t aware of the new exchange, but you can use Pulse Inspector to listen on a specified exchange and routing key pattern to see the messages as they’re received by Pulse. If you want to listen to all messages published to the Firefox Test Engineering exchange for Treeherder jobs, enter the exchange exchange/fxtesteng/jobs and click ‘Start Listening’. Note that traffic on this exchange is currently very low, but it’s useful if you’ve triggered a job and want to see what’s being sent.

Registering with Treeherder

Now that we’re sending messages for each job, all that’s left is to tell Treeherder about your exchange. Once that’s done, the next job published to Pulse should be picked up by Treeherder and displayed under the appropriate repository. The following screenshot shows FxAPOM results in Treeherder.

You can see these for yourself, and take a look at the log files, HTML reports and other details for the results.

What’s next?

At the time of writing, only FxAPOM is configured to submit results to Treeherder. This repository has a low volume of commits, and the tests are run once a day. Next, we’d like to submit results for more of our repositories. As we’re using our shared library, this is a relatively small change for most of our projects. There are also a number of enhancements filed for the shared library, which will improve the display of the results in Treeherder. If you’re interested in working on any of these, please add a comment and I’d be happy to mentor you.

I’d like to think that this model could be repeated for other continuous integration services. It’s already provided by TaskCluster, but there’s certainly some value in submitting results from Travis CI or CircleCI. That said, there are no current plans to implement this (that I’m aware of). If this is something you’d like to work on, let me know!

Acknowledgements

I would like to say thanks to the Treeherder team for their assistance and considerable patience whilst I worked on this. I’d also like to thank Andrew Bayer from the Jenkins team for his encouragement and assistance while I battled through a few Groovy and Jenkins pipeline issues. Whenever we’re in the same city, I definitely owe that guy a drink!

Migrated content from seleniumexamples.com

2017-03-24T00:00:00+00:00

I ran a blog at seleniumexamples.com from 2009-2011, where I posted examples for using Selenium. This content has been unavailable for some time, so I decided to migrate it here. Many of the examples are unlikely to work, and I didn’t bother to migrate comments, but it might at least prove interesting to some. Personal highlights include playing Pacman, the first London meetup, and cheesecake. You can find the entire archive here.

Migrating to declarative Jenkins pipelines

2017-03-23T00:00:00+00:00

Last year I shared my thoughts on Jenkins pipelines and provided a walkthrough of how we’re using pipelines at Mozilla. Since then, the Pipeline Model Definition plugin has came out of beta, and we’ve been migrating our pipelines to the new declarative syntax with a shared library.

Shared libraries

As soon as you’re implementing similar steps in multiple Jenkins pipelines, it makes sense to consider writing a shared library. Our fxtest-jenkins-pipeline library allows us to centrally maintain custom steps such as IRC notifications, or creating variables files with desired capabilities for Selenium, rather than implementing these in every pipeline.

Pipeline options

In our original pipelines it was necessary to wrap steps in order to configure timeouts, ANSI colours, and timestamps. With declarative this is made so much better by allowing pipeline-specific options to be configured. The following snippet demonstrates enabling all three of these:

options {
  ansiColor('xterm')
  timestamps()
  timeout(time: 1, unit: 'HOURS')
}

Everything defined in one place, and with less nesting/indentation vastly improves the readability and maintainability of the pipelines.

Environment variables

Another huge improvement is the handling of environment variables variables and accessing credentials. Previously it was necessary to wrap steps in withEnv for environment variables, and withCredentials for credentials. This is what we previously would have needed:

withCredentials([[
  $class: 'StringBinding',
  credentialsId: 'SAUCELABS_API_KEY',
  variable: 'SAUCELABS_API_KEY']]) {
  withEnv(["PYTEST_ADDOPTS=" +
    "-n=${processes} " +
    "--driver=SauceLabs " +
    "--variables=capabilities.json " +
    "--color=yes"]) {
      // ...
}

With declarative, this can be replaced with the following:

environment {
  PYTEST_ADDOPTS =
    "-n=10 " +
    "--tb=short " +
    "--color=yes " +
    "--driver=SauceLabs " +
    "--variables=capabilities.json"
  SAUCELABS_API_KEY = credentials('SAUCELABS_API_KEY')
}

Note that there’s a regression in version 1.1.1 of the plugin that prevents strings from being split over multiple lines, but this is already fixed and will be included in the next release.

Post build steps

Possibly the most valuable addition to declarative piplines are the post build steps. It’s now possible to define steps to execute after each stage or pipeline depending on the current status of the build. Previously we were using try/catch/finally for our tests step ensure we reported our results, and a broader try/catch to send failure notifications.

We now have a post section immediately after our test stage that publishes artifacts, similar to the following snippet:

stage('Test') {
  steps {
    // ...
  }
  post {
    always {
      archiveArtifacts 'results/*'
      junit 'results/*.xml'
    }
  }
}

We also have a post section for the entire pipeline for notifications. This means that we send notifications when any stage fails, which is another improvement over our previous pipelines.

Conclusion

In my opinion declarative pipelines are a huge improvement over the original scripted pipelines. The new syntax is succinct and easier to read and maintain. As a result of migrating to declarative and our shared library, one of our pipelines had a 60% reduction in the lines of code but with more functionality!

Here’s a full example of one of our declarative Jenkins pipelines.