Posts
-
State of Performance Test Engineering (H2/2019)
It’s a new year, and time for me to post an update on the state of Firefox performance test engineering. The last update was in July 2019 and covered the first half of the year. This update covers the second half of 2019.
-
State of Performance Test Engineering (H1/2019)
Late in 2018 I stepped out of the familiar position of automation engineer, and into the unknown as an engineering manager. A new team was formed for me to manage, focusing on performance test engineering. Now here we are, just over six months in, and I’m excited to share some updates!
-
EuroPython 2018
In July I took the train up to beautiful Edinburgh to attend the EuroPython 2018 conference. Despite using Python professionally for almost 8 years, this was my first experience of a Python conference. The schedule was packed, and it was challenging deciding what talks to attend, but I had a great time and enjoyed the strong community feeling of the event. We even went for a group run around Holyrood Park and Arthur’s Seat, which I hope is included in the schedule for future years.
-
Python unit tests now running with Python 3 at Mozilla
I’m excited to announce that you can now run the Python unit tests for packages in the Firefox source code against Python 3! This will allow us to gradually build support for Python 3, whilst ensuring that we don’t later regress. Any tests not currently passing in Python 3 are skipped with the condition
skip-if = python == 3in the manifest files, so if you’d like to see how they fail (and maybe provide a patch to fix some!) then you will need to remove that condition locally. Once you’ve done this, use themach python-testcommand with the new optional argument--python. This will accept a version number of Python or a path to the binary. You will need to make sure you have the appropriate version of Python installed. -
Prototype multi-device Firefox tests
With Firefox Accounts, you can access your tabs, history, and bookmarks from any device. You can even send tabs from one device to another, which is great when I find myself on a page that’s not optimised for mobile, or if I get distracted at the weekend and find something I want to pick up when I get to work on Monday morning. While these features are awesome, I’ve had issues when the sync isn’t triggered, or things don’t go as expected. Some of these issues are known (and are being addressed), but currently it’s too easy for regressions to be introduced.
-
Command line tool for Firefox Accounts
When testing services that depend on Firefox Accounts, it’s useful to be able to create disposable test accounts. Fortunately we’ve had this ability from the very early days of the service, and our automated tests make heavy use of PyFxA to create, verify, and ultimately destroying accounts. As useful as this is, it hasn’t been particularly easy to create accounts for the purposes of manual testing. For the rare occasion that I’ve needed an account, I’ve either created them manually via main user interface with a disposable email account, or I’ve created a simple one-off script to create a batch of accounts. As I had this need again recently, I decided to write a simple command line tool for creating verified accounts and subsequently destroying them.
-
Effective CI for Firefox projects developed in GitHub
Whilst the canonical repository for the Firefox source code uses Mercurial, it’s becoming increasingly popular for Firefox projects to use GitHub for development. When it’s time to ship, many of these projects will land their code inside the canonical repository for inclusion in the upcoming Firefox release. There are a few challenges that come with this approach.
-
Reporting test results to Treeherder
Many of the web and services automated tests at Mozilla run in Jenkins, and until recently our instance was public. This meant it was easy for both paid and volunteer contributors to discover test failures, file issues, and provide fixes either for the tests or the projects they serve. Unfortunately, just like any software, Jenkins has had some security vulnerabilities. Last year, one of these prompted us to remove public access to our instance.
-
Migrated content from seleniumexamples.com
I ran a blog at seleniumexamples.com from 2009-2011, where I posted examples for using Selenium. This content has been unavailable for some time, so I decided to migrate it here. Many of the examples are unlikely to work, and I didn’t bother to migrate comments, but it might at least prove interesting to some. Personal highlights include playing Pacman, the first London meetup, and cheesecake. You can find the entire archive here.
-
Migrating to declarative Jenkins pipelines
Last year I shared my thoughts on Jenkins pipelines and provided a walkthrough of how we’re using pipelines at Mozilla. Since then, the Pipeline Model Definition plugin has came out of beta, and we’ve been migrating our pipelines to the new declarative syntax with a shared library.
-
Firefox: The Puppet Show
This year marked my return to FOSDEM, which I last attended in 2013. On my previous visit I did a joint talk with Henrik Skupin on Automating Firefox OS and we thought it would be fun to return with another talk on automating the desktop Firefox application using Selenium.
-
Analysing pytest results using ActiveData
When attending Selenium Conference 2016 in London, I was particularly interested in any talks on how others are analysing their test results. There have been many times I would like to things such as how many tests we’re running, what percentage of them are failing, and which tests take the longest to run. Rather than implement something from scratch, I was hoping that I could take inspiration from others. I wasn’t disappointed, as both Dave Cadwallader and Hugh McCamphill gave presentations on how they gather and analyse their test results.
-
Using Jekyll plugins with GitHub pages
I was a little disappointed to discover that Jekyll doesn’t offer a way to view all posts with a particular tag. Fortunately there’s a comprehensive library of plugins, including jekyll-tagging, however as I was letting GitHub pages take care of building my site, I was limited to the list of supported plugins. To work around this, I moved from a User Pages site to a Project Page site so that I could use separate branches for the source and generated content (User Pages only allows the
masterbranch to be used). Then, I found the neat jekyll-github-deploy tool, which builds the site and pushes to thegh-pagesbranch. So now each post includes a list of tags, and each tag links to a page listing the posts associated with that tag. -
Jenkins pipeline walkthrough
I’ve recently been migrating Mozilla’s traditional Jenkins jobs for functional UI testing of our web properties into pipelines. The following describes some of the common features of these pipelines. I’ve included my reasons for these design choices, and highlighted limitations that I’m hoping will be resolved as pipelines evolve. I’ve also written a post on my thoughts on Jenkins pipelines and IRC notifications in Jenkins pipelines.
-
My thoughts on Jenkins pipelines
Before Jenkins pipelines, all job configuration was stored as XML on the server, and only configurable via the web dashboard. There are a number of limitations to this approach.
-
Using Firefox options in pytest-selenium
I’m pleased to announce that pytest-selenium v1.7.0 has been released, and there’s a change to the way Firefox options are handled. Previously, the profile and binary path for Firefox were passed as keyword arguments when instantiating the Firefox driver.
-
IRC notifications from Jenkins pipelines
I’ve been migrating several of our Jenkins jobs to pipelines, and one of the challenges was preserving our IRC notifications whenever a build result changes. At this time, the IRC plugin for Jenkins doesn’t include support for pipelines, however it is still possible to trigger a notification using the
shstep. -
Lessons from mentoring
Now that my first two mentoring experiences have come to an end, I wanted to reflect a little on the experience. Before I do, I want to say that I’m incredibly proud of what Justin and Ana were able to achieve this summer. I feel lucky to have had the opportunity to mentor such talented, enthusiastic, and motivated individuals. I hope they feel they’ve learned something of value during their projects - I most certainly have!
-
A summer to mentor
This summer I am mentoring Justin Potts – a university intern working on improving Mozilla’s add-ons related test automation, and Ana Ribeiro – an Outreachy participant working on enhancing the pytest-html plugin.
-
Experimenting with Jekyll
I’ve decided that Wordpress is really overkill for what I need, so I’m going to experiment with using Jekyll with GitHub Pages to maintain a simple blog using Markdown. The idea of Blogging Like a Hacker certainly appeals to me, and if this experiment is successful I will likely migrate previous blogs into this one over time.
-
London Selenium Users Event #4
In a couple of weeks the 4th event of the London Selenium users meetup takes place. This time it will be hosted by Thoughtworks, and has a loose theme of extending on top of Selenium. I’m pleased to announce the following speakers, in addition to a talk from Google on mobile testing with Selenium.
-
Selenium Conference 2011 - Survey
The first ever Selenium conference is currently being planned for sometime in April next year. Get involved by responding to this short survey. Your responses will be used to help plan the conference.
-
Videos now online for 'Advanced Test Automation with Firefox'
The videos from the third London Selenium users event are now available on the London Selenium YouTube channel.
-
Meetup.com Attendance Lists
For the recent London Selenium Users Meetup Event I was asked if I could provide the attendance list in a suitable format for creating labels for the guests when they arrive. Given the short timeframe I did this simply by highlighting the names on meetup.com, copying them, and pasting them into a simple text editor. I then quickly cleaned this up before sending the list on. Within a short while the list was out of date due to some members dropping out.
-
Cheesecake
Here’s another one of my odd Selenium examples! Last month I finally got to eat at a ‘Cheesecake Factory’ restaurant, after first hearing about them a few years back from an American friend. There’s an overwhelming choice of 30+ cheesecakes to choose from, and I’m really not good at making decisions when there’s so much choice. I decided to go with ‘The Original’, and that every subsequent visit to the chain I would work my way down the menu, knowing I’d probably never get through all of them in my lifetime!
-
How Mozilla Uses Selenium
Here are the slides from Stephen Donner’s talk on how Mozilla uses Selenium.
-
London Selenium Meetup Event #3
This week saw the third London Selenium Users Meetup event, with presentations from Mozilla and an update from Simon Stewart on the progress of Selenium 2. You can read my full write-up of the event on my personal blog.
-
New London Users Event: Advanced Test Automation with Firefox
Although it’s been scheduled for a while, the next London Selenium Users meetup event has now been officially announced. As before, Google will be generously hosting the event, which this time is Firefox flavoured! There will be four presentations, mostly by folks from Mozilla, but also an update from Simon Stewart on the progress of Selenium 2. Head over to the event page on meetup.com for all the latest details.
-
Checking Wedding Gifts with Selenium
Admittedly this one is unlikely to be useful to many, but it could still serve as a useful example for Selenium 2. Also, I haven’t posted an example since alpha 5 was released.
-
Proposal for new Selenium Q&A site
If you’re not already aware of the dedicated Selenium Q&A site that I have proposed, then it’s definitely worth checking it out. If you’re interested in supporting the proposal, and helping it to reach beta then please commit. By committing you are stating that you are prepared to both ask and answer questions on the proposed site.
-
Selenium 2 in .NET framework
I’m not Dave but you might have met me at one of the London Selenium meets. My name is Zac and I have several years of automated testing experience with Selenium RC in Java and C# and am now intending to be one of the early adopters of Selenium 2 using the .NET version.
Selenium 2 and .NET are up to alpha4 already but thorough examples of code on the internet are a bit thin on the ground so far compared to the Java equivalent. In my time using Selenium RC and now Selenium 2 I have built up a framework to support it.
-
Introducing Zac
As he’s preparing his first post for SeleniumExamples, it’s probably a good time to introduce Zac! I have worked with Zac in the past and he recently mentioned having some suitable content for this blog. I thought it’d be easiest to just give him an account.
I suspect most of Zac’s contributions will be .Net or C# examples, which is something I have little experience of myself. If you’re also interested in submitting content to the blog, get in touch with details of what you’d like to contribute and a bit about yourself.
-
Improving Selenium IDE
I’ve been working on a new formatter plugin for Selenium IDE, and along the way I discovered some quirks (not necessarily bugs) in the code. After a few discussions with some of the Selenium community, I decided to get stuck in and see if I couldn’t make some improvements. In the interest of sharing my experiences, here is what I did.
-
Get a new iPhone with help from Selenium
I’ve been looking forward to the iPhone 4 ever since the 3GS was released, as I was in contract at the time and decided to wait another year for whatever Apple decided to release next. Due to unexpected (but very cool) circumstances, I was out of the country on the day Apple released the iPhone 4 and so was unable to queue up for one of the devices, and now due to the demand it’s very difficult to get hold of one. Since getting back to the UK I have been visiting my two nearest O2 stores to check if they have had a delivery, and today they pointed me towards their online stock checker…
-
Play Pacman with Selenium 2
When I saw Google’s recent interactive Pacman doodle to celebrate the game’s 30th anniversary, my first thought was ‘Wouldn’t it be cool to automate playing Pacman using Selenium’. I know - I’m a geek!
-
Mozilla & LinkedIn using Selenium
Last night was the May Selenium Users meeting in San Francisco, where Mozilla and LinkedIn presented their use of Selenium. Adam Goucher (Selenium IDE maintainer) was there, and has provided his notes on his blog. It sounds like it was a great event, hopefully there’ll be some videos online soon!
-
GTAC 2010 Announced
The Google Test Automation Conference (GTAC) for 2010 has now been announced on the Google Testing Blog. This year the event will be at the Novotel in Hyderabad, India. More details can be found by following updates on the official GTAC website.
-
Survey Results from London Selenium Users
Before holding the second London Selenium Users meetup we asked several questions of the Selenium users in London
-
Selenium 2 User Interactions API
Here are the slides from Eran Messeri’s talk on the advanced user interactions API that he’s currently working on for Selenium 2.
-
What does Selenium 2 mean to the average QA?
The slides and demos from David Burns’ talk at the second London Selenium Users meetup are now available on his blog.
-
From IDE to Selenium 2
Here are the slides from my ‘From IDE to Selenium 2’ talk that I gave at the second London Selenium Users meetup. Slides from the other talks to follow.
-
Another successful Selenium meetup in London!
Last night was the second London Selenium Users meetup. In contrast to the last one - a purely social event in a pub in Fitzrovia - this one was hosted by Google with talks from myself, David Burns (The Automated Tester), Eran Messeri (Google), and a Q&A with Simon Stewart (Google).
-
Selenium IDE - My plugin baby
With last month’s release of Selenium IDE 1.0.4 came initial support for plugins, and this month’s release of 1.0.5 continues to build on the new plugin API as well as fixing a few bugs.
-
New London Users Event: Introducing Selenium 2
Today the ‘Introducing Selenium 2’ London Selenium Users event was tentatively announced for April 20th 2010. Please visit the event page to register your interest. The date will be confirmed when a venue has been decided. If you would like to suggest a venue, please send me a message.
-
Selenium 2 Examples
Yesterday Selenium 2 (alpha 1) was released. This is the first release since the Selenium and WebDriver projects started to merge. The main difference is the inclusion of the WebDriver API into Selenium. I’ve put together a small example below that uses the new API to log into two web based e-mail clients and send an e-mail.
-
Improving Selenium Grid
Hopefully Selenium Grid 1.0.5 will soon be released, with the much anticipated self-healing features that will save me so much time when RCs go AWOL. Looking further ahead, I would like to see some minor improvements to the Selenium Grid Console such as integrating this very handy Greasemonkey script for unregistering Remote Controls and sortable columns.
-
A Successful First London Selenium Meetup
The first London Selenium meetup took place on the 11th November. If you weren’t able to make it then due to the success I will be scheduling another one early in the new year.
-
Construct unique safe filenames in TestNG
In order to save files for investigating failures from within TestNG it’s important to have a safe filename that is unique to the test - otherwise you may overwrite important files. I have written the following simple method in Java that is called from a listener with an
ITestResultparameter to construct a unique file name that should be safe on most file systems. -
Capturing screenshots from remote Selenium RC
Despite the name, the Selenium RC (Remote Control) server is often run on the same machine as the testing framework, which makes saving screenshots to disk quite easy. If however you are running Selenium RC on a separate machine, or are using Selenium Grid it can become more difficult as the screenshots are also saved on the remote machines.
-
Using soft assertions in TestNG
One of the big differences between Selenium IDE and a Selenium RC solution is the ability to perform ‘soft’ assertions. Selenium IDE users can append commands with
verifyorassertto determine whether the test execution should stop when a failure is observed. A popular use for this is to first assert that you are on the correct page (assertTitle) and then verify elements on the page. If you were only able toassertthen your tests may fail early on, not revealing further failures that may exist. -
London Selenium Meetup increases RSVP limit
Due to the impressive response to the first London Selenium Meetup event, I am pleased to announce that I have increased the RSVP limit from 30 to 40. Head over to the Meetup or Event page to join and RSVP. If the 40 places are already full then add yourself to the waiting list and you’ll be notified if anybody drops out.
-
Drag and drop to object within a scrolling block
The
dragAndDropToObjectcommand works really well in Selenium, however it does have some limitations. One such limitation I came across recently while writing tests for an ExtJS web application: when your destination object is in a scrolling box (and not in view) the command fails. -
Select an option from an ExtJS ComboBox
Automating an ExtJS web application can be difficult due to the dynamic nature of the page. For example, the majority of unique ID attributes in the HTML will be different between builds, which causes problems locating elements reliably. Another issue is selected items from a ComboBox, which is not a normal HTML
<select>element, but an<input>that is populated from data in a completely separate section of the DOM.
subscribe via RSS