The objective of this post is to get a simple test running in BrowserStack, SauceLabs, and CrossBrowserTesting. In doing so, I want to compare the features and document the experience of running my first test from a trial account.
BrowserStack includes the following services: Live, Automate, and Screenshots & Responsive.
This allows you to open a VM within the browser in different OS's on different versions of IE, Firefox, Chrome, etc. and manually click test.
This will only sites that are live on the web. To use this on a local server, you'll have to install the Chrome Extension
This allows you to run your local selenium tests on BrowserStack. There are a quite a few options depending on which testing tool you'd like to use. When clicking on the
Docs tab in the menu, there are docs for setting up in each of the following platforms:
Using the API credentials in Account Settings, you can get a test running on BrowserStack within minutes. If you're testing in ruby, BrowserStack has a repo with some sample tests in rspec, capybara, and cucumber watir.
When you run your tests on BrowserStack, you have the ability to watch a recording of them running. You can see the test loading a webpage, clicking, filling in forms, etc. If a test is failing, you can try to debug it in the
Screenshots & Responsive
This feature allows you to provide a URL where BrowserStack will visit the site with a multitude of operating system & browser combinations.
Here's a sample test that logs into PagerDuty and asserts that the h1 text is correct.
require_relative './spec_helper.rb' require 'selenium-webdriver' describe "Visiting Sign in page" do it "loads the form successfully", :run_on_browserstack => true do @driver.manage.timeouts.implicit_wait = 10 @driver.navigate.to "https://pdt-browserstack.pagerduty.com" raise "Unable to load PagerDuty." unless @driver.title.include? "PagerDuty" query = @driver.find_element :id, "user_email" query.send_keys ENV['username'] query = @driver.find_element :id, "user_password" query.send_keys ENV['password'] query.submit expect(@driver.title).to eql("Incidents - PagerDuty") header = @driver.find_element :css, '.page-header h1' expect(header.text).to eql("Incidents on All Teams") end end
You can see this gist for more details on the driver/browserstack setup. BrowserStack also provides this tool to help you construct your
caps config which is a config where you set
version, etc. You would then enter these values in your driver.
BrowserStack can run up to 10 parallel tests depending on the plan you choose.
I reached out to BrowserStack and they currently do not have any way to host a test suite and automatically run it. The recommended approach is to have your CI run the BrowserStack tests.
It seems like billing is based on the feature set. There are different pricing plans if you'd like the Live feature set vs Automate. The plan types are Live, Automate, and Screenshots & Responsive.
Each plan for Automate comes with a limit on how many hours per day you can use, which is their "Fair Policy". Here's the definition: https://www.browserstack.com/question/619
After playing with BrowserStack for a little over a day, it is a solid tool. The documentation is really good and easy to follow.
The one thing I don't like though is having Live and Automate as separate plans. To me they seem to go hand in hand. When an automated test fails, you will probably want to login to a VM and try the same steps to see why it's failing, which you'd have to pay extra money for if you were on the Automate plan.
I do believe that you would get a certain amount of minutes for Live if you were on Automate. The free plans have minute limitations, but once you run out, you're screwed.
SauceLabs provides manual and automated testing on what they call "The World's Largest Cloud Based Selenium Grid".
Just like BrowserStack, SauceLabs has the ability to execute your automated tests on their cloud infrastructure and the ability to fire up a VM in different OS/browser combinations and click around.
SauceLabs does not have the feature to take screenshots like BrowserStack does, although you can see screenshots in the individual test runs as well as a recording of the test run.
This is comparable to BrowserStack's Live functionality. To connect to a local server, you'll have to setup a tunnel.
Getting a test running on SauceLabs is very similar to BrowserStack. You basically just need to grab the access key from your account and plug that in to the driver for your test.
SauceLabs also provides the ability to see a video recording of your test run.
In my opinion, it's a little harder to navigate the docs as it's written wiki style. Things are organized more as guides and how-to's rather than by testing tool/technology.
I was able to use the exact same test, but just had to change the driver. You can use this tool to gather the
caps config options to enter into your driver.
Here's a gist that runs the same PagerDuty login test, but in SauceLabs.
SauceLabs has a JIRA plugin where it can file a ticket when a build fails. It will prefill the ticket with all of the metadata around the test failture. This is very cool if your team uses JIRA.
SauceLabs currently has SauceLabs runner in Beta. This is a feature that will allow you to provide SauceLabs with a github repo URL that contains your SauceLabs tests. SauceLabs would then run these whenever you login and manually trigger the build. They also provide a REST Curl command where you can call it from your CI server or what scheduled process you may have. Note that this is still in beta and is only offered to some users.
SauceLabs can run up to 8 concurrent sessions depending on the plan. Or if you choose the Enterprise plan, then the sky's the limit.
SauceLabs provides manual and automated testing on all plans, but will limit the amount of concurrent sessions and automated testing minutes based on the plan you choose. The minutes are also on a monthly basis as opposed to daily.
I like SauceLabs and the how the plans include unlimited manual testing. To me, this is a crucial feature to have so that you can troubleshoot why an automated test may be failing.
Sauce Runner will also be a handy feature once released. I'm curious if they will allow a schedule to be set.
The JIRA plugin is also a nice to have. This will allow a smooth workflow when a build fails.
As a developer, I'd be more inclined to choose SauceLabs over BrowserStack, but we still have to check out Cross Browser Testing first.
Cross Browser Testing
CrossBrowserTesting touts itself by the fact that they use real, physical devices. They have the usual manual testing and automated testing.
CBT also has the ability to take screenshots, which seems comparable to BrowserStack's screenshot functionality.
CBT has a neat bookmarklet that will allow you to take screenshots in various OS/browser combinations or hop in a live test session (in the OS/browser of your choice) of the current page that you're in. So for example if I was on google.com and I fired the bookmarklet, I can see google on Windows 7 IE 10 assuming I chose that.
Testing against a local server
CBT provides the option to run tests again a local server either by downloading their Chrome extension or using their Java or node.js module in the command line.
CBT can run up to 20 concurrent tests (business plan).
Because of the lack of documentation, this test took a little extra time to setup (like knowing which CrossBrowserTesting capabilities are required). There are also some manual things you'd have to do if you want CrossBrowserTesting to know if a suite fails.
Upon failure, you're supposed to send a
POST to the score API endpoint and set the score to
unset. Also if you want a screenshot upon failure, you have to send a
POST to the screenshot API.
Here's is a gist with an example rspec test logging into PagerDuty on CrossBrowserTesting.
After execution, you can see a video recording of the test run.
Their billing model is similar to SauceLabs in that manual testing is unlimited across all plans, but users, automated testing minutes, and concurrent tests are increased as you move up the tiers.
Cross Browser Testing seems like a young product, but if testing with real, physical devices as opposed to VMs is a big deal to you, then this would be a good choice.
The product actually works well and the UX is clean, but the lack of documentation makes me a little nervous if I were to choose CBT.
Running a simple bash one-liner, I recorded the execution time of each test run. I ran the test 3 times on SauceLabs, CrossBrowserTesting, and BrowserStack. I would've ran it more times but I only get a limited set of minutes on a trial account. Anyways, here are the results:
The test basically visits a PageDuty sign in page, logs in, and asserts the correct title. The test is run in Windows 7 on IE10.
Take these benchmarks with a grain of salt. Maybe one of the services is having a bad day. I did only run the test 3 times.
I think BrowserStack, SauceLabs, and CrossBrowserTesting are all good in their own right. If I were a UI designer, I would love the screenshot capability that BrowserStack and CBT provide. If you're on a budget, then you also have to pay attention to the fact that BrowserStack charges separately for manual and automated testing. But I would say that BrowserStack is ideal for UI designers because of it's support, documentation, and good product UX.
As a developer, I like SauceLabs and BrowserStack. If you're going to write tests that run in a multitude of browsers/platform combinations then concurrency will reduce the amount of execution time. If I'm concerned with budget however, then I would go with SauceLabs. They seem to provide more for less money, just by comparing their pricing plans (looking at testing minutes and the fact that all plans come with unlimited manual testing).
The only edge that CrossBrowserTesting has is the fact that they run on real, physical devices. This has its drawbacks in speed and parallelization though. If this is what you're looking for, then this is the service for you.