Cypress vs Selenium

Selenium based test automation framework:

In our little QA world, we have built a Selenium-based test automation framework to automate End-to-End GUI tests. So far 100 test cases are automated and integrated with Jenkins. Test cases are getting executed every day on the dev and the production environment. The daily run has caught 60+ issues so far!

Selenium is just a library, around which we have built a framework. We have leveraged TestNG - a java based framework. We have incorporated logging, extent reporting, screenshot taking capability, screen recording capability. The framework is well modularized with the POM design pattern. We could easily extend this framework to develop non-selenium functionalities such as Facebook graph API interface, Twitter API interface, WO API interface, Gmail API interface etc

The automation framework once developed was fine-tuned to overcome the challenges thrown by Selenium. Explicit wait (Webdriver wait) was added to wait for a certain amount of time until it finds the desired element. Several retry mechanisms such as retrying a test case if it fails, retrying finding element or clicking element with a different method etc. were also incorporated.

Well! A lot of effort went in to stabilize! And to maintain locators which get changed very frequently!

Okay! Then what is the problem here to continue to use? And why did we start hunting for a better tool?

Test execution is time-consuming. It takes a whole night to complete the execution on the dev and the production environment. As the test case base grows, we need more machines to run! Or We need to execute the test cases in parallel. New test case development time is considerably more. We can not get instantaneous test reports!

No doubt Selenium’s market share is still high and it is a very popular test automation framework. But there is a new claim to the throne! CYPRESS! Cypress aims to overcome the hurdles faced while using Selenium. I started researching about Cypress, did a POC and really liked it. I also did research on all other tools in the market to find a better tool than Cypress.

Overview of different test automation frameworks:

Cucumber(BDD): Cucumber is a testing approach which supports Behavior Driven Development (BDD). It explains the behavior of the application in a simple English text using Gherkin language. Most of the organizations use Selenium for functional testing. These organizations which are using Selenium want to integrate Cucumber with selenium as Cucumber makes it easy to read and to understand the application flow.

Capybara (BDD): Capybara is a web-based test automation software that simulates scenarios for user stories and automates web application testing for behavior-driven software development. It is written in the Ruby. Capybara can mimic the actions of real users interacting with web-based applications. It can receive pages, parse the HTML and submit forms.Capybara can be used with RackTest (default driver). It can also be used with Selenium WebDriverDevelopers describe Capybara as an "Acceptance test framework for web applications".

Mocha: Mocha is a feature-rich JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple and fun. Mocha tests run serially, allowing for flexible and accurate reporting while mapping uncaught exceptions to the correct test cases. Hosted on GitHub.

Chai: Chai is a BDD / TDD assertion library for node and the browser that can be delightfully paired with any javascript testing framework. We can download chai for Node. Chai package is available on npm. Developers can develop plugins, plugins help extend Chai’s assertions to new contexts. One such featured plugin is chai webdriver. This helps to integrate chai with selenium webdriver.

Robot Framework(Keyword driven): Robot Framework is a generic open source automation framework. It can be used for test automation. The framework has a rich ecosystem of libraries and more custom libraries can be added. It is a keyword driver framework. Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development(ATDD).

Protractor: Protractor is an end-to-end test framework for Angular and AngularJS applications. Protractor runs tests against your application running in a real browser, interacting with it as a user would. Protractor is a library that wraps Selenium and provides us with improved syntax and special built-in hooks for Angular.

Watir: An open source Ruby library for automating tests. Watir interacts with a browser the same way people do: clicking links, filling out forms and validating text. Again, Selenium under the hood. Better API’s are provided but less community support.

Nightwatch: Nightwatch has its own implementation of the selenium WebDriver. And provides its own testing framework with a test server, assertions, and tools.

TestCafe (Not Selenium based): TestCafe is a great alternative to Selenium-Based tools. It was rewritten and open-sourced at the end of 2016. TestCafe injects itself into the website as JavaScript scripts instead of controlling the browsers themselves like Selenium does. This allows it to run on any browser, including on mobile devices, and have full control over the JavaScript execution loop.

A note about Cypress:

Cypress is a direct competitor of TestCafe. They are doing relatively the same, which is injecting tests into a website, but they try to do it in a more modern, flexible and convenient way. The difference between them is that Cypress.io runs itself in the browser and controls your tests from there where TestCafe runs in Node.js and controls the tests through a serialized communication with its injected script in the browser.

Conclusion: I found out that most tools in the market are either selenium based, built as a wrapper around selenium or used the same principles. Cypress architecture and concepts are very different from Selenium.

Selenium Architecture:

Let us understand the Selenium architecture first before going into the Cypress architecture. WebDriver drives a browser natively, as a user would, either locally or on a remote machine using the Selenium server, marks a leap forward in terms of browser automation.Selenium WebDriver refers to both the language bindings and the implementations of the individual browser controlling code.

Advantages of selenium:

Open source and a large community support
Multi-browser support
Support across various operating systems
Language support
Easy to learn and easy to use
Parallel test execution

Disadvantages of selenium:

Supports web testing only (We can use Appium for mobile testing)
Test case writing is time consuming(We are coding more as per automation standards!)
Locator selection gets tricky
Limited support for image testing
Framework creation around Selenium is time consuming
Selenium tests are flaky

Cypress: A modern test automation framework:

Cypress is a next generation front end testing tool built for the modern web. Developers or QA engineers can use this to do the following

End-to-end tests
Integration tests
Unit tests
API tests

Cypress can test anything that runs in a browser.

Cypress ecosystem: Cypress consists of a free, open source, locally installed Test Runner and a dashboard service for recording tests. Cypress can be integrated with CI providers.

Cypress Features:

Time travel: Cypress takes screenshots while running tests
Debug ability: Readable errors and stack traces are useful
Automatic Waiting: Never add waits or sleeps to your tests
Spies, Stubs, and Clocks: Verify and control the behaviour of functions, server responses, or timers.
Network Traffic Control: Easily control, stub, and test edge cases without involving your server.
Consistent Results: Cypress architecture doesn’t use Selenium or WebDriver. Consistent and reliable tests that are flake-free.
Screenshots and Videos: View screenshots taken automatically on failure, or videos of your entire test suite when running from the CLI.

Cypress Architecture:

Most testing tools (like Selenium) operate by running outside of the browser and executing remote commands across the network. Cypress is the exact opposite. Cypress is executed in the same run loop as your application.

Behind Cypress is a Node server process. Cypress and the Node process constantly communicate, synchronize, and perform tasks on behalf of each other.

Cypress is built on top of Mocha and Chai. Cypress supports both Chai’s BDD and TDD assertion styles. Tests you write in Cypress will mostly adhere to this style.