Head to head: Playwright, Cypress und Selenium

Automated UI-tests have become an essential part of the software development process. They serve as regression tests and thus help developers to ensure that their applications are stable, reliable and perform as expected. As the complexity of web applications increases, so does the need for powerful and efficient test automation (TA) frameworks for the system under test (SUT). For years, Selenium has been the number 1 player when it comes to TA frameworks in web. Many other well-known TA frameworks and tools like Tosca and Katalon use or used Selenium WebDriver under the hood for browser automation. However, for the last few years there are new players in the field which are gaining a lot of interest lately and are increasingly being used more and more. 

In this blog post, we compare three popular open-source-frameworks for test automation: Playwright, Cypress and Selenium’ We’ll go over their strengths and weaknesses, their compatibility and in which application areas they make the most sense.  
But first, ‘let’s go back to the beginning. 
 

Selenium

The first version of Selenium was released in 2004 by Jason Huggins. At that time, it still ran with the so-called Selenium RC. Today’s Selenium is different: It runs outside the browser and controls it automatically with an external interface (WebDriver).  

Apart from Selenium-Framework, there is also a browser plugin called Selenium IDE. This runs directly in the browser; tests can be recorded and replayed directly. An export to the Selenium-Framework is also possible. 

Back to the Selenium Framework itself.  
This can be used in a large number of programming languages: Java, C#, Python, Ruby and several other languages.  
Likewise, Selenium also supports quite a few different test frameworks due to this language diversity. Here is an selection:  

• JUnit (Java)
• TestNG (Java)
• Cucumber (BDD, Java)
• MSTest (C#)
• XUnit (C#)
• SpecFlow (BDD, C#)
• PyUnit (Python)
• UnitTest (Python)
• Behave (BDD, C#)
• RSpec (BDD, Ruby)

This wide compatibility/versatility is one of the reasons why Selenium is the leading web automation framework. Selenium is also ranked #1 in the Swiss market, as our Trends & Benchmarks Report this year shows. Let’s move on to the weaknesses: 
Selenium can be used well for large and complex projects.  
BUT: The architecture is outdated. Communication with Selenium runs via a http-request. First a http-request is sent directly from the Selenium-API to the Selenium Webdriver, which then communicates directly with the respective browser. All drivers can also be implemented in a tool called “Selenium Grid” where you can easily switch drivers and run tests on other machines. Through these different webdrivers it is also possible to test native Android/iOS apps (Appium). For each driver-action a separate http-request is created and terminated at the end. (well explained again here) This takes a lot of time. Other frameworks like Playwright solve this more efficient like with websockets. (see Playwright). 

In contrast to Cypress or Playwright, the user has to take care of much more here: starting and stopping the browser correctly etc., this sometimes turns out to be difficult, very rudimentary: usually a custom wrapper framework is built around it.

Cypress

Developed in 2014 by Brian Mann, the framework called Cypress was the first notable competitor to Selenium. This Chromium-based project runs directly in the browser, unlike Selenium:  

It creates a sandbox, in which the web pages are tested. Test automation and front-end developers can use it to write automated tests in JavaScript or TypeScript. Cypress is under the MIT-license, so it can be used for free, while Cypress Cloud can also be used for free, but with better support and more features, it has to be paid for. As foreshadowed in Selenium, with Cypress the user does not have to worry about starting or stopping the browser correctly.  

This and the fact that everything takes place in the same browser ensure a certain speed. Cypress specializes in E2E and component testing. There is an eco-system of frameworks that are perfectly aligned: The user can start automating very quickly. 

Cypress also offers the possibility to manipulate requests during test execution, i.e. mocking and stubbing. 
For this you can use different methods like “cy.intercept” or “cy.request”. With these functions you can e.g. use mock data or simulate test cases for different network conditions. The whole concept is called request intercepting. Furthermore Cypress includes an own trace viewer (Time Travel) with fantastic functions for debugging. In this way, every action that was taken in the browser and which requests took place can be traced back. Another big advantage is the documentation and the community support of Cypress. This is strongly given. With Cypress Cloud it is even possible to run tests in parallel. 

But here we are already directly at the disadvantages: Only one tab is supported for an execution, i.e. tests, that open new tabs, are not possible. Only one concurrent browser instance per test execution is supported. This will not change. This is because Cypress focuses on applications where all actions take place only within the same tab (such as SPAs). Older applications that open their own tabs can therefore not be tested by Cypress. With large SPAs or SUTs, it can also happen that the browser crashes because both the SPA/SUT and the test runner/execution are running in one tab at the same time and it’s too much for the memory. In the past only Chrome was supported, now it runs the most common browsers (Chrome, Firefox, Edge, Safari Webkit).

Other programming languages or test frameworks are not supported. 

But that’s where a new player comes into play that, at least currently, has the potential to possibly even replace Cypress.  

Playwright

Playwright is the youngest player in the round. Developed by Microsoft in 2019, it is currently experiencing strong growth. JavaScript frameworks like Vue.js offer Playwright integration out of the box. It specializes in Chrome, WebKit(Safari) and the Firefox browser. Playwright tests can be written in various programming languages (TypeScript/JavaScript/Python/ C# and Java) and various testing frameworks are supported: e.g. Jest, Jasmine, Mocha, Vitest. Playwright supports out of the box parallel execution of tests on the local machine. For CI, the testsuite must be splitted over a number of CI runners.  

Like Selenium, Playwright runs outside the browser. Unlike Selenium however, websockets rather than http requests are used for communication. With websockets a connection is established and it remains open. In this open stream data can be exchanged in both directions at any time. Not like with http-requests, where every time a connection is established and closed afterwards.

Playwright uses the so-called ChromeDevTools for each browser to communicate with the browser. This technology is already used by the browsers themselves, which means that Playwright does not need a WebDriver. Multiple browsers at the same time are also supported!

Playwright thus offers the best of both the Selenium and Cypress worlds: 
The speed of Cypress and the language support of Selenium. 

Just like Cypress, Playwright also includes its own trace viewer with fantastic debugging features. 

Our colleagues from Qxperts have published a great video about this:  

Now for the weaknesses: 
Playwright is still very young. Accordingly, the community and its support is still very limited. In our report about the Swiss market, Playwright could not yet crack the 5% mark. Furthermore, only the most common browsers are supported. Chromium-based browsers run out of the box.

Tests on WebKit and Firefox are also possible. Unlike Cypress, Playwright supports multiple browser instances (called browser context) within a test. However, tests on native cell phones are only supported to a limited extent (there are more possibilities with Selenium). 
In contrast to Cypress/Selenium, which has been on the market for much longer, Playwright is still quite new and the community support is therefore somewhat limited. 

Conclusion

If we compare the different frameworks, we notice that especially Cypress and Playwright are becoming more and more popular. The difference between Cypress and Playwright is that Cypress comes as a “package”, i.e. there are certain default structures in a precisely defined eco-system (test framework, programming language, order structure, etc.).  
Playwright offers more freedom (different test frameworks / programming languages). Cypress runs with Mocha as test framework, Playwright supports Mocha (but also Jest, Jasmine…).  

Between Cypress and Playwright should be decided by structure and flexibility for automated testing. If we are in a Javascript technology stack with Mocha as the testing framework; Cypress for example would be a good choice. However, if we are using a completely different technology stack such as one from Microsoft, Playwright with its technology openness makes more sense. It must always be considered: what technologies are already in use and are there already defined framework standards in the company?  

At this point, all 3 frameworks can be used for small to large projects. For larger and complex projects (several different browsers, several tabs, connections to external APIs, etc.), however, you would currently still have to work with Selenium and Playwright if you do not want any restrictions in this respect: Tests should preferably remain in the same technology stack in which the SUT is located. A program in a JavaScript stack should rather be tested in JavaScript technologies. Cypress or Playwright is a great choice for that. A program in a Microsoft stack should rather be tested in Microsoft technologies etc. Selenium or Playwright is a good choice for that. The technologies are built to work optimally with each other. Only the support is more given with Selenium than with Playwright. 

A big difference between Selenium and Cypress compared to Playwright is that the former two have a very strong and large community and documentation support. Playwright is just too young and new to the market for that.

Speaking of limitations: Both Cypress and Playwright are still rolling out new features all the time. We at SwissQ/Xebia think it is important to look into both. Especially since the usage of both frameworks is increasing rapidly. Playwright has a simple architecture, is fast and reliable, and just like Cypress, there are new features coming out all the time. For this reason, we also work with the frameworks and also give Playwright and Cypress courses in our Academy: 

• https://swissq.it/course/end-2-end-testing-with-playwright/
• https://swissq.it/course/test-automation-using-cypress/

For courses in Netherlands see here.

In summary, it is an exciting head-to-head race. As our benchmark analysis shows, Selenium is still the big player number one.  Appium and Tosca also use(d) Selenium. 

Next to Selenium, Cypress is catching up very strongly and so is Playwright. Cypress has doubled in size within a year. Playwright has not (yet) made it over the 5% in our swiss market statistics this year. 

So, what’s for the future?  
Of course, there are many other aspects that one could go into, e.g., pipeline support, but they would go beyond the scope of this blog. In any case, we are excited to see what the future will unfold. 
If you have any further questions, don’t hesitate to contact us.