Table of results from running the same regression suite in Playwright, Selenium and TestCafe

Regression Testing Tools (2026): We Ran the Same Suite in 3

Regression testing tools re-run the checks you already trust every time the code changes, so a new release does not quietly break an old feature. Most guides list tools and quote the vendors. On 17 November 2026 we did something different: we wrote the same three regression checks for a small demo shop in Playwright, Selenium and TestCafe, ran each suite 10 times, and then renamed one button to see what broke.

Quick answer

For a new JavaScript or TypeScript suite, pick Playwright. In our run it passed 10 of 10 times in about 2.5 seconds with no waits written by us. Selenium also passed 10 of 10 and was the fastest, but only after we added explicit waits; our first version passed 1 time in 10. TestCafe passed 10 of 10 and needed no waits, but took about 8.4 seconds. None of the three repaired a test after we renamed a button.

What we tested and how

The target was our AITG Demo Shop, a tiny fake store with fixed data that we run on our own computer. It has a product list, a cart, a checkout form and an orders page. We wrote three regression checks and implemented them once per tool:

  • Buy a keyboard end to end: open the shop, add the Mechanical keyboard, check the cart total is $49, check out with an email, see “Order #1”.
  • Reject an empty-cart order: send a checkout request with no items and expect HTTP 400.
  • Order appears on the orders page: buy a USB-C hub, open the orders page, find “#1 – USB-C hub – $29”.
ItemVersion or setting
Playwright@playwright/test 1.65.1
Seleniumselenium-webdriver 4.52.0, ChromeDriver 153.0.8010.52
TestCafetestcafe 3.7.7
BrowserChrome 153.0.8010.12, headless, same version for all three
MachineOne Linux cloud sandbox, Node 22, run on 17 Nov 2026
MethodThe whole suite 10 times per tool, one after another; each run starts the browser from scratch

Timing includes starting the browser, so it is the time you would wait for a small suite in a script, not a speed benchmark. Run on one machine with three checks, it says nothing about large suites, parallel runs or CI.

Results: pass rate, time and code size

ToolPassed (10 runs)Average timeTest codeWaits we had to write
Playwright10 of 102.5 s (2.4 to 2.6)25 linesNone
Selenium, first version1 of 101.6 s41 linesOnly one explicit wait
Selenium, after waits for links and form5 of 101.7 s42 linesWaits before each click
Selenium, final version10 of 101.9 s42 lines8 explicit waits
TestCafe10 of 108.4 s (8.3 to 8.6)25 linesNone

The Selenium row is the one to read carefully. Our first version failed 9 times in 10 with no such element on the Checkout link, because the browser had not finished loading the next page when the script looked for it. We added waits for the link and the form, and it still failed half the time, because the test jumped to the orders page before the order was saved. Only after we waited for the confirmation did it pass every time. The tool did what it was told; the tests were missing waits. Playwright and TestCafe wait for elements automatically, so the same flows passed on the first try.

// Playwright: no waits needed, it retries until the element is ready
await page.getByRole('button', { name: 'Add to cart' }).click();
await expect(page.getByTestId('total')).toHaveText('Total: $49');

// Selenium: you write the wait yourself
await (await d.wait(until.elementLocated(By.linkText('Checkout')), 5000)).click();

TestCafe was the slowest. Its own report shows about 6 seconds for the three checks, and browser start-up adds the rest of the 8.4 second average. It ran the same checks without a single wait. If your suite is small and you run it often, that overhead adds up. On a larger suite it matters less.

What happens when the page changes

Regression suites spend most of their life being repaired, so we tested that too. We restarted the demo shop with one change: the button that says “Add to cart” now says “Add to basket”. The product still works exactly as before. This is the kind of harmless change that breaks tests in real projects.

ToolTests failedTime to failWhat the error said
PlaywrightThe 2 that click the button62 s, which matches two tests each hitting the 30 s default timeoutWaiting for getByRole button named “Add to cart”
SeleniumThe 2 that click the button0.8 sno such element for the XPath of “Add to cart”
TestCafeThe 2 that click the button27.6 sThe specified selector does not match any element in the DOM tree (it shows the Add to cart line)

Two lessons. First, none of the three repaired the test, so any claim of self-healing has to come from a different kind of tool, and we have not tested one for this article. Second, how fast a tool fails matters in CI: Selenium told us in under a second, while Playwright made us wait a full minute unless we shorten its timeout. All three named the thing they could not find, which is what you need to fix it quickly.

Which tool should you choose?

Other kinds of regression testing tools

The open-source tools above run your own scripts. Other products add a layer on top. We did not run any of the tools below for this article, so we only point you to our existing pages for each kind.

KindWhat it addsWhere to read more
Codeless and AI-assistedRecord or describe tests instead of coding them; some claim self-healing locatorsmabl review, Testim review, Katalon review
Visual regressionCompare screenshots between releases to catch layout changesVisual regression testing guide, Applitools alternatives
Cross-browser cloudsRun the same suite on many browsers and real devicesBrowserStack review, BrowserStack vs Sauce Labs

Not sure where to start? Our tool finder and best AI testing tools page narrow the list by what you need.

What we did not test

  • Cypress: its installer could not download its program in our sandbox, so it is not included.
  • Large suites, parallel runs, CI pipelines and cloud browsers.
  • Any commercial tool, including self-healing and AI features.
  • Firefox and Safari. All runs used Chrome 153.

Frequently Asked Questions

What is a regression testing tool?

It is software that re-runs saved checks after each code change to confirm that existing features still work. Some tools run scripts you write, such as Playwright or Selenium. Others record tests or compare screenshots.

Which regression testing tool is best for beginners?

In our run Playwright needed the least code and no waits, so it was the easiest to get a stable result. TestCafe was also simple. Selenium works well but you have to add waits yourself.

Is Selenium still worth using for regression tests?

Yes, if your team already uses it or needs its language support. It was the fastest to run in our test. Expect to write explicit waits, or the tests will fail at random.

Can regression tests fix themselves?

Not the three open-source tools we ran. After a button was renamed, all three failed the tests that used it. Some commercial tools say they repair locators automatically, but we have not tested that.

How many times should I run a new test before trusting it?

Run it at least 10 times in a row. The flaky Selenium version in our test passed once in 10 runs, and a single run would have shown either result.

Sources

Was this helpful?Suggest an edit
Zain Ul Rehman
Zain Ul Rehman

QA automation engineer. Each guide says what I ran myself and what I only researched, and I update it when things change. About me

One useful email a week

New guides, tool price changes and AI testing news. No spam.

Free. Confirm by email, unsubscribe in one click. Privacy

Scroll to Top