Updated October 2026: added a section on where CI flakiness comes from, based on public Playwright issues and engineering write-ups from 2025 and 2026.
The biggest lie sold to SDETs today is that AI will magically generate stable, bug-finding Playwright tests. This fantasy leads teams down a costly rabbit hole, chasing automated test creation when their real problem isn't a lack of tests, but a profound lack of understanding why their existing tests constantly break.
We've been there. The initial allure of "AI-powered test generation" promised to offload the grunt work. We experimented, like many, with various tools claiming to produce Playwright scripts from user flows or existing documentation. What we got was generally brittle, context-agnostic code that collapsed under the slightest UI change, adding more noise than signal to our CI pipelines. The true value of AI in Playwright isn't in test generation; it's in intelligent analysis, pattern recognition, and augmentation of the SDET's debugging workflow.
Generative AI For Tests: A Costly Distraction
Let's be blunt: most generative AI for end-to-end tests is a glorified record-and-playback tool with a fancier UI and a large language model bolted on. It excels at creating tests that assert the happy path in a static environment, which is precisely where most of our real bugs don't live. The moment you introduce dynamic data, async operations, or nuanced user interactions, these "AI-generated" tests fall apart.
The problem isn't the AI's intelligence; it's the inherent complexity of web application testing. An LLM trained on billions of code snippets can write syntactically correct Playwright code, sure. But it can't infer the intent of a user beyond simple clicks, nor can it anticipate the race conditions, network latencies, or data dependencies that plague real-world applications. Relying on it for test creation is like asking a chef to bake a cake without knowing the ingredients, only the recipe.
The Illusion of AI Test Stability
Many vendors are pushing "self-healing" tests, where AI supposedly adapts selectors or steps when the UI changes. This, too, is largely an illusion. At best, it's a brittle heuristic that postpones failure, often by selecting the wrong element or masking a genuine bug. At worst, it creates a false sense of security, allowing broken functionality to ship because the "AI" test silently passed by interacting with the wrong part of the DOM.
Take a critical Playwright login test. A minor UI refactor changes a button's data-test-id. An "AI-enhanced" test framework, instead of failing, silently finds a different button with a similar text label and clicks it, leaving a partial login state that nobody notices. The test passes, but the application is broken. This "stability" is a dangerous lie. We need failures to be loud, clear, and actionable, not quietly swept under the rug by an overzealous algorithm.
Playwright Traces: The Undervalued Goldmine
The real power of AI with Playwright lies not in generating tests, but in making sense of their failures. Playwright 1.4x's trace viewer is already an indispensable debugging tool, showing screenshots, DOM snapshots, network requests, and console logs at every step. But even with this wealth of data, manually sifting through a complex trace for an intermittent failure can take hours. This is where LLMs shine.
Imagine feeding an entire Playwright trace – the HAR file, the DOM snapshots, the console output – directly to a sophisticated LLM like Claude 3.5 Sonnet or GPT-4o. Instead of guessing, you're asking an expert system to analyze the sequence of events, identify anomalies, and pinpoint potential root causes that a human might miss or take ages to uncover.
Consider a scenario where a test fails because an element isn't visible. A human might check the screenshot. An LLM, given the full trace, can correlate that invisibility with a preceding network request that timed out, a console error indicating a component failed to render, or even a specific CSS animation that didn't complete.
Augmenting Debugging: The True Power of LLMs
A better use is to integrate LLMs into the Playwright failure analysis pipeline. When a Playwright test fails in GitHub Actions, don't stop at a stack trace and a screenshot. Capture the full Playwright trace, aggregate relevant logs from Testcontainers-spun environments, and then feed this structured data to an LLM.
Here's a simplified Python snippet demonstrating how you might extract data from a Playwright trace (assuming you've processed the trace.zip into a structured format) and prompt an LLM:
import json
import base64
from openai import OpenAI # Or Anthropic client
import os
# Assume 'trace_data' is a dictionary parsed from the Playwright trace.json
# and 'console_logs' is a list of log entries.
# 'screenshot_base64' is the base64 encoded image of the failure step.
def analyze_playwright_failure_with_llm(trace_data, console_logs, screenshot_base64, test_description):
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) # Or use Anthropic client
messages = [
{"role": "system", "content": "You are an expert SDET. Analyze the provided Playwright test failure details. Identify the most likely root cause, considering network, DOM, and application logic issues."},
{"role": "user", "content": f"Test Description: {test_description}"},
{"role": "user", "content": f"Playwright Trace Events (simplified, actual would be more detailed):\n{json.dumps(trace_data.get('actions', [])[:5], indent=2)}"},
{"role": "user", "content": f"Console Logs:\n{json.dumps(console_logs, indent=2)}"},
{"role": "user", "content": "Here is a screenshot of the failure state:"},
{"role": "user", "content": {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{screenshot_base64}"}}}
]
response = client.chat.completions.create(
model="gpt-4o", # Or "claude-3-5-sonnet-20240620"
messages=messages,
max_tokens=1500
)
return response.choices[0].message.content
# Example usage (in a real scenario, these would come from your CI pipeline)
# mock_trace_data = { "actions": [{"name": "click", "selector": "button#submit", "status": "failed", "error": "Element not visible"}] }
# mock_console_logs = ["Error: Cannot read properties of undefined (reading 'data')", "Network request to /api/data failed with 500"]
# mock_screenshot_b64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=" # A tiny placeholder image
# mock_test_desc = "Verify user can log in with valid credentials"
# analysis = analyze_playwright_failure_with_llm(mock_trace_data, mock_console_logs, mock_screenshot_b64, mock_test_desc)
# print(analysis)
This kind of analysis shortens debugging of intermittent Playwright failures. Instead of staring at a trace for an hour, an SDET gets a concise summary and pointed suggestions within minutes. It's not fixing the bug, it's making the SDET dramatically more efficient at finding it.
From Noise to Signal: AI-Powered Allure Reports
Another area where AI transforms Playwright testing is in reporting. Allure reports are fantastic, but can become overwhelming with hundreds or thousands of tests. We've integrated LLM capabilities to generate dynamic summaries for test suites or even individual test runs.
Instead of just a count of passed/failed tests, our Allure reports now include an AI-generated executive summary highlighting common failure patterns, identifying tests that consistently fail for similar reasons (e.g., "5 tests failed due to TimeoutError on network requests to /api/v2/users"), and even suggesting potential areas of application instability. This transforms a static report into an interactive diagnostic tool, helping product owners and developers quickly grasp the quality state without diving into individual test results. This has significantly improved our signal-to-noise ratio in daily quality discussions.
Update, October 2026: Where CI Flakiness Actually Comes From
Since writing this I went through the public Playwright issue tracker and a handful of engineering write-ups from 2025 and 2026 to see which flakiness problems keep coming back. Four patterns came up again and again. None of them need a model to diagnose, and all of them should be fixed before you hand a trace to anything.
1. Green locally, red in CI
This is the most repeated complaint in the tracker. Issue #34060, "Tests flaky on GitHub Actions", is the classic shape: works locally, times out on CI most of the time. #35543 reports a final test.step that is occasionally flaky on CI only.
The causes people cite are boring and consistent:
- CI runners with less CPU and memory than a developer laptop
- fixed waits that happen to be long enough locally
- more workers than the runner can handle
networkidlewaits that never settle- shared state between tests running in parallel
If a test only fails in CI, check those five before reading a single trace.
2. Retries hide flakes, and you need a policy
By default, Playwright passes a run whose only problem is a flaky test: it failed, was retried, and passed. Unless you set failOnFlakyTests, that build is green.
Teams keep asking for finer control, which tells me they are fighting this in their own configs. #34397 asks for --fail-on-flaky-tests in the config file. #37002 asks to fail on a maximum number of flaky tests, and a later comment asks for a flaky rate threshold, because raw counts stop meaning anything as a suite grows. #40414 asked to let reporters change the exit code for known flaky tests and was closed with "write a custom reporter".
The cost of not having a policy is concrete. SpecStory describes one cleanup of 63 flaky tests in which 5 turned out to be real product bugs that retries had hidden for months.
My take: keep retries, because the retry is what gives you the trace. But decide what a flake means for the build, write it down, and make the config enforce it. failOnFlakyTests on CI is the simplest version. Publishing the flaky count on every run is the minimum.
3. Lost trust is the real cost
The worst outcome is not a red build. It is a team that re-runs or ignores red because it has stopped believing the suite.
Atlassian put numbers on it in Taming Test Flakiness (December 2025): flaky tests caused up to 21% of master build failures in the Jira Frontend repo and about 15% in the backend, where reruns cost over 150,000 developer hours a year. They built an internal tool to manage it.
A EuroSTAR article from August 2026 (written by the CEO of a testing vendor, so read it with that in mind) splits the cost into rerun time, investigation time, escaped defects and lost trust, and puts the goal well: "the goal was never zero flaky tests. It's a suite whose failures you believe."
It even shows up in hiring. A 2026 KORE1 hiring guide uses a model job ad that reads "340 Playwright tests with about a 4% flake rate ... reducing flakiness ... is your first 90 days".
4. Slow suites and unbalanced shards
In my experience, a slow suite raises the price of every flake: the longer a run takes, the more tempting it is to re-run instead of investigate. Sharding is the usual answer, and it has had its own problems. #36253 asks to distribute projects evenly among shards, #36955 reports a sharding algorithm that leaves one shard empty, and #35066 reports incorrect sharding of --last-failed. The maintainers are working on balance in the tool itself: PR #38624, merged in December 2025, adds custom sharding weights.
Sometimes the bigger win is deleting tests. One team wrote up how they halved their Playwright E2E suite: runtime went from about 37 to about 17 minutes, from about 50% flaky runs to stable, after removing about a third of the tests.
What this changes about the rest of this post
Trace analysis is still where I would spend effort on intermittent failures. But it is the second step. The first is making the pipeline honest: CI resources that match the worker count, no fixed waits, a written flake policy that the config enforces, and a suite fast enough that people investigate red instead of re-running it.
Stop Generating, Start Analyzing
The future of AI in Playwright automation isn't about replacing SDETs; it's about empowering them to tackle increasingly complex systems. It's about moving beyond simplistic test generation and focusing on sophisticated analysis. This means leveraging AI to contextualize failures, identify systemic issues across test runs, and provide actionable insights that would take a human exponentially longer to discover.
We've pivoted our focus entirely from "how can AI write our tests?" to "how can AI make our SDETs orders of magnitude more effective at maintaining and understanding our Playwright test suites?" The latter is where the tangible ROI lies.
This week, pick one notoriously flaky Playwright test from your CI pipeline. Capture its full Playwright trace (trace.zip). Then, use a tool like Playwright's trace viewer to extract key console logs, network requests, and the final DOM snapshot. Feed this structured data, along with the test's purpose and expected behavior, into Claude 3.5 Sonnet (or GPT-4o). Ask it to identify potential root causes of the flakiness based on the provided context. Don't expect it to fix the test, expect a deeper, AI-assisted insight into the problem that empowers you to engineer a robust solution.