Skip to main content
Logo, a drawing of a warm coffee vgagaleski
Back to all blogs

Playwright 1.51 release, "Copy prompt"

test automation
Published on by Viktor Gagaleski · 10 min read

Table of Contents

Show more

Playwright 1.51 has just landed, and I’m excited about one standout feature: the new “Copy prompt” button. While it appears as a simple addition to the Playwright report, this functionality can significantly boost reporting capabilities, shorten debugging time, and streamline error resolution. By creating a bridge between Playwright and language models, it opens the way for innovative features that could transform how we leverage AI and LLMs in our testing workflows. Let’s explore what’s new and discuss how this addition could benefit test automation going forward.

What’s New in Playwright 1.51?

Before we dive deep into my favorite new feature, let’s quickly summarize what’s packed into this release:

  • IndexedDB Storage State: You can now save and restore IndexedDB contents with browserContext.storageState(), which is particularly useful when your app uses IndexedDB for auth tokens.
  • Filter Visible Elements: The new visible option for locator.filter() lets you match only visible elements, making your selectors more reliable.
  • Git Information in HTML Reports: Setting testConfig.captureGitInfo captures git information into your test reports, providing better context for test results.
  • Test Step Improvements: A new TestStepInfo object is now available in test steps, allowing you to add attachments or conditionally skip steps. No need to comment out failing steps anymore.
  • Miscellaneous Updates: New options like contrast for media emulation and failOnStatusCode for API requests, plus updated browser versions (Chromium 134, Firefox 135, WebKit 18.4).

Copy prompt

The new Copy prompt button appears on errors in the HTML report, trace viewer, and UI mode. With a single click, you can copy a pre-filled LLM prompt containing the error message and useful context for fixing the issue.

Prompt details

I tested this using the initial playwright setup tests where I introduced a bug in the test on purpose. Here is how the report looked like, once the tests finished, notice the Copy prompt button.

Copy prompt example

When you click the button, you will get copy of quite extensive prompt. I was even surprised by the detail of the prompt, and in the following text I will go over each of the sections.

The prompt starts with instructions, which cannot be changed and provide guidance on analyzing the failed Playwright test, emphasizing conciseness, best practices, and including a fix if possible. Next, we have the test info, which contains metadata about the test, including its name and the file location where the failure occurred.

# Instructions

- Following Playwright test failed.
- Explain why, be concise, respect Playwright best practices.
- Provide a snippet of code with the fix, if possible.

# Test info

- Name: New Todo >> should allow me to add todo items
- Location: /Users/viktorgagaleski/Code/astro/vgagaleski-blog/tests/demo-todo-app.spec.ts:14:8

Error details section shows the error message from Playwright and highlights the difference between expected and received values. This is the same error that we see in the HTML report.

Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)

Locator: getByTestId('todo-title')
- Expected  - 1
+ Received  + 1

  Array [
    "buy some cheese",
-   "book a doctors appointment",
+   "feed the cat",
  ]
Call log:
  - expect.toHaveText with timeout 5000ms
  - waiting for getByTestId('todo-title')
    9 × locator resolved to 2 elements

    at /Users/viktorgagaleski/Code/astro/vgagaleski-blog/tests/demo-todo-app.spec.ts:30:50

The page snapshot captures the rendered state of the page at the time of failure, including the full accessibility tree in a structured YAML format, showing the UI elements present.

- text: This is just a demo of TodoMVC for testing, not the
- link "real TodoMVC app."
- heading "todos" [level=1]
- textbox "What needs to be done?"
- checkbox "❯Mark all as complete"
- text: ❯Mark all as complete
- list:
    - listitem:
        - checkbox "Toggle Todo"
        - text: buy some cheese
    - listitem:
        - checkbox "Toggle Todo"
        - text: feed the cat
- strong: '2'
- text: items left
- list:
    - listitem:
        - link "All"
    - listitem:
        - link "Active"
    - listitem:
        - link "Completed"
- contentinfo:
    - paragraph: Double-click to edit a todo
    - paragraph:
        - text: Created by
        - link "Remo H. Jansen"
    - paragraph:
        - text: Part of
        - link "TodoMVC"

The test file includes the actual test code, pinpointing the line that caused the failure and providing context on how the test is structured.

   1 | import { expect, test, type Page } from '@playwright/test'
   2 |
   3 | test.beforeEach(async ({ page }) => {
   4 |   await page.goto('https://demo.playwright.dev/todomvc')
   5 | })
   6 |
   7 | const TODO_ITEMS = [
   8 |   'buy some cheese',
   9 |   'feed the cat',
  10 |   'book a doctors appointment'
  11 | ] as const
  12 |
  13 | test.describe('New Todo', () => {
  14 |   test.only('should allow me to add todo items', async ({ page }) => {
  15 |     // create a new todo locator
  16 |     const newTodo = page.getByPlaceholder('What needs to be done?')
  17 |
  18 |     // Create 1st todo.
  19 |     await newTodo.fill(TODO_ITEMS[0])
  20 |     await newTodo.press('Enter')
  21 |
  22 |     // Make sure the list only has one todo item.
  23 |     await expect(page.getByTestId('todo-title')).toHaveText([TODO_ITEMS[0]])
  24 |
  25 |     // Create 2nd todo.
  26 |     await newTodo.fill(TODO_ITEMS[1])
  27 |     await newTodo.press('Enter')
  28 |
  29 |     // Make sure the list now has two todo items.
> 30 |     await expect(page.getByTestId('todo-title')).toHaveText([
     |                                                  ^ Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)
  31 |       TODO_ITEMS[0],
  32 |       TODO_ITEMS[2]
  33 |     ])
  34 |
  35 |     await checkNumberOfTodosInLocalStorage(page, 2)
  36 |   })
  37 | })
  38 |
  39 | async function checkNumberOfTodosInLocalStorage(page: Page, expected: number) {
  40 |   return await page.waitForFunction((e) => {
  41 |     return JSON.parse(localStorage['react-todos']).length === e
  42 |   }, expected)
  43 | }
  44 |

Finally, the prompt displays a Git diff of recent changes, showing modifications in playwright.config.ts, you get this only if you add captureGitInfo: { commit: true, diff: true } as part of your playwright.config.ts file.

diff --git a/playwright.config.ts b/playwright.config.ts
index 48f71d2..bef1e34 100644
--- a/playwright.config.ts
+++ b/playwright.config.ts
@@ -12,6 +12,7 @@ export default defineConfig({
   workers: process.env.CI ? 1 : undefined,
   /* Reporter to use. See https://playwright.dev/docs/test-reporters */
   reporter: 'html',
+  captureGitInfo: { commit: true, diff: true },
   /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
   use: {
     /* Base URL to use in actions like `await page.goto('/')`. */

This report provides a complete view of what went wrong, making debugging and fixing issues more efficient. Let’s put this to the test.

I ran the prompt with ChatGPT and got the output shown bellow. I was to some extend expecting the result, because of the amount of detail we provided and the fact that this is really basic test. I’d definitelly like to test this in a more complex scenario that better reflects a real-world project. I’ve worked on projects with Page Object Model (POM), nesting, API clients, and multiple dynamic project configurations that can add quite a lot of complexity in the error message. I’m quite certain that with such a setup, we won’t get the same results.

Issue Explanation
The test fails because it expects the second todo to be "book a doctors appointment", but the actual second item in the list is "feed the cat".

This happens due to an indexing mistake in the test:

await expect(page.getByTestId('todo-title')).toHaveText([
  TODO_ITEMS[0],
  TODO_ITEMS[2] // Incorrect index, should be TODO_ITEMS[1]
])

TODO_ITEMS[2] refers to "book a doctors appointment", which was never added in this test.

Fix
Update the expectation to check for the correct second todo:

await expect(page.getByTestId('todo-title')).toHaveText([
  TODO_ITEMS[0],
  TODO_ITEMS[1] // Correct index
])

Why am I so excited about this seemingly simple button?

Because it represents a significant step toward integrating AI capabilities directly into our testing workflow. As someone who spends a lot of time optimizing CI/CD pipelines and building test automation frameworks, I can immediately see how this feature can save precious debugging time. Instead of staring at an error trying to decipher what went wrong, I can now leverage LLMs to help analyze and propose solutions.

What’s particularly interesting is that this feature follows community-led initiatives like the “Fix with AI” button for Playwright HTML reports by Vitaliy Potapov. This open-source solution, developed by the community, already shows the value of using LLMs to get insights into test failures.

The main benefit of the community solution is the ability to construct custom prompts as part of your code, giving you more control over what information is sent to the AI and how questions are framed. This customization could be a great next step for Playwright’s native “Copy prompt” feature to consider in future releases.

”Copy prompt” feature improvement ideas

The “Copy prompt” feature inspired me to think how this integration could evolve to further transform and enhance the Playwright report. Here are some interesting ideas that came to mind while thinking on how we can get more out of the “Copy prompt” button. I hope some of them can get on the list of features for the Playwrigth team.

Configurable Prompts in HTML Reports

We could see configuration options added to the HTML report setup that allow defining custom prompt templates:

// playwright.config.ts
import { defineConfig } from '@playwright/test'

export default defineConfig({
  reporter: [
    [
      'html',
      {
        promptTemplates: {
          locatorFailure:
            "Analyze this locator failure: {error}. Here's the DOM snapshot: {dom}",
          apiFailure:
            'Analyze this API response error: {error}. Status code: {status}'
          // Custom templates for different error types
        }
      }
    ]
  ]
})

This is similar to the “Fix with AI” implementation, where we can specify our own prompt.

Direct LLM integration

The next step can be direct integration with various LLM providers:

// playwright.config.ts
import { defineConfig } from '@playwright/test'

export default defineConfig({
  reporter: [
    [
      'html',
      {
        ai: {
          provider: 'anthropic', // or 'openai', etc.
          apiKey: process.env.AI_API_KEY,
          automaticAnalysis: true // automatically analyze failures
        }
      }
    ]
  ]
})

Instead of us manually, copy-pasting the code to the LLM chat, this can be done automatically and the results could be part of the report.

Custom self-healing reporter (or pipeline step)

I am definitelly going to explore this more. A specialized self-healing could run after tests complete in your CI/CD pipeline. This AI reporter could:

  • Analyze failed tests using an LLM
  • Determine if the issue can be easily fixed (e.g., updating a selector)
  • Generate code fixes automatically
  • Create a pull request with the proposed changes
  • Add appropriate reviewers for human validation
// playwright.config.ts
import { defineConfig } from '@playwright/test'

export default defineConfig({
  reporter: [
    [
      'html',
      {
        /* html options */
      }
    ],
    [
      'ai-analyzer',
      {
        // Only analyze specific error types
        errorTypes: ['locator', 'assertion', 'timeout'],
        // Attempt self-healing for these issues
        attemptFix: ['locator-updates'],
        // Create PRs for suggested fixes
        createPullRequests: true,
        // Require human approval before merges
        requireApproval: true,
        // GitHub integration for PR creation
        githubToken: process.env.GITHUB_TOKEN,
        repository: 'org/repo'
      }
    ]
  ]
})

For example, if a test fails because a button’s label changed from “Submit” to “Send”, the AI reporter could:

  • Detect that the locator is the issue
  • Generate an updated locator
  • Create a PR with the fix
  • Request review from the appropriate team member

As you can see, I’m quite excited about the potential of AI in test automation. These ideas might seem ambitious, but they offer valuable starting points for how we could evolve our testing approaches. The possibilities at the intersection of AI and test automation are definitely worth exploring further.

Conclusion

The “Copy prompt” feature in Playwright 1.51 might seem modest, but it represents an important step toward AI-enhanced test automation. By bridging the gap between test failures and AI assistance, Playwright is opening doors to more efficient debugging and smarter test frameworks. We can now envision a future where Playwright integrates seamlessly with AI to provide:

  • Automated error analysis
  • Self-healing tests
  • Proactive test maintenance

As testing professionals, we should embrace these AI integrations while maintaining our critical judgment. The combination of human expertise and AI assistance will ultimately lead to more robust, maintainable test suites and higher quality software.

Next blog
A complete guide to organizing your Playwright project
Previous blog
Why every team should consider Playwright for test automation