Quickstart: Troubleshoot tests with Microsoft Playwright Testing Preview

In this quickstart, you learn how to troubleshoot your Playwright tests easily using reports and artifacts published on Microsoft Playwright Testing Preview. Additionally, this guide demonstrates how to utilize the reporting feature, regardless of whether you're running tests on the cloud-hosted browsers provided by the service.

After you complete this quickstart, you'll have a Microsoft Playwright Testing workspace to view test results and artifacts in the service portal.

Background

Microsoft Playwright Testing service enables you to:

• Accelerate build pipelines by running tests in parallel using cloud-hosted browsers.
• Simplify troubleshooting by publishing test results and artifacts to the service, making them accessible through the service portal.

These two features of the service can be used independently of each other and each has its own pricing plan. This means you can:

• Expedite test runs and streamline troubleshooting by running tests in cloud-hosted browsers and publishing results to the service.
• Run tests only in cloud-hosted browsers to finish test runs faster.
• Publish test results to the service while continuing to run tests locally for efficient troubleshooting.

Prerequisites

• An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
• Your Azure account needs the Owner, Contributor, or one of the classic administrator roles.
• A Playwright project. If you don't have project, create one by using the Playwright getting started documentation or use our Microsoft Playwright Testing sample project.
• Azure CLI. If you don't have Azure CLI, see Install Azure CLI.

Create a workspace

To get started with publishing test results on Playwright Testing service, first create a Microsoft Playwright Testing workspace in the Playwright portal.

1. Sign in to the Playwright portal with your Azure account.

2. If you already have a workspace, select an existing workspace, and move to the next step.

   Tip

   If you have multiple workspaces, you can switch to another workspace by selecting the workspace name at the top of the page, and then select Manage all workspaces.

3. If you don't have a workspace yet, select + New workspace, and then provide the following information:

   Field	Description
   Workspace name	Enter a unique name to identify your workspace. The name can only consist of alphanumerical characters, and have a length between 3 and 64 characters.
   Azure subscription	Select the Azure subscription that you want to use for this Microsoft Playwright Testing workspace.
   Region	Select a geographic location to host your workspace. This is the location where the test run data is stored for the workspace.

   

4. Select Create workspace to create the workspace in your subscription.

   During the workspace creation, a new resource group and a Microsoft Playwright Testing Azure resource are created in your Azure subscription.

When the workspace creation finishes, you're redirected to the setup guide.

Install Microsoft Playwright Testing package

To use the service, install the Microsoft Playwright Testing package.

npm init @azure/microsoft-playwright-testing

This command generates playwright.service.config.ts file which serves to:

• Direct and authenticate your Playwright client to the Microsoft Playwright Testing service.
• Adds a reporter to publish test results and artifacts.

If you already have this file, the prompt asks you to override it.

To use only reporting feature for the test run, disable cloud-hosted browsers by setting useCloudHostedBrowsers as false.

export default defineConfig(
  config,
  getServiceConfig(config, {
    timeout: 30000,
    os: ServiceOS.LINUX,
	useCloudHostedBrowsers: false // Do not use cloud hosted browsers
  }),
  {
    reporter: [['list'], ['@azure/microsoft-playwright-testing/reporter']], // Reporter for Microsoft Playwright Testing service
  }
);

Setting the value as false ensures that cloud-hosted browsers aren't used to run the tests. The tests run on your local machine but the results and artifacts are published on the service.

Configure the service region endpoint

In your setup, you have to provide the region-specific service endpoint. The endpoint depends on the Azure region you selected when creating the workspace.

To get the service endpoint URL:

1. In Add region endpoint in your setup, copy the region endpoint for your workspace.

   The endpoint URL matches the Azure region that you selected when creating the workspace.

   

Set up your environment

To set up your environment, you have to configure the PLAYWRIGHT_SERVICE_URL environment variable with the value you obtained in the previous steps.

We recommend that you use the dotenv module to manage your environment. With dotenv, you define your environment variables in the .env file.

1. Add the dotenv module to your project:

   npm i --save-dev dotenv
   

2. Create a .env file alongside the playwright.config.ts file in your Playwright project:

   PLAYWRIGHT_SERVICE_URL={MY-REGION-ENDPOINT}
   

   Make sure to replace the {MY-REGION-ENDPOINT} text placeholder with the value you copied earlier.

Set up authentication

To publish test results and artifacts to your Microsoft Playwright Testing workspace, you need to authenticate the Playwright client where you're running the tests with the service. The client could be your local dev machine or CI machine.

The service offers two authentication methods: Microsoft Entra ID and Access Tokens.

Microsoft Entra ID uses your Azure credentials, requiring a sign-in to your Azure account for secure access. Alternatively, you can generate an access token from your Playwright workspace and use it in your setup.

Set up authentication using Microsoft Entra ID

Microsoft Entra ID is the default and recommended authentication for the service. From your local dev machine, you can use Azure CLI to sign-in

az login

Set up authentication using access tokens

You can generate an access token from your Playwright Testing workspace and use it in your setup. However, we strongly recommend Microsoft Entra ID for authentication due to its enhanced security. Access tokens, while convenient, function like long-lived passwords and are more susceptible to being compromised.

1. Authentication using access tokens is disabled by default. To use, Enable access-token based authentication

2. Set up authentication using access tokens

Enable artifacts in Playwright configuration

In the playwright.config.ts file of your project, make sure you're collecting all the required artifacts.

  use: {
    trace: 'on-first-retry',
    video:'retain-on-failure',
    screenshot:'on'
  },

Run your tests and publish results on Microsoft Playwright Testing

You've now prepared the configuration for publishing test results and artifacts with Microsoft Playwright Testing. Run tests using the newly created playwright.service.config.ts file and publish test results and artifacts to the service.

   npx playwright test --config=playwright.service.config.ts

After the test completes, you can view the test status in the terminal.

Running 6 test using 2 worker
    5 passed, 1 failed (20.2s)
    
Test report: https://playwright.microsoft.com/workspaces/<workspace-id>/runs/<run-id>

View test runs and results in the Playwright portal

You can now troubleshoot the failed test cases in the Playwright portal.

1. After your test run completes, a link to the Playwright Portal is generated. Open this link to view detailed test results and associated artifacts. The portal displays essential information, including:

   • CI build details
   • Overall test run status
   • The commit ID linked to the test run

   

2. The Playwright portal provides all the necessary information for troubleshooting. You can:

   • Switch between retries.
   • View detailed error logs, test steps, and attached artifacts such as screenshots or videos.
   • Navigate directly to the Trace Viewer for deeper analysis.

   

3. The Trace Viewer allows you to step through your test execution visually. You can:

   • Use the timeline to hover over individual steps, revealing the page state before and after each action.
   • Inspect detailed logs, DOM snapshots, network activity, errors, and console output for each step.

   

Next step

You've successfully created a Microsoft Playwright Testing workspace in the Playwright portal and run your Playwright tests on cloud browsers.

Advance to the next quickstart to set up continuous end-to-end testing by running your Playwright tests in your CI/CD workflow.