Quickstart: Set up continuous end-to-end testing with Microsoft Playwright Testing Preview

Add service configuration file

If you haven't configured your Playwright tests yet for running them with service, add a service configuration file to your repository. In the next step, you then specify this service configuration file on the Playwright CLI.

1. Create a new file playwright.service.config.ts alongside the playwright.config.ts file.

   Optionally, use the playwright.service.config.ts file in the sample repository.

2. Add the following content to it:

   import { defineConfig } from '@playwright/test';
   import { getServiceConfig, ServiceOS } from '@azure/microsoft-playwright-testing';
   import config from './playwright.config';
   
   /* Learn more about service configuration at https://aka.ms/mpt/config */
   export default defineConfig(
     config,
     getServiceConfig(config, {
       exposeNetwork: '<loopback>',
       timeout: 30000,
       os: ServiceOS.LINUX,
       useCloudHostedBrowsers: true
     }),
     {
       /* 
       Playwright Testing service reporter is added by default.
       This will override any reporter options specified in the base playwright config.
       If you are using more reporters, please update your configuration accordingly.
       */
       reporter: [['list'], ['@azure/microsoft-playwright-testing/reporter']],
     }
   );
   

   By default, the service configuration enables you to:

   • Accelerate build pipelines by running tests in parallel using cloud-hosted browsers.
   • Simplify troubleshooting with easy access to test results and artifacts published to the service.

   However, you can choose to use either of these features or both. See How to use service features and update the service configuration file as per your requirements.

3. Save and commit the file to your source code repository.

Update package.json file

Update the package.json file in your repository to add details about Microsoft Playwright Testing service package in devDependencies section.

"devDependencies": {
    "@azure/microsoft-playwright-testing": "^1.0.0-beta.6"
}

Enable artifacts in Playwright configuration

In the playwright.config.ts file of your project, make sure you are collecting all the required artifacts.

  use: {
    trace: 'on-first-retry',
    video:'retain-on-failure',
    screenshot:'on'
  },

Set up service configuration

1. Create a new file PlaywrightServiceSetup.cs in the root directory of your project. This file facilitates authentication of your client with the service.

2. Add the following content to it:

   using Azure.Developer.MicrosoftPlaywrightTesting.NUnit;
   using NUnit.Framework;
   
   namespace PlaywrightTests; // Remember to change this as per your project namespace
   
   [SetUpFixture]
   public class PlaywrightServiceSetup : PlaywrightServiceNUnit { };
   

3. Save and commit the file to your source code repository.

Install service package

In your project, install Microsoft Playwright Testing package.

dotnet add package Azure.Developer.MicrosoftPlaywrightTesting.NUnit --prerelease

This updates your project's csproj file by adding the service package details to the ItemGroup section. Remember to commit these changes.

  <ItemGroup>
    <PackageReference Include="Azure.Developer.MicrosoftPlaywrightTesting.NUnit" Version="1.0.0-beta.2" />
  </ItemGroup>

Add or update .runsettings file for your project.

If you haven't configured your Playwright tests yet for running them with service, add .runsettings file to your repository. In the next step, you then specify this service configuration file on the Playwright CLI.

1. Create a new .runsettings file.

   Optionally, use the .runsettings file in the sample repository.

2. Add the following content to it:

   <?xml version="1.0" encoding="utf-8"?>
   <RunSettings>
       <TestRunParameters>
           <!-- The below parameters are optional -->
           <Parameter name="Os" value="linux" />
           <!--<Parameter name="RunId" value="sample-run-id1" />-->
           <Parameter name="ServiceAuthType" value="EntraId" />
           <Parameter name="UseCloudHostedBrowsers" value="true" />
           <Parameter name="AzureTokenCredentialType" value="DefaultAzureCredential" />
           <!--<Parameter name="ManagedIdentityClientId" value="{clientId-value}" />-->
           <Parameter name="EnableGitHubSummary" value="false" />
           <!--<Parameter name="ExposeNetwork" value="*" />-->
       </TestRunParameters>
     <!-- NUnit adapter -->  
     <NUnit>
       <!-- Adjust parallel workers, parallel worker would also be bound by number of unit test files -->
       <NumberOfTestWorkers>10</NumberOfTestWorkers>
     </NUnit>
     <!-- General run configuration -->
     <RunConfiguration>
       <EnvironmentVariables>
         <!-- For debugging selectors, it's recommend to set the following environment variable -->
           <!--<DEBUG>pw:api*</DEBUG>--> 
       </EnvironmentVariables>
     </RunConfiguration>
     <!-- Playwright -->  
     <Playwright>
       <BrowserName>chromium</BrowserName>
       <ExpectTimeout>5000</ExpectTimeout>
       <LaunchOptions>
         <Headless>false</Headless>
         <!--Channel>msedge</Channel-->
       </LaunchOptions>
     </Playwright>
       <LoggerRunSettings>
           <Loggers>
               <!--microsoft playwright testing service logger for reporting -->
               <Logger friendlyName="microsoft-playwright-testing" enabled="True" />
               <!--could enable any logger additionally -->
               <Logger friendlyName="trx" enabled="false" />
           </Loggers>
       </LoggerRunSettings>
   </RunSettings>
   

   The settings in this file enable you to:

   • Accelerate build pipelines by running tests in parallel using cloud-hosted browsers.
   • Publish test results and artifacts to the service for faster troubleshooting.

   However, you can choose to use either of these features or both. See How to use service features and update the service configuration file as per your requirements.

3. Save and commit the file to your source code repository.

Enable artifacts in your Playwright setup

Enable artifacts such as screenshot, videos and traces to be captured by Playwright.

• For screenshots, see capture screenshots
• For videos, see record videos for your tests
• For traces, see recording a trace

Once you collect these artifacts, attach them to the TestContext to ensure they are available in your test reports. For more information, see our sample project for NUnit

Update the CI workflow definition to run your Playwright tests with the Playwright CLI. Pass the service configuration file as an input parameter for the Playwright CLI. You configure your environment by specifying environment variables.

1. Open the CI workflow definition

2. Add the following steps to run your Playwright tests in Microsoft Playwright Testing.

   The following steps describe the workflow changes for GitHub Actions or Azure Pipelines. Similarly, you can run your Playwright tests by using the Playwright CLI in other CI platforms.

   • GitHub Actions
   • Azure Pipelines

   
     # This step is to sign-in to Azure to run tests from GitHub Action workflow. 
     # Choose how to set up authentication to Azure from GitHub Actions. This is one example. 
   on:
     push:
       branches: [ main, master ]
     pull_request:
       branches: [ main, master ]
   permissions: # Required when using Microsoft Entra ID to authenticate
     id-token: write
     contents: read
   jobs:
     test:
       timeout-minutes: 60
       runs-on: ubuntu-latest
         steps:
         - uses: actions/checkout@v4    
   - name: Login to Azure with AzPowershell (enableAzPSSession true) 
     uses: azure/login@v2 
     with: 
       client-id: ${{ secrets.AZURE_CLIENT_ID }} 
       tenant-id: ${{ secrets.AZURE_TENANT_ID }}  
       subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}  
       enable-AzPSSession: true 
   
   - name: Install dependencies
       working-directory: path/to/playwright/folder # update accordingly
     run: npm ci
   
   - name: Run Playwright tests
       working-directory: path/to/playwright/folder # update accordingly
     env:
       # Regional endpoint for Microsoft Playwright Testing
       PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
       PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}
     run: npx playwright test -c playwright.service.config.ts --workers=20
   
   - name: Upload Playwright report
     uses: actions/upload-artifact@v3
     if: always()
     with:
       name: playwright-report
       path: path/to/playwright/folder/playwright-report/ # update accordingly
       retention-days: 10
   

Update the CI workflow definition to run your Playwright tests with the Playwright NUnit CLI. Pass the .runsettings file as an input parameter for the Playwright CLI. You configure your environment by specifying environment variables.

1. Open the CI workflow definition.

2. Add the following steps to run your Playwright tests in Microsoft Playwright Testing.

   The following steps describe the workflow changes for GitHub Actions or Azure Pipelines. Similarly, you can run your Playwright tests by using the Playwright CLI in other CI platforms.

   • GitHub Actions
   • Azure Pipelines

   on:
     push:
       branches: [ main, master ]
     pull_request:
       branches: [ main, master ]
   permissions: # Required when using AuthType as EntraId
     id-token: write
     contents: read
   jobs:
     test:
       timeout-minutes: 60
       runs-on: ubuntu-latest
         steps:
         - uses: actions/checkout@v4
       # This step is to sign-in to Azure to run tests from GitHub Action workflow. 
       # Choose how to set up authentication to Azure from GitHub Actions. This is one example. 
   
         - name: Login to Azure with AzPowershell (enableAzPSSession true) 
           uses: azure/login@v2 
           with: 
             client-id: ${{ secrets.AZURE_CLIENT_ID }} 
             tenant-id: ${{ secrets.AZURE_TENANT_ID }}  
             subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}  
             enable-AzPSSession: true 
   
         - name: Setup .NET
           uses: actions/setup-dotnet@v4
           with:
             dotnet-version: 8.0.x
   
         - name: Restore dependencies
           run: dotnet restore
           working-directory: path/to/playwright/folder # update accordingly
   
         - name: Build
           run: dotnet build --no-restore
           working-directory: path/to/playwright/folder # update accordingly
   
         - name: Run Playwright tests
             working-directory: path/to/playwright/folder # update accordingly
           env:
             # Regional endpoint for Microsoft Playwright Testing
             PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
             PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}
           run: dotnet test --settings:.runsettings --logger "microsoft-playwright-testing" -- NUnit.NumberOfTestWorkers=20
   
         - name: Upload Playwright report
           uses: actions/upload-artifact@v3
           if: always()
           with:
             name: playwright-report
             path: path/to/playwright/folder/playwright-report/ # update accordingly
             retention-days: 10
   

You can run a single test with the service by using the following command-line:

npx playwright test {name-of-file.spec.ts} --config=playwright.service.config.ts

1. After your test run completes, a link to the Playwright Portal is generated. Open this link to view detailed test results and associated artifacts. The portal displays essential information, including:

   • CI build details
   • Overall test run status
   • The commit ID linked to the test run

   

2. The Playwright portal provides all the necessary information for troubleshooting. You can:

   • Switch between retries.
   • View detailed error logs, test steps, and attached artifacts such as screenshots or videos.
   • Navigate directly to the Trace Viewer for deeper analysis.

   

3. The Trace Viewer allows you to step through your test execution visually. You can:

   • Use the timeline to hover over individual steps, revealing the page state before and after each action.
   • Inspect detailed logs, DOM snapshots, network activity, errors, and console output for each step.

   

1. After your test run completes, you get a link to the Playwright portal in your terminal. Open this link to view detailed test results and associated artifacts. The portal displays essential information, including:

   • CI build details
   • Overall test run status
   • The commit ID linked to the test run

   

2. The Playwright portal provides all the necessary information for troubleshooting. You can:

   • View detailed error logs, and attached artifacts such as screenshots or videos.
   • Navigate directly to the Trace Viewer for deeper analysis.

   

4. The Trace Viewer allows you to step through your test execution visually. You can:

   • Use the timeline to hover over individual steps, revealing the page state before and after each action.
   • Inspect detailed logs, DOM snapshots, network activity, errors, and console output for each step.