Test Automation with GitHub Actions and Playwright for Microsoft 365 solutions

Use Playwright to test:

• Navigation elements, element’s values
• the existence of certain elements by making use of the so called “Pixel matching” (e.g. used for template testing)
• API mocking: mock REAL apis by defining URL pattern to match by return a predefined set of returns

Writing and running the tests

First, you will define your tests by describing them:

test.describe(('...' => {

})

Handling MFA-secured logins for tests within M365

Every test that was set up along the above description can be run by calling npx run test from within its Playwright testing project. Keep in mind that for M365, any MFA setup will ‘interfere’ with your any test runs. To address this, there are two options (see Elios Blog for reference):

1. Using a time-based one-time code (TOTP); make sure to setup in Microsoft Entra ID or via the Security info profile of the user

   Note

   Note on this: the account name and the secret key together will generate a specific token that can only be used on the specific device)

2. using an authenticated state (literally a json file that contains all the necessary values for the session keys)

Automated tesings through Pipelines (example: Github Actions)

For the purpose of automated testing scenarios, appropriate workflows can be setup through pipelines (this is also possible for Azure DevOps).

Github Actions are workflows defined in YAML files which are triggered by events. Actions are resuable tasks.

Variables and secrets

Variables and secrets can be stored and referenced with brackets. They can be definde on org / repository / environment or workflow (env variables only) level. Repository secrets are stored under Security > Secrets and variables under the respoitory settings.

Make sure that you install all dependencies, such as npm ci as well as the browsers by using npx playright install --with-deps

Browsers

Installing browsers is time consuming. Therefore, consider define caching by using actions/cache.

Deployment artifacts

Artifacts can be stored as well, such as :

• reports (HTML, traces)
• snapshots (used for pixel matching)
• pictures

It’s recommended to split a long-running / complex workflow into dedicated workflow “chunks” (in separate jobs) and call them sequentially.

Doing “better” test reports

Consider using a Github Reporter config.reporter.push(...) by adding Elios Gitub Actions Reporter (@estruyf/github-actions-reporter) as well as a bunch of quick additions to the workflow definition file.

Moreover, you could (should!) add post operations for notification purposes (e.g. by calling a Power Automate webhook) on the tests’ end. To do so, there’s a ready-to-use template available for an http reporter that can trigger any HTTP webhook.

Automated deployment

Release workflow (process):