test: 💍 local visual regressions testing support

.gitignore

@@ -12,3 +12,8 @@ vite.config.ts.timestamp*
 .yarn/*
 !.yarn/releases
 !.yarn/plugins
+
+**.ts-snapshots
+tests/**/*-actual.png
+tests/**/*-diff.png
+test-results

README.md

@@ -22,6 +22,8 @@ You can find the official docs for the Click UI design system and component libr
   - [Generating design tokens](#generating-design-tokens)
   - [Local development server](#local-development-server)
 * [Tests](#Tests)
+  - [Functional tests](#functional-tests)
+  - [Visual regression tests](#visual-regression-tests)
 * [Storybook](#storybook)
   - [Stories development server](#stories-development-server)
   - [Public static site](#public-static-site)
@@ -68,12 +70,47 @@ It'll default to the location [http://localhost:5173](http://localhost:5173), if
 
 ## Tests
 
+### Functional tests
+
 The package uses [vitest](https://vitest.dev/) and [react testing library](https://testing-library.com) for tests, e.g. functional tests.
 
 ```sh
 yarn test
 ```
 
+### Visual regression tests
+
+The project uses [Chromatic](https://www.chromatic.com/) for visual regression testing of UI components.
+
+It captures screenshots of Storybook and compares them across builds to detect unintended visual changes by:
+
+- Automated visual testing in GitHub CI/CD pipeline, e.g. storybook publish, UI tests
+- Leveraging storybook stories
+- Provides visual diff reviews and approval workflows
+- Helps catch UI bugs
+
+To setup, you must get a team member project token.
+
+Add the token as an environment variable to your environment preference or profile, e.g. `~/.zshrc`:
+
+```sh
+export CHROMATIC_PROJECT_TOKEN=<YOUR-TOKEN-HERE>
+```
+
+Once ready, you can run tests by:
+
+```sh
+yarn test:chromatic
+```
+
+> [!NOTE]
+> Chromatic does NOT generate a report in the terminal due to its cloud nature, which only outputs:
+> - Build status, e.g. uploading or testing
+> - Link to the cloud runner or dashboard
+> - Exit code
+
+If you need quicker iteration feedback, or more testing control during local development, read [here](./docs/tests/playwright.md)
+
 ## Storybook
 
 The component library provides a collection of ready-to-use components. We use [Storybook](#storybook) to showcase and document our components.
@@ -102,7 +139,7 @@ Once built, you can serve the static files by:
 yarn storybook:serve
 ```
 
-### Public static version
+### Public static site
 
 The latest static version's built and deployed automatically when contributing to `main` of [Click UI](https://github.com/ClickHouse/click-ui).
 

docs/tests/playwright.md

@@ -0,0 +1,123 @@
+# Playwright
+
+Playwright provide us the ability to do end-to-end testing, headless browser testing, amongst others; local visual regression testing as an alternative to Chromatic's cloud-based approach.
+
+Use Playwright when you need to:
+
+- Faster feedback loops during active development
+- Full control over test execution and baselines, e.g. snapshots or screenshots
+- OS specific visual testing
+
+> [!TIP]
+> Use Playwright for rapid local development, (e.g. if you need fast feedback loop) and Chromatic for team reviews and CI/CD.
+
+## Visual regression testing locally
+
+Visual regression tests are operating system dependent.
+
+Screenshots generated on macOS will differ from Linux or Windows due to font rendering, anti-aliasing, and browser engine differences.
+
+For this reason at time of writing, the following snaptshots are ignored:
+
+```sh
+# The snapshots which are generally persisted for comparison
+**.ts-snapshots
+
+# These are generated during test runner, generally ignored
+tests/**/*-actual.png
+tests/**/*-diff.png
+test-results
+```
+
+### Generating snapshots for comparison
+
+You MUST generate the snapshots you want to compare against. At time of writing, the team's using Chromatic for visual regression testing.
+
+> [!WARNING]
+> If you are reading this document, you should be aware that this provides you with custom control for advanced needs only! It does not provide you with the setup for cross-operating system, e.g. docker linux.
+> This workflow does NOT expect you to store your favourite OS image/snapshots in the repository, it's for your own usage, or fast feedback loop only!
+
+To generate snapshots, you'll have to manually checkout/switch to the target commit in history and run the `test:visual` to generate it and return back to your ongoing feature branch. Alternatively, you can store snapshots separatily and place them at your need.
+
+Hypothetically, you could use the following workflow to facilitate:
+
+```sh
+# Switch to a target/stable version of your liking
+git checkout <COMMIT-STABLE-VERSION>
+# Generate the snapshots against your target version
+yarn test:visual
+# Check if snapshots generated?
+find ./tests -type f -name '**.ts-snapshots*'
+# Once happy return to feature branch
+git checkout <FEATURE-BRANCH>
+# Run the tests
+yarn test:visual
+```
+
+When intentional visual changes are made, update baselines:
+
+> [!WARNING]
+> Only update baselines when visual changes are intentional and reviewed.
+
+```sh
+yarn test:visual:update
+```
+
+### Running tests locally
+
+Execute all visual regression tests by running:
+
+```sh
+yarn test:visual
+```
+
+Alternatively, launch UI mode for a browser like experience with timeline, console logs, etc.
+
+```sh
+yarn test:visual:ui
+```
+
+### Run specific test files
+
+```sh
+yarn test:visual tests/buttons/overview.spec.ts
+```
+
+### Show report
+
+To review test reports as HTML run:
+
+```sh
+yarn test:visual:report
+```
+
+### Configuration
+
+The Playwright configuration file contains default configuration values, which you can extend, modify or personalize.
+
+For example, you can modify the default number of workers:
+
+```sh
+  ...
+
+  workers: process.env.CI ? 4 : undefined,
+```
+
+### Migration to Chromatic
+
+> [!NOTE]
+> At time of writing, there aren't any test source files for Chromatic. Should assume that these are based on Storybook stories: available properties/parameters and tests are not curated.
+
+Playwright tests can be easily migrated to Chromatic as currently (per the example provided), both leverage Storybook stories. Meaning that you can start with Playwright for local development and transition to Chromatic as collaboration or any requirements demand.
+
+Here's a quick example of migration switch:
+
+```ts
+import { test, expect } from "@playwright/test"; 
+import { test, expect } from "@chromatic-com/playwright"; 
+```
+
+## Learn more
+
+- [Test snapshots](https://playwright.dev/docs/test-snapshots)
+- [Best practices](https://playwright.dev/docs/best-practices)

package.json

@@ -41,7 +41,6 @@
     "changeset:status": "yarn changeset status",
     "changeset:version": "yarn changeset version",
     "changeset:verify": ".scripts/changeset-verification",
-    "chromatic": "yarn dlx chromatic",
     "dev": "vite",
     "generate-tokens": "node build-tokens.js && prettier --write \"src/theme/tokens/*.ts\" --config .prettierrc",
     "lint": "eslint src --report-unused-disable-directives --max-warnings 0",
@@ -54,6 +53,11 @@
     "storybook:build": "storybook build -o .storybook/out",
     "storybook:serve": "yarn dlx http-server .storybook/out",
     "test": "vitest",
+    "test:chromatic": "yarn dlx chromatic",
+    "test:visual": "playwright test",
+    "test:visual:ui": "playwright test --ui",
+    "test:visual:update": "playwright test --update-snapshots",
+    "test:visual:report": "playwright show-report",
     "typecheck": "tsc --noEmit",
     "watch": "vitest --watch",
     "prepare": "husky"
@@ -85,6 +89,7 @@
   },
   "devDependencies": {
     "@changesets/cli": "^2.29.8",
+    "@playwright/test": "^1.57.0",
     "@storybook/addon-a11y": "^10.1.10",
     "@storybook/addon-docs": "^10.1.10",
     "@storybook/addon-links": "^10.1.10",

playwright.config.ts

@@ -0,0 +1,32 @@
+import { defineConfig, devices } from "@playwright/test";
+
+const webServer = {
+  command: "yarn storybook",
+  url: "http://localhost:6006",
+  reuseExistingServer: !process.env.CI,
+  timeout: 120000,
+};
+
+export default defineConfig({
+  testDir: "./tests",
+  fullyParallel: true,
+  forbidOnly: !!process.env.CI,
+  retries: process.env.CI ? 2 : 0,
+  workers: process.env.CI ? 4 : undefined,
+  reporter: "list",
+
+  use: {
+    baseURL: "http://localhost:6006",
+    trace: "on-first-retry",
+    screenshot: "only-on-failure",
+  },
+
+  projects: [
+    {
+      name: "chromium",
+      use: { ...devices["Desktop Chrome"] },
+    },
+  ],
+
+  webServer,
+});

tests/buttons/overview.spec.ts

@@ -0,0 +1,82 @@
+import { test as it, expect } from '@playwright/test';
+
+const { describe, beforeEach, afterEach } = it;
+
+const getStory = (id: string) => ({
+  id,
+  pathname: `/iframe.html?path=/story/${id}`,
+});
+
+describe('Buttons', () => {
+  describe('Button', () => {
+    const { id, pathname } = getStory('buttons-button--playground');
+
+    beforeEach(async ({ page }) => {
+      await page.goto(pathname, { waitUntil: 'domcontentloaded' });
+      await page.waitForLoadState('networkidle');
+    });
+
+    afterEach(async ({ page }) => {
+      await page.close();
+    });
+
+    it(`should render ${id}`, async ({ page }) => {
+      const button = page.getByRole('button');
+
+      await expect(button).toBeVisible({ timeout: 10_000 });
+
+      await expect(button).toHaveScreenshot(`${id}.png`, {
+        timeout: 10_000,
+        maxDiffPixels: 100,
+      });
+    });
+  });
+
+  describe('ButtonGroup', () => {
+    const { id, pathname } = getStory('buttons-buttongroup--playground');
+
+    beforeEach(async ({ page }) => {
+      await page.goto(pathname, { waitUntil: 'domcontentloaded' });
+      await page.waitForLoadState('networkidle');
+    });
+
+    afterEach(async ({ page }) => {
+      await page.close();
+    });
+
+    it(`should render ${id}`, async ({ page }) => {
+      const button = page.locator('[class*="ButtonGroupWrapper"]');
+
+      await expect(button).toBeVisible({ timeout: 10_000 });
+
+      await expect(button).toHaveScreenshot(`${id}.png`, {
+        timeout: 10_000,
+        maxDiffPixels: 100,
+      });
+    });
+
+    it(`should render ${id}, on first element click`, async ({ page }) => {
+      const button = page.locator('[role="button"]:nth-child(1)');
+
+      await expect(button).toBeVisible({ timeout: 10_000 });
+
+      await button.click();
+
+      await expect(button).toHaveScreenshot(`${id}-pressed-first.png`, {
+        timeout: 10_000,
+        maxDiffPixels: 100,
+      });
+    });
+
+    it(`should ${id}, on second element click have aria-pressed`, async ({ page }) => {
+      const button = page.locator('[role="button"]:nth-child(2)');
+
+      await expect(button).toBeVisible({ timeout: 10_000 });
+
+      await button.click();
+
+      await expect(button).toHaveAttribute('aria-pressed', 'true');
+    });
+  });
+});
+