test: Implement shared tests from spec submodule by caitlynstocker · Pull Request #47 · OctopusDeploy/openfeature-provider-ts-web

caitlynstocker · 2026-03-30T06:00:43Z

Background 🌇

As per this ADR, we are adding a shared testing fixture to ensure consistency across all of our OpenFeature implementations.

Each test case contains a description, associated toggle slug, evaluation context, and expected outcome, as well as a list of toggles in the shape of V1FeatureToggleEvaluation to evaluate against.

What's this? 🌵

This PR adds a test which stands up a fake HTTP server which serves a list of toggles (keyed by a client id generated in each test) then runs each case as a separate test.

The shared data is accessed as a Git submodule from our specifications repository. The current code is pointing to an older version of the test fixtures as we are yet to implement functionality (fractional segment rollout) already covered by the latest test fixtures.

How to review? 🔍

☑️ Check for consistency with .NET and Java implementations

Completes DEVEX-139

caitlynstocker · 2026-03-30T06:02:01Z

This is pointed to an old version of the specifications submodule as we have not yet implemented fractional rollout to segments.

caitlynstocker · 2026-03-30T06:02:45Z

+        return new Promise((resolve) => {
+            this.server = http.createServer((req, res) => {
+                // URL will be updated to /api/toggles/evaluations/v3/ in a later PR
+                if (req.method === "GET" && req.url === "/api/featuretoggles/v3/") {


URL will be updated to /api/toggles/evaluations/v3/ when we switch over to the new endpoint.

Copilot

Pull request overview

Adds fixture-driven “specification” tests by pulling fixtures from a git submodule and running them against the provider via a lightweight local HTTP server.

Changes:

Added a specification git submodule to source shared JSON fixtures.
Added a test harness (Server) plus a Jest suite to run all fixture cases against OctopusFeatureProvider.
Updated CI workflow checkout to include submodules so fixtures are available in CI.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

File	Description
`src/specificationTests/server.ts`	Introduces a local HTTP server used by fixture-based tests.
`src/specificationTests/fixtureEvaluationTests.test.ts`	Loads JSON fixtures from the submodule and runs them as parameterized Jest tests.
`.gitmodules`	Adds the `specification` submodule definition.
`.github/workflows/build-test-release.yml`	Ensures CI checks out submodules for the test job.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

liamhughes

Looking good!

A few optional improvement questions, plus one about localStorage.

liamhughes · 2026-04-01T01:41:40Z

+import { Server } from "./server";
+
+interface Fixture {
+    response: V1FeatureToggleEvaluation[];


Can declare response as unknown given we don't actually use any of its fields.

Makes sense 🙂

liamhughes · 2026-04-01T01:42:53Z

+    for (const file of fixtureFiles) {
+        const json = fs.readFileSync(path.join(fixturesDir, file), "utf-8");
+        const fixture: Fixture = JSON.parse(json);
+        const testResponse = JSON.stringify(fixture.response);


In the other libraries we are avoiding deserialising and then reserialising the response. It probably matters less in JavaScript/TypeScript given the objects aren't strongly typed, but it might be nice to read response as a raw string if it's not too tricky/hacky?

I've had a look at how we might do this and it's possible but I think it would be overkill considering that parsing and stringifying the JSON should be lossless in TS.

Can add a comment explaining the difference between this and the .NET/Java implementations 👍

Yep, that's fair.

liamhughes · 2026-04-01T01:48:15Z

+});
+
+// Requires `specification` submodule to be initialized.
+test.each(testCases)("$testCase.description", async ({ testResponse, testCase }) => {


Could we include the source JSON filename in the test name? Makes it easier to track where cases are configured.

Good point! But I think there are pros and cons to this 😊 Having the file name in the test name could make it easier to find the file, but we already have the case description in test name, and that's easy to paste into a search box.

If we added the file name to the test names, it might actually make it harder to search on the description as we wouldn't be able to just bung the whole test name into a search box to find the right case.

Okay, may need to put it to the vote when Dylan's back; I added them in the Java implementation. 😅

liamhughes · 2026-04-01T01:57:14Z

+});
+
+beforeEach(() => {
+    localStorage.clear();


I'm not sure why we are referencing localStorage here?

If I'm understanding this correctly, we need to clear local storage between test runs because we are caching the manifest when we call getEvaluationContext, and we don't want this to stick around between tests.

Ahh, okay. Feels a bit like the abstraction is leaking, and the tests are fine without it. The cache only appears to be used if the HTTP client is unable to reach the OctoToggle service (see getFeatureManifest).

I'm happy to take it out if it's not needed 👍

liamhughes

Approved, but if it were me, I'd remove the localStorage reset.

caitlynstocker added 2 commits March 30, 2026 14:15

Add submodule

4fdea2f

Claude first pass at adding tests

f3fe69c

caitlynstocker marked this pull request as draft March 30, 2026 06:00

caitlynstocker commented Mar 30, 2026

View reviewed changes

Small fixes tidying claude code

fdc5b79

caitlynstocker force-pushed the cat/devex-138/add-specifications-tests branch from f6b1ca6 to fdc5b79 Compare March 30, 2026 06:03

Unflatten test input

70d2769

caitlynstocker changed the title ~~Cat/devex 138/add specifications tests~~ test: Implement shared tests from spec submodule Mar 30, 2026

caitlynstocker requested a review from Copilot March 30, 2026 06:31

Copilot started reviewing on behalf of caitlynstocker March 30, 2026 06:31 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

Comment thread src/specificationTests/server.ts

Comment thread src/specificationTests/server.ts

Comment thread src/specificationTests/server.ts

Comment thread src/specificationTests/fixtureEvaluationTests.test.ts

Comment thread src/specificationTests/fixtureEvaluationTests.test.ts Outdated

caitlynstocker added 2 commits March 30, 2026 17:03

Reorganise for consistency with .NET implementation

7170976

Small fixes as per copilot comments

5a97a43

caitlynstocker force-pushed the cat/devex-138/add-specifications-tests branch from 4fbe6df to 5a97a43 Compare March 31, 2026 06:10

caitlynstocker marked this pull request as ready for review March 31, 2026 06:26

caitlynstocker requested a review from a team March 31, 2026 06:27

liamhughes reviewed Apr 1, 2026

View reviewed changes

Small fixes based on Liam's comments

b45e154

caitlynstocker requested a review from liamhughes April 1, 2026 05:05

liamhughes approved these changes Apr 1, 2026

View reviewed changes

Remove storage reset

a5026da

caitlynstocker merged commit b0c11c4 into main Apr 1, 2026
7 checks passed

caitlynstocker deleted the cat/devex-138/add-specifications-tests branch April 1, 2026 05:34

Conversation

caitlynstocker commented Mar 30, 2026 • edited by liamhughes Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background 🌇

What's this? 🌵

How to review? 🔍

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liamhughes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caitlynstocker Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caitlynstocker Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liamhughes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

caitlynstocker commented Mar 30, 2026 •

edited by liamhughes

Loading

caitlynstocker Apr 1, 2026 •

edited

Loading

caitlynstocker Apr 1, 2026 •

edited

Loading