Skip to content

Add extension contract tests#3645

Merged
zackverham merged 6 commits intomainfrom
zverham/extension-api-contract-tests
Mar 17, 2026
Merged

Add extension contract tests#3645
zackverham merged 6 commits intomainfrom
zverham/extension-api-contract-tests

Conversation

@zackverham
Copy link
Copy Markdown
Collaborator

@zackverham zackverham commented Mar 3, 2026

Summary

  • Adds test/extension-contract-tests/ — a contract test suite that validates which VSCode and Positron APIs the extension calls, with what arguments, and what it expects back
  • Tests mock the vscode and positron modules, import actual extension source code against those mocks, and assert API interactions
  • Includes compile-time conformance checks (tsc) that verify mock properties exist in the real @types/vscode and Positron type definitions
  • No Go binary, no HTTP server, no network — tests run entirely in-process using Vitest with module aliasing
  • Adds just test-extension-contracts and just check-extension-contract-conformance recipes
  • Adds extension-contract-tests workflow to CI (pull request, main, nightly)

What's tested

Test File Extension Source APIs Validated
activation extension.ts activate() wiring: trust, URI handler, commands, contexts
auth-provider authProvider.ts authentication.registerAuthenticationProvider
connect-filesystem connect_content_fs.ts workspace.registerFileSystemProvider, FileSystemError
dialogs dialogs.ts window.showInformationMessage (modal), l10n.t
document-tracker entrypointTracker.ts Editor/document change events, commands.executeCommand("setContext")
extension-settings extension.ts workspace.getConfiguration("positPublisher")
file-watchers watchers.ts workspace.createFileSystemWatcher, RelativePattern
interpreter-discovery utils/vscode.ts commands.executeCommand, workspace.getConfiguration, Positron runtime
llm-tools llm/index.ts lm.registerTool
open-connect open_connect.ts window.showInputBox, workspace.updateWorkspaceFolders
positron-settings utils/positronSettings.ts workspace.getConfiguration("positron.r")
window-utils utils/window.ts window.showErrorMessage, withProgress, createTerminal

Test plan

  • just test-extension-contracts passes
  • just check-extension-contract-conformance passes
  • CI extension-contract-tests workflow passes

🤖 Generated with Claude Code

@zackverham zackverham requested a review from a team as a code owner March 3, 2026 21:45
@zackverham zackverham marked this pull request as draft March 4, 2026 15:51
Contract tests that validate which VSCode and Positron APIs the
extension calls, with what arguments, and what it expects back.
Tests mock the vscode and positron modules, import actual extension
code against those mocks, and assert API interactions.

Includes compile-time conformance checks that verify mock properties
exist in the real @types/vscode and positron type definitions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zackverham zackverham force-pushed the zverham/extension-api-contract-tests branch from 727db02 to 6cf3d08 Compare March 9, 2026 17:00
@zackverham zackverham changed the title Add extension API contract tests (extension-api-contracts) Add extension contract tests Mar 9, 2026
@zackverham zackverham marked this pull request as ready for review March 9, 2026 18:10
Copy link
Copy Markdown
Collaborator

@dotNomad dotNomad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this idea is great. I had a couple ideas that I think are worth digging into a bit to simplify


I wonder if we can simplify this a bit by using a __mocks__ folder

If there is a mocks folder alongside a file that you are mocking, and the factory is not provided, Vitest will try to find a file with the same name in the mocks subfolder and use it as an actual module. If you are mocking a dependency, Vitest will try to find a mocks folder in the root of the project (default is process.cwd()). You can tell Vitest where the dependencies are located through the deps.moduleDirectories config option.

There is an example of __mocks__ in the File System example: https://vitest.dev/guide/mocking/file-system.html#example

"devDependencies": {
"@types/vscode": "^1.87.0",
"typescript": "^5.7.0",
"vitest": "^3.0.0"
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could bump up to the latest vitest here.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we use this tsconfig.json at all? I only see the conformance one being used in the package.json

Comment on lines +28 to +32
export enum TreeItemCollapsibleState {
None = 0,
Collapsed = 1,
Expanded = 2,
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For these enum should we just import them from vscode? I can't see a reason for us to duplicate them here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to claude, we shouldn't / can't do this:

No, you shouldn't import enums from vscode directly in the mock file. Here's why:

  The vscode module is not a real npm package — it's a virtual module provided by the VS Code runtime at extension load time. In Vitest (which runs outside VS Code), there is no real vscode module to import
  from. That's exactly why every test file uses vi.mock("vscode", () => { ... }) to provide a fake implementation.

zackverham and others added 4 commits March 13, 2026 14:20
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the unused tsconfig.json from extension-contract-tests (only
tsconfig.conformance.json and vitest.config.ts are referenced). Bump
vitest from ^3.0.0 to ^4.1.0.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ibility

Vitest 4.x passes `new` calls through to the implementation function.
Arrow functions cannot be constructors, so `new vi.fn(() => ...)` throws
TypeError. Switch to regular function expressions for all mocks that are
instantiated with `new`.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zackverham zackverham merged commit 80c72e6 into main Mar 17, 2026
40 checks passed
@zackverham zackverham deleted the zverham/extension-api-contract-tests branch March 17, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants