@askable-ui/mcp

MCP server and browser bridge primitives for askable UI context.

What this repo is for

This package turns askable's semantic UI context into MCP-friendly resources and tools.

Instead of relying on generic browser scraping, screenshot OCR, or DOM reconstruction, askable-mcp exposes the app-authored meaning of the UI:

• current focus (ui_context)
• recent interaction history (ui_history)
• visible annotated elements (ui_viewport)
• a command channel for explicit focus actions (select_element)

Why this is different from generic browser MCP

Generic browser MCP servers help agents inspect or control web pages.

askable-mcp is a different layer:

• browser MCP reconstructs meaning from DOM/layout/screenshots
• askable-mcp exposes semantic context the app already knows

Example:

{
  "metric": "revenue",
  "delta": "-12%",
  "period": "Q3",
  "screen": "RevenueScreen"
}

That makes the MCP surface much higher-signal for copilots, IDE agents, and desktop AI tools.

Current slice

This repo currently ships:

• createAskableMcpSnapshot() — serialize an AskableContext into an MCP-friendly session snapshot
• createAskableMcpBridge() — send snapshot updates through an injected transport
• AskableMcpServer — in-memory session/resource/tool state
• createAskableMcpSdkServer() — MCP SDK adapter exposing askable resources and the select_element tool
• askable-mcp CLI — runnable stdio MCP server with a local bridge HTTP sidecar

Install

npm install @askable-ui/mcp @askable-ui/core

Run the local stdio server

npm run build
npm run start

By default this starts:

• an MCP stdio server on stdin / stdout for Claude Desktop, Cursor, or other process-spawned MCP clients
• a local bridge server on http://127.0.0.1:4318

Available local bridge endpoints:

• POST /bridge — ingest serialized snapshot messages from the browser/app bridge
• GET /sessions — inspect known askable sessions
• GET /commands?sessionId=... — drain queued select_element commands for a browser session
• GET /health — basic health check

You can change the bridge listener with CLI flags:

node dist/cli.js --host 127.0.0.1 --bridge-port 4318

Claude Desktop configuration

Build the package, then point Claude Desktop at the stdio entrypoint:

{
  "mcpServers": {
    "askable": {
      "command": "node",
      "args": [
        "/absolute/path/to/askable-mcp/dist/cli.js",
        "--bridge-port",
        "4318"
      ]
    }
  }
}

After Claude Desktop launches the server, ui_context, ui_history, ui_viewport, and select_element become available once your app starts posting snapshots to the local bridge.

Browser / app bridge example

import { createAskableContext } from '@askable-ui/core';
import { createAskableMcpBridge } from '@askable-ui/mcp';

const ctx = createAskableContext({ viewport: true });
const sessionId = 'dashboard-tab';

createAskableMcpBridge({
  ctx,
  sessionId,
  pageUrl: window.location.href,
  pageTitle: document.title,
  includeViewport: true,
  transport: {
    send(message) {
      void fetch('http://127.0.0.1:4318/bridge', {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: message,
      });
    },
  },
});

async function pollCommands() {
  const response = await fetch(
    `http://127.0.0.1:4318/commands?sessionId=${encodeURIComponent(sessionId)}`
  );
  const payload = (await response.json()) as {
    commands: Array<{ type: 'select_element'; elementId: string }>;
  };

  for (const command of payload.commands) {
    if (command.type === 'select_element') {
      console.log('focus element in app', command.elementId);
    }
  }
}

setInterval(() => {
  void pollCommands();
}, 500);

Library example

import { createAskableContext } from '@askable-ui/core';
import {
  AskableMcpServer,
  createAskableMcpBridge,
  createAskableMcpSdkServer,
} from '@askable-ui/mcp';

const ctx = createAskableContext({ viewport: true });
const state = new AskableMcpServer({
  onCommand(command) {
    console.log('bridge command', command);
  },
});

createAskableMcpBridge({
  ctx,
  sessionId: 'dashboard-tab',
  pageUrl: 'https://app.example.com/revenue',
  pageTitle: 'Revenue dashboard',
  includeViewport: true,
  transport: {
    send(message) {
      state.ingestBridgeMessage(message);
    },
  },
});

const mcpServer = createAskableMcpSdkServer(state, {
  name: 'askable-mcp',
  version: '0.1.0',
});

Planned follow-up work

• browser-side helpers for handling select_element automatically
• streamable HTTP deployment story
• multi-session routing and active-session selection UX
• richer Claude Desktop / Cursor setup docs
• optional auth / origin controls for the local bridge endpoint

Development

npm install
npm test
npm run build