Skip to content

Commit 0abcc6e

Browse files
Sg312icecrasher321waleedlatif1claudeTheodoreSpeaks
authored
improvement(mothership): restructured stream, tool structures, code typing, file write/patch/append tools, timing issues (#4090)
* fix build error * improvement(mothership): new agent loop (#3920) * feat(transport): replace shared chat transport with mothership-stream module * improvement(contracts): regenerate contracts from go * feat(tools): add tool catalog codegen from go tool contracts * feat(tools): add tool-executor dispatch framework for sim side tool routing * feat(orchestrator): rewrite tool dispatch with catalog-driven executor and simplified resume loop * feat(orchestrator): checkpoint resume flow * refactor(copilot): consolidate orchestrator into request/ layer * refactor(mothership): reorganize lib/copilot into structured subdirectories * refactor(mothership): canonical transcript layer, dead code cleanup, type consolidation * refactor(mothership): rebase onto latest staging * refactor(mothership): rename request continue to lifecycle * feat(trace): add initial version of request traces * improvement(stream): batch stream from redis * fix(resume): fix the resume checkpoint * fix(resume): fix resume client tool * fix(subagents): subagent resume should join on existing subagent text block * improvement(reconnect): harden reconnect logic * fix(superagent): fix superagent integration tools * improvement(stream): improve stream perf * Rebase with origin dev * fix(tests): fix failing test * fix(build): fix type errors * fix(build): fix build errors * fix(build): fix type errors * feat(mothership): add cli execution * fix(mothership): fix function execute tests * Force redeploy * feat(motheship): add docx support * feat(mothership): append * Add deps * improvement(mothership): docs * File types * Add client retry logic * Fix stream reconnect * Eager tool streaming * Fix client side tools * Security * Fix shell var injection * Remove auto injected tasks * Fix 10mb tool response limit * Fix trailing leak * Remove dead tools * file/folder tools * Folder tools * Hide function code inline * Dont show internal tool result reads * Fix spacing * Auth vfs * Empty folders should show in vfs * Fix run workflow * change to node runtime * revert back to bun runtime * Fix * Appends * Remove debug logs * Patch * Fix patch tool * Temp * Checkpoint * File writes * Fix * Remove tool truncation limits * Bad hook * replace react markdown with streamdown * Checkpoitn * fix code block * fix stream persistence * temp * Fix file tools * tool joining * cleanup subagent + streaming issues * streamed text change * Tool display intetns * Fix dev * Fix tests * Fix dev * Speed up dev ci * Add req id * Fix persistence * Tool call names * fix payload accesses * Fix name * fix snapshot crash bug * fix * Fix * remove worker code * Clickable resources * Options ordering * Folder vfs * Restore and mass delete tools * Fix * lint * Update request tracing and skills and handlers * Fix editable * fix type error * Html code * fix(chat): make inline code inherit parent font size in markdown headers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * improved autolayout * durable stream for files * one more fix * POSSIBLE BREAKAGE: SCROLLING * Fixes * Fixes * Lint fix * fix(resource): fix resource view disappearing on ats (#4103) Co-authored-by: Theodore Li <theo@sim.ai> * Fixes * feat(mothership): add execution logs as a resource type Adds `log` as a first-class mothership resource type so copilot can open and display workflow execution logs as tabs alongside workflows, tables, files, and knowledge bases. - Add `log` to MothershipResourceType, all Zod enums, and VALID_RESOURCE_TYPES - Register log in RESOURCE_REGISTRY (Library icon) and RESOURCE_INVALIDATORS - Add EmbeddedLog and EmbeddedLogActions components in resource-content - Export WorkflowOutputSection from log-details for reuse in EmbeddedLog - Add log resolution branch in open_resource handler via new getLogById service - Include log id in get_workflow_logs response and extract resources from output - Exclude log from manual add-resource dropdown (enters via copilot tools only) - Regenerate copilot contracts after adding log to open_resource Go enum * Fix perf and message queueing * Fix abort * fix(ui): dont delete resource on clearing from context, set resource closed on new task (#4113) Co-authored-by: Theodore Li <theo@sim.ai> * improvement(mothership): structure sim side typing * address comments * reactive text editor tweaks * Fix file read and tool call name persistence bug * Fix code stream + create file opening resource * fix use chat race + headless trace issues * Fix type issue * Fix mothership block req lifecycle * Fix build * Move copy reqid * Fix * fix(ui): fix resource tag transition from home to task (#4132) Co-authored-by: Theodore Li <theo@sim.ai> * Fix persistence --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Waleed Latif <walif6@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: Theodore Li <theodoreqili@gmail.com>
1 parent d238052 commit 0abcc6e

File tree

389 files changed

+37921
-18979
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

389 files changed

+37921
-18979
lines changed

.github/workflows/ci.yml

Lines changed: 69 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ permissions:
1616
jobs:
1717
test-build:
1818
name: Test and Build
19+
if: github.ref != 'refs/heads/dev' || github.event_name == 'pull_request'
1920
uses: ./.github/workflows/test-build.yml
2021
secrets: inherit
2122

@@ -45,11 +46,66 @@ jobs:
4546
echo "ℹ️ Not a release commit"
4647
fi
4748
48-
# Build AMD64 images and push to ECR immediately (+ GHCR for main)
49+
# Dev: build all 3 images for ECR only (no GHCR, no ARM64)
50+
build-dev:
51+
name: Build Dev ECR
52+
needs: [detect-version]
53+
if: github.event_name == 'push' && github.ref == 'refs/heads/dev'
54+
runs-on: blacksmith-8vcpu-ubuntu-2404
55+
permissions:
56+
contents: read
57+
id-token: write
58+
strategy:
59+
fail-fast: false
60+
matrix:
61+
include:
62+
- dockerfile: ./docker/app.Dockerfile
63+
ecr_repo_secret: ECR_APP
64+
- dockerfile: ./docker/db.Dockerfile
65+
ecr_repo_secret: ECR_MIGRATIONS
66+
- dockerfile: ./docker/realtime.Dockerfile
67+
ecr_repo_secret: ECR_REALTIME
68+
steps:
69+
- name: Checkout code
70+
uses: actions/checkout@v4
71+
72+
- name: Configure AWS credentials
73+
uses: aws-actions/configure-aws-credentials@v4
74+
with:
75+
role-to-assume: ${{ secrets.DEV_AWS_ROLE_TO_ASSUME }}
76+
aws-region: ${{ secrets.DEV_AWS_REGION }}
77+
78+
- name: Login to Amazon ECR
79+
id: login-ecr
80+
uses: aws-actions/amazon-ecr-login@v2
81+
82+
- name: Login to Docker Hub
83+
uses: docker/login-action@v3
84+
with:
85+
username: ${{ secrets.DOCKERHUB_USERNAME }}
86+
password: ${{ secrets.DOCKERHUB_TOKEN }}
87+
88+
- name: Set up Docker Buildx
89+
uses: useblacksmith/setup-docker-builder@v1
90+
91+
- name: Build and push
92+
uses: useblacksmith/build-push-action@v2
93+
with:
94+
context: .
95+
file: ${{ matrix.dockerfile }}
96+
platforms: linux/amd64
97+
push: true
98+
tags: ${{ steps.login-ecr.outputs.registry }}/${{ secrets[matrix.ecr_repo_secret] }}:dev
99+
provenance: false
100+
sbom: false
101+
102+
# Main/staging: build AMD64 images and push to ECR + GHCR
49103
build-amd64:
50104
name: Build AMD64
51-
needs: [detect-version]
52-
if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging' || github.ref == 'refs/heads/dev')
105+
needs: [test-build, detect-version]
106+
if: >-
107+
github.event_name == 'push' &&
108+
(github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging')
53109
runs-on: blacksmith-8vcpu-ubuntu-2404
54110
permissions:
55111
contents: read
@@ -75,8 +131,8 @@ jobs:
75131
- name: Configure AWS credentials
76132
uses: aws-actions/configure-aws-credentials@v4
77133
with:
78-
role-to-assume: ${{ github.ref == 'refs/heads/main' && secrets.AWS_ROLE_TO_ASSUME || github.ref == 'refs/heads/dev' && secrets.DEV_AWS_ROLE_TO_ASSUME || secrets.STAGING_AWS_ROLE_TO_ASSUME }}
79-
aws-region: ${{ github.ref == 'refs/heads/main' && secrets.AWS_REGION || github.ref == 'refs/heads/dev' && secrets.DEV_AWS_REGION || secrets.STAGING_AWS_REGION }}
134+
role-to-assume: ${{ github.ref == 'refs/heads/main' && secrets.AWS_ROLE_TO_ASSUME || secrets.STAGING_AWS_ROLE_TO_ASSUME }}
135+
aws-region: ${{ github.ref == 'refs/heads/main' && secrets.AWS_REGION || secrets.STAGING_AWS_REGION }}
80136

81137
- name: Login to Amazon ECR
82138
id: login-ecr
@@ -106,26 +162,20 @@ jobs:
106162
ECR_REPO="${{ secrets[matrix.ecr_repo_secret] }}"
107163
GHCR_IMAGE="${{ matrix.ghcr_image }}"
108164
109-
# ECR tags (always build for ECR)
110165
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
111166
ECR_TAG="latest"
112-
elif [ "${{ github.ref }}" = "refs/heads/dev" ]; then
113-
ECR_TAG="dev"
114167
else
115168
ECR_TAG="staging"
116169
fi
117170
ECR_IMAGE="${ECR_REGISTRY}/${ECR_REPO}:${ECR_TAG}"
118171
119-
# Build tags list
120172
TAGS="${ECR_IMAGE}"
121173
122-
# Add GHCR tags only for main branch
123174
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
124175
GHCR_AMD64="${GHCR_IMAGE}:latest-amd64"
125176
GHCR_SHA="${GHCR_IMAGE}:${{ github.sha }}-amd64"
126177
TAGS="${TAGS},$GHCR_AMD64,$GHCR_SHA"
127178
128-
# Add version tag if this is a release commit
129179
if [ "${{ needs.detect-version.outputs.is_release }}" = "true" ]; then
130180
VERSION="${{ needs.detect-version.outputs.version }}"
131181
GHCR_VERSION="${GHCR_IMAGE}:${VERSION}-amd64"
@@ -256,6 +306,14 @@ jobs:
256306
docker manifest push "${IMAGE_BASE}:${VERSION}"
257307
fi
258308
309+
# Run database migrations for dev
310+
migrate-dev:
311+
name: Migrate Dev DB
312+
needs: [build-dev]
313+
if: github.event_name == 'push' && github.ref == 'refs/heads/dev'
314+
uses: ./.github/workflows/migrations.yml
315+
secrets: inherit
316+
259317
# Check if docs changed
260318
check-docs-changes:
261319
name: Check Docs Changes

.github/workflows/migrations.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,5 +38,5 @@ jobs:
3838
- name: Apply migrations
3939
working-directory: ./packages/db
4040
env:
41-
DATABASE_URL: ${{ github.ref == 'refs/heads/main' && secrets.DATABASE_URL || secrets.STAGING_DATABASE_URL }}
41+
DATABASE_URL: ${{ github.ref == 'refs/heads/main' && secrets.DATABASE_URL || github.ref == 'refs/heads/dev' && secrets.DEV_DATABASE_URL || secrets.STAGING_DATABASE_URL }}
4242
run: bunx drizzle-kit migrate --config=./drizzle.config.ts

README.md

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -74,10 +74,6 @@ docker compose -f docker-compose.prod.yml up -d
7474

7575
Open [http://localhost:3000](http://localhost:3000)
7676

77-
#### Background worker note
78-
79-
The Docker Compose stack starts a dedicated worker container by default. If `REDIS_URL` is not configured, the worker will start, log that it is idle, and do no queue processing. This is expected. Queue-backed API, webhook, and schedule execution requires Redis; installs without Redis continue to use the inline execution path.
80-
8177
Sim also supports local models via [Ollama](https://ollama.ai) and [vLLM](https://docs.vllm.ai/) — see the [Docker self-hosting docs](https://docs.sim.ai/self-hosting/docker) for setup details.
8278

8379
### Self-hosted: Manual Setup
@@ -123,12 +119,10 @@ cd packages/db && bun run db:migrate
123119
5. Start development servers:
124120

125121
```bash
126-
bun run dev:full # Starts Next.js app, realtime socket server, and the BullMQ worker
122+
bun run dev:full # Starts Next.js app and realtime socket server
127123
```
128124

129-
If `REDIS_URL` is not configured, the worker will remain idle and execution continues inline.
130-
131-
Or run separately: `bun run dev` (Next.js), `cd apps/sim && bun run dev:sockets` (realtime), and `cd apps/sim && bun run worker` (BullMQ worker).
125+
Or run separately: `bun run dev` (Next.js) and `cd apps/sim && bun run dev:sockets` (realtime).
132126

133127
## Copilot API Keys
134128

apps/sim/app/(landing)/components/features/components/features-preview.tsx

Lines changed: 17 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
import { type SVGProps, useEffect, useRef, useState } from 'react'
44
import { AnimatePresence, motion, useInView } from 'framer-motion'
5-
import ReactMarkdown, { type Components } from 'react-markdown'
6-
import remarkGfm from 'remark-gfm'
5+
import { Streamdown } from 'streamdown'
6+
import 'streamdown/styles.css'
77
import { ChevronDown } from '@/components/emcn'
88
import { Database, File, Library, Table } from '@/components/emcn/icons'
99
import {
@@ -557,26 +557,32 @@ The team agreed to prioritize the new onboarding flow. Key decisions:
557557
558558
Follow up with engineering on the timeline for the API v2 migration. Draft the proposal for the board meeting next week.`
559559

560-
const MD_COMPONENTS: Components = {
561-
h1: ({ children }) => (
560+
const MD_COMPONENTS = {
561+
h1: ({ children }: { children?: React.ReactNode }) => (
562562
<p
563563
role='presentation'
564564
className='mb-4 border-[#E5E5E5] border-b pb-2 font-semibold text-[#1C1C1C] text-[20px]'
565565
>
566566
{children}
567567
</p>
568568
),
569-
h2: ({ children }) => (
569+
h2: ({ children }: { children?: React.ReactNode }) => (
570570
<h2 className='mt-5 mb-3 border-[#E5E5E5] border-b pb-1.5 font-semibold text-[#1C1C1C] text-[16px]'>
571571
{children}
572572
</h2>
573573
),
574-
ul: ({ children }) => <ul className='mb-3 list-disc pl-6'>{children}</ul>,
575-
ol: ({ children }) => <ol className='mb-3 list-decimal pl-6'>{children}</ol>,
576-
li: ({ children }) => (
574+
ul: ({ children }: { children?: React.ReactNode }) => (
575+
<ul className='mb-3 list-disc pl-6'>{children}</ul>
576+
),
577+
ol: ({ children }: { children?: React.ReactNode }) => (
578+
<ol className='mb-3 list-decimal pl-6'>{children}</ol>
579+
),
580+
li: ({ children }: { children?: React.ReactNode }) => (
577581
<li className='mb-1 text-[#1C1C1C] text-[14px] leading-[1.6]'>{children}</li>
578582
),
579-
p: ({ children }) => <p className='mb-3 text-[#1C1C1C] text-[14px] leading-[1.6]'>{children}</p>,
583+
p: ({ children }: { children?: React.ReactNode }) => (
584+
<p className='mb-3 text-[#1C1C1C] text-[14px] leading-[1.6]'>{children}</p>
585+
),
580586
}
581587

582588
function MockFullFiles() {
@@ -618,9 +624,9 @@ function MockFullFiles() {
618624
transition={{ duration: 0.4, delay: 0.5 }}
619625
>
620626
<div className='h-full overflow-auto p-6'>
621-
<ReactMarkdown remarkPlugins={[remarkGfm]} components={MD_COMPONENTS}>
627+
<Streamdown mode='static' components={MD_COMPONENTS}>
622628
{source}
623-
</ReactMarkdown>
629+
</Streamdown>
624630
</div>
625631
</motion.div>
626632
</div>
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
import { db } from '@sim/db'
2+
import { user } from '@sim/db/schema'
3+
import { eq } from 'drizzle-orm'
4+
import { type NextRequest, NextResponse } from 'next/server'
5+
import { getSession } from '@/lib/auth'
6+
import { env } from '@/lib/core/config/env'
7+
8+
const ENV_URLS: Record<string, string | undefined> = {
9+
dev: env.MOTHERSHIP_DEV_URL,
10+
staging: env.MOTHERSHIP_STAGING_URL,
11+
prod: env.MOTHERSHIP_PROD_URL,
12+
}
13+
14+
function getMothershipUrl(environment: string): string | null {
15+
return ENV_URLS[environment] ?? null
16+
}
17+
18+
async function isAdminRequestAuthorized() {
19+
const session = await getSession()
20+
if (!session?.user?.id) return false
21+
22+
const [currentUser] = await db
23+
.select({ role: user.role })
24+
.from(user)
25+
.where(eq(user.id, session.user.id))
26+
.limit(1)
27+
28+
return currentUser?.role === 'admin'
29+
}
30+
31+
/**
32+
* Proxy to the mothership admin API.
33+
*
34+
* Query params:
35+
* env - "dev" | "staging" | "prod"
36+
* endpoint - the admin endpoint path, e.g. "requests", "licenses", "traces"
37+
*
38+
* The request body (for POST) is forwarded as-is. Additional query params
39+
* (e.g. requestId for GET /traces) are forwarded.
40+
*/
41+
export async function POST(req: NextRequest) {
42+
if (!(await isAdminRequestAuthorized())) {
43+
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
44+
}
45+
46+
const adminKey = env.MOTHERSHIP_API_ADMIN_KEY
47+
if (!adminKey) {
48+
return NextResponse.json({ error: 'MOTHERSHIP_API_ADMIN_KEY not configured' }, { status: 500 })
49+
}
50+
51+
const { searchParams } = new URL(req.url)
52+
const environment = searchParams.get('env') || 'dev'
53+
const endpoint = searchParams.get('endpoint')
54+
55+
if (!endpoint) {
56+
return NextResponse.json({ error: 'endpoint query param required' }, { status: 400 })
57+
}
58+
59+
const baseUrl = getMothershipUrl(environment)
60+
if (!baseUrl) {
61+
return NextResponse.json(
62+
{ error: `No URL configured for environment: ${environment}` },
63+
{ status: 400 }
64+
)
65+
}
66+
67+
const targetUrl = `${baseUrl}/api/admin/${endpoint}`
68+
69+
try {
70+
const body = await req.text()
71+
const upstream = await fetch(targetUrl, {
72+
method: 'POST',
73+
headers: {
74+
'Content-Type': 'application/json',
75+
'x-api-key': adminKey,
76+
},
77+
...(body ? { body } : {}),
78+
})
79+
80+
const data = await upstream.json()
81+
return NextResponse.json(data, { status: upstream.status })
82+
} catch (error) {
83+
return NextResponse.json(
84+
{
85+
error: `Failed to reach mothership (${environment}): ${error instanceof Error ? error.message : 'Unknown error'}`,
86+
},
87+
{ status: 502 }
88+
)
89+
}
90+
}
91+
92+
export async function GET(req: NextRequest) {
93+
if (!(await isAdminRequestAuthorized())) {
94+
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
95+
}
96+
97+
const adminKey = env.MOTHERSHIP_API_ADMIN_KEY
98+
if (!adminKey) {
99+
return NextResponse.json({ error: 'MOTHERSHIP_API_ADMIN_KEY not configured' }, { status: 500 })
100+
}
101+
102+
const { searchParams } = new URL(req.url)
103+
const environment = searchParams.get('env') || 'dev'
104+
const endpoint = searchParams.get('endpoint')
105+
106+
if (!endpoint) {
107+
return NextResponse.json({ error: 'endpoint query param required' }, { status: 400 })
108+
}
109+
110+
const baseUrl = getMothershipUrl(environment)
111+
if (!baseUrl) {
112+
return NextResponse.json(
113+
{ error: `No URL configured for environment: ${environment}` },
114+
{ status: 400 }
115+
)
116+
}
117+
118+
const forwardParams = new URLSearchParams()
119+
searchParams.forEach((value, key) => {
120+
if (key !== 'env' && key !== 'endpoint') {
121+
forwardParams.set(key, value)
122+
}
123+
})
124+
125+
const qs = forwardParams.toString()
126+
const targetUrl = `${baseUrl}/api/admin/${endpoint}${qs ? `?${qs}` : ''}`
127+
128+
try {
129+
const upstream = await fetch(targetUrl, {
130+
method: 'GET',
131+
headers: { 'x-api-key': adminKey },
132+
})
133+
134+
const data = await upstream.json()
135+
return NextResponse.json(data, { status: upstream.status })
136+
} catch (error) {
137+
return NextResponse.json(
138+
{
139+
error: `Failed to reach mothership (${environment}): ${error instanceof Error ? error.message : 'Unknown error'}`,
140+
},
141+
{ status: 502 }
142+
)
143+
}
144+
}

apps/sim/app/api/billing/update-cost/route.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ import { type NextRequest, NextResponse } from 'next/server'
44
import { z } from 'zod'
55
import { recordUsage } from '@/lib/billing/core/usage-log'
66
import { checkAndBillOverageThreshold } from '@/lib/billing/threshold-billing'
7-
import { checkInternalApiKey } from '@/lib/copilot/utils'
7+
import { checkInternalApiKey } from '@/lib/copilot/request/http'
88
import { isBillingEnabled } from '@/lib/core/config/feature-flags'
99
import { generateRequestId } from '@/lib/core/utils/request'
1010

0 commit comments

Comments
 (0)