Skip to content

Commit 591467b

Browse files
ijboijbo
committed
feat: migrate model hosting to textagent HuggingFace org
- Duplicated 7 ONNX models to textagent HuggingFace org - All model IDs changed from onnx-community/ to textagent/ - Fallback swaps to onnx-community/ if textagent models unavailable - Removed GitLab mirror from all runtime code - model-hosts.js exports MODEL_ORG + MODEL_ORG_FALLBACK - Updated ai-worker, speech-worker, tts-worker, ai-models, ai-assistant
1 parent 7b9f846 commit 591467b

13 files changed

Lines changed: 261 additions & 101 deletions

‎CHANGELOG-model-hosting.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Model Hosting Migration — HuggingFace textagent Org
2+
3+
- All ONNX model IDs changed from `onnx-community/` to `textagent/` namespace
4+
- 7 models duplicated to `textagent` HuggingFace org via `duplicate_repo` API
5+
- `model-hosts.js` simplified: single HuggingFace CDN, org-based primary/fallback
6+
- All workers (`ai-worker.js`, `speech-worker.js`, `tts-worker.js`) updated to use `textagent/` models with `onnx-community/` fallback
7+
- Fixed: removed all GitLab model hosting references from runtime code
8+
- `mirror-models.sh` retained as GitLab backup utility (downloads from `onnx-community`)
9+
10+
---
11+
12+
## Summary
13+
Migrated all ONNX model references from `onnx-community` to a self-owned `textagent` HuggingFace organization. Models are server-side duplicated (not re-uploaded), ensuring identical content. If any `textagent/` model is unavailable, workers automatically fall back to the original `onnx-community/` namespace on the same HuggingFace CDN.
14+
15+
---
16+
17+
## 1. HuggingFace Org Setup
18+
**What:** Created `textagent` organization on HuggingFace Hub; duplicated 7 ONNX models server-side using `huggingface_hub.duplicate_repo()`.
19+
**Impact:** TextAgent now owns its model namespace — insulated from upstream renames/deletions by `onnx-community`.
20+
21+
### Models Duplicated
22+
| Model | Size | URL |
23+
|-------|------|-----|
24+
| Qwen 3.5 0.8B ONNX | ~500 MB | `huggingface.co/textagent/Qwen3.5-0.8B-ONNX` |
25+
| Qwen 3.5 2B ONNX | ~1.2 GB | `huggingface.co/textagent/Qwen3.5-2B-ONNX` |
26+
| Qwen 3.5 4B ONNX | ~2.5 GB | `huggingface.co/textagent/Qwen3.5-4B-ONNX` |
27+
| Qwen 3 4B Thinking ONNX | ~2.5 GB | `huggingface.co/textagent/Qwen3-4B-Thinking-2507-ONNX` |
28+
| Whisper Large V3 Turbo | ~1.5 GB | `huggingface.co/textagent/whisper-large-v3-turbo` |
29+
| Kokoro 82M v1.0 ONNX | ~80 MB | `huggingface.co/textagent/Kokoro-82M-v1.0-ONNX` |
30+
| Kokoro 82M v1.1-zh ONNX | ~80 MB | `huggingface.co/textagent/Kokoro-82M-v1.1-zh-ONNX` |
31+
32+
## 2. Model Hosts Configuration
33+
**Files:** `js/model-hosts.js`
34+
**What:** Replaced GitLab primary + HuggingFace fallback with single HuggingFace CDN; exports `MODEL_HOST`, `MODEL_ORG` (`textagent`), and `MODEL_ORG_FALLBACK` (`onnx-community`).
35+
**Impact:** Simplified host config; no more GitLab dependency for model downloads.
36+
37+
## 3. AI Worker Updates
38+
**Files:** `ai-worker.js`, `public/ai-worker.js`
39+
**What:** Changed `MODEL_HOST` from GitLab to HuggingFace; fallback now swaps model ID prefix from `textagent/` to `onnx-community/` instead of switching hosts.
40+
**Impact:** Workers download models from HuggingFace CDN with automatic namespace fallback.
41+
42+
## 4. Speech Worker Update
43+
**Files:** `js/speech-worker.js`
44+
**What:** Removed GitLab host; Whisper model ID uses `textagent/whisper-large-v3-turbo` with `onnx-community/` fallback on failure.
45+
**Impact:** Speech-to-text downloads from HuggingFace with org-based fallback.
46+
47+
## 5. TTS Worker Update
48+
**Files:** `js/tts-worker.js`
49+
**What:** Removed GitLab comments; model ID changed to `textagent/Kokoro-82M-v1.1-zh-ONNX`.
50+
**Impact:** TTS model downloads from textagent HuggingFace org.
51+
52+
## 6. Model Registry Update
53+
**Files:** `js/ai-models.js`, `js/ai-assistant.js`
54+
**What:** All `localModelId` values changed from `onnx-community/` to `textagent/` prefix.
55+
**Impact:** Model selector dropdown and download flows use textagent namespace.
56+
57+
---
58+
59+
## Files Changed (10 total)
60+
61+
| File | Lines Changed | Type |
62+
|------|:---:|------|
63+
| `js/model-hosts.js` | +12 −11 | Simplified: HF-only, org-based fallback |
64+
| `ai-worker.js` | +9 −9 | HF host, org-based fallback |
65+
| `public/ai-worker.js` | +9 −9 | Same as above (public copy) |
66+
| `js/speech-worker.js` | +11 −10 | HF host, org-based fallback |
67+
| `js/tts-worker.js` | +3 −6 | Removed GitLab comments |
68+
| `js/ai-models.js` | +5 −5 | Model IDs → textagent/ |
69+
| `js/ai-assistant.js` | +1 −1 | Default model ID → textagent/ |
70+
| `js/ai-actions.js` | +2 −2 | Minor |
71+
| `js/ai-docgen.js` | +2 −2 | Minor |
72+
| `js/ai-docgen-generate.js` | +3 −3 | Minor |

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -455,7 +455,8 @@ TextAgent has undergone significant evolution since its inception. What started
455455

456456
| Date | Commits | Feature / Update |
457457
|------|---------|-----------------|
458-
| **2026-03-12** | — | 🔊 **Kokoro TTS** — hybrid text-to-speech engine: English/Chinese via [Kokoro 82M v1.1-zh ONNX](https://huggingface.co/onnx-community/Kokoro-82M-v1.1-zh-ONNX) (~80 MB, off-thread WebWorker via `kokoro-js`), Japanese & 10+ languages via Web Speech API fallback; hover preview text → click 🔊 for pronunciation; voice auto-selection by language; `textToSpeech.js` main module + `tts-worker.js` WebWorker + `tts.css` styling; model-hosts.js for configurable hosting with auto-fallback to HuggingFace |
458+
| **2026-03-12** | — | 🏠 **Model Hosting Migration** — all 7 ONNX models (Qwen 3.5 0.8B/2B/4B, Qwen 3 4B Thinking, Whisper Large V3 Turbo, Kokoro 82M v1.0/v1.1-zh) duplicated to self-owned [`textagent` HuggingFace org](https://huggingface.co/textagent); model IDs updated from `onnx-community/` to `textagent/` across all workers; automatic fallback to `onnx-community/` namespace if textagent models unavailable; GitLab mirror removed from runtime code |
459+
| **2026-03-12** | — | 🔊 **Kokoro TTS** — hybrid text-to-speech engine: English/Chinese via [Kokoro 82M v1.1-zh ONNX](https://huggingface.co/textagent/Kokoro-82M-v1.1-zh-ONNX) (~80 MB, off-thread WebWorker via `kokoro-js`), Japanese & 10+ languages via Web Speech API fallback; hover preview text → click 🔊 for pronunciation; voice auto-selection by language; `textToSpeech.js` main module + `tts-worker.js` WebWorker + `tts.css` styling; model-hosts.js for configurable hosting with auto-fallback |
459460
| **2026-03-12** | — | 📷 **OCR Tag** — new `{{@OCR:}}` document tag for image-to-text extraction; amber-accented card with mode pills (Text/Math/Table); 📎 image upload with `@upload:` editor sync; Qwen model default; vision-capable model flags (`supportsVision`) on Qwen 3.5 Flash, 35B-A3B, and DeepSeek V3.2 |
460461
| **2026-03-12** | — | 🏗️ **Model Architecture** — ai-worker.js refactored for architecture-aware loading (`qwen3` text-only vs `qwen3_5` vision); `setModelId` accepts `architecture` + `dtype` params; automatic fallback to HuggingFace when primary host fails; `moonshine-medium-worker.js` deleted (replaced by unified `speech-worker.js`); Language Learning template with TTS pronunciation tips; SQLite-compatible SQL in Technical template |
461462
| **2026-03-11** | — | ▶ **Run All Notebook Engine** — one-click `▶ Run All` button executes every code/tag block in document order; 11 runtime adapters (bash, math, python, html, js, sql, docgen-ai, docgen-image, docgen-agent, api, linux-script); Block Registry with FNV-1a stable IDs; Execution Controller with fixed-bottom progress bar, per-block status badges (pending/running/done/error), and abort support; SQLite `_exec_results` context store for cross-block data sharing; DocGen/API adapters use auto-accept mode (skip review panel); Linux adapter submits to Judge0 CE; deferred adapter queue for module loading order; `exec-engine.css` styling; 12 new Playwright tests (191 total) |

‎ai-worker.js‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -19,15 +19,15 @@ let RawImage = null;
1919

2020
const TRANSFORMERS_URL = "https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.0.0-next.6";
2121

22-
// Self-hosted model mirror — downloads ONNX models from GitLab instead of HuggingFace
23-
const MODEL_HOST = "https://gitlab.com/textagent/models/-/raw/main";
24-
const MODEL_HOST_FALLBACK = "https://huggingface.co";
22+
// Model host — downloads ONNX models from textagent HuggingFace org
23+
const MODEL_HOST = "https://huggingface.co";
24+
const MODEL_ORG_FALLBACK = "onnx-community";
2525

2626
// Reference to Transformers.js env (set after dynamic import)
2727
let transformersEnv = null;
2828

2929
// Model config — default to 0.8B, overridable via setModelId message
30-
let MODEL_ID = "onnx-community/Qwen3.5-0.8B-ONNX";
30+
let MODEL_ID = "textagent/Qwen3.5-0.8B-ONNX";
3131
let MODEL_LABEL = "Qwen 3.5";
3232
let MODEL_ARCH = "qwen3_5"; // 'qwen3_5' or 'qwen3'
3333
let MODEL_DTYPE = "q4"; // 'q4' or 'q4f16'
@@ -73,7 +73,7 @@ async function loadModel() {
7373
TextStreamer = transformers.TextStreamer;
7474
RawImage = transformers.RawImage;
7575

76-
// Point model downloads to self-hosted GitLab mirror
76+
// Point model downloads to HuggingFace
7777
transformersEnv = transformers.env;
7878
transformersEnv.remoteHost = MODEL_HOST;
7979
} catch (importError) {
@@ -152,13 +152,13 @@ async function loadModel() {
152152
try {
153153
await loadFromHost();
154154
} catch (primaryErr) {
155-
// Primary host failed — fall back to HuggingFace
156-
console.warn(`Primary model host failed: ${primaryErr.message}. Falling back to HuggingFace…`);
155+
// Primary org (textagent) failed — fall back to onnx-community
156+
console.warn(`textagent model failed: ${primaryErr.message}. Falling back to onnx-community…`);
157157
self.postMessage({
158158
type: "status",
159-
message: `Primary host unavailable — falling back to HuggingFace…`,
159+
message: `Falling back to onnx-community models…`,
160160
});
161-
transformersEnv.remoteHost = MODEL_HOST_FALLBACK;
161+
MODEL_ID = MODEL_ID.replace('textagent/', MODEL_ORG_FALLBACK + '/');
162162
processor = null;
163163
model = null;
164164
await loadFromHost();

‎changelogs/CHANGELOG-model-hosting.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Model Hosting Migration — HuggingFace textagent Org
2+
3+
- All ONNX model IDs changed from `onnx-community/` to `textagent/` namespace
4+
- 7 models duplicated to `textagent` HuggingFace org via `duplicate_repo` API
5+
- `model-hosts.js` simplified: single HuggingFace CDN, org-based primary/fallback
6+
- All workers (`ai-worker.js`, `speech-worker.js`, `tts-worker.js`) updated to use `textagent/` models with `onnx-community/` fallback
7+
- Fixed: removed all GitLab model hosting references from runtime code
8+
- `mirror-models.sh` retained as GitLab backup utility (downloads from `onnx-community`)
9+
10+
---
11+
12+
## Summary
13+
Migrated all ONNX model references from `onnx-community` to a self-owned `textagent` HuggingFace organization. Models are server-side duplicated (not re-uploaded), ensuring identical content. If any `textagent/` model is unavailable, workers automatically fall back to the original `onnx-community/` namespace on the same HuggingFace CDN.
14+
15+
---
16+
17+
## 1. HuggingFace Org Setup
18+
**What:** Created `textagent` organization on HuggingFace Hub; duplicated 7 ONNX models server-side using `huggingface_hub.duplicate_repo()`.
19+
**Impact:** TextAgent now owns its model namespace — insulated from upstream renames/deletions by `onnx-community`.
20+
21+
### Models Duplicated
22+
| Model | Size | URL |
23+
|-------|------|-----|
24+
| Qwen 3.5 0.8B ONNX | ~500 MB | `huggingface.co/textagent/Qwen3.5-0.8B-ONNX` |
25+
| Qwen 3.5 2B ONNX | ~1.2 GB | `huggingface.co/textagent/Qwen3.5-2B-ONNX` |
26+
| Qwen 3.5 4B ONNX | ~2.5 GB | `huggingface.co/textagent/Qwen3.5-4B-ONNX` |
27+
| Qwen 3 4B Thinking ONNX | ~2.5 GB | `huggingface.co/textagent/Qwen3-4B-Thinking-2507-ONNX` |
28+
| Whisper Large V3 Turbo | ~1.5 GB | `huggingface.co/textagent/whisper-large-v3-turbo` |
29+
| Kokoro 82M v1.0 ONNX | ~80 MB | `huggingface.co/textagent/Kokoro-82M-v1.0-ONNX` |
30+
| Kokoro 82M v1.1-zh ONNX | ~80 MB | `huggingface.co/textagent/Kokoro-82M-v1.1-zh-ONNX` |
31+
32+
## 2. Model Hosts Configuration
33+
**Files:** `js/model-hosts.js`
34+
**What:** Replaced GitLab primary + HuggingFace fallback with single HuggingFace CDN; exports `MODEL_HOST`, `MODEL_ORG` (`textagent`), and `MODEL_ORG_FALLBACK` (`onnx-community`).
35+
**Impact:** Simplified host config; no more GitLab dependency for model downloads.
36+
37+
## 3. AI Worker Updates
38+
**Files:** `ai-worker.js`, `public/ai-worker.js`
39+
**What:** Changed `MODEL_HOST` from GitLab to HuggingFace; fallback now swaps model ID prefix from `textagent/` to `onnx-community/` instead of switching hosts.
40+
**Impact:** Workers download models from HuggingFace CDN with automatic namespace fallback.
41+
42+
## 4. Speech Worker Update
43+
**Files:** `js/speech-worker.js`
44+
**What:** Removed GitLab host; Whisper model ID uses `textagent/whisper-large-v3-turbo` with `onnx-community/` fallback on failure.
45+
**Impact:** Speech-to-text downloads from HuggingFace with org-based fallback.
46+
47+
## 5. TTS Worker Update
48+
**Files:** `js/tts-worker.js`
49+
**What:** Removed GitLab comments; model ID changed to `textagent/Kokoro-82M-v1.1-zh-ONNX`.
50+
**Impact:** TTS model downloads from textagent HuggingFace org.
51+
52+
## 6. Model Registry Update
53+
**Files:** `js/ai-models.js`, `js/ai-assistant.js`
54+
**What:** All `localModelId` values changed from `onnx-community/` to `textagent/` prefix.
55+
**Impact:** Model selector dropdown and download flows use textagent namespace.
56+
57+
---
58+
59+
## Files Changed (10 total)
60+
61+
| File | Lines Changed | Type |
62+
|------|:---:|------|
63+
| `js/model-hosts.js` | +12 −11 | Simplified: HF-only, org-based fallback |
64+
| `ai-worker.js` | +9 −9 | HF host, org-based fallback |
65+
| `public/ai-worker.js` | +9 −9 | Same as above (public copy) |
66+
| `js/speech-worker.js` | +11 −10 | HF host, org-based fallback |
67+
| `js/tts-worker.js` | +3 −6 | Removed GitLab comments |
68+
| `js/ai-models.js` | +5 −5 | Model IDs → textagent/ |
69+
| `js/ai-assistant.js` | +1 −1 | Default model ID → textagent/ |
70+
| `js/ai-actions.js` | +2 −2 | Minor |
71+
| `js/ai-docgen.js` | +2 −2 | Minor |
72+
| `js/ai-docgen-generate.js` | +3 −3 | Minor |

‎js/ai-actions.js‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@
3131

3232
var isReady = _ai.isCurrentModelReady();
3333
if (!isReady) {
34-
M.openAiPanel();
34+
if (M.showModelDownloadPopup) M.showModelDownloadPopup();
3535
return;
3636
}
3737

@@ -93,7 +93,7 @@
9393
e.preventDefault();
9494
var isReady = _ai.isCurrentModelReady();
9595
if (!isReady) {
96-
M.openAiPanel();
96+
if (M.showModelDownloadPopup) M.showModelDownloadPopup();
9797
return;
9898
}
9999
if (!_ai.panelOpen) M.openAiPanel();

0 commit comments

Comments
 (0)