ydy0615/llm-in-text
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: ydy0615/llm-in-text:5e258015096c44e969de76d7fc4e8cb861a6a42e
Branches Tags
ydy0615/llm-in-text:main ydy0615/llm-in-text:lopsided-roadrunner
..
compare: ydy0615/llm-in-text:lopsided-roadrunner
Branches Tags
ydy0615/llm-in-text:main ydy0615/llm-in-text:lopsided-roadrunner

1 Commits
5e25801509 .. lopsided-roadrunner

Author SHA1 Message Date
ydy0615 05b1cbf80d chore: 更新项目配置和前后端代码优化 2026-04-04 20:10:24 +08:00
201 changed files with 9377 additions and 85660 deletions
Expand all files Collapse all files
.cline/kanban/config.json
+9
View File
@@ -0,0 +1,9 @@
{
"shortcuts": [
{
"label": "Run",
"command": "npm run dev",
"icon": "play"
}
]
}
.coveragerc
-13
View File
@@ -1,13 +0,0 @@
[run]
source = backend
omit =
backend/tests/*
backend/test_*.py
backend/__pycache__/*
[report]
exclude_lines =
pragma: no cover
if TYPE_CHECKING:
raise NotImplementedError
if __name__ == .__main__.:
.dockerignore
-14
View File
@@ -1,14 +0,0 @@
node_modules
dist
htmlcov
.pytest_cache
.coverage
.build-check
reports
.git
docker-data
backend/.env
backend/models
backend/__pycache__
backend/tests/__pycache__
**/.DS_Store
.env.example
+1 -18
View File
@@ -1,21 +1,4 @@
VITE_API_BASE_URL=https://api.imageteach.tech:8002
VITE_API_BASE_URL=
VITE_API_URL=
VITE_OCR_URL=
VITE_CONVERT_URL=
VITE_PRO_URL=
VITE_TTS_URL=
VITE_TTS_STATUS_URL=
VITE_TTS_CONFIG_URL=
VITE_ASR_URL=
VITE_JOB_LOAD_URL=
VITE_DOCS_NODES_URL=
VITE_DOCS_FOLDERS_URL=
VITE_DOCS_TEXT_FILES_URL=
VITE_DOCS_UPLOAD_URL=
VITE_DOCS_BLOB_BASE_URL=
VITE_DOCS_NODES_BASE_URL=
VITE_PRO_FRONTEND_TIMEOUT_MS=3660000
VITE_API_KEY=
# Document block compression context limit (characters)
VITE_DOC_COMPRESS_CONTEXT_LIMIT=128000
.gitignore
-21
View File
@@ -13,7 +13,6 @@ dist-ssr
*.local
# Python
backend/models/
__pycache__/
*.py[cod]
*.pyc.*
@@ -24,9 +23,6 @@ env/
.pytest_cache/
.mypy_cache/
.ruff_cache/
htmlcov/
.coverage
api_performance_report.md
# Env files
.env
@@ -38,25 +34,8 @@ api_performance_report.md
!.vscode/extensions.json
.idea
.DS_Store
**/xcuserdata/
*.xcuserstate
DerivedData/
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?
# IDE directories
.kilocode/
.kilo/
.codex/
# Agent/runtime state and local verification artifacts
.omx/
.tmp-*.png
tmp-*.txt
# Docker runtime data must live under /Users/allenyuan/lit, never in this repo.
docker-data/
.kilocode/rules/rules.md
+11
View File
@@ -0,0 +1,11 @@
# rules.md
在构建这个LLM应用网页时，你需要基于VUE3开发。我需要前端只运行渲染和数据回传，后端负责llm api调用，类似copilet的auto inline suggustions实现和数据解析。
## 指导原则
- 不要擅自用npm或者yarn运行网页，你既看不到网页的内容，也无法阻止命令暂停。但是，你可以用npm run build检查代码。
- 应该保证代码效率，不多定义变量，不写冗余注释，把降低延迟放在第一位。
- 每次完成任务前都要反复阅读检查代码，确保代码准确无误。
- 尽量不要搜索关键字，而是了解代码结构后查询整个问题代码明确问题所在。
- @/milkdown-docs/ 代表milkdown的最新官方文档，不要修改，涉及到前端编辑器的指令时要核对官方文档。
.playwright-cli/page-2026-05-30T04-32-21-011Z.yml
-91
View File
@@ -1,91 +0,0 @@
- generic [ref=e3]:
- generic:
- button "设置" [ref=e5] [cursor=pointer]:
- img [ref=e6]
- generic [ref=e9]:
- generic [ref=e10]:
- heading "设置" [level=2] [ref=e11]
- button "关闭" [ref=e12] [cursor=pointer]:
- img [ref=e13]
- generic [ref=e16]:
- generic [ref=e17]:
- heading "外观" [level=3] [ref=e18]
- generic [ref=e19]:
- generic [ref=e20]: 外观
- combobox [ref=e21]:
- option "深色"
- option "浅色"
- option "跟随系统" [selected]
- option "暖色调"
- option "读书灯"
- option "自定义图片"
- generic [ref=e22]:
- heading "视图" [level=3] [ref=e23]
- generic [ref=e24]:
- button "编辑器" [ref=e25] [cursor=pointer]:
- img [ref=e26]
- generic [ref=e28]: 编辑器
- button "文档" [ref=e29] [cursor=pointer]:
- img [ref=e30]
- generic [ref=e33]: 文档
- generic [ref=e34]:
- heading "PRO 模式" [level=3] [ref=e35]
- generic [ref=e36]:
- generic [ref=e37]: PRO 思考程度
- generic [ref=e38]:
- button "低" [ref=e39] [cursor=pointer]
- button "中" [ref=e40] [cursor=pointer]
- button "高" [ref=e41] [cursor=pointer]
- paragraph [ref=e42]: PRO 模式使用独立思考强度，普通补全设置不会影响它。
- generic [ref=e43]:
- heading "模型智能" [level=3] [ref=e44]
- generic [ref=e45]:
- generic [ref=e46]: 思考程度
- generic [ref=e47]:
- button "低" [ref=e48] [cursor=pointer]
- button "中" [ref=e49] [cursor=pointer]
- button "高" [ref=e50] [cursor=pointer]
- paragraph [ref=e51]: 直接补全（最快）
- generic [ref=e52]:
- generic [ref=e53]: "防抖时间: 1000ms"
- slider [ref=e54]: "1000"
- generic [ref=e55]:
- heading "隐私与偏好" [level=3] [ref=e56]
- generic [ref=e57]:
- generic [ref=e58]: 隐私模式
- button [ref=e59] [cursor=pointer]
- paragraph [ref=e61]: 不向 AI 发送 IP 地址和偏好设置
- generic [ref=e62]:
- generic [ref=e63]: 语言
- combobox [disabled] [ref=e64]:
- option "自动检测" [selected]
- option "Chinese"
- option "English"
- option "Japanese"
- option "Korean"
- option "German"
- option "French"
- generic [ref=e65]:
- generic [ref=e66]: 货币
- combobox [disabled] [ref=e67]:
- option "自动检测" [selected]
- option "CNY (¥)"
- option "USD ($)"
- option "EUR (€)"
- option "JPY (¥)"
- option "KRW (₩)"
- option "GBP (£)"
- option "AUD ($)"
- option "CAD ($)"
- generic [ref=e68]:
- heading "语音设置" [level=3] [ref=e69]
- generic [ref=e70]:
- generic [ref=e71]: 声音描述
- textbox "例如：用温柔的语气说" [ref=e72]
- paragraph [ref=e73]: 描述你想要的声音风格，如语气、情感等
- generic [ref=e74]:
- heading "关于我们" [level=3] [ref=e75]
- generic [ref=e76]:
- heading "llm-in-text" [level=4] [ref=e77]
- paragraph [ref=e78]: A smart Markdown editor with local LLM intelligence.
- paragraph [ref=e79]: v0.0.0
.playwright-cli/page-2026-05-30T04-52-35-877Z.yml
-124
View File
@@ -1,124 +0,0 @@
- generic [ref=e3]:
- generic [ref=e80]:
- textbox [active] [ref=e83]:
- heading "欢迎使用 LLM-IN-TEXT" [level=1] [ref=e84]
- paragraph [ref=e85]: 即时可用的 LLM 系统
- paragraph [ref=e86]: 在下方开始创作吧...
- generic [ref=e87]:
- button "Undo" [disabled] [ref=e88]:
- img [ref=e89]
- button "Redo" [disabled] [ref=e92]:
- img [ref=e93]
- generic [ref=e96]:
- button "上传" [ref=e97] [cursor=pointer]:
- img [ref=e98]
- generic: 上传
- button "导入 Markdown" [ref=e104] [cursor=pointer]:
- img [ref=e105]
- generic: 导入 Markdown
- button "导出 Markdown" [ref=e109] [cursor=pointer]:
- img [ref=e110]
- generic: 导出 Markdown
- button "禁用 AI" [ref=e114] [cursor=pointer]:
- img [ref=e115]
- generic: 禁用 AI
- generic [ref=e119]:
- img [ref=e120]
- text: 0 KB
- generic [ref=e122]:
- button "模板" [ref=e124] [cursor=pointer]:
- img [ref=e125]
- generic: 模板
- button "清除" [ref=e128] [cursor=pointer]:
- img [ref=e129]
- generic: 清除文档
- generic:
- button "设置" [ref=e5] [cursor=pointer]:
- img [ref=e6]
- generic [ref=e9]:
- generic [ref=e10]:
- heading "设置" [level=2] [ref=e11]
- button "关闭" [ref=e12] [cursor=pointer]:
- img [ref=e13]
- generic [ref=e16]:
- generic [ref=e17]:
- heading "外观" [level=3] [ref=e18]
- generic [ref=e19]:
- generic [ref=e20]: 外观
- combobox [ref=e21]:
- option "深色"
- option "浅色"
- option "跟随系统" [selected]
- option "暖色调"
- option "读书灯"
- option "自定义图片"
- generic [ref=e22]:
- heading "视图" [level=3] [ref=e23]
- generic [ref=e24]:
- button "编辑器" [ref=e25] [cursor=pointer]:
- img [ref=e26]
- generic [ref=e28]: 编辑器
- button "文档" [ref=e29] [cursor=pointer]:
- img [ref=e30]
- generic [ref=e33]: 文档
- generic [ref=e34]:
- heading "PRO 模式" [level=3] [ref=e35]
- generic [ref=e36]:
- generic [ref=e37]: PRO 思考程度
- generic [ref=e38]:
- button "低" [ref=e39] [cursor=pointer]
- button "中" [ref=e40] [cursor=pointer]
- button "高" [ref=e41] [cursor=pointer]
- paragraph [ref=e42]: PRO 模式使用独立思考强度，普通补全设置不会影响它。
- generic [ref=e43]:
- heading "模型智能" [level=3] [ref=e44]
- generic [ref=e45]:
- generic [ref=e46]: 思考程度
- generic [ref=e47]:
- button "低" [ref=e48] [cursor=pointer]
- button "中" [ref=e49] [cursor=pointer]
- button "高" [ref=e50] [cursor=pointer]
- paragraph [ref=e51]: 直接补全（最快）
- generic [ref=e52]:
- generic [ref=e53]: "防抖时间: 1000ms"
- slider [ref=e54]: "1000"
- generic [ref=e55]:
- heading "隐私与偏好" [level=3] [ref=e56]
- generic [ref=e57]:
- generic [ref=e58]: 隐私模式
- button [ref=e59] [cursor=pointer]
- paragraph [ref=e61]: 不向 AI 发送 IP 地址和偏好设置
- generic [ref=e62]:
- generic [ref=e63]: 语言
- combobox [disabled] [ref=e64]:
- option "自动检测" [selected]
- option "Chinese"
- option "English"
- option "Japanese"
- option "Korean"
- option "German"
- option "French"
- generic [ref=e65]:
- generic [ref=e66]: 货币
- combobox [disabled] [ref=e67]:
- option "自动检测" [selected]
- option "CNY (¥)"
- option "USD ($)"
- option "EUR (€)"
- option "JPY (¥)"
- option "KRW (₩)"
- option "GBP (£)"
- option "AUD ($)"
- option "CAD ($)"
- generic [ref=e68]:
- heading "语音设置" [level=3] [ref=e69]
- generic [ref=e70]:
- generic [ref=e71]: 声音描述
- textbox "例如：用温柔的语气说" [ref=e72]
- paragraph [ref=e73]: 描述你想要的声音风格，如语气、情感等
- generic [ref=e74]:
- heading "关于我们" [level=3] [ref=e75]
- generic [ref=e76]:
- heading "llm-in-text" [level=4] [ref=e77]
- paragraph [ref=e78]: A smart Markdown editor with local LLM intelligence.
- paragraph [ref=e79]: v0.0.0
.prompt.md
-129
View File
@@ -1,129 +0,0 @@
# 前端代码简化与优化提示词
## 目标
深入分析并简化前端代码结构，提升执行效率、可维护性和类型安全性。
## 优化原则
### 1. 代码简化
- **消除冗余**：识别并移除重复的逻辑、条件判断和错误处理模式
- **函数拆分**：将大型函数拆分为职责单一的小函数（每个函数只做一件事）
- **提取常量**：将魔法数字、字符串字面量提取为命名常量
- **减少嵌套**：使用早期返回（early return）替代深层 if/else 嵌套
### 2. 类型安全
- **明确类型**：为所有函数参数和返回值添加 TypeScript 类型注解
- **接口定义**：为复杂对象结构定义 interface，避免 `any` 类型
- **联合类型**：使用 discriminated unions 替代运行时 typeof 检查
### 3. 性能优化
- **懒加载**：对非核心模块使用动态 import()
- **防抖节流**：对频繁触发的事件（输入、滚动）添加 debounce/throttle
- **计算缓存**：对纯函数的重复计算结果进行 memoization
- **条件渲染**：使用 v-if/v-show 控制不必要的 DOM 操作
### 4. 错误处理
- **统一错误边界**：集中处理 fetch/API 调用异常
- **有意义错误信息**：避免空 catch，提供具体的失败原因
- **降级策略**：关键功能失败时有优雅的 fallback
### 5. 状态管理
- **最小化状态**：只存储必要的响应式数据
- **派生状态**：使用 computed 替代手动监听 + 条件赋值
- **作用域限制**：将状态定义在尽可能小的组件范围内
## 检查清单
### api.js 优化点
```typescript
// ❌ 问题：过多的条件分支和嵌套
function getCancelUrl(apiUrl) {
const normalized = String(apiUrl || '').replace(/\/+$/, '')
if (/\/v1\/pro\/completions$/i.test(normalized)) { /*...*/ }
if (/\/v1\/web-search$/i.test(normalized)) { /*...*/ }
// ...
}
// ✅ 优化：使用映射表替代条件链
const CANCEL_PATH_MAP = {
'/v1/pro/completions': '/v1/pro/completions/cancel',
'/v1/web-search': '/v1/web-search/cancel',
'/v1/completions': '/v1/completions/cancel',
}
function getCancelUrl(apiUrl) {
const base = new URL(apiUrl).pathname.replace(/\/+$/, '')
return CANCEL_PATH_MAP[base] || `${base}/cancel`
}
```
### 通用模式识别
1. **重复的 fetch 包装**：提取统一的 `safeFetch()` 函数处理认证头、错误和超时
2. **SSE 解析器重复**：将 `parseSseEvent` 抽象为可复用的 stream 处理器
3. **设置状态访问**：避免在每次 API 调用时重新创建 settings store 实例
4. **条件类型检查**：用 TypeScript discriminated union 替代运行时 `typeof x === 'string'`
## 执行步骤
### 第一步：分析
1. 使用 `grep_search` 查找重复模式（相同的 if/else 块、try/catch）
2. 使用 `semantic_search` 查找相似功能的不同实现
3. 识别高频调用的函数（API 请求、事件处理器）
### 第二步：重构
1. **提取纯函数**：将副作用（fetch、DOM操作）与数据处理分离
2. **创建工具库**：将通用逻辑移至 `src/utils/` 下的独立模块
3. **添加类型定义**：在 `src/plugins/types.ts` 中集中管理接口
4. **简化条件逻辑**：用策略模式或映射表替代 switch/if 链
### 第三步：验证
1. 运行 `npm run build` 确认无类型错误
2. 检查 `get_errors` 确保没有引入新问题
3. 手动测试关键路径（补全、OCR、上传）
## 输出格式
每次优化后提供：
```markdown
### 优化项: [函数名/文件名]
**问题**: [简要描述当前代码的问题]
**改动**:
```diff
- // 旧代码
+ // 新代码
```
**收益**:
- 行数减少: X%
- 时间复杂度: O(n) → O(1)
- 可读性提升: [具体说明]
```
## 示例调用
```bash
# 简化 api.js 中的 URL 处理逻辑
"简化 src/utils/api.js 中 getCancelUrl() 函数的条件分支，使用映射表替代正则匹配"
# 优化 copilotPlugin.ts 的类型定义
"为 src/plugins/copilotPlugin.ts 添加完整的 TypeScript 类型注解，消除所有 any 类型"
# 提取重复的错误处理
"将 src/utils/api.js 中分散的 try/catch 错误处理提取为统一的 errorBoundary() 高阶函数"
```
## 注意事项
- ✅ 保持向后兼容：不破坏现有 API 接口和事件流
- ✅ 小步快跑：每次只优化一个函数或模块，验证后再继续
- ❌ 避免过度优化：不要为了炫技引入复杂的函数式编程模式
- ⚠️ 测试覆盖：修改核心路径（补全、取消请求）前确保有对应测试
- 📝 文档同步：更新 CLAUDE.md 和 AGENTS.md 中的架构描述
## 参考文件
- `src/utils/api.js` - API 请求层，存在多处可简化的条件逻辑
- `src/plugins/copilotPlugin.ts` - 补全插件，类型定义不完整
- `src/stores/settings.js` - 状态管理，可优化响应式依赖
- `backend/prompt.py` - Prompt 组装逻辑（后端参考）
AGENTS.md
+44 -188
View File
@@ -1,196 +1,52 @@
# LLM in Text 仓库指引 (v0.2.0)
# 仓库指南
本文件适用于整个仓库。进入更深层目录后，子目录中的 AGENTS.md 优先于本文件。
## 语言约定
项目文档、日志、错误提示以及对外返回的文字信息统一使用 **中文**。前端 UI 默认展示中文，若需多语言支持请在相应模块实现。
## 项目定位
## 项目结构 \& 模块组织
```
backend/ # FastAPI 后端（Python）
├─ main.py # API 入口
├─ llm.py # LLM 包装工具
├─ prompt.py # Prompt 构建辅助
└─ tests/ # pytest 测试套件
public/ # 前端静态资源
src/ # 前端源码（Vite + React）
dist/ # 构建产出（生成文件）
```
生产代码主要位于 `backend/`（Python）和 `src/`（JS/TS）。测试文件与被测模块并置。
- 这是一个智能 Markdown 编辑器，前端负责编辑器 UI、上传导出、补全交互和设置状态，后端负责 LLM、OCR、文件转换和 TTS 接口。
- 前端技术栈：Vue 3 + Vite + Milkdown/Crepe + Pinia + Vue Router。
- 后端技术栈：FastAPI + Python + OpenAI-compatible LLM endpoint + Redis Streams。
- 仓库内另有 `InTEX/` iOS 客户端工程，技术栈为 SwiftUI + SwiftData；当前仍是独立的 Xcode 初始工程，尚未接入 Web/Docker 后端。
- 项目版本：v0.2.0（自 b82c6d3 之后的全栈架构升级版本）。
## 构建、测试、开发命令
| 命令 | 说明 |
|----------------------------------------------|--------------------------------------------------|
| `npm install` | 安装前端依赖 |
| `npm run dev` | 启动 Vite 开发服务器 |
| `uvicorn backend.main:app --reload` | 本地运行 FastAPI 服务 |
| `pytest` | 运行 Python 测试套件 |
| `npm run build` | 生成生产环境构建产物至 `dist/` |
## 功能块系统（核心概念）
## 编码风格 \& 命名约定
- **Python**：使用 4 空格缩进，`snake_case` 命名函数/变量，`PascalCase` 命名类。提交前请使用 `ruff`/`black` 格式化。
- **JavaScript/TypeScript**：使用 2 空格缩进，`camelCase` 命名变量/函数，`PascalCase` 命名 React 组件。使用 `eslint` 与 `prettier` 检查。
- 文件名采用全小写加短横线，例如 `my-module.py`、`my-component.tsx`。
- **统一命名**：文档块、PRO 块、上传块统称为"功能块"。
- **禁止嵌套**：所有功能块的 schema 均设 `atom: true, isolating: true`，ProseMirror 层面强制禁止互相嵌套。DocBlockCrepe.vue 的嵌套 Crepe 编辑器不注册任何功能块插件，从架构上杜绝深层嵌套。
- **无数量限制**：一篇文档可包含任意数量的功能块，彼此独立存在。
- **导入自动解析**：从 Markdown 文件导入后，各功能块语法必须通过对应的 Remark/parseMarkdown 解析器自动识别并还原为交互卡片。
- **导出可复原**：通过 toMarkdown/leafText 序列化器将功能块还原为 Markdown 语法，确保导出后再次导入能完整复原。doc_block toMarkdown 输出 legacy HTML tag，`getExportMarkdown()` 中通过 `transformLegacyDocBlocksForExport()` 转换为 fenced code block。
## 测试指南
- 后端使用 **pytest**，测试文件放在对应模块目录下，命名为 `test_<module>.py`。
- 目标覆盖率 ≥ 80%（`pytest --cov=backend`）。
- 在虚拟环境中运行：`pip install -r backend/requirements.txt && pytest`。
| 类型 | Node Type | Markdown 语法 | Plugin 文件 | Utility 文件 |
|------|-----------|---------------|-------------|--------------|
| 文档块 | `doc_block` | \`\`\`llm-file fenced code / `<doc_type=...>` legacy HTML tag | `plugins/docBlockPlugin.ts` | `utils/docBlock.js` |
| PRO 块 | `pro_block` | `[PRO]` / `[PRO]{指令}` | `plugins/proBlockPlugin.ts` | `utils/proBlock.js` |
| 上传块 | `upload_block` | `{{{}}}` / `{{{upload file type:...}}}` | `plugins/uploadBlockPlugin.ts` | `utils/uploadBlock.js` |
## 提交 \& Pull Request 规范
- 提交信息遵循 **Conventional Commits**：`feat:` 新功能、`fix:` 修复、`docs:` 文档、`refactor:` 重构等。
- PR 必须包含：
- 与提交信息匹配的标题。
- 关联的 Issue（如 `Fixes #123`）。
- UI 变更或 API 示例的截图/示例。
- 所有 CI 检查（代码检查、测试、类型检查）均通过。
- **已验证**：当前代码完全符合"禁止嵌套、自动解析复原"的要求。修改功能块相关逻辑时需验证：1) schema 的 atom/isolating 属性不被移除；2) Remark/parseMarkdown/toMarkdown 解析链路完整。
- 当前代码中可以确认的主功能是：AI 补全、OCR、文档转 Markdown、TTS/ASR、Markdown/DOCX/PDF 导入导出。
- 历史文档中有一部分 TTS/ASR、Apple Silicon、Whisper、离线模式说明已经落后于当前代码；出现冲突时以实际代码和测试为准。
## 安全 \& 配置建议
- 敏感信息请放入 `.env` 并确保已在 `.gitignore` 中。
- 按照 `backend/main.py` 中的实现，对上传文件的大小和类型进行校验，防止滥用。
- 定期审计依赖安全（`npm audit`、`pip-audit`）。
## 先看哪里
- 项目概览和运行说明：README.md
- 前端入口：src/main.js
- 路由：src/router/index.js
- 编辑器主组件：src/components/MilkdownEditor.vue
- AI 补全核心：src/plugins/copilotPlugin.ts、src/plugins/copilotTypes.ts（类型定义）
- 前端请求层：src/utils/api.js、src/utils/fetch.js（安全 fetch wrapper）
- 前端配置：src/utils/config.js
- 设置状态：src/stores/settings.js
- 后端入口和主路由：backend/main.py
- LLM 和 OCR 调用：backend/llm.py
- Prompt 组装：backend/prompt.py（含 PRO 模式模板）
- TTS 路由：backend/tts_asr.py
- **任务队列系统**：backend/job_system.py（Redis Streams 异步任务管理）
- **Worker 进程入口**：backend/worker.py（消费 Redis Streams 的独立 worker）
- **任务处理器注册表**：backend/job_handlers.py（completion/PRO/web_search/compress/OCR/convert/TTS/ASR 处理器）
- **会话管理**：backend/session_store.py（内存 + PostgreSQL 双后端）
- **API/LLM 审计日志**：backend/audit_store.py（PostgreSQL 持久化）
- **风控配置**：backend/risk_config.py（环境变量驱动的配置数据类）
- **风控引擎**：backend/risk_control.py（速率限制、并发控制、熔断器、预算追踪）
- **验证码路由**：backend/captcha_api.py（FastAPI router）
- **文档存储**：backend/docs_store.py（MIME 类型检测、文本/二进制分类）
- **LLM 策略解析**：backend/llm_policy.py（按 job_type 解析模型配置）
- **网页搜索块组件**：src/components/WebSearchBlockCrepe.vue（可折叠/压缩/删除的搜索结果卡片）
- **网页搜索块插件**：src/plugins/webSearchBlockPlugin.ts（```llm-websearch fenced code 语法）
- **OCR 图片包装器**：src/components/OCRImageWrapper.vue（加载/成功/失败状态覆盖层）
- **验证码组件**：src/components/CaptchaComponent.vue（vue3-captcha 封装）
- **SSE 流式解析**：src/utils/sse.ts（Server-Sent Events 事件解析）
- **文档管理 API**：src/utils/docsApi.js（上传/预览/删除接口客户端）
- **PRO 功能接受追踪**：src/utils/proAccept.js（使用分析和统计）
- **Cookie 策略管理**：src/utils/cookie_policy.js（SameSite/Secure 标志处理）
- **字符串工具**：src/utils/string.ts（长度计算、编码检测）
- **网页搜索上下文提取**：src/utils/webSearch.js（搜索结果 Markdown 构建与解析）
- **测试配置和入口**：pytest.ini、backend/tests/run_tests.py
- **iOS 客户端工程**：InTEX/InTEX.xcodeproj（scheme: `InTEX`）
- **iOS 客户端入口**：InTEX/InTEX/InTEXApp.swift、InTEX/InTEX/ContentView.swift
## 稳定事实
- **功能块禁止嵌套**：`doc_block`、`pro_block`、`upload_block` 的 schema 均设 `atom: true, isolating: true`，ProseMirror 层面强制禁止互相嵌套。DocBlockCrepe.vue 的嵌套 Crepe 编辑器仅注册 copilotPlugin + hiddenText*，不注册任何功能块插件。修改时不得移除 atom/isolating 属性或在嵌套编辑器中引入功能块插件。
- **功能块导入解析**：各功能块的 Remark 插件（`docBlockRemark`, `proBlockRemark`, `uploadBlockRemark`）负责从 Markdown AST 识别对应语法并转换为节点。doc_block Remark 同时支持 fenced code (`llm-file`) 和 legacy HTML tag (`<doc_type=...>`)。解析链路必须保持完整，否则导入后无法自动复原。
- **功能块导出序列化**：doc_block 的 `toMarkdown` runner 输出 legacy HTML tag，但 `getExportMarkdown()` 中通过 `transformLegacyDocBlocksForExport()` 转换为 fenced code block。pro_block/upload_block 的 toMarkdown/leafText 直接输出标准语法。修改时需验证导出后再次导入能完整复原。
- **Web Search 块语法**：web_search_block 使用 fenced code ```llm-websearch date=... 语法，与 doc_block(llm-file)、pro_block([PRO])、upload_block({{{}}}) 并列。触发语法为 [WEBSEARCH]（不区分大小写），由 `webSearchBlockPlugin.ts` 处理。
- **Store 同步格式**：`scheduleMarkdownSync()` emit markdown 不做转换（doc_block 为 legacy HTML tag），store 中始终是 legacy format。`syncInitialMarkdown()` / `getExportMarkdown()` 负责格式转换。
- **上传块生命周期**：upload_block 是临时占位符，文件上传后被替换为 image node（图片）或 doc_block（文档），不会与 pro_block 共存。
- **PRO block escape/unescape**：`escapeProBlockContent()` / `unescapeProBlockSyntax()` 处理 `\`, `]`, `}`, newline，确保指令 round-trip 正确。
- **补全接口当前不是 SSE**；前端用普通 POST 请求拿 JSON 响应。
- **任务队列架构（v0.2.0 新增）**：后端从同步端点转向 Redis Streams 异步队列。`job_system.py` 定义 JOB_TYPES（completion/pro_completion/web_search/compress/ocr/convert/tts/asr），`worker.py` 消费队列，`job_handlers.py` 注册各类型处理器。所有任务支持并发控制、速率限制和熔断器模式。
- **会话追踪**：`session_store.py` 提供 InMemorySessionStore（开发）和 PostgresSessionStore（生产），通过 session_hash + ip_hash 追踪请求身份。
- **风控系统**：`risk_config.py` + `risk_control.py` 实现速率限制（滑动窗口）、并发控制、熔断器模式和预算追踪，所有阈值通过环境变量配置。
- 前端会生成 X-Request-Id，并在请求被中止时额外调用 /v1/completions/cancel。
- 文档超过 32 KB 时，AI 补全会在前端和插件层被禁用。
- OCR 文本和文档块内容会被注入补全上下文，但这些内容属于隐藏上下文，不应被直接当作用户可见文本重复输出。
- /v1/convert 当前支持 txt、docx、pptx、pdf，非 txt 文件通过 MarkItDown 转成 Markdown，之后会清理图片标记。
- **AI 开关是全局广播状态**：MilkdownEditor.vue 通过 `llm-in-text:copilot-toggle` 同步主编辑器、文档块嵌套编辑器、网页搜索块嵌套编辑器；修 ghost text 时要同时检查这三处。
- **设置项统一使用 country**：前后端请求、prompt、store、设置面板只使用 `country`。
- **DOCX/PDF 导出改为纯前端**：不再依赖 `/v1/export/pdf`。当前策略是先展开所有功能块，再从编辑器 HTML 构建导出内容；`src/utils/richExport.js` 负责 HTML -> PDF / DOCX。
- **上传单文件限制统一为 100MB**：前端校验和后端 OCR 风控上限都按 100MB 处理。
- **视频解析策略**：上传视频时，后端 `/v1/ocr` 接收 `media_type=video`，视频画面走 OCR 模型，音轨通过 ffmpeg 抽取后走 ASR 模型，最终合并为“视频画面 OCR + 视频音频 ASR”文本。
- **OCR 明确关闭思考**：backend/llm.py 的 OCR payload 显式下发 `options.think = False` 与 `temperature = 0`。
- **LLM/PRO/OCR 模型统一**：`LLM_MODEL`、`PRO_LLM_MODEL`、`VLM_MODEL` 的当前默认值和 Docker 运行值统一为 `Nex-N2-mini-mlx-OptiQ-8bit-MTP`。网页搜索模型仍可通过 `RISK_WEB_SEARCH_MODEL` 独立配置。
- **OCR 完成状态不等于识别正确**：Redis 任务、审计入库和 SSE `done` 只能证明链路完成。2026-06-27 的清晰文本图片真实测试中，Nex OCR 仍可能返回未识别文本；用户已明确暂停继续修复 OCR 精度，不要把链路完成误报为 OCR 正常。
- **审计 SQL 参数必须严格对齐**：`llm_call_audit` 当前包含 `queue_ms`、`run_ms`、`total_ms`，修改 INSERT 时必须保持列数、`%s` 占位符数和参数数一致；`backend/tests/test_audit_store.py` 覆盖此回归。
- **TTS/ASR 当前真实实现**：backend/tts_asr.py 统一通过 `LLM_BASE_URL` + `LLM_API_KEY` 调用 OpenAI-compatible Speech API，默认模型为 `Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit` 与 `Qwen3-ASR-0.6B-8bit`。VoiceDesign 请求字段必须使用复数 `instructions`；空声音描述必须回退到 `TTS_DEFAULT_INSTRUCTIONS`，不能向上游发送空指令。
- **六按钮收缩菜单**：MilkdownEditor.vue 的 upload/import/export/AI/template/clear 六个动作统一收进右下角 `.more-actions`。视口宽度 `>= 520px` 横向展开，窄屏纵向展开；再次点击主按钮、再次点击已打开的 export/template、点击 backdrop 或按 Escape 都必须完整收缩回右下角省略号。
- **iOS 工程边界**：`InTEX/` 当前是 SwiftUI + SwiftData 示例骨架，包含 `InTEX`、`InTEXTests`、`InTEXUITests` 三个 target。不要声称它已接入编辑器或 Docker API；提交时禁止加入 `xcuserdata`、`*.xcuserstate` 和 DerivedData。
## 常用命令
- 前端安装：npm install
- 前端开发：npm run dev
- 前端构建：npm run build
- 后端安装：pip install -r backend/requirements.txt
- 后端启动：python backend/main.py
- 可选启动方式：uvicorn backend.main:app --reload --port 8001
- iOS 工程检查：xcodebuild -project InTEX/InTEX.xcodeproj -list
- iOS 模拟器构建：优先使用 XcodeBuildMCP；命令行回退可用 `xcodebuild -project InTEX/InTEX.xcodeproj -scheme InTEX -sdk iphonesimulator CODE_SIGNING_ALLOWED=NO build`
- 全量测试：pytest
- 常用窄测试：
- pytest backend/tests/test_main_endpoints.py -v
- pytest backend/tests/test_main_cancel.py -v
- pytest backend/tests/test_prompt.py -v
- pytest backend/tests/test_llm.py -v
## Docker 部署约定
- 本机部署目录固定在 `/Users/allenyuan/lit/` 下，不在仓库外再散落数据库或 Docker 持久化目录。
- 当前推荐的部署工作目录是 `/Users/allenyuan/lit/llm-in-text/`；把仓库同步到该目录后，从该目录执行 `docker compose up -d --build`。
- 不再使用 `/Volumes/New Volume/lit/` 部署本项目；迁移时可放弃旧 PostgreSQL 数据，从新部署目录初始化空数据库。
- **docker-compose.yml 定义的服务**：api（FastAPI）、worker（任务消费者）、frontend（Nginx）、postgres（审计+会话存储）、redis（任务队列）、searxng（网页搜索后端）、firecrawl（网页内容抓取）。每次 `docker compose up --build` 后，必须验证所有目标服务（api/worker/frontend/postgres/redis）是否处于 Up 状态。
- **前端网络硬约定**：前端 API 必须调用 `https://api.imageteach.tech:8002/` 反向代理，不要让前端调用 Docker 内 `api` 服务、本机 `localhost:8001`、`localhost:8081` 或同源 `/v1` 代理。网络和反向代理由外部配置处理，除非用户明确要求，不要新增或恢复前端到 Docker 后端的代理。
- 每次修改会影响 Docker 运行效果的代码后，不能只停留在本地测试；必须同步更新当前 Docker 环境中的代码，并验证容器内代码已经变化。
- 首选更新方式：
1. 在仓库根目录执行 `docker compose up -d --build`。
2. 执行 `docker compose ps` 确认 `api`、`worker`、`frontend` 等目标服务已重新创建并处于 Up。
3. 对关键修复点执行容器内验证，例如 `docker compose exec -T worker sh -lc "python - <<'PY'\nfrom pathlib import Path\nprint('_normalize_preferences' in Path('/app/backend/prompt.py').read_text())\nPY"`。
- 如果 `docker compose build` 因 Docker Hub、镜像源、网络 token 超时等外部原因无法拉基础镜像，仍然必须更新正在运行的 Docker 环境。可用应急方式：
1. 先执行 `npm run build` 生成最新前端产物。
2. 用 `docker cp` 将改动后的后端文件复制到 `api` 和 `worker` 容器的 `/app/backend/`，必要时将 `dist/` 复制到 `frontend` 容器的 `/usr/share/nginx/html/`。
3. 执行 `docker compose restart api worker frontend` 重启受影响服务。
4. 执行 `docker commit llm-in-text-api-1 llm-in-text-api:latest`、`docker commit llm-in-text-worker-1 llm-in-text-worker:latest`；如果更新了前端，也执行 `docker commit llm-in-text-frontend-1 llm-in-text-frontend:latest`。
5. 执行 `docker compose up -d --no-build --force-recreate api worker frontend`，确保新容器来自已更新镜像。
6. 再次用 `docker compose exec -T ...` 验证容器内文件和行为，不能只看本地文件。
- Docker 持久化数据统一落在部署目录内的 `docker-data/`，包括 PostgreSQL、Redis 和任务共享临时目录。
- Docker BuildKit 导出缓存统一落在 `docker-data/build-cache/{frontend,api,worker}`。`docker-compose.yml` 的 `cache_from/cache_to` 与 Dockerfile 的 apt/pip/npm cache mount 必须保留；首次构建填充缓存，后续相同源码构建应全部命中缓存。
- `.dockerignore` 必须排除 `backend/models`、`backend/.env`、`docker-data`、`node_modules`、`.git` 等大目录或敏感文件。`backend/models` 是约 4.9GB 的旧本地模型目录，当前 API-based TTS/ASR 不应把它发送进构建上下文。
- 容器内访问宿主机模型服务时，不要继续使用 `localhost`；应改成 `host.docker.internal` 之类的容器可达地址。
- 当前 Docker 部署的 `backend/requirements.docker.txt` 已包含 OCR、转换、队列和基础 API 依赖；`backend/Dockerfile` 额外安装 `ffmpeg` 以支持视频拆音轨。
- **Worker 容器**：worker.py 作为独立服务运行，通过 Redis Streams 消费任务队列。修改 job_handlers.py 或 worker.py 后需要验证 worker 容器内的代码已更新，可通过 `docker compose exec -T worker sh -lc "python -c 'from backend.job_handlers import get_handler; print(get_handler(\"completion\").__name__)'"` 验证。
- **Redis Streams 架构**：任务队列使用 Redis Streams，支持并发控制、速率限制和熔断器。job_system.py 定义 JOB_TYPES 和队列配置，worker.py 注册处理器并运行事件循环。
- 修改 Docker 相关文件时，除了代码本身，还要同步检查：
- `docker-compose.yml`
- `backend/Dockerfile`
- `backend/requirements.docker.txt`
- `Dockerfile.frontend`
- `docker/nginx.conf`
- `backend/.env.example` 与实际部署用 `backend/.env`
## 代码约定
- 不要把整个仓库当成“全小写+短横线命名”项目。当前实际情况是：
- Vue 组件和视图多为 PascalCase
- 前端工具模块多为小写 .js
- 插件层使用 TypeScript
- Python 使用 snake_case
- 以就地风格为准，不要顺手做全仓格式统一。
- UI 文案和代理回复默认使用中文。
- 不要修改 milkdown-docs/，它是只读参考资料。
- 不要新增硬编码密钥、空 catch/except、as any、@ts-ignore 之类的扩散式技术债。
- 代理在这个仓库里应优先做局部、可验证的修改，不要做无关重构。
## 调试路径
- 补全问题：
src/components/MilkdownEditor.vue
-> src/plugins/copilotPlugin.ts
-> src/utils/api.js
-> backend/main.py
-> backend/prompt.py / backend/llm.py
- OCR 问题：
src/components/MilkdownEditor.vue
-> backend/main.py
-> backend/llm.py
- 文档转换问题：
src/utils/convert.js
-> backend/main.py
- TTS 问题：
src/components/TTSMenu.vue / src/components/TTSPlayer.vue / src/components/MilkdownEditor.vue
-> src/utils/api.js
-> backend/tts_asr.py
## 测试和产物
- pytest.ini 对 backend.main、backend.llm、backend.prompt、backend.geoip、backend.prompts、backend.tts_asr 设了覆盖率门槛，低于 90% 会失败。
- 默认测试目录是 backend/tests。
- 常见生成产物包括 dist、htmlcov、.pytest_cache、api_performance_report.md；它们不是源代码。
## 文档注意事项
- README.md 对产品功能有参考价值，但其中补全、TTS/ASR 和部分接口说明已经比代码旧。
- backend/TTS_ASR_MACOS_FIX.md 和 backend/tests/TESTING_GUIDE.md 更适合作为历史背景，不应在与代码冲突时被当成事实来源。
- 修改行为时，优先参考实现代码和对应测试，再决定是否同步普通文档。
---
以上指南旨在保持贡献一致性并维护代码库健康，欢迎通过 Pull Request 提出改进。
Dockerfile.frontend
-21
View File
@@ -1,21 +0,0 @@
ARG DOCKER_REGISTRY_PREFIX=
FROM ${DOCKER_REGISTRY_PREFIX}node:22-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
npm ci --prefer-offline --no-audit
COPY index.html vite.config.js ./
COPY public ./public
COPY src ./src
RUN npm run build
FROM ${DOCKER_REGISTRY_PREFIX}nginx:1.27-alpine
COPY docker/nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=build /app/dist /usr/share/nginx/html
EXPOSE 80
FIX_LIST_ANONYMOUS_PRODUCTION.md
+814
View File
@@ -0,0 +1,814 @@
# llm-in-text 修复清单（匿名可用版）
## 说明
这不是审计报告。
这份文档只回答三件事：
1. 现在具体哪里有问题
2. 问题为什么会发生
3. 应该怎么改
前提按你的要求处理：
- 网站是匿名可用的
- 不做用户登录
- 不做用户身份体系
- 但仍然要防止接口被滥用、站点被刷爆、服务被恶意调用
匿名可用不等于完全不做保护。
对于这种网站，正确做法通常是：
- 不做用户登录
- 不在前端放任何真正的服务端秘密
- 用服务端限流、来源限制、请求大小限制、网关策略保护接口
- 必要时用站点级防刷手段，而不是用户级登录
---
## 1. 前端硬编码了服务端 API Key
### 具体问题
- [src/utils/api.js:4](/C:/Users/ydy/Desktop/llm-in-text/src/utils/api.js#L4)
- [src/utils/convert.js:3](/C:/Users/ydy/Desktop/llm-in-text/src/utils/convert.js#L3)
代码里把：
```js
const API_KEY = 'your-secret-key-here'
```
直接写进了前端源码。
### 错误原因
前端代码最终会发到浏览器里。
只要用户能打开网站，就一定能在浏览器开发者工具、打包产物、网络请求里看到这个 key。
所以前端里的“密钥”根本不是密钥，只是公开字符串。
### 会导致什么
- 任何人都可以绕过你的网站，直接写脚本刷你的后端
- 这个 key 一旦被复制，就等于后端公开可调用
### 整改方式
你的场景不做登录，所以最简单、正确的做法是：
1. 删除前端里的 `API_KEY`
2. 后端不要再要求前端传固定共享 key
3. 改成下面这套匿名保护方案：
- 只允许来自你站点域名的浏览器请求
- 网关层限流
- 接口级限流
- 请求体大小限制
- 必要时加站点级验证码或 challenge，而不是登录
### 你应该改成什么
- `src/utils/api.js` 不再发 `X-API-Key`
- `src/utils/convert.js` 不再发 `X-API-Key`
- `backend/main.py` 删除固定 `API_KEY` 和对应校验逻辑
### 验收标准
- 全仓库搜不到 `your-secret-key-here`
- 前端请求头中不再包含固定共享 key
---
## 2. 后端 CORS 过宽
### 具体问题
- [backend/main.py:34](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L34)
- [backend/main.py:35](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L35)
- [backend/main.py:36](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L36)
- [backend/main.py:37](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L37)
现在配置是：
- `allow_origins=["*"]`
- `allow_credentials=True`
- `allow_methods=["*"]`
- `allow_headers=["*"...]`
### 错误原因
这是开发期常见的“先全开让它跑起来”的写法。
但生产里这样做会让任何站点都能更容易发起跨域调用。
### 会导致什么
- 其他网站更容易借你的浏览器接口能力
- 以后一旦加 cookie、session、任何凭据，会立刻放大风险
### 整改方式
既然你是匿名站点，不做登录，那就更应该把跨域收紧：
1. 只允许你的正式域名和本地开发域名
2. 不要开 `allow_credentials=True`，匿名站一般不需要
3. 只开放需要的方法和头
### 建议改法
把：
```python
allow_origins=["*"]
allow_credentials=True
allow_methods=["*"]
allow_headers=["*", "X-API-Key", "X-Client-IP", "X-Request-Id"]
```
改成类似：
```python
allow_origins=[
"https://your-domain.com",
"https://www.your-domain.com",
"http://localhost:5173",
]
allow_credentials=False
allow_methods=["POST", "OPTIONS"]
allow_headers=["Content-Type", "X-Request-Id"]
```
### 验收标准
- 非你自己域名的网页无法直接跨域调用你的接口
- 不再开放无用头和无用方法
---
## 3. 默认会去拿用户公网 IP，并发送给后端
### 具体问题
- [src/stores/settings.js:16](/C:/Users/ydy/Desktop/llm-in-text/src/stores/settings.js#L16)
- [src/utils/api.js:54](/C:/Users/ydy/Desktop/llm-in-text/src/utils/api.js#L54)
- [src/utils/api.js:100](/C:/Users/ydy/Desktop/llm-in-text/src/utils/api.js#L100)
- [backend/main.py:110](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L110)
流程是：
1. 前端默认 `privacyMode = false`
2. 前端请求 `https://api.ipify.org?format=json`
3. 获取公网 IP
4. 放进 `X-Client-IP`
5. 后端再做地理位置推断
### 错误原因
这是把“个性化上下文”做成了默认行为。
但对匿名站点来说，这不是必要信息。
### 会导致什么
- 页面会额外访问第三方服务
- 用户 IP 会进入你的请求链路
- 模型上下文中会混入地理位置信息
### 整改方式
如果你不需要真正的按地理位置个性化，就最简单：
1. 删除 `getClientIP()`
2. 删除调用 `api.ipify.org`
3. 删除 `X-Client-IP`
4. 后端删除 GeoIP 逻辑
5. `privacyMode` 可以保留，但默认应是更安全的行为
### 你应该删什么
- `src/utils/api.js` 中的 `getClientIP`
- `headers['X-Client-IP'] = clientIP`
- `backend/main.py` 中 `get_client_ip`
- `location = get_ip_location_text(client_ip)`
- `geoip.py` 如果以后不用可以移除
### 验收标准
- 前端网络面板中不再出现 `api.ipify.org`
- 后端不再接收 `X-Client-IP`
- prompt 不再包含用户位置
---
## 4. 后端把内部异常原样返回给前端
### 具体问题
- [backend/main.py:184](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L184)
- [backend/main.py:251](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L251)
- [backend/main.py:303](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L303)
现在写法是：
```python
return JSONResponse(content={"error": str(e)}, status_code=500)
```
### 错误原因
这是开发期为了调试方便常见的写法。
但线上不应该把真实异常直接发给浏览器。
### 会导致什么
- 暴露内部实现细节
- 暴露依赖报错、路径、上游信息
### 整改方式
统一改成：
1. 前端只收到固定错误码和通用提示
2. 后端日志里保留详细异常
3. 返回 request id 方便排查
### 建议响应格式
```json
{
"error": {
"code": "UPSTREAM_TIMEOUT",
"message": "Service temporarily unavailable",
"request_id": "xxxx"
}
}
```
### 验收标准
- 前端不再收到 Python 原始报错
- 日志可通过 request id 查到真实错误
---
## 5. `/v1/convert` 和 `/v1/ocr` 没有文件安全边界
### 具体问题
- [backend/main.py:239](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L239)
- [backend/main.py:268](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L268)
- [backend/main.py:275](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L275)
- [backend/main.py:281](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L281)
- [src/utils/convert.js:22](/C:/Users/ydy/Desktop/llm-in-text/src/utils/convert.js#L22)
现在的问题是：
- 直接 base64 解码
- 没有严格文件大小限制
- 没有严格文件类型白名单
- 没有魔数校验
- 没有超时和并发保护
### 错误原因
当前实现是功能优先，默认相信前端传来的内容。
但上传链路是最容易出问题的地方之一。
### 会导致什么
- 大文件压垮内存
- 恶意文件拖慢 CPU
- 第三方库处理异常文件时出故障
### 整改方式
#### 对 `ocr`
1. 图片大小上限先改为 5MB 或 10MB
2. 只允许 `jpg/png/webp`
3. 服务端校验 MIME 和魔数
4. 增加请求超时
5. 增加并发限制
#### 对 `convert`
1. 只允许明确白名单格式
2. 每种格式单独设大小上限
3. 服务端检查扩展名和文件头
4. `markitdown` 执行增加超时
5. 临时文件放到独立目录
6. 临时文件异常时也要清理
### 建议白名单
- `.pdf`
- `.docx`
- `.pptx`
- `.xlsx`
- `.md`
- `.txt`
### 建议直接拒绝
- 可执行文件
- 压缩包
- 未知二进制
- 超大图片
### 验收标准
- 超限文件返回 413
- 非法类型返回 415
- OCR/convert 高并发下不会拖垮服务
---
## 6. 没有限流，匿名站点很容易被刷
### 具体问题
- 当前代码里没有 rate limit
- 没有按 IP、UA、路径、时间窗做限制
- `ACTIVE_COMPLETIONS` 只处理取消，不是限流器。[backend/main.py:29](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py#L29)
### 错误原因
因为现在的代码默认是“正常用户正常使用”。
但匿名公网站点上线后，必须假设会被脚本反复调用。
### 会导致什么
- 模型成本失控
- CPU、内存、连接数被耗尽
- 服务变慢甚至不可用
### 整改方式
匿名站点不做登录，标准保护方式是限流：
1. 反向代理层限流
2. 应用层限流
3. 高成本接口单独限流
### 建议策略
#### `/v1/completions`
- 单 IP 每分钟 20 到 60 次
- 同时进行中的请求数限制 2 到 4 个
#### `/v1/ocr`
- 单 IP 每分钟 5 到 10 次
- 同时进行中的 OCR 限制更低
#### `/v1/convert`
- 单 IP 每分钟 3 到 5 次
- 强并发限制 1 到 2
### 还可以加什么
- Cloudflare Turnstile / hCaptcha 这类站点级防刷
- 对明显机器人流量加 challenge
这不需要登录，也不需要用户体系。
### 验收标准
- 连续脚本请求会命中 429
- 单个来源无法无限刷接口
---
## 7. 模型调用没有明确超时与失败策略
### 具体问题
- [backend/llm.py:15](/C:/Users/ydy/Desktop/llm-in-text/backend/llm.py#L15)
当前 `ollama.AsyncClient` 调用没有明显的统一超时和失败分类。
### 错误原因
开发阶段一般默认上游会正常返回。
但生产里，上游模型服务经常会出现：
- 变慢
- 卡住
- 连接失败
- 超时
### 会导致什么
- 请求挂很久
- 连接被占住
- 用户看起来像页面没响应
### 整改方式
1. completions 设置明确超时，例如 15 到 30 秒
2. OCR 设置更短或更明确的处理时限
3. convert 设置文件转换超时
4. 把错误分成：
- timeout
- unavailable
- bad response
5. 前端对这些错误做不同提示
### 验收标准
- 上游模型挂掉时，接口会快速失败而不是一直卡住
---
## 8. 缺少健康检查接口
### 具体问题
当前没有明确的：
- `/health/live`
- `/health/ready`
### 错误原因
项目还是开发态，没有进入正式部署思路。
### 会导致什么
- 你很难判断服务是否真的可用
- 容器/进程平台无法正确探活
### 整改方式
增加两个接口：
#### `/health/live`
只表示“应用进程活着”
#### `/health/ready`
表示“应用准备好服务请求”
这个接口至少检查：
- 模型上游是否可连接
- 关键配置是否存在
### 验收标准
- 反向代理或容器平台能用它判断是否接流量
---
## 9. 预览组件有 XSS 风险
### 具体问题
- [src/components/MarkdownPreview.vue:2](/C:/Users/ydy/Desktop/llm-in-text/src/components/MarkdownPreview.vue#L2)
- [src/components/MarkdownPreview.vue:19](/C:/Users/ydy/Desktop/llm-in-text/src/components/MarkdownPreview.vue#L19)
现在做法是：
- `v-html`
- `html: true`
### 错误原因
这意味着 markdown 里的原始 HTML 会被直接渲染。
如果内容来源不完全可信，这就是典型 XSS 入口。
### 会导致什么
- 恶意脚本执行
- 页面被注入恶意 DOM
### 整改方式
你有两个选择：
#### 方案 A：最简单，直接关掉 HTML
把：
```js
html: true
```
改成：
```js
html: false
```
#### 方案 B：保留 HTML，但做净化
1. 引入 DOMPurify
2. `md.render()` 后先 sanitize
3. 再给 `v-html`
### 推荐
如果你不是必须支持原始 HTML，直接用方案 A。
### 验收标准
- 恶意 markdown/HTML 不会执行脚本
---
## 10. 前端默认配置会误连固定服务地址
### 具体问题
- [src/utils/config.js:1](/C:/Users/ydy/Desktop/llm-in-text/src/utils/config.js#L1)
- [.env.example:1](/C:/Users/ydy/Desktop/llm-in-text/.env.example#L1)
- [backend/llm.py:12](/C:/Users/ydy/Desktop/llm-in-text/backend/llm.py#L12)
默认值里有固定公网域名和固定内网 IP。
### 错误原因
这是把“某次部署环境”写成了“代码默认值”。
### 会导致什么
- 本地开发可能误连生产或旧环境
- 新服务器部署时容易配错
### 整改方式
1. 默认值改成本地开发地址或空值
2. 关键配置不存在时直接报错
3. `.env.example` 只放模板，不放真实地址
### 建议
- `VITE_API_BASE_URL` 默认走同域，如 `''`
- 前端优先使用 `/v1/...` 反代
- 后端 `OLLAMA_HOST` 必须来自环境变量
### 验收标准
- 不配置环境变量时，不会误连旧服务
---
## 11. 包版本和界面版本不一致
### 具体问题
- [package.json:4](/C:/Users/ydy/Desktop/llm-in-text/package.json#L4) 是 `0.0.0`
- [src/components/SettingsPanel.vue:275](/C:/Users/ydy/Desktop/llm-in-text/src/components/SettingsPanel.vue#L275) 写的是 `v0.1.0-beta`
### 错误原因
一个是包元数据，一个是手写展示文案，没人保证同步。
### 会导致什么
- 发布后你都不确定线上到底是哪版
### 整改方式
1. 统一从 `package.json` 注入版本
2. 前端不要手写版本号
### 验收标准
- 页面显示版本和构建版本完全一致
---
## 12. `package.json` 缺少质量脚本
### 具体问题
- [package.json:6](/C:/Users/ydy/Desktop/llm-in-text/package.json#L6)
当前只有：
- `dev`
- `build`
- `preview`
没有：
- `test`
- `lint`
- `check`
### 错误原因
项目还停留在“能运行”的阶段，没有建立质量门禁。
### 会导致什么
- 任何改动都只能靠手工试
- 回归问题容易漏
### 整改方式
至少补这些脚本：
```json
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview",
"test": "pytest backend/tests -q",
"lint": "eslint src",
"check": "npm run lint && npm run build && pytest backend/tests -q"
}
```
如果前端暂时没配 ESLint，也至少先把 `test` 和 `check` 建起来。
### 验收标准
- 以后每次改代码前后都能统一执行 `check`
---
## 13. 测试覆盖不够，缺关键路径
### 具体问题
当前测试主要是：
- prompt 构造
- 取消逻辑
- LLM 消息结构
缺少：
- 匿名访问基本流程
- 错误响应格式
- OCR 文件限制
- convert 文件限制
- 限流
- XSS
### 错误原因
现有测试更偏功能开发时的局部验证，不是上线前测试矩阵。
### 整改方式
补这些测试：
1. completions 正常返回
2. completions 上游超时
3. completions 请求超长
4. OCR 非法类型
5. OCR 超大图片
6. convert 非法类型
7. convert 超大文件
8. 限流命中
9. 未授权方案移除后，匿名访问可正常工作
10. markdown 预览 XSS 样例
### 验收标准
- 关键错误分支都有自动化测试
---
## 14. Service Worker 缓存策略还不够稳
### 具体问题
- [src/main.js:13](/C:/Users/ydy/Desktop/llm-in-text/src/main.js#L13)
- [public/sw.js:1](/C:/Users/ydy/Desktop/llm-in-text/public/sw.js#L1)
当前是手写缓存逻辑，版本固定写死。
### 错误原因
这是一个能用的基础实现，但不适合长期生产维护。
### 会导致什么
- 更新后可能缓存混乱
- 老版本资源残留
### 整改方式
如果你不强依赖离线能力：
1. 先临时关闭 SW
2. 等核心功能稳定后再重做 PWA
如果要保留：
1. 用成熟方案接管，如 Vite PWA / Workbox
2. 资源按 hash 控制
3. 做更新提示
### 推荐
如果现在重点是先上线稳定版，先停掉 SW 更省事。
### 验收标准
- 用户刷新后不会出现随机旧资源
---
## 15. 构建体积偏大
### 具体问题
本次构建已经出现大 chunk 警告，尤其是 Mermaid 相关包比较重。
### 错误原因
图表、编辑器、语法高亮、数学渲染这类库本身就大。
现在又没有足够按功能懒加载。
### 会导致什么
- 首屏慢
- 弱网体验差
### 整改方式
1. Mermaid 按需加载
2. 预览按需加载
3. OCR / convert 相关 UI 按需加载
4. 收敛 `manualChunks`
### 验收标准
- 首页首次加载明显更轻
---
## 16. 匿名站点应该怎么做保护，而不是登录
这是你这个项目最关键的方向问题。
你不想做用户级网站，这完全可以。
那就按匿名站点的标准做：
### 必做
1. 去掉前端共享密钥
2. 收紧 CORS
3. 加 Nginx / Cloudflare / 网关限流
4. 应用层再做限流
5. 限制请求体大小
6. 限制 OCR/convert 并发
7. 错误信息脱敏
8. 加健康检查
9. 处理 XSS
### 可选
1. Cloudflare Turnstile
2. 简单的人机验证 challenge
3. 对高频匿名流量启用冷却时间
### 不必做
1. 登录
2. 注册
3. 用户系统
4. JWT
只要你的目标是匿名工具站，而不是多租户平台，上面这套就够了。
---
## 最简修复顺序
如果你要最低成本把项目拉到“能较安全公开上线”的程度，建议顺序是：
1. 删除前端 API key 和后端固定 key 校验
2. 删除 IP 获取和地理位置推断
3. 收紧 CORS
4. 统一错误响应
5. 给 OCR/convert 加大小、类型、超时限制
6. 加限流
7. 修掉 `MarkdownPreview` 的 XSS 风险
8. 增加 `/health/live` 和 `/health/ready`
9. 补 `test` / `check` 脚本
10. 视情况先关闭 service worker
---
## 这份清单对应的文件
- [backend/main.py](/C:/Users/ydy/Desktop/llm-in-text/backend/main.py)
- [backend/llm.py](/C:/Users/ydy/Desktop/llm-in-text/backend/llm.py)
- [src/utils/api.js](/C:/Users/ydy/Desktop/llm-in-text/src/utils/api.js)
- [src/utils/convert.js](/C:/Users/ydy/Desktop/llm-in-text/src/utils/convert.js)
- [src/components/MarkdownPreview.vue](/C:/Users/ydy/Desktop/llm-in-text/src/components/MarkdownPreview.vue)
- [src/stores/settings.js](/C:/Users/ydy/Desktop/llm-in-text/src/stores/settings.js)
- [src/components/SettingsPanel.vue](/C:/Users/ydy/Desktop/llm-in-text/src/components/SettingsPanel.vue)
- [public/sw.js](/C:/Users/ydy/Desktop/llm-in-text/public/sw.js)
- [package.json](/C:/Users/ydy/Desktop/llm-in-text/package.json)
InTEX/InTEX.xcodeproj/project.pbxproj
-592
View File
@@ -1,592 +0,0 @@
// !$*UTF8*$!
{
archiveVersion = 1;
classes = {
};
objectVersion = 77;
objects = {
/* Begin PBXContainerItemProxy section */
A75D91532FF0146B00EE6196 /* PBXContainerItemProxy */ = {
isa = PBXContainerItemProxy;
containerPortal = A75D91392FF0146A00EE6196 /* Project object */;
proxyType = 1;
remoteGlobalIDString = A75D91402FF0146A00EE6196;
remoteInfo = InTEX;
};
A75D915D2FF0146B00EE6196 /* PBXContainerItemProxy */ = {
isa = PBXContainerItemProxy;
containerPortal = A75D91392FF0146A00EE6196 /* Project object */;
proxyType = 1;
remoteGlobalIDString = A75D91402FF0146A00EE6196;
remoteInfo = InTEX;
};
/* End PBXContainerItemProxy section */
/* Begin PBXFileReference section */
A75D91412FF0146A00EE6196 /* InTEX.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = InTEX.app; sourceTree = BUILT_PRODUCTS_DIR; };
A75D91522FF0146B00EE6196 /* InTEXTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = InTEXTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
A75D915C2FF0146B00EE6196 /* InTEXUITests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = InTEXUITests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
/* End PBXFileReference section */
/* Begin PBXFileSystemSynchronizedBuildFileExceptionSet section */
A75D91642FF0146B00EE6196 /* Exceptions for "InTEX" folder in "InTEX" target */ = {
isa = PBXFileSystemSynchronizedBuildFileExceptionSet;
membershipExceptions = (
Info.plist,
);
target = A75D91402FF0146A00EE6196 /* InTEX */;
};
/* End PBXFileSystemSynchronizedBuildFileExceptionSet section */
/* Begin PBXFileSystemSynchronizedRootGroup section */
A75D91432FF0146A00EE6196 /* InTEX */ = {
isa = PBXFileSystemSynchronizedRootGroup;
exceptions = (
A75D91642FF0146B00EE6196 /* Exceptions for "InTEX" folder in "InTEX" target */,
);
path = InTEX;
sourceTree = "<group>";
};
A75D91552FF0146B00EE6196 /* InTEXTests */ = {
isa = PBXFileSystemSynchronizedRootGroup;
path = InTEXTests;
sourceTree = "<group>";
};
A75D915F2FF0146B00EE6196 /* InTEXUITests */ = {
isa = PBXFileSystemSynchronizedRootGroup;
path = InTEXUITests;
sourceTree = "<group>";
};
/* End PBXFileSystemSynchronizedRootGroup section */
/* Begin PBXFrameworksBuildPhase section */
A75D913E2FF0146A00EE6196 /* Frameworks */ = {
isa = PBXFrameworksBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
A75D914F2FF0146B00EE6196 /* Frameworks */ = {
isa = PBXFrameworksBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
A75D91592FF0146B00EE6196 /* Frameworks */ = {
isa = PBXFrameworksBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXFrameworksBuildPhase section */
/* Begin PBXGroup section */
A75D91382FF0146A00EE6196 = {
isa = PBXGroup;
children = (
A75D91432FF0146A00EE6196 /* InTEX */,
A75D91552FF0146B00EE6196 /* InTEXTests */,
A75D915F2FF0146B00EE6196 /* InTEXUITests */,
A75D91422FF0146A00EE6196 /* Products */,
);
sourceTree = "<group>";
};
A75D91422FF0146A00EE6196 /* Products */ = {
isa = PBXGroup;
children = (
A75D91412FF0146A00EE6196 /* InTEX.app */,
A75D91522FF0146B00EE6196 /* InTEXTests.xctest */,
A75D915C2FF0146B00EE6196 /* InTEXUITests.xctest */,
);
name = Products;
sourceTree = "<group>";
};
/* End PBXGroup section */
/* Begin PBXNativeTarget section */
A75D91402FF0146A00EE6196 /* InTEX */ = {
isa = PBXNativeTarget;
buildConfigurationList = A75D91652FF0146B00EE6196 /* Build configuration list for PBXNativeTarget "InTEX" */;
buildPhases = (
A75D913D2FF0146A00EE6196 /* Sources */,
A75D913E2FF0146A00EE6196 /* Frameworks */,
A75D913F2FF0146A00EE6196 /* Resources */,
);
buildRules = (
);
dependencies = (
);
fileSystemSynchronizedGroups = (
A75D91432FF0146A00EE6196 /* InTEX */,
);
name = InTEX;
packageProductDependencies = (
);
productName = InTEX;
productReference = A75D91412FF0146A00EE6196 /* InTEX.app */;
productType = "com.apple.product-type.application";
};
A75D91512FF0146B00EE6196 /* InTEXTests */ = {
isa = PBXNativeTarget;
buildConfigurationList = A75D916A2FF0146B00EE6196 /* Build configuration list for PBXNativeTarget "InTEXTests" */;
buildPhases = (
A75D914E2FF0146B00EE6196 /* Sources */,
A75D914F2FF0146B00EE6196 /* Frameworks */,
A75D91502FF0146B00EE6196 /* Resources */,
);
buildRules = (
);
dependencies = (
A75D91542FF0146B00EE6196 /* PBXTargetDependency */,
);
fileSystemSynchronizedGroups = (
A75D91552FF0146B00EE6196 /* InTEXTests */,
);
name = InTEXTests;
packageProductDependencies = (
);
productName = InTEXTests;
productReference = A75D91522FF0146B00EE6196 /* InTEXTests.xctest */;
productType = "com.apple.product-type.bundle.unit-test";
};
A75D915B2FF0146B00EE6196 /* InTEXUITests */ = {
isa = PBXNativeTarget;
buildConfigurationList = A75D916D2FF0146B00EE6196 /* Build configuration list for PBXNativeTarget "InTEXUITests" */;
buildPhases = (
A75D91582FF0146B00EE6196 /* Sources */,
A75D91592FF0146B00EE6196 /* Frameworks */,
A75D915A2FF0146B00EE6196 /* Resources */,
);
buildRules = (
);
dependencies = (
A75D915E2FF0146B00EE6196 /* PBXTargetDependency */,
);
fileSystemSynchronizedGroups = (
A75D915F2FF0146B00EE6196 /* InTEXUITests */,
);
name = InTEXUITests;
packageProductDependencies = (
);
productName = InTEXUITests;
productReference = A75D915C2FF0146B00EE6196 /* InTEXUITests.xctest */;
productType = "com.apple.product-type.bundle.ui-testing";
};
/* End PBXNativeTarget section */
/* Begin PBXProject section */
A75D91392FF0146A00EE6196 /* Project object */ = {
isa = PBXProject;
attributes = {
BuildIndependentTargetsInParallel = 1;
LastSwiftUpdateCheck = 2650;
LastUpgradeCheck = 2650;
TargetAttributes = {
A75D91402FF0146A00EE6196 = {
CreatedOnToolsVersion = 26.5;
};
A75D91512FF0146B00EE6196 = {
CreatedOnToolsVersion = 26.5;
TestTargetID = A75D91402FF0146A00EE6196;
};
A75D915B2FF0146B00EE6196 = {
CreatedOnToolsVersion = 26.5;
TestTargetID = A75D91402FF0146A00EE6196;
};
};
};
buildConfigurationList = A75D913C2FF0146A00EE6196 /* Build configuration list for PBXProject "InTEX" */;
developmentRegion = en;
hasScannedForEncodings = 0;
knownRegions = (
en,
Base,
);
mainGroup = A75D91382FF0146A00EE6196;
minimizedProjectReferenceProxies = 1;
preferredProjectObjectVersion = 77;
productRefGroup = A75D91422FF0146A00EE6196 /* Products */;
projectDirPath = "";
projectRoot = "";
targets = (
A75D91402FF0146A00EE6196 /* InTEX */,
A75D91512FF0146B00EE6196 /* InTEXTests */,
A75D915B2FF0146B00EE6196 /* InTEXUITests */,
);
};
/* End PBXProject section */
/* Begin PBXResourcesBuildPhase section */
A75D913F2FF0146A00EE6196 /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
A75D91502FF0146B00EE6196 /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
A75D915A2FF0146B00EE6196 /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXResourcesBuildPhase section */
/* Begin PBXSourcesBuildPhase section */
A75D913D2FF0146A00EE6196 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
A75D914E2FF0146B00EE6196 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
A75D91582FF0146B00EE6196 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXSourcesBuildPhase section */
/* Begin PBXTargetDependency section */
A75D91542FF0146B00EE6196 /* PBXTargetDependency */ = {
isa = PBXTargetDependency;
target = A75D91402FF0146A00EE6196 /* InTEX */;
targetProxy = A75D91532FF0146B00EE6196 /* PBXContainerItemProxy */;
};
A75D915E2FF0146B00EE6196 /* PBXTargetDependency */ = {
isa = PBXTargetDependency;
target = A75D91402FF0146A00EE6196 /* InTEX */;
targetProxy = A75D915D2FF0146B00EE6196 /* PBXContainerItemProxy */;
};
/* End PBXTargetDependency section */
/* Begin XCBuildConfiguration section */
A75D91662FF0146B00EE6196 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
CODE_SIGN_ENTITLEMENTS = InTEX/InTEX.entitlements;
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
ENABLE_PREVIEWS = YES;
GENERATE_INFOPLIST_FILE = YES;
INFOPLIST_FILE = InTEX/Info.plist;
INFOPLIST_KEY_UIApplicationSceneManifest_Generation = YES;
INFOPLIST_KEY_UIApplicationSupportsIndirectInputEvents = YES;
INFOPLIST_KEY_UILaunchScreen_Generation = YES;
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPhone = "UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
);
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = imageteach.InTEX;
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = YES;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor;
SWIFT_EMIT_LOC_STRINGS = YES;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TARGETED_DEVICE_FAMILY = "1,2";
};
name = Debug;
};
A75D91672FF0146B00EE6196 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
CODE_SIGN_ENTITLEMENTS = InTEX/InTEX.entitlements;
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
ENABLE_PREVIEWS = YES;
GENERATE_INFOPLIST_FILE = YES;
INFOPLIST_FILE = InTEX/Info.plist;
INFOPLIST_KEY_UIApplicationSceneManifest_Generation = YES;
INFOPLIST_KEY_UIApplicationSupportsIndirectInputEvents = YES;
INFOPLIST_KEY_UILaunchScreen_Generation = YES;
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
INFOPLIST_KEY_UISupportedInterfaceOrientations_iPhone = "UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
);
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = imageteach.InTEX;
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = YES;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor;
SWIFT_EMIT_LOC_STRINGS = YES;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TARGETED_DEVICE_FAMILY = "1,2";
};
name = Release;
};
A75D91682FF0146B00EE6196 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
ALWAYS_SEARCH_USER_PATHS = NO;
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES;
CLANG_ANALYZER_NONNULL = YES;
CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
CLANG_CXX_LANGUAGE_STANDARD = "gnu++20";
CLANG_ENABLE_MODULES = YES;
CLANG_ENABLE_OBJC_ARC = YES;
CLANG_ENABLE_OBJC_WEAK = YES;
CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES;
CLANG_WARN_BOOL_CONVERSION = YES;
CLANG_WARN_COMMA = YES;
CLANG_WARN_CONSTANT_CONVERSION = YES;
CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES;
CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR;
CLANG_WARN_DOCUMENTATION_COMMENTS = YES;
CLANG_WARN_EMPTY_BODY = YES;
CLANG_WARN_ENUM_CONVERSION = YES;
CLANG_WARN_INFINITE_RECURSION = YES;
CLANG_WARN_INT_CONVERSION = YES;
CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES;
CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES;
CLANG_WARN_OBJC_LITERAL_CONVERSION = YES;
CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR;
CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER = YES;
CLANG_WARN_RANGE_LOOP_ANALYSIS = YES;
CLANG_WARN_STRICT_PROTOTYPES = YES;
CLANG_WARN_SUSPICIOUS_MOVE = YES;
CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE;
CLANG_WARN_UNREACHABLE_CODE = YES;
CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
COPY_PHASE_STRIP = NO;
DEBUG_INFORMATION_FORMAT = dwarf;
ENABLE_STRICT_OBJC_MSGSEND = YES;
ENABLE_TESTABILITY = YES;
ENABLE_USER_SCRIPT_SANDBOXING = YES;
GCC_C_LANGUAGE_STANDARD = gnu17;
GCC_DYNAMIC_NO_PIC = NO;
GCC_NO_COMMON_BLOCKS = YES;
GCC_OPTIMIZATION_LEVEL = 0;
GCC_PREPROCESSOR_DEFINITIONS = (
"DEBUG=1",
"$(inherited)",
);
GCC_WARN_64_TO_32_BIT_CONVERSION = YES;
GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;
GCC_WARN_UNDECLARED_SELECTOR = YES;
GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;
GCC_WARN_UNUSED_FUNCTION = YES;
GCC_WARN_UNUSED_VARIABLE = YES;
IPHONEOS_DEPLOYMENT_TARGET = 26.5;
LOCALIZATION_PREFERS_STRING_CATALOGS = YES;
MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE;
MTL_FAST_MATH = YES;
ONLY_ACTIVE_ARCH = YES;
SDKROOT = iphoneos;
SWIFT_ACTIVE_COMPILATION_CONDITIONS = "DEBUG $(inherited)";
SWIFT_OPTIMIZATION_LEVEL = "-Onone";
};
name = Debug;
};
A75D91692FF0146B00EE6196 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
ALWAYS_SEARCH_USER_PATHS = NO;
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES;
CLANG_ANALYZER_NONNULL = YES;
CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
CLANG_CXX_LANGUAGE_STANDARD = "gnu++20";
CLANG_ENABLE_MODULES = YES;
CLANG_ENABLE_OBJC_ARC = YES;
CLANG_ENABLE_OBJC_WEAK = YES;
CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES;
CLANG_WARN_BOOL_CONVERSION = YES;
CLANG_WARN_COMMA = YES;
CLANG_WARN_CONSTANT_CONVERSION = YES;
CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES;
CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR;
CLANG_WARN_DOCUMENTATION_COMMENTS = YES;
CLANG_WARN_EMPTY_BODY = YES;
CLANG_WARN_ENUM_CONVERSION = YES;
CLANG_WARN_INFINITE_RECURSION = YES;
CLANG_WARN_INT_CONVERSION = YES;
CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES;
CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES;
CLANG_WARN_OBJC_LITERAL_CONVERSION = YES;
CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR;
CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER = YES;
CLANG_WARN_RANGE_LOOP_ANALYSIS = YES;
CLANG_WARN_STRICT_PROTOTYPES = YES;
CLANG_WARN_SUSPICIOUS_MOVE = YES;
CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE;
CLANG_WARN_UNREACHABLE_CODE = YES;
CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
COPY_PHASE_STRIP = NO;
DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym";
ENABLE_NS_ASSERTIONS = NO;
ENABLE_STRICT_OBJC_MSGSEND = YES;
ENABLE_USER_SCRIPT_SANDBOXING = YES;
GCC_C_LANGUAGE_STANDARD = gnu17;
GCC_NO_COMMON_BLOCKS = YES;
GCC_WARN_64_TO_32_BIT_CONVERSION = YES;
GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;
GCC_WARN_UNDECLARED_SELECTOR = YES;
GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;
GCC_WARN_UNUSED_FUNCTION = YES;
GCC_WARN_UNUSED_VARIABLE = YES;
IPHONEOS_DEPLOYMENT_TARGET = 26.5;
LOCALIZATION_PREFERS_STRING_CATALOGS = YES;
MTL_ENABLE_DEBUG_INFO = NO;
MTL_FAST_MATH = YES;
SDKROOT = iphoneos;
SWIFT_COMPILATION_MODE = wholemodule;
VALIDATE_PRODUCT = YES;
};
name = Release;
};
A75D916B2FF0146B00EE6196 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
GENERATE_INFOPLIST_FILE = YES;
IPHONEOS_DEPLOYMENT_TARGET = 26.5;
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = imageteach.InTEXTests;
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = NO;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TARGETED_DEVICE_FAMILY = "1,2";
TEST_HOST = "$(BUILT_PRODUCTS_DIR)/InTEX.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/InTEX";
};
name = Debug;
};
A75D916C2FF0146B00EE6196 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
GENERATE_INFOPLIST_FILE = YES;
IPHONEOS_DEPLOYMENT_TARGET = 26.5;
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = imageteach.InTEXTests;
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = NO;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TARGETED_DEVICE_FAMILY = "1,2";
TEST_HOST = "$(BUILT_PRODUCTS_DIR)/InTEX.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/InTEX";
};
name = Release;
};
A75D916E2FF0146B00EE6196 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
GENERATE_INFOPLIST_FILE = YES;
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = imageteach.InTEXUITests;
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = NO;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TARGETED_DEVICE_FAMILY = "1,2";
TEST_TARGET_NAME = InTEX;
};
name = Debug;
};
A75D916F2FF0146B00EE6196 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
GENERATE_INFOPLIST_FILE = YES;
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = imageteach.InTEXUITests;
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = NO;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TARGETED_DEVICE_FAMILY = "1,2";
TEST_TARGET_NAME = InTEX;
};
name = Release;
};
/* End XCBuildConfiguration section */
/* Begin XCConfigurationList section */
A75D913C2FF0146A00EE6196 /* Build configuration list for PBXProject "InTEX" */ = {
isa = XCConfigurationList;
buildConfigurations = (
A75D91682FF0146B00EE6196 /* Debug */,
A75D91692FF0146B00EE6196 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
A75D91652FF0146B00EE6196 /* Build configuration list for PBXNativeTarget "InTEX" */ = {
isa = XCConfigurationList;
buildConfigurations = (
A75D91662FF0146B00EE6196 /* Debug */,
A75D91672FF0146B00EE6196 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
A75D916A2FF0146B00EE6196 /* Build configuration list for PBXNativeTarget "InTEXTests" */ = {
isa = XCConfigurationList;
buildConfigurations = (
A75D916B2FF0146B00EE6196 /* Debug */,
A75D916C2FF0146B00EE6196 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
A75D916D2FF0146B00EE6196 /* Build configuration list for PBXNativeTarget "InTEXUITests" */ = {
isa = XCConfigurationList;
buildConfigurations = (
A75D916E2FF0146B00EE6196 /* Debug */,
A75D916F2FF0146B00EE6196 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
/* End XCConfigurationList section */
};
rootObject = A75D91392FF0146A00EE6196 /* Project object */;
}
InTEX/InTEX.xcodeproj/project.xcworkspace/contents.xcworkspacedata
Generated
-7
View File
@@ -1,7 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<Workspace
version = "1.0">
<FileRef
location = "self:">
</FileRef>
</Workspace>
InTEX/InTEX/Assets.xcassets/AccentColor.colorset/Contents.json
-11
View File
@@ -1,11 +0,0 @@
{
"colors" : [
{
"idiom" : "universal"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
InTEX/InTEX/Assets.xcassets/AppIcon.appiconset/Contents.json
-35
View File
@@ -1,35 +0,0 @@
{
"images" : [
{
"idiom" : "universal",
"platform" : "ios",
"size" : "1024x1024"
},
{
"appearances" : [
{
"appearance" : "luminosity",
"value" : "dark"
}
],
"idiom" : "universal",
"platform" : "ios",
"size" : "1024x1024"
},
{
"appearances" : [
{
"appearance" : "luminosity",
"value" : "tinted"
}
],
"idiom" : "universal",
"platform" : "ios",
"size" : "1024x1024"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
InTEX/InTEX/Assets.xcassets/Contents.json
-6
View File
@@ -1,6 +0,0 @@
{
"info" : {
"author" : "xcode",
"version" : 1
}
}
InTEX/InTEX/ContentView.swift
-61
View File
@@ -1,61 +0,0 @@
//
// ContentView.swift
// InTEX
//
// Created by allenyuan on 2026/6/27.
//
import SwiftUI
import SwiftData
struct ContentView: View {
@Environment(\.modelContext) private var modelContext
@Query private var items: [Item]
var body: some View {
NavigationSplitView {
List {
ForEach(items) { item in
NavigationLink {
Text("Item at \(item.timestamp, format: Date.FormatStyle(date: .numeric, time: .standard))")
} label: {
Text(item.timestamp, format: Date.FormatStyle(date: .numeric, time: .standard))
}
}
.onDelete(perform: deleteItems)
}
.toolbar {
ToolbarItem(placement: .navigationBarTrailing) {
EditButton()
}
ToolbarItem {
Button(action: addItem) {
Label("Add Item", systemImage: "plus")
}
}
}
} detail: {
Text("Select an item")
}
}
private func addItem() {
withAnimation {
let newItem = Item(timestamp: Date())
modelContext.insert(newItem)
}
}
private func deleteItems(offsets: IndexSet) {
withAnimation {
for index in offsets {
modelContext.delete(items[index])
}
}
}
}
#Preview {
ContentView()
.modelContainer(for: Item.self, inMemory: true)
}
InTEX/InTEX/InTEX.entitlements
-14
View File
@@ -1,14 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>aps-environment</key>
<string>development</string>
<key>com.apple.developer.icloud-container-identifiers</key>
<array/>
<key>com.apple.developer.icloud-services</key>
<array>
<string>CloudKit</string>
</array>
</dict>
</plist>
InTEX/InTEX/InTEXApp.swift
-32
View File
@@ -1,32 +0,0 @@
//
// InTEXApp.swift
// InTEX
//
// Created by allenyuan on 2026/6/27.
//
import SwiftUI
import SwiftData
@main
struct InTEXApp: App {
var sharedModelContainer: ModelContainer = {
let schema = Schema([
Item.self,
])
let modelConfiguration = ModelConfiguration(schema: schema, isStoredInMemoryOnly: false)
do {
return try ModelContainer(for: schema, configurations: [modelConfiguration])
} catch {
fatalError("Could not create ModelContainer: \(error)")
}
}()
var body: some Scene {
WindowGroup {
ContentView()
}
.modelContainer(sharedModelContainer)
}
}
InTEX/InTEX/Info.plist
-10
View File
@@ -1,10 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>UIBackgroundModes</key>
<array>
<string>remote-notification</string>
</array>
</dict>
</plist>
InTEX/InTEX/Item.swift
-18
View File
@@ -1,18 +0,0 @@
//
// Item.swift
// InTEX
//
// Created by allenyuan on 2026/6/27.
//
import Foundation
import SwiftData
@Model
final class Item {
var timestamp: Date
init(timestamp: Date) {
self.timestamp = timestamp
}
}
InTEX/InTEXTests/InTEXTests.swift
-19
View File
@@ -1,19 +0,0 @@
//
// InTEXTests.swift
// InTEXTests
//
// Created by allenyuan on 2026/6/27.
//
import Testing
@testable import InTEX
struct InTEXTests {
@Test func example() async throws {
// Write your test here and use APIs like `#expect(...)` to check expected conditions.
// Swift Testing Documentation
// https://developer.apple.com/documentation/testing
}
}
InTEX/InTEXUITests/InTEXUITests.swift
-43
View File
@@ -1,43 +0,0 @@
//
// InTEXUITests.swift
// InTEXUITests
//
// Created by allenyuan on 2026/6/27.
//
import XCTest
final class InTEXUITests: XCTestCase {
override func setUpWithError() throws {
// Put setup code here. This method is called before the invocation of each test method in the class.
// In UI tests it is usually best to stop immediately when a failure occurs.
continueAfterFailure = false
// In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
}
override func tearDownWithError() throws {
// Put teardown code here. This method is called after the invocation of each test method in the class.
}
@MainActor
func testExample() throws {
// UI tests must launch the application that they test.
let app = XCUIApplication()
app.launch()
// Use XCTAssert and related functions to verify your tests produce the correct results.
// XCUIAutomation Documentation
// https://developer.apple.com/documentation/xcuiautomation
}
@MainActor
func testLaunchPerformance() throws {
// This measures how long it takes to launch your application.
measure(metrics: [XCTApplicationLaunchMetric()]) {
XCUIApplication().launch()
}
}
}
InTEX/InTEXUITests/InTEXUITestsLaunchTests.swift
-35
View File
@@ -1,35 +0,0 @@
//
// InTEXUITestsLaunchTests.swift
// InTEXUITests
//
// Created by allenyuan on 2026/6/27.
//
import XCTest
final class InTEXUITestsLaunchTests: XCTestCase {
override class var runsForEachTargetApplicationUIConfiguration: Bool {
true
}
override func setUpWithError() throws {
continueAfterFailure = false
}
@MainActor
func testLaunch() throws {
let app = XCUIApplication()
app.launch()
// Insert steps here to perform after app launch but before taking a screenshot,
// such as logging into a test account or navigating somewhere in the app
// XCUIAutomation Documentation
// https://developer.apple.com/documentation/xcuiautomation
let attachment = XCTAttachment(screenshot: app.screenshot())
attachment.name = "Launch Screen"
attachment.lifetime = .keepAlways
add(attachment)
}
}
PRODUCTION_REMEDIATION_CHECKLIST.md
+579
View File
@@ -0,0 +1,579 @@
# llm-in-text 生产环境修复清单
## 文档目的
本文档是针对当前仓库在 2026-04-01 状态下基于代码的生产就绪性审查。
它回答三个问题：
1. 当前项目是否已准备好投入生产？
2. 自上次审查以来已修复了哪些问题？
3. 还有什么因素阻碍安全上线？
本次审查仅限于仓库中已有的内容和本地直接验证的内容，不包括外部基础设施、反向代理配置、云资源、CI 平台密钥或运行时运维的完整审计。
## 审查范围
- 前端构建和运行时入口
- 后端 FastAPI 端点和模型集成
- 补全、OCR 和文件转换请求路径
- 隐私相关行为和本地存储
- 基础测试覆盖率和构建验证
- PWA/Service Worker 状态
- 仓库中存在的部署和运维工件
## 已验证的事实
以下项目已针对当前仓库直接验证：
- 前端生产构建通过 `npm.cmd run build` 成功
- 后端测试通过 `pytest backend/tests -q`
- 当前后端测试结果：`8 passed, 1 skipped`
- 存在健康检查端点：`/health/live` 和 `/health/ready`
- 前端不再硬编码 API 密钥
- 后端不再需要旧的 `X-API-Key` 头
- 前端隐私模式现在默认为启用
- 后端错误响应现在已规范化，不再返回原始异常字符串
- OCR 和转换端点现在强制执行基本的文件大小和扩展名检查
## 当前评估
项目尚未准备好投入生产。
当前状态更接近于：
- 一个可用的原型
- 内部演示
- 具有部分加固的预发布候选版本
由于缺少几个核心生产基线，它尚未准备好面向互联网的生产使用：
- 真正的限流和并发保护
- 正式的部署工件和运行时拓扑
- CI/CD 和自动化质量门禁
- 结构化的可观测性和告警
- 更强的请求验证和更安全的文件处理隔离
- 前端自动化测试和端到端验证
## 已修复的问题
与之前的清单相比，以下项目不再作为阻碍因素：
### 已修复：硬编码的前端/后端共享 API 密钥
- `src/utils/api.js` 不再发送 `X-API-Key`
- `src/utils/convert.js` 不再发送 `X-API-Key`
- `backend/main.py` 不再强制执行旧的静态密钥
这消除了上次审查中最严重的问题之一。
遗留问题：
- `backend/tests/test_main_cancel.py` 仍然包含过时的 `X-API-Key` 头，但它们目前是无效的，表明测试假设已过时而非活动中的认证逻辑
### 已修复：危险的通配符 CORS 配置
当前后端 CORS 限制为：
- `http://localhost:5173`
- `http://localhost:3000`
并使用：
- `allow_credentials=False`
- `allow_methods=["POST", "OPTIONS"]`
- `allow_headers=["Content-Type", "X-Request-Id"]`
这比之前的通配符配置安全得多。
遗留问题：
- CORS 仍然针对本地开发硬编码，对于 staging/生产环境不是环境驱动的
### 已修复：默认收集前端公网 IP
- `src/stores/settings.js` 现在将 `privacyMode` 默认为 `true`
- `src/utils/api.js` 不再调用 `ipify`
- `src/utils/api.js` 不再发送 `X-Client-IP`
- `backend/main.py` 不再将 IP 派生的位置注入提示词
遗留问题：
- `backend/geoip.py` 和 GeoLite 数据库仍然存在于仓库中，这可能会造成对当前隐私模型的混淆
### 已修复：向客户端暴露原始异常字符串
当前后端响应使用 `_error_response(...)` 和结构化载荷，例如：
- `INTERNAL_ERROR`
- `OCR_FAILED`
- `CONVERT_FAILED`
- `FILE_TOO_LARGE`
- `INVALID_FILE_TYPE`
这比直接返回 `str(exception)` 更好。
遗留问题：
- 日志仍然记录用户派生内容的预览，这是另一个隐私/可观测性问题
### 部分修复：上传和转换输入边界
`backend/main.py` 中的当前后端保护包括：
- OCR 大小上限：10 MB
- 转换大小上限：50 MB
- OCR 和转换的扩展名白名单
- 在 `finally` 中清理临时转换文件
这是有意义的进展，但不足以投入生产。
## 生产阻碍因素
优先级含义：
- `P0`：必须在生产上线前完成
- `P1`：应在公开发布或广泛推广前完成
- `P2`：重要的后续加固和维护工作
---
## P0 阻碍因素
### P0-01 声明了限流和并发控制但未强制执行
当前状态：
- `backend/main.py` 定义了 `MAX_CONCURRENT_COMPLETIONS = 4`
- `backend/main.py` 定义了 `COMPLETION_RATE_LIMIT = 60`
- 没有任何实际的限流器使用这两个值
- OCR 和转换端点也没有真正的每客户端节流
风险：
- 容易滥用昂贵的补全/OCR/转换端点
- 可避免地对模型主机、CPU、内存和临时存储造成过载
- 突发流量下无法控制降级
必需的修复：
1. 在应用或网关中强制执行真正的每路由限流
2. 为补全作业添加真正的并发保护
3. 为 `/v1/completions`、`/v1/ocr`、`/v1/convert` 添加单独预算
4. 达到限制时返回明确的 `429`
5. 为节流的请求和队列深度发出指标
验收标准：
- 重复的突发流量触发确定性的 `429`
- 并发补全不能超过配置的预算
- 负载测试下服务保持稳定
---
### P0-02 仓库中不存在生产部署基线
当前状态：
- 后端仅通过 `uvicorn.run(...)` 暴露开发式启动
- 没有 `Dockerfile`
- 没有 compose 文件
- 没有 Kubernetes 清单或 Helm chart
- 没有 systemd 单元
- 没有反向代理参考配置
- 没有记录的生产环境契约
风险：
- 没有可复现的部署路径
- 没有明确的过程监督、重启或优雅的发布模型
- 没有记录的 ingress/请求体大小/超时/TLS 姿态
必需的修复：
1. 定义一个官方部署目标
2. 为该目标添加部署工件
3. 记录所需的环境变量、端口、探针和存储
4. 定义优雅关闭和发布行为
5. 记录反向代理限制和信任边界
验收标准：
- 新环境可以从仓库文档和工件部署
- 健康探针已接入所选运行时
- 回滚路径已记录
---
### P0-03 缺少 CI/CD 和仓库质量门禁
当前状态：
- 仓库级别没有找到 `.github/workflows`
- `package.json` 没有真正的前端测试脚本
- 没有 lint 脚本
- 没有类型检查脚本
- 没有依赖扫描或密钥扫描工作流
风险：
- 回归只能手动捕获
- 安全和打包漂移很可能发生
- 生产就绪性取决于本地开发者的规范
必需的修复：
1. 为构建和测试添加 CI 工作流
2. 添加前端自动化测试
3. 在适用的情况下添加 lint 和类型检查门禁
4. 添加依赖漏洞扫描
5. 添加密钥扫描和基本 SAST
验收标准：
- 每个 PR 都运行构建、后端测试、前端测试和 lint
- 失败的检查阻止合并
---
### P0-04 文件处理路径仍然缺乏生产级隔离
当前状态：
- 后端将完整 base64 载荷解码到内存中
- 转换写入临时文件并将其传递给 `markitdown`
- OCR 和转换主要依赖扩展名检查，而不是内容嗅探
- 没有工作进程隔离或用于转换的单独沙箱
- 转换任务没有队列或资源预算
风险：
- 大型或并发上传导致内存峰值
- 畸形或对抗性文档会给解析器带来压力
- 转换工作负载可能干扰核心补全可用性
必需的修复：
1. 明确验证 base64 解码失败
2. 添加 MIME/内容嗅探，而不仅仅是扩展名检查
3. 在入口和应用层添加更低的、路由特定的请求体限制
4. 将转换隔离到单独的工作进程/进程边界
5. 添加超时、并发上限和队列深度控制
验收标准：
- 畸形载荷失败并返回明确的 4xx 响应
- 转换不能饿死补全服务
- 压力下临时文件和内存增长保持有界
---
### P0-05 环境配置不一致且不安全
当前状态：
- 前端 `.env.example` 相当安全
- 后端 `.env.example` 过时且与代码不一致
- 代码读取 `OLLAMA_HOST`
- 后端示例仍然使用 `OLLAMA_BASE_URL`
- 后端示例仍然定义 `OPENAI_API_KEY=ollama`，这是误导性的
风险：
- 新环境配置不正确
- 运营商可能假设不支持的认证/配置行为
- staging/生产漂移很可能发生
必需的修复：
1. 用代码实际使用的变量替换后端环境示例
2. 在启动时验证所需的环境变量
3. 分离开发、staging 和生产环境契约
4. 缺失关键配置时快速失败
验收标准：
- 示例环境文件匹配真实运行时行为
- 无效或缺失关键配置导致启动失败
---
## P1 高优先级差距
### P1-01 日志仍然捕获用户派生内容预览
当前状态：
- `backend/main.py` 记录提示词派生的前缀和后缀预览
- 补全结果记录包含内容预览
- OCR 和转换记录文本预览长度和片段
风险：
- 日志可能包含敏感文档内容
- 隐私姿态与应用可见的隐私设置不一致
- 难以证明保留/合规姿态
必需的修复：
1. 默认情况下停止记录用户内容正文和预览
2. 仅保留请求元数据：路由、请求 ID、状态、延迟、大小
3. 引入结构化 JSON 日志
4. 脱敏或哈希任何敏感标识符
验收标准：
- 默认日志不包含用户文档文本
- 请求关联仍可通过请求 ID 和元数据工作
### P1-02 请求验证仍然过于宽松
当前状态：
- Pydantic 模型定义了字段但没有长度或枚举约束
- `prefix`、`suffix`、`filename` 和 `reason` 受到极小约束
- base64 字段在路由特定检查之前仍然可能非常大
- 前端转换路径在将完整文件读入 base64 之前不执行预验证
风险：
- 过大或畸形的请求太容易到达昂贵的逻辑
- 端点之间的 4xx 行为不一致
必需的修复：
1. 为 Pydantic 字段添加长度和枚举约束
2. 明确验证 base64 格式
3. 更防御性地规范化文件名处理
4. 添加前端预检查大小/类型作为 UX
验收标准：
- 畸形请求尽早失败并返回确定性的 4xx 响应
### P1-03 前端自动化覆盖率基本缺失
当前状态：
- 后端有针对性的单元/集成风格测试
- 前端没有配置测试运行器
- 核心用户路径没有 E2E 覆盖率
重要说明：
- `backend/tests/test_main_cancel.py` 仍然发送过时的 `X-API-Key` 头；测试通过仅仅是因为后端忽略它们
风险：
- 编辑器、上传、OCR、转换和设置的回归将会遗漏
必需的修复：
1. 添加前端单元/组件测试
2. 为补全、取消、上传、OCR 和转换添加 E2E 覆盖率
3. 从后端测试中移除过时的认证假设
验收标准：
- 核心用户旅程在 CI 中自动覆盖
### P1-04 健康检查端点存在，但就绪性浅且缺少可观测性
当前状态：
- `/health/live` 存在
- `/health/ready` 存在
- 就绪性实际上不检查上游模型可用性
- 没有指标端点
- 没有追踪
- 没有告警定义
风险：
- 运行时故障检测太晚
- 平台探针可能报告健康而上游依赖不可用
必需的修复：
1. 使就绪性反映关键依赖状态
2. 添加请求/延迟/错误指标
3. 为上游故障和饱和添加告警阈值
4. 为关键路由定义仪表板
验收标准：
- 运营商可以快速检测模型依赖故障
- 请求成功率和延迟可观测
### P1-05 Service Worker 实现存在但被禁用
当前状态：
- `public/sw.js` 存在
- `src/main.js` 用 `&& false` 硬禁用注册
- Service Worker 策略是手写的并通过静态缓存名称版本化
风险：
- 当前仓库包含未被实际使用的休眠 PWA 逻辑
- 如果随意重新启用，更新和缓存行为可能很脆弱
必需的修复：
1. 确定 PWA 是否在生产范围内
2. 如果是，采用维护的策略如 Vite PWA/Workbox
3. 如果不是，移除无用的 Service Worker 代码以减少混淆
验收标准：
- PWA 行为要么被有意支持和测试，要么被完全移除
### P1-06 背景图像持久化可能导致本地存储和内存膨胀
当前状态：
- 设置面板将上传的背景图像读取为 data URL
- 背景图像数据存储在 localStorage 中
- 没有对背景资产强制执行明确的大小上限
风险：
- 存储配额耗尽
- 大图像导致 UI 缓慢
- 跨浏览器的持久化行为脆弱
必需的修复：
1. 读取前在客户端添加大小上限
2. 持久化前调整大小/压缩
3. 对于较大的资产，优先使用 blob/object URL 或 IndexedDB
4. 为存储溢出添加迁移/错误处理
验收标准：
- 大图像不能降低启动或破坏设置持久化
---
## P2 重要后续工作
### P2-01 构建成功，但 bundle/chunk 策略仍然粗糙
验证的构建输出显示：
- `manualChunks` 生成许多空 chunk
- 一个与 Mermaid 相关的大型 chunk 超过 1 MB 压缩后
风险：
- 不必要的 chunk 开销
- 较弱的设备上冷启动较慢
必需的修复：
1. 简化 `manualChunks`
2. 延迟加载重型可选功能
3. 清理 chunk 后重新测量首次加载成本
### P2-02 OCR 缓存和图像哈希缓存没有明确的驱逐策略
当前状态：
- OCR 数据存储在内存中的 `Map`
- 哈希缓存也在内存中
- 没有 TTL
- 没有最大条目数
风险：
- 长时间会话会累积内存
必需的修复：
1. 添加 TTL 和条目边界
2. 如需要，暴露缓存指标用于调试
### P2-03 仓库仍然包含过时和混淆的工件
示例：
- `backend/geoip.py` 和 GeoLite DB 仍然存在，尽管 IP 地理定位在请求流程中不再活跃
- `backend/.env.example` 记录的变量不是代码使用的
- 后端测试仍然包含过时的 `X-API-Key`
风险：
- 未来维护者可能无意中重新引入已移除的行为
必需的修复：
1. 移除死代码和过时配置
2. 使测试和文档与当前实现保持一致
---
## 上线前必需的缺失证据
仓库目前不提供以下生产能力的证据：
- staging 部署管道
- 回滚程序
- 流量/负载测试结果
- 故障注入或混沌测试
- 备份/恢复程序
- 事件响应运行手册
- SLO/SLA 定义
- 安全扫描基线
- 依赖更新策略
- 隐私/数据保留文档
目前应将证据缺失视为未就绪，而不是隐式完成。
## 推荐的修复顺序
### 第一阶段：解除生产上线阻碍
1. 实现真正的限流和并发强制执行
2. 定义官方部署拓扑和工件
3. 添加 CI/CD 质量门禁
4. 加固和隔离文件处理工作负载
5. 修复后端环境配置契约
### 第二阶段：稳定运维和隐私姿态
1. 移除承载内容的日志
2. 加强请求验证
3. 深化就绪检查和指标
4. 添加前端和 E2E 自动化测试
### 第三阶段：性能和可维护性清理
1. 清理 chunk 策略
2. 限制 OCR/图像缓存
3. 移除过时代码和配置
4. 决定 PWA 支持是保留还是移除
## 最低上线门槛
至少在以下所有条件都满足之前，不应称该项目为生产就绪：
- 所有 `P0` 项目都已完成
- 日志不再捕获用户内容
- 前端和端到端自动化测试存在并在 CI 中运行
- 就绪性反映真实的上游依赖状态
- 部署和回滚已记录且可重现
- staging 环境已通过集成验证
- 至少执行了一次受控负载测试并经过审查
## 最终评估
与之前的清单相比，该项目已有实质性改进。几个严重的早期发现不再成立，特别是：
- 硬编码的认证密钥暴露
- 通配符式 CORS 姿态
- 默认公网 IP 收集
- 原始异常泄漏
然而，这一进展并不意味着已准备好投入生产。
当前仓库展示了有用的加固工作，但仍然缺乏生产服务预期的运维、测试、节流、部署和可观测性基线。
README.md
+221 -142
View File
@@ -1,185 +1,264 @@
# LLM in Text - 智能写作助手
基于 Vue3 和 FastAPI 的智能 Markdown 编辑器，集成大语言模型（LLM）实时补全建议功能。
基于 Vue3 和 FastAPI 的智能 Markdown 编辑器，集成大语言模型（LLM）实时补全建议功能，提供类似 GitHub Copilot 的 Ghost Text 体验。
## 功能特性
### Markdown 编辑器
- 基于 Milkdown Crepe 的所见即所得编辑体验
- 支持 Markdown 语法和 LaTeX 公式
- 支持 Mermaid 图表渲染
- 支持完整 Markdown 语法和 LaTeX 公式
- 导入/导出 Markdown 文件
- 导出 DOCX 和 PDF 格式
### AI 智能补全
- 实时生成文本补全建议（灰色显示）
- 流式响应，低延迟体验
- 多种交互方式：Tab接受、Esc拒绝、点击接受
- 多种交互方式：
- **Tab 键**：接受建议
- **Esc 键**：拒绝建议
- **点击灰色文本**：接受建议
- **继续输入**：自动拒绝建议
### 功能块系统
编辑器提供三种**功能块**，统一为顶层原子节点（`atom: true, isolating: true`），通过 ProseMirror schema 强制禁止互相嵌套，无数量限制。导入 Markdown 后自动解析还原为交互卡片，导出后可完整复原：
| 功能块 | Markdown 语法 | 作用 |
|--------|---------------|------|
| **文档块** (`doc_block`) | \`\`\`llm-file fenced code / `<doc_type=...>` legacy HTML tag | 上传的 PDF/DOCX/PPTX/TXT 等文件以可折叠卡片嵌入编辑器，支持内联编辑和 AI 补全 |
| **PRO 块** (`pro_block`) | `[PRO]` / `[PRO]{指令}` | 基于全文上下文进行深度 AI 思考并流式生成 Markdown，`Ctrl+Shift+P` 快速插入 |
| **上传块** (`upload_block`) | `{{{}}}` / `{{{upload file type:pdf,docx}}}` | 文件上传占位符，支持按类型过滤（PDF/DOCX/PPTX/TXT/JSON/YAML/图片等） |
### 文档处理（历史名称，已整合入功能块系统）
- OCR 图片识别：上传图片自动识别文字（OCR 结果注入 AI 补全和 PRO 块上下文）
- 智能大小限制：32KB自动禁用AI
### 设置面板
- 外观主题：亮色/暗色/跟随系统
- 背景模式：默认/暖色/阅读灯/自定义图片
- 模型智能：低/中/高思考级别
- 隐私控制：隐私模式防止发送IP
- 多语言界面：中英日韩德法
### 语音功能
- TTS文字转语音（macOS优化，支持Apple Silicon M1/M2/M3）
- STT语音转文字（支持多种模型大小和量化）
- 自动设备检测（MPS/CUDA/CPU智能切换）
- 离线模式支持（模型缓存检查）
### AI 开关控制
- 右下角 AI 开关按钮
- 白色 = AI 启用，黑色 = AI 禁用
- 禁用时自动清除灰色文本并停止 API 调用
## 技术架构
前端: Vue3 + Vite + Milkdown/Crepe + ProseMirror
后端: FastAPI + Python（OpenAI 兼容端点）
```mermaid
flowchart TB
subgraph Frontend["前端 (Vue3 + Vite)"]
A[App.vue] --> B[MilkdownEditor.vue]
B --> C[Crepe Editor]
C --> D[ProseMirror]
D --> E[copilotPlugin.ts]
E --> F[copilotGhostMark]
E --> G[api.js]
end
### 功能块架构
三种功能块统一为顶层原子节点（`atom: true, isolating: true`），通过 ProseMirror schema 强制禁止嵌套：
- **文档块** (`doc_block`) — `src/plugins/docBlockPlugin.ts`，Markdown 语法：\`\`\`llm-file fenced code block
- **PRO 块** (`pro_block`) — `src/plugins/proBlockPlugin.ts`，Markdown 语法：`[PRO]` / `[PRO]{指令}`
- **上传块** (`upload_block`) — `src/plugins/uploadBlockPlugin.ts`，Markdown 语法：`{{{}}}` / `{{{upload file type:...}}}`
subgraph Backend["后端 (FastAPI + Python)"]
H[main.py<br/>FastAPI Server] --> I[prompt.py<br/>Prompt 构建]
H --> J[llm.py<br/>Ollama 调用]
J --> K[Ollama API]
end
每个功能块配备独立的 Remark 解析器和序列化器，确保 Markdown 导入导出时自动识别和还原。
G -->|POST /v1/completions<br/>SSE 流式响应| H
K -->|LLM 响应| J
```
## 项目结构
```
llm-in-text/
├── src/
│ ├── components/
│ │ └── MilkdownEditor.vue # 主编辑器组件
│ ├── plugins/
│ │ ├── copilotPlugin.ts # ProseMirror AI 补全插件
│ │ ├── types.ts # 类型定义
│ │ └── index.ts # 插件导出
│ ├── utils/
│ │ ├── api.js # API 调用封装
│ │ ├── config.js # 配置文件
│ │ └── ocrCache.js # OCR 缓存管理
│ ├── App.vue
│ └── main.js
├── backend/
│ ├── main.py # FastAPI 服务器
│ ├── llm.py # LLM API 调用
│ ├── prompt.py # Prompt 构建
│ └── requirements.txt
└── README.md
```
## 快速开始
环境: Node.js 18+、Python 3.8+
### 环境要求
- Node.js 18+
- Python 3.8+
- Ollama 服务（或其他兼容 OpenAI API 的服务）
安装:
- 前端: npm install
- 后端: pip install -r backend/requirements.txt
启动:
- 后端: python backend/main.py (端口8001)
- 前端: npm run dev (端口5173)
## Docker 部署
将整个项目目录放进本机 `~/lit/llm-in-text` 后，在项目根目录执行：
### 安装
```bash
cp backend/.env.example backend/.env
docker compose up -d --build
# 前端
npm install
# 后端
cd backend
pip install -r requirements.txt
```
默认对外端口：
- 前端: `http://localhost:8080`
- 后端: `http://localhost:8001`
### 配置
持久化目录全部位于当前项目下的 `docker-data/`：
- PostgreSQL: `docker-data/postgres`
- Redis: `docker-data/redis`
- 任务共享临时目录: `docker-data/jobs`
在 `backend/.env` 中配置：
部署前至少需要修改这些环境变量：
- `backend/.env` 中的 `LLM_BASE_URL`
- `backend/.env` 中的 `LLM_API_KEY`
- `backend/.env` 或 shell 环境中的 `DATABASE_URL`
- `backend/.env` 中的 `API_KEY`
```env
OLLAMA_MODEL=gpt-oss:20b
OLLAMA_HOST=http://localhost:11434
```
## API接口
### 启动
- POST /v1/completions 流式补全建议
- POST /v1/ocr 图片文字识别
- POST /v1/convert 文档转换
- POST /v1/completions/cancel 取消请求
- GET /v1/docs/nodes 文档空间节点列表
- POST /v1/docs/folders 创建文件夹
- POST /v1/docs/files/text 创建文本文件
- POST /v1/docs/files/upload 上传文件到文档空间
- PATCH /v1/docs/nodes/{id} 更新节点
- PUT /v1/docs/files/{id}/blob 替换文件二进制内容
- DELETE /v1/docs/nodes/{id} 删除节点
- GET /v1/docs/files/{id}/blob 下载或预览原文件
- GET /v1/tts-asr/status TTS/ASR状态
- GET /v1/tts-asr/config TTS/ASR配置信息
- POST /v1/tts-asr/tts 文字转语音
- POST /v1/tts-asr/asr 语音转文字
```bash
# 后端（端口 8000）
cd backend
python main.py
## TTS/ASR环境变量配置
# 前端（端口 5173）
npm run dev
```
支持以下环境变量来配置TTS/ASR模块：
访问 http://localhost:5173
| 变量名 | 说明 | 默认值 |
|--------|------|--------|
| `LLM_BASE_URL` | OpenAI-compatible 上游地址 | 必填 |
| `LLM_API_KEY` | OpenAI-compatible 上游密钥 | 必填 |
| `TTS_MODEL_ID` | TTS 模型名 | `Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit` |
| `ASR_MODEL_ID` | ASR 模型名 | `Qwen3-ASR-0.6B-8bit` |
| `TTS_ASR_TTS_TIMEOUT_SECONDS` | TTS 上游超时(秒) | 180 |
| `TTS_ASR_ASR_TIMEOUT_SECONDS` | ASR 上游超时(秒) | 300 |
| `TTS_ASR_MAX_CONNECTIONS` | Speech API 连接池上限 | 24 |
| `TTS_ASR_MAX_KEEPALIVE_CONNECTIONS` | Speech API keepalive 连接数 | 12 |
## API 接口
### POST /v1/completions
流式获取补全建议
**请求：**
```json
{
"prefix": "# Title\n\nContent ",
"suffix": "",
"languageId": "markdown"
}
```
**响应（SSE）：**
```
data: {"content": "here"}
data: {"content": "here is"}
data: {"done": true}
```
## 核心实现
### 后端
- main.py: FastAPI服务器、SSE流式响应
- llm.py: 异步LLM调用（OpenAI兼容）、超时控制
- prompt.py: 7条Prompt规则
- tts_asr.py: 基于共享 OpenAI-compatible Speech API 的 TTS/ASR 适配层
- 统一使用 `LLM_BASE_URL` 和 `LLM_API_KEY`
- 通过 `/audio/speech` 与 `/audio/transcriptions` 调用上游
- 内建连接池、超时、音频时长估算和上游请求 ID 透传
### 后端设计
### 前端
- copilotPlugin.ts: ProseMirror Mark系统
- 关键函数: scheduleFetch、insertGhostText
- Pinia Store状态管理
#### main.py - FastAPI 服务器
- 定义 `/v1/completions` 端点
- 使用 `StreamingResponse` 返回 SSE 流式响应
- CORS 配置允许跨域请求
#### llm.py - LLM 调用封装
- 使用 `ollama.AsyncClient` 异步调用
- 支持 `think='high'` 思考模式
- 返回 `content` 和 `thinking` 字段
#### prompt.py - Prompt 工程
精心设计的 Prompt 模板，包含 7 条核心规则：
| 规则 | 说明 |
|------|------|
| RULE #1 | 无缝连接 - 不重复 suffix 内容，避免"复读机"错误 |
| RULE #2 | 空白处理 - 避免双空格，正确对接标点 |
| RULE #3 | 缩进对齐 - 匹配当前缩进级别和类型 |
| RULE #4 | 列表维护 - 识别并继续任务列表、有序列表、无序列表 |
| RULE #5 | 语法闭合 - 自动闭合未完成的 Markdown 语法 |
| RULE #6 | 输出格式 - 仅输出续写文本，无解释无注释 |
| RULE #7 | 必须输出 - 始终提供有用的续写建议 |
### 前端设计
#### ProseMirror Mark 系统
使用 ProseMirror 的 Mark 系统实现灰色建议文本：
```typescript
// 定义 ghost mark
export const copilotGhostMark = $markSchema('copilot_ghost', () => ({
excludes: '_',
inclusive: true,
toDOM: () => ['span', {
'data-copilot-ghost': '',
class: 'copilot-ghost-text'
}, 0]
}))
// CSS 样式
.copilot-ghost-text {
color: #999;
opacity: 0.6;
}
```
#### copilotPlugin 核心逻辑
```mermaid
flowchart LR
A[用户输入] --> B{文档变化?}
B -->|是| C[清除旧建议]
C --> D[防抖 1000ms]
D --> E[发送 API 请求]
E --> F[收到建议]
F --> G[插入 Ghost Text]
G --> H{用户操作}
H -->|Tab| I[接受建议<br/>移除 mark]
H -->|Esc| J[拒绝建议<br/>删除文本]
H -->|点击 Ghost| I
H -->|继续输入| J
```
#### 关键函数
| 函数 | 作用 |
|------|------|
| `scheduleFetch` | 防抖调度 API 请求 |
| `insertGhostText` | 插入带 mark 的建议文本 |
| `acceptSuggestion` | Tab 接受建议 |
| `rejectSuggestion` | Esc 拒绝建议 |
| `clearGhostText` | 清除当前建议 |
### 数据流
```mermaid
sequenceDiagram
participant U as 用户
participant E as Editor (ProseMirror)
participant P as copilotPlugin
participant A as api.js
participant B as Backend
participant L as LLM
U->>E: 输入文本
E->>P: view.update()
P->>P: 清除旧建议
P->>P: 防抖 1000ms
P->>A: fetchSuggestion(prefix, suffix)
A->>B: POST /v1/completions
B->>B: build_prompt()
B->>L: ollama.chat()
L-->>B: {content, thinking}
B-->>A: SSE stream
A-->>P: suggestion text
P->>E: insertGhostText()
E-->>U: 显示灰色建议
alt Tab 键
U->>P: Tab
P->>E: acceptSuggestion()
E-->>U: 建议变为正常文本
else Esc 键
U->>P: Esc
P->>E: rejectSuggestion()
E-->>U: 建议消失
else 继续输入
U->>E: 输入其他字符
E->>P: handleKeyDown()
P->>E: clearGhostText()
end
```
## 设计亮点
1. 前后端分离
2. 低延迟优化：防抖+SSE+AbortController
3. ProseMirror Mark系统
4. 多种交互方式
5. 智能大小限制
6. 隐私保护
7. 多语言支持
8. 主题定制
9. 文档处理
10. 语音功能
## 开发指南
代码风格: Python(4空格,snake_case) JS/TS(2空格,camelCase)
测试: pytest
构建: npm run build
### 运行测试
项目提供完整的测试套件，包括单元测试、集成测试和macOS环境模拟测试：
```bash
# 快速运行单元测试
python backend/tests/run_tests.py unit
# 运行集成测试（需要启动后端服务）
python backend/tests/run_tests.py integration
# 运行macOS环境模拟测试（在非Mac环境测试）
python backend/tests/run_tests.py simulate
# 运行所有测试
python backend/tests/run_tests.py all
```
详细测试说明请参考: [测试指南](backend/tests/TESTING_GUIDE.md)
1. **前后端分离**：前端只负责渲染和数据回传，后端负责 LLM 调用、Prompt 构建和数据解析
2. **低延迟优化**：防抖机制 (1000ms) + SSE 流式响应 + AbortController 取消过期请求
3. **ProseMirror Mark 系统**：与编辑器状态完美集成，支持 Undo/Redo
4. **多种交互方式**：Tab/Esc/点击/输入，用户体验友好
5. **智能大小限制**：文档超过 32KB 自动禁用 AI 功能
## 许可证
backend/.env.example
+4 -141
View File
@@ -1,141 +1,4 @@
# OpenAI-compatible endpoint
# In Docker, use host.docker.internal instead of localhost for a model service on the host.
LLM_BASE_URL=https://api.openai.com/v1/
LLM_API_KEY=sk-your-key
# Default model for inline completions
LLM_MODEL=Nex-N2-mini-mlx-OptiQ-8bit-MTP
# Pro-tier model (defaults to LLM_MODEL if unset)
PRO_LLM_MODEL=Nex-N2-mini-mlx-OptiQ-8bit-MTP
# Vision model for OCR
VLM_MODEL=Nex-N2-mini-mlx-OptiQ-8bit-MTP
# API key for the FastAPI app (change in production)
API_KEY=your-secret-key-here
# Browser origins allowed to send anonymous session cookies
CORS_ALLOW_ORIGINS=https://imageteach.tech,https://www.imageteach.tech,http://localhost:5173,http://127.0.0.1:5173
# Anonymous session cookie
SESSION_COOKIE_NAME=llm_anonymous_session
SESSION_COOKIE_SECURE=true
SESSION_COOKIE_SAMESITE=none
SESSION_COOKIE_DOMAIN=
SESSION_COOKIE_PATH=/
SESSION_COOKIE_MAX_AGE_SECONDS=2592000
SESSION_ROTATION_SECONDS=86400
# Job backend
JOB_BACKEND=redis
REDIS_URL=redis://localhost:6379/0
DATABASE_URL=postgresql://llm_in_text:llm_in_text_change_me@localhost:5432/llm_in_text
DOCS_BACKEND=postgres
JOB_REDIS_PREFIX=llmtext:jobs
JOB_CONSUMER_NAME=
JOB_SHARED_TEMP_DIR=/tmp/llm-in-text-jobs
JOB_STATE_TTL_SECONDS=600
JOB_EVENT_TTL_SECONDS=600
JOB_EVENT_STREAM_MAXLEN=512
JOB_CANCEL_POLL_SECONDS=0.5
JOB_BUSY_NORMAL_THRESHOLD=0.25
JOB_BUSY_HIGH_THRESHOLD=0.75
JOB_BUSY_FULL_THRESHOLD=1.0
# Per-queue concurrency and capacity
JOB_COMPLETION_CONCURRENCY=2
JOB_COMPLETION_MAX_QUEUE=16
JOB_PRO_COMPLETION_CONCURRENCY=1
JOB_PRO_COMPLETION_MAX_QUEUE=8
JOB_WEB_SEARCH_CONCURRENCY=1
JOB_WEB_SEARCH_MAX_QUEUE=4
JOB_COMPRESS_CONCURRENCY=1
JOB_COMPRESS_MAX_QUEUE=8
JOB_OCR_CONCURRENCY=1
JOB_OCR_MAX_QUEUE=8
JOB_CONVERT_CONCURRENCY=1
JOB_CONVERT_MAX_QUEUE=8
JOB_TTS_CONCURRENCY=4
JOB_TTS_MAX_QUEUE=16
JOB_ASR_CONCURRENCY=2
JOB_ASR_MAX_QUEUE=8
# Timeouts (seconds)
LLM_COMPLETION_TIMEOUT=600
LLM_OCR_TIMEOUT=600
# Compression limit
DOC_COMPRESS_CONTEXT_LIMIT=128000
# Risk control
RISK_API_WINDOW_SECONDS=60
RISK_API_SOFT_LIMIT=90
RISK_API_HARD_LIMIT=180
RISK_LLM_WINDOW_SECONDS=600
RISK_LLM_SOFT_LIMIT=8
RISK_LLM_HARD_LIMIT=16
RISK_SESSION_CONCURRENCY_LIMIT=2
RISK_GLOBAL_CONCURRENCY_LIMIT=12
RISK_DAILY_BUDGET_GLOBAL_USD=20
RISK_DAILY_BUDGET_SESSION_USD=2
RISK_DAILY_BUDGET_IP_USD=5
RISK_SINGLE_REQUEST_MAX_COST_USD=0.8
RISK_DELAY_STEP_MS=2500
RISK_DELAY_CAP_MS=30000
RISK_MODEL_CIRCUIT_FAILURES=8
RISK_MODEL_CIRCUIT_TTL_SECONDS=300
RISK_ENFORCE_REDIS_FAIL_CLOSED=false
# Backend-controlled model policy
RISK_COMPLETION_MODEL=Nex-N2-mini-mlx-OptiQ-8bit-MTP
RISK_PRO_MODEL=Nex-N2-mini-mlx-OptiQ-8bit-MTP
RISK_VISION_MODEL=Nex-N2-mini-mlx-OptiQ-8bit-MTP
RISK_WEB_SEARCH_MODEL=Nex-N2-mini-mlx-OptiQ-8bit-MTP
RISK_SPEECH_TTS_MODEL=Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit
RISK_SPEECH_ASR_MODEL=Qwen3-ASR-0.6B-8bit
RISK_COMPLETION_MAX_INPUT_CHARS=24000
RISK_COMPLETION_MAX_OUTPUT_TOKENS=768
RISK_COMPLETION_TEMPERATURE=0.4
RISK_PRO_MAX_INPUT_CHARS=48000
RISK_PRO_MAX_OUTPUT_TOKENS=2048
RISK_PRO_TEMPERATURE=0.6
RISK_WEB_SEARCH_MAX_INPUT_CHARS=128000
RISK_WEB_SEARCH_MAX_OUTPUT_TOKENS=4096
RISK_WEB_SEARCH_TEMPERATURE=0.4
RISK_COMPRESS_MAX_INPUT_CHARS=128000
RISK_COMPRESS_MAX_OUTPUT_TOKENS=1536
RISK_OCR_MAX_INPUT_BYTES=104857600
RISK_SPEECH_TTS_MAX_INPUT_CHARS=4096
RISK_SPEECH_ASR_MAX_INPUT_BYTES=104857600
# Web search providers
SEARXNG_BASE_URL=http://searxng:8080
SEARXNG_RESULT_LIMIT=10
FIRECRAWL_BASE_URL=http://firecrawl:3002
FIRECRAWL_API_KEY=change-me
WEB_SEARCH_QUERY_COUNT=4
WEB_SEARCH_SELECTED_URL_LIMIT=10
# Estimated pricing for budget control
RISK_COMPLETION_INPUT_COST_PER_1K=0.0004
RISK_COMPLETION_OUTPUT_COST_PER_1K=0.0016
RISK_PRO_INPUT_COST_PER_1K=0.003
RISK_PRO_OUTPUT_COST_PER_1K=0.012
RISK_VISION_INPUT_COST_PER_1K=0.0008
RISK_VISION_OUTPUT_COST_PER_1K=0.0024
RISK_SPEECH_TTS_INPUT_COST_PER_1K_CHARS=0
RISK_SPEECH_TTS_OUTPUT_COST_PER_MINUTE_AUDIO=0
RISK_SPEECH_ASR_INPUT_COST_PER_MB=0
# Shared speech API settings (uses LLM_BASE_URL + LLM_API_KEY)
TTS_MODEL_ID=Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit
TTS_DEFAULT_INSTRUCTIONS=A clear, natural voice speaking Mandarin Chinese.
ASR_MODEL_ID=Qwen3-ASR-0.6B-8bit
TTS_ASR_MAX_TEXT_CHARS=4096
ASR_MAX_AUDIO_BYTES=104857600
TTS_ASR_TTS_TIMEOUT_SECONDS=180
TTS_ASR_ASR_TIMEOUT_SECONDS=300
TTS_ASR_HEALTHCHECK_TIMEOUT_SECONDS=5
TTS_ASR_MAX_CONNECTIONS=24
TTS_ASR_MAX_KEEPALIVE_CONNECTIONS=12
OPENAI_API_KEY=ollama
OLLAMA_BASE_URL=http://192.168.0.120:11434/v1/
OLLAMA_MODEL=gpt-oss:20b
VLM_MODEL=qwen3-vl:30b
backend/AGENTS.md
-186
View File
@@ -1,186 +0,0 @@
# Backend 后端指引 (v0.2.0)
本文件适用于 backend/ 下的后端实现。进入 backend/tests/ 后，以子目录 AGENTS.md 为准。
## 后端职责
- 对外提供补全、取消补全、OCR、文档转换和 TTS/ASR 相关接口。
- 组织 Prompt，上下文清洗，调用 OpenAI-compatible 模型接口。
- **通过 Redis Streams 异步任务队列处理各类作业（completion/PRO/web_search/compress/OCR/convert/TTS/ASR）。**
- 负责 API Key 校验、日志记录和队列任务路由。
## 先看哪里
- API 入口和路由：main.py
- **任务队列系统**：job_system.py（Redis Streams 异步任务管理）
- **Worker 进程入口**：worker.py（消费 Redis Streams 的独立 worker）
- **任务处理器注册表**：job_handlers.py（completion/PRO/web_search/compress/OCR/convert/TTS/ASR 处理器）
- **会话管理**：session_store.py（内存 + PostgreSQL 双后端）
- **API/LLM 审计日志**：audit_store.py（PostgreSQL 持久化）
- **风控配置**：risk_config.py（环境变量驱动的配置数据类）
- **风控引擎**：risk_control.py（速率限制、并发控制、熔断器、预算追踪）
- **验证码路由**：captcha_api.py（FastAPI router）
- **文档存储**：docs_store.py（MIME 类型检测、文本/二进制分类）
- **LLM 策略解析**：llm_policy.py（按 job_type 解析模型配置）
- Ollama 调用封装：llm.py
- Prompt 清洗和拼装：prompt.py
- 数据模型：models.py
- 地理位置：geoip.py
- TTS 路由：tts_asr.py
- Prompt 模板：prompts/
- 后端测试：tests/
## 当前接口面
- POST /v1/completions
- POST /v1/completions/cancel
- POST /v1/ocr
- POST /v1/convert
- **POST /v1/web-search** - 网页搜索（SearXNG + Firecrawl）
- **POST /v1/compress** - 文本压缩（用于 OCR/文档转换后的大段文本）
- **POST /captcha/generate** - 验证码生成
- **POST /captcha/verify** - 验证码验证
- /v1/tts-asr/* 由 tts_asr.py 延迟注册
## 请求流转
### /v1/completions
- 读取或生成 request_id。
- privacy_mode 为 false 时，尝试根据客户端 IP 生成 location 文本。
- 调用 prepare_prompt_context 清洗 prefix 和 suffix。
- 调用 build_completion_prompts 生成 system_prompt 和 user_prompt。
- **通过 job_system.py 提交到 Redis Streams 队列，由 worker.py 消费。**
- **成功时返回 JSON：content 和 request_id。**
### /v1/completions/cancel
- 通过 request_id 在 ACTIVE_COMPLETIONS 中查找任务。
- 未找到返回 not_found。
- 已完成返回 already_done。
- 仍在执行则调用 task.cancel() 并返回 ok。
### /v1/ocr
- 把 base64 媒体内容解码成字节。
- 支持 `media_type=image|video` 和 `mime_type`。
- 图片直接调用 call_vlm_ocr。
- 视频先把整段视频送入 OCR 模型，再用 ffmpeg 抽取音轨交给 ASR，最后合并文本。
- **结果通过 job_handlers.py ocr_handler 处理。**
- 返回识别文本、原始文件名，以及视频场景下的 `ocr_text` / `asr_text`。
### /v1/convert
- 接收 base64 文件内容和文件名。
- 当前允许的扩展名只有 txt、docx、pptx、pdf。
- txt 直接解码后清洗。
- 其他格式写入临时文件，用 MarkItDown 转换，再做 Markdown 清洗。
- **结果通过 job_handlers.py convert_handler 处理。**
- 清洗逻辑会移除图片 Markdown 和 img HTML 标签，并压缩多余空行。
### /v1/web-search (新增)
- 接收搜索查询词和可选的引擎列表。
- **通过 job_handlers.py web_search_handler 处理。**
- **调用 SearXNG 搜索 API，再通过 Firecrawl 抓取页面内容。**
- **返回结构化的搜索结果（标题、链接、摘要）。**
### /v1/compress (新增)
- 接收长文本（OCR 结果、文档转换结果等）。
- **通过 job_handlers.py compress_handler 处理。**
- **调用 LLM 进行文本压缩，保留关键信息，移除冗余内容。**
- **返回压缩后的文本和压缩率统计。**
### /captcha/generate (新增)
- 生成随机验证码图片（4-6 位字母数字组合）。
- **通过 captcha_api.py captcha_generate_handler 处理。**
- **返回 base64 编码的验证码图片和明文答案。**
### /captcha/verify (新增)
- 验证用户输入的验证码是否正确。
- **通过 captcha_api.py captcha_verify_handler 处理。**
- **比对用户输入与存储的验证码答案，返回验证结果和错误信息（如有）。**
### /v1/tts-asr/*
- 通过 _register_tts_asr_routes 延迟导入并挂到主应用。
- **TTS 请求通过 job_handlers.py tts_handler 处理。**
- **ASR 请求通过 job_handlers.py asr_handler 处理。**
- **当前实现统一通过 `LLM_BASE_URL` + `LLM_API_KEY` 调用共享 Speech API，默认模型为 `Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit` 和 `Qwen3-ASR-0.6B-8bit`。**
## 开发命令
- 安装依赖：pip install -r backend/requirements.txt
- **Docker 部署**：docker compose up -d --build（在 /Users/allenyuan/lit/llm-in-text/ 目录下执行）
- 启动：python backend/main.py
- **Worker 进程**：python backend/worker.py（独立运行，消费 Redis Streams）
- 开发启动：uvicorn backend.main:app --reload --port 8001
- 路由相关测试：
- pytest backend/tests/test_main_endpoints.py -v
- pytest backend/tests/test_main_cancel.py -v
- Prompt 测试：
- pytest backend/tests/test_prompt.py -v
- pytest backend/tests/test_prompt_extended.py -v
- LLM 测试：
- pytest backend/tests/test_llm.py -v
- pytest backend/tests/test_llm_extended.py -v
- **新增测试**：
- pytest backend/tests/test_web_search.py -v（网页搜索功能）
- pytest backend/tests/test_compress.py -v（文本压缩功能）
## 编码约定
- Python 使用 4 空格缩进。
- 函数、变量使用 snake_case，类使用 PascalCase。
- **新逻辑优先保留显式类型和明确的输入输出。**
- **异步边界要清晰；阻塞操作优先放进 asyncio.to_thread，而不是直接阻塞事件循环。**
- **异常要么转成 HTTPException，要么转成结构化 JSONResponse；不要静默吞掉后端错误。**
- **日志尽量带 request_id 或短 tag，便于把前后端一次请求串起来。**
## 容易误判的点
- **任务队列架构（v0.2.0 新增）**：后端从同步端点转向 Redis Streams 异步队列。job_system.py 定义 JOB_TYPES，worker.py 消费队列，job_handlers.py 注册各类型处理器。
- **会话追踪**：session_store.py 提供 InMemorySessionStore（开发）和 PostgresSessionStore（生产），通过 session_hash + ip_hash 追踪请求身份。
- **风控系统**：risk_config.py + risk_control.py 实现速率限制（滑动窗口）、并发控制、熔断器模式和预算追踪，所有阈值通过环境变量配置。
- **验证码路由**：captcha_api.py 提供 /captcha/generate 和 /captcha/verify 端点，用于前端验证用户输入。
- **文档存储**：docs_store.py 提供 MIME 类型检测、文本/二进制分类和预览提取（8MB 限制）。
- **LLM 策略解析**：llm_policy.py 按 job_type 解析模型配置（模型名、温度、最大 token、思考级别）。
- **OCR 明确关闭思考**：llm.py 的 `call_vlm_ocr` 对 OCR 请求显式设置 `options.think = False` 和 `temperature = 0`。
- **补全接口当前不是流式响应，不要按 SSE 方式改造周边代码。**
- **ACTIVE_COMPLETIONS 在补全和取消路径里都被读写，任务生命周期要谨慎处理。**
- **`/v1/export/pdf` 不是当前主链路**；DOCX/PDF 导出已经转到前端 `src/utils/richExport.js`。
- **API_KEY 存在占位默认值，这更像本地开发兜底，不是推荐的安全模式。**
- **历史 TTS/ASR 文档和部分测试覆盖的是旧实现；代码与文档冲突时，先确认产品方向，再决定修代码还是修文档。**
## 改动时的定位建议
- **如果问题是补全结果不对，先查 prompt.py，再查 llm.py，不要只盯着 main.py。**
- **如果问题是取消不生效，先查 main.py 里的 request_id 生命周期，再对照前端的 X-Request-Id 和 cancel 调用。**
- **如果问题是 OCR 识别为空，先看 main.py 的 base64 解码，再看 llm.py 的 call_vlm_ocr。**
- **如果问题是转换结果脏，重点看 main.py 里的 _sanitize_converted_markdown。**
- **如果问题是 TTS 行为和文档不一致，以 tts_asr.py 为准，不要以 README 为准。**
- **如果问题是 Web Search 失败，检查 SearXNG 和 Firecrawl 服务状态（docker compose ps）。**
- **如果问题是任务队列积压，检查 worker.py 日志和 Redis Streams 长度（INFO keys=stream）。**
- **如果问题是会话丢失，检查 session_store.py 的 InMemory/Postgres 后端切换逻辑。**
- **如果问题是风控触发，检查 risk_config.py 的环境变量配置和风险阈值。**
## 测试映射
- **路由主行为**：tests/test_main_endpoints.py
- **取消逻辑**：tests/test_main_cancel.py
- **Prompt 逻辑**：tests/test_prompt.py、tests/test_prompt_extended.py
- **LLM 包装层**：tests/test_llm.py、tests/test_llm_extended.py
- **GeoIP**：tests/test_geoip.py
- **TTS 相关**：tests/test_tts_asr_*.py
- **网页搜索**：tests/test_web_search.py（新增）
- **文本压缩**：tests/test_compress.py（新增）
## 文档使用原则
- README.md、TTS_ASR_MACOS_FIX.md、tests/TESTING_GUIDE.md 可以作为背景材料。
- **一旦这些文档和 main.py、llm.py、prompt.py、tts_asr.py 冲突，以代码为准。**
- **新增模块（job_system、worker、job_handlers、session_store、audit_store、risk_config、risk_control、captcha_api、docs_store、llm_policy）的文档需与代码保持同步。**
- **Docker 部署相关文档（docker-compose.yml、backend/Dockerfile、Dockerfile.frontend、.dockerignore）需与 docker/ 目录下的配置保持一致。**
backend/Dockerfile
-22
View File
@@ -1,22 +0,0 @@
ARG DOCKER_REGISTRY_PREFIX=
FROM ${DOCKER_REGISTRY_PREFIX}python:3.11-slim
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /app/backend
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg
COPY backend/requirements.docker.txt /tmp/requirements.docker.txt
RUN --mount=type=cache,target=/root/.cache/pip \
pip install -r /tmp/requirements.docker.txt
COPY backend /app/backend
EXPOSE 8001
CMD ["python", "main.py"]
backend/__init__.py
-1
View File
@@ -1 +0,0 @@
"""Backend package marker for tests and patch targets."""
backend/audit_store.py
-251
View File
@@ -1,251 +0,0 @@
import json
import os
import threading
from typing import Any
try:
import psycopg
except Exception: # pragma: no cover
psycopg = None
class BaseAuditStore:
def record_api_request(self, payload: dict[str, Any]) -> None:
return None
def record_llm_call(self, payload: dict[str, Any]) -> None:
return None
def upsert_daily_usage(self, payload: dict[str, Any]) -> None:
return None
class NullAuditStore(BaseAuditStore):
pass
class PostgresAuditStore(BaseAuditStore):
def __init__(self, database_url: str) -> None:
if psycopg is None:
raise RuntimeError("psycopg 未安装，无法使用 PostgreSQL 审计存储")
self.database_url = database_url
self._init_lock = threading.Lock()
self._initialized = False
def _connect(self):
return psycopg.connect(self.database_url, autocommit=True)
def _ensure_initialized(self) -> None:
if self._initialized:
return
with self._init_lock:
if self._initialized:
return
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
CREATE TABLE IF NOT EXISTS api_request_audit (
id BIGSERIAL PRIMARY KEY,
request_id TEXT NOT NULL,
session_hash TEXT NOT NULL,
ip_hash TEXT NOT NULL,
route TEXT NOT NULL,
method TEXT NOT NULL,
status_code INTEGER NOT NULL,
decision TEXT NOT NULL,
delay_ms INTEGER NOT NULL DEFAULT 0,
queue_ms INTEGER NOT NULL DEFAULT 0,
error_code TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb
)
"""
)
cur.execute(
"""
CREATE TABLE IF NOT EXISTS llm_call_audit (
id BIGSERIAL PRIMARY KEY,
request_id TEXT NOT NULL,
session_hash TEXT NOT NULL,
ip_hash TEXT NOT NULL,
job_type TEXT NOT NULL,
model TEXT NOT NULL,
estimated_input_tokens INTEGER NOT NULL DEFAULT 0,
max_output_tokens INTEGER NOT NULL DEFAULT 0,
estimated_cost NUMERIC(18, 8) NOT NULL DEFAULT 0,
actual_output_chars INTEGER NOT NULL DEFAULT 0,
actual_cost NUMERIC(18, 8) NOT NULL DEFAULT 0,
queue_ms INTEGER NOT NULL DEFAULT 0,
run_ms INTEGER NOT NULL DEFAULT 0,
total_ms INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL,
error_code TEXT NOT NULL DEFAULT '',
started_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
finished_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb
)
"""
)
cur.execute(
"ALTER TABLE llm_call_audit ADD COLUMN IF NOT EXISTS queue_ms INTEGER NOT NULL DEFAULT 0"
)
cur.execute(
"ALTER TABLE llm_call_audit ADD COLUMN IF NOT EXISTS run_ms INTEGER NOT NULL DEFAULT 0"
)
cur.execute(
"ALTER TABLE llm_call_audit ADD COLUMN IF NOT EXISTS total_ms INTEGER NOT NULL DEFAULT 0"
)
cur.execute(
"""
CREATE TABLE IF NOT EXISTS risk_events (
id BIGSERIAL PRIMARY KEY,
session_hash TEXT NOT NULL,
ip_hash TEXT NOT NULL,
event_type TEXT NOT NULL,
severity TEXT NOT NULL,
reason TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb
)
"""
)
cur.execute(
"""
CREATE TABLE IF NOT EXISTS daily_budget_usage (
usage_day DATE NOT NULL,
scope TEXT NOT NULL,
scope_hash TEXT NOT NULL,
estimated_cost NUMERIC(18, 8) NOT NULL DEFAULT 0,
actual_cost NUMERIC(18, 8) NOT NULL DEFAULT 0,
request_count INTEGER NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (usage_day, scope, scope_hash)
)
"""
)
cur.execute(
"CREATE INDEX IF NOT EXISTS idx_api_request_audit_request_id ON api_request_audit (request_id)"
)
cur.execute(
"CREATE INDEX IF NOT EXISTS idx_api_request_audit_route_created_at ON api_request_audit (route, created_at DESC)"
)
cur.execute(
"CREATE INDEX IF NOT EXISTS idx_llm_call_audit_request_id ON llm_call_audit (request_id)"
)
cur.execute(
"CREATE INDEX IF NOT EXISTS idx_llm_call_audit_job_type_started_at ON llm_call_audit (job_type, started_at DESC)"
)
cur.execute(
"CREATE INDEX IF NOT EXISTS idx_llm_call_audit_model_started_at ON llm_call_audit (model, started_at DESC)"
)
self._initialized = True
def record_api_request(self, payload: dict[str, Any]) -> None:
self._ensure_initialized()
metadata = payload.get("metadata") or {}
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO api_request_audit (
request_id, session_hash, ip_hash, route, method, status_code,
decision, delay_ms, queue_ms, error_code, metadata_json
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s::jsonb)
""",
(
payload["request_id"],
payload["session_hash"],
payload["ip_hash"],
payload["route"],
payload["method"],
int(payload["status_code"]),
payload["decision"],
int(payload.get("delay_ms", 0)),
int(payload.get("queue_ms", 0)),
payload.get("error_code", ""),
json.dumps(metadata, ensure_ascii=False),
),
)
def record_llm_call(self, payload: dict[str, Any]) -> None:
self._ensure_initialized()
metadata = payload.get("metadata") or {}
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO llm_call_audit (
request_id, session_hash, ip_hash, job_type, model,
estimated_input_tokens, max_output_tokens, estimated_cost,
actual_output_chars, actual_cost, queue_ms, run_ms, total_ms,
status, error_code, metadata_json
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s::jsonb)
""",
(
payload["request_id"],
payload["session_hash"],
payload["ip_hash"],
payload["job_type"],
payload["model"],
int(payload.get("estimated_input_tokens", 0)),
int(payload.get("max_output_tokens", 0)),
float(payload.get("estimated_cost", 0.0)),
int(payload.get("actual_output_chars", 0)),
float(payload.get("actual_cost", 0.0)),
int(payload.get("queue_ms", 0)),
int(payload.get("run_ms", 0)),
int(payload.get("total_ms", 0)),
payload["status"],
payload.get("error_code", ""),
json.dumps(metadata, ensure_ascii=False),
),
)
def upsert_daily_usage(self, payload: dict[str, Any]) -> None:
self._ensure_initialized()
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO daily_budget_usage (
usage_day, scope, scope_hash, estimated_cost, actual_cost, request_count
)
VALUES (%s, %s, %s, %s, %s, %s)
ON CONFLICT (usage_day, scope, scope_hash)
DO UPDATE SET
estimated_cost = daily_budget_usage.estimated_cost + EXCLUDED.estimated_cost,
actual_cost = daily_budget_usage.actual_cost + EXCLUDED.actual_cost,
request_count = daily_budget_usage.request_count + EXCLUDED.request_count,
updated_at = CURRENT_TIMESTAMP
""",
(
payload["usage_day"],
payload["scope"],
payload["scope_hash"],
float(payload.get("estimated_cost", 0.0)),
float(payload.get("actual_cost", 0.0)),
int(payload.get("request_count", 0)),
),
)
_audit_store: BaseAuditStore | None = None
def get_audit_store(database_url: str | None = None) -> BaseAuditStore:
global _audit_store
if _audit_store is not None:
return _audit_store
if database_url or os.getenv("DATABASE_URL"):
_audit_store = PostgresAuditStore(database_url or os.getenv("DATABASE_URL", ""))
else:
_audit_store = NullAuditStore()
return _audit_store
def reset_audit_store() -> None:
global _audit_store
_audit_store = None
backend/captcha_api.py
-256
View File
@@ -1,256 +0,0 @@
"""
验证码和 Cookie 策略管理模块
提供功能:
1. 图形验证码生成与验证 API
2. 现代 Cookie 策略管理 (HttpOnly, Secure, SameSite)
3. 验证码结果持久化到 Cookie
"""
import random
import string
import json
from typing import Optional
from fastapi import APIRouter, HTTPException, Request, Response
from fastapi.responses import JSONResponse
router = APIRouter(prefix="/captcha", tags=["验证码"])
# ==================== 数据模型 ====================
class CaptchaConfig:
"""验证码配置"""
LENGTH = 6 # 验证码长度
CHARSET = string.ascii_letters + string.digits # 字符集: 大小写字母+数字
EXPIRE_SECONDS = 3600 # 过期时间: 1小时
class CaptchaResult:
"""验证码结果"""
def __init__(self, text: str):
self.text = text
self.created_at = int(__import__('time').time())
@property
def is_expired(self) -> bool:
now = int(__import__('time').time())
return (now - self.created_at) > CaptchaConfig.EXPIRE_SECONDS
# ==================== 全局状态 ====================
# 内存中的验证码存储 (生产环境建议用 Redis)
_active_captchas: dict[str, CaptchaResult] = {}
# ==================== 验证码 API ====================
@router.get("/generate", summary="生成新验证码")
async def generate_captcha(
response: Response,
use_cookie: bool = False, # 是否通过 Cookie 传递验证码文本
length: int = CaptchaConfig.LENGTH,
):
"""
生成新的验证码
- **use_cookie**: 是否同时设置 Cookie (方便前端读取)
- **length**: 验证码长度 (4-10)
返回:
- **request_id**: 验证码请求 ID
- **expires_in**: 过期时间(秒)
"""
# 生成随机字符串
chars = CaptchaConfig.CHARSET
captcha_text = ''.join(random.choices(chars, k=length))
# 存储到内存
request_id = f"captcha_{int(__import__('time').time() * 1000)}"
_active_captchas[request_id] = CaptchaResult(captcha_text)
# 如果请求使用 Cookie，设置 HttpOnly Cookie
if use_cookie:
response.set_cookie(
key="llm_captcha_text",
value=captcha_text,
max_age=CaptchaConfig.EXPIRE_SECONDS,
httponly=False, # 允许前端读取
secure=False, # HTTP/HTTPS 都适用
samesite="Lax", # 防止 CSRF
domain=".imageteach.tech",
path="/"
)
return {
"request_id": request_id,
"expires_in": CaptchaConfig.EXPIRE_SECONDS,
"cookie_set": use_cookie
}
@router.post("/validate", summary="验证用户输入的验证码")
async def validate_captcha(
request: Request,
user_input: str,
request_id: Optional[str] = None,
):
"""
验证用户输入的验证码
- **user_input**: 用户输入的验证码文本
- **request_id**: 可选，指定验证哪个验证码
返回:
- **is_valid**: 是否验证成功
- **submitted**: 用户提交的文本
"""
if not user_input:
raise HTTPException(status_code=400, detail="缺少验证码输入")
# 从请求头或 Cookie 获取 request_id
rid = request_id or request.headers.get("X-Captcha-Request-Id")
if not rid or rid not in _active_captchas:
raise HTTPException(
status_code=404,
detail="未找到验证码，请先生成"
)
captcha_result = _active_captchas[rid]
# 检查是否过期
if captcha_result.is_expired:
del _active_captchas[rid]
raise HTTPException(
status_code=410, # Gone
detail="验证码已过期，请重新生成"
)
# 不区分大小写比较
is_valid = captcha_result.text.lower() == user_input.strip().lower()
# 验证成功后删除该验证码 (一次性使用)
if is_valid:
del _active_captchas[rid]
return {
"is_valid": is_valid,
"submitted": user_input,
"matched": is_valid
}
@router.delete("/clear", summary="清除验证码 Cookie")
async def clear_captcha_cookie(response: Response):
"""清除所有验证码相关的 Cookie"""
response.delete_cookie(key="llm_captcha_text")
response.delete_cookie(key="llm_captcha_result")
return {"message": "验证码 Cookie 已清除"}
# ==================== Cookie 策略工具类 ====================
class CookiePolicy:
"""
现代 Cookie 策略管理器
支持的属性:
- **HttpOnly**: 防止 XSS 读取 Cookie
- **Secure**: 仅 HTTPS 传输 (当前设为 False 以支持 HTTP)
- **SameSite**: Lax/Strict/None (控制跨域行为)
- **Domain**: 指定域名 (.imageteach.tech)
- **Path**: 路径 (/)
- **Max-Age**: 过期时间 (秒)
"""
# 默认 Cookie 配置
DEFAULT_CONFIG = {
"llm_session": {
"max_age": 86400 * 7, # 7天
"httponly": True, # 防止 XSS
"secure": False, # HTTP/HTTPS 都适用
"samesite": "Lax", # 防止 CSRF
"domain": ".imageteach.tech",
"path": "/"
},
"llm_captcha": {
"max_age": 3600, # 1小时
"httponly": False,
"secure": False,
"samesite": "Lax",
"domain": ".imageteach.tech",
"path": "/"
},
"llm_preferences": {
"max_age": 86400 * 30, # 30天
"httponly": False,
"secure": True,
"samesite": "None", # 跨域场景
"domain": ".imageteach.tech",
"path": "/"
}
}
@classmethod
def set_cookie(cls, response: Response, name: str, value: str, override: dict = None):
"""
设置 Cookie
Args:
response: FastAPI Response 对象
name: Cookie 名称
value: Cookie 值
override: 可选的覆盖配置
"""
config = cls.DEFAULT_CONFIG.get(name, {})
if override:
config.update(override)
response.set_cookie(
key=name,
value=value,
max_age=config.get("max_age", 3600),
httponly=config.get("httponly", False),
secure=config.get("secure", False),
samesite=config.get("samesite", "Lax"),
domain=config.get("domain", ".imageteach.tech"),
path=config.get("path", "/")
)
@classmethod
def get_cookie_config(cls, name: str) -> dict:
"""获取 Cookie 配置"""
return cls.DEFAULT_CONFIG.get(name, {})
# ==================== 前端可用的 API ====================
@router.get("/cookies/list", summary="列出所有验证码相关 Cookie")
async def list_captcha_cookies(request: Request):
"""返回当前请求携带的所有验证码相关 Cookie"""
cookies = {
k: v for k, v in request.cookies.items()
if k.startswith("llm_")
}
return {
"cookies": cookies,
"has_captcha": "llm_captcha_text" in cookies,
"has_session": "llm_session" in cookies
}
@router.post("/cookies/set", summary="设置测试 Cookie")
async def set_test_cookie(
response: Response,
cookie_name: str = "llm_test",
cookie_value: str = "test_value"
):
"""设置一个测试用的 Cookie"""
CookiePolicy.set_cookie(response, cookie_name, cookie_value)
return {
"message": f"Cookie '{cookie_name}' 已设置",
"name": cookie_name,
"value": cookie_value
}
backend/docs_store.py
-668
View File
@@ -1,668 +0,0 @@
import mimetypes
import os
import threading
import uuid
from dataclasses import dataclass
from typing import Any
try: # pragma: no cover - optional in some test paths
import psycopg
from psycopg.rows import dict_row
except Exception: # pragma: no cover
psycopg = None
dict_row = None
MAX_TEXT_SIZE = 8 * 1024 * 1024
PREVIEW_TEXT_SIZE = 2 * 1024 * 1024
TEXT_EXTENSIONS = {
"md", "markdown", "txt", "json", "js", "jsx", "ts", "tsx",
"css", "scss", "less", "html", "htm", "py", "vue", "xml",
"yaml", "yml", "csv", "log", "sql", "toml", "ini", "cfg",
"conf", "sh", "bat", "ps1", "java", "c", "cpp", "h", "hpp",
"go", "rs", "swift", "kt", "rb", "php", "pl", "r", "scala",
"gradle", "properties", "env", "gitignore", "dockerfile",
}
BINARY_EXTENSIONS = {
"exe", "dll", "so", "dylib", "bin", "dat", "obj", "o", "a",
"doc", "docx", "xls", "xlsx", "ppt", "pptx", "odt", "ods", "odp",
"pdf", "zip", "rar", "7z", "tar", "gz", "bz2", "xz",
"png", "jpg", "jpeg", "gif", "bmp", "ico", "webp", "svg",
"mp3", "mp4", "wav", "avi", "mov", "mkv", "flv", "wmv",
"ttf", "otf", "woff", "woff2", "eot",
"class", "pyc", "pyo", "jar", "war", "ear",
"db", "sqlite", "mdb", "accdb",
"pem", "key", "crt", "cer", "p12", "pfx", "jks",
"msg", "eml", "pst", "ost",
"dwg", "dxf", "step", "stl", "fbx", "3ds", "blend",
}
DEFAULT_MIME_TYPES = {
"avi": "video/x-msvideo",
"md": "text/markdown",
"markdown": "text/markdown",
"mkv": "video/x-matroska",
"mov": "video/quicktime",
"mp4": "video/mp4",
"txt": "text/plain",
"json": "application/json",
"js": "text/javascript",
"jsx": "text/javascript",
"ts": "text/typescript",
"tsx": "text/typescript",
"css": "text/css",
"html": "text/html",
"htm": "text/html",
"py": "text/x-python",
"vue": "text/plain",
"xml": "application/xml",
"yaml": "text/yaml",
"yml": "text/yaml",
"csv": "text/csv",
"log": "text/plain",
"sql": "text/plain",
"toml": "text/plain",
"ini": "text/plain",
"cfg": "text/plain",
"conf": "text/plain",
"sh": "text/plain",
"bat": "text/plain",
"ps1": "text/plain",
"jpg": "image/jpeg",
"jpeg": "image/jpeg",
"flv": "video/x-flv",
"m4v": "video/x-m4v",
"png": "image/png",
"gif": "image/gif",
"webp": "image/webp",
"svg": "image/svg+xml",
"pdf": "application/pdf",
"ogg": "video/ogg",
"ogv": "video/ogg",
"webm": "video/webm",
"wmv": "video/x-ms-wmv",
}
def _now_sql() -> str:
return "CURRENT_TIMESTAMP"
def get_extension(name: str = "") -> str:
value = str(name or "")
if "." not in value:
return value.lower() if value.lower() == "dockerfile" else ""
return value.rsplit(".", 1)[-1].lower()
def infer_mime_type(name: str, fallback: str = "") -> str:
if fallback:
return fallback
ext = get_extension(name)
guessed = DEFAULT_MIME_TYPES.get(ext)
if guessed:
return guessed
guessed, _ = mimetypes.guess_type(name)
return guessed or "application/octet-stream"
def is_text_file(name: str, mime_type: str = "") -> bool:
ext = get_extension(name)
if ext in BINARY_EXTENSIONS:
return False
mime_value = str(mime_type or "").lower()
return ext in TEXT_EXTENSIONS or mime_value.startswith("text/") or "json" in mime_value or "xml" in mime_value
def _text_payload(text: str) -> dict[str, Any]:
encoded = text.encode("utf-8")
preview = encoded[:PREVIEW_TEXT_SIZE].decode("utf-8", errors="ignore")
return {
"storage_kind": "text",
"size": len(encoded),
"content_text": text,
"preview_text": preview if len(encoded) > PREVIEW_TEXT_SIZE else text,
"is_truncated_preview": len(encoded) > PREVIEW_TEXT_SIZE,
"blob_data": None,
}
def prepare_file_payload(name: str, raw_bytes: bytes, mime_type: str = "") -> dict[str, Any]:
resolved_mime = infer_mime_type(name, mime_type)
if is_text_file(name, resolved_mime):
text = raw_bytes.decode("utf-8", errors="ignore")
return {
"mime_type": resolved_mime,
**_text_payload(text),
}
return {
"mime_type": resolved_mime,
"storage_kind": "blob",
"size": len(raw_bytes),
"content_text": "",
"preview_text": "",
"is_truncated_preview": False,
"blob_data": raw_bytes,
}
@dataclass
class BlobPayload:
content: bytes
mime_type: str
filename: str
class BaseDocumentStore:
_UNSET = object()
def list_nodes(self) -> list[dict[str, Any]]:
raise NotImplementedError
def create_folder(self, name: str, parent_id: str | None) -> dict[str, Any]:
raise NotImplementedError
def create_text_file(self, name: str, parent_id: str | None, content: str = "") -> dict[str, Any]:
raise NotImplementedError
def upload_file(self, name: str, parent_id: str | None, raw_bytes: bytes, mime_type: str = "") -> dict[str, Any]:
raise NotImplementedError
def update_node(self, node_id: str, *, name: str | None | object = _UNSET, parent_id: str | None | object = _UNSET, content: str | None | object = _UNSET) -> dict[str, Any]:
raise NotImplementedError
def replace_blob(self, node_id: str, filename: str, raw_bytes: bytes, mime_type: str = "") -> dict[str, Any]:
raise NotImplementedError
def delete_node(self, node_id: str) -> None:
raise NotImplementedError
def get_blob(self, node_id: str) -> BlobPayload:
raise NotImplementedError
class InMemoryDocumentStore(BaseDocumentStore):
def __init__(self) -> None:
self.nodes: dict[str, dict[str, Any]] = {}
def _serialize(self, node: dict[str, Any]) -> dict[str, Any]:
return {
"id": node["id"],
"parentId": node["parent_id"],
"type": node["type"],
"name": node["name"],
"mimeType": node.get("mime_type") or "",
"storageKind": node.get("storage_kind") or "text",
"size": int(node.get("size") or 0),
"content": node.get("content_text") or "",
"previewText": node.get("preview_text") or "",
"isTruncatedPreview": bool(node.get("is_truncated_preview")),
"createdAt": int(node["created_at"]),
"updatedAt": int(node["updated_at"]),
}
def _timestamp(self) -> int:
import time
return int(time.time() * 1000)
def _get(self, node_id: str) -> dict[str, Any]:
node = self.nodes.get(node_id)
if node is None:
raise KeyError(node_id)
return node
def list_nodes(self) -> list[dict[str, Any]]:
ordered = sorted(
self.nodes.values(),
key=lambda item: (item["type"] != "folder", item["name"].lower(), item["created_at"]),
)
return [self._serialize(node) for node in ordered]
def create_folder(self, name: str, parent_id: str | None) -> dict[str, Any]:
now = self._timestamp()
node = {
"id": str(uuid.uuid4()),
"parent_id": parent_id,
"type": "folder",
"name": name,
"mime_type": "",
"storage_kind": "text",
"size": 0,
"content_text": "",
"preview_text": "",
"is_truncated_preview": False,
"blob_data": None,
"created_at": now,
"updated_at": now,
}
self.nodes[node["id"]] = node
return self._serialize(node)
def create_text_file(self, name: str, parent_id: str | None, content: str = "") -> dict[str, Any]:
now = self._timestamp()
payload = _text_payload(content)
node = {
"id": str(uuid.uuid4()),
"parent_id": parent_id,
"type": "file",
"name": name,
"mime_type": infer_mime_type(name),
**payload,
"created_at": now,
"updated_at": now,
}
self.nodes[node["id"]] = node
return self._serialize(node)
def upload_file(self, name: str, parent_id: str | None, raw_bytes: bytes, mime_type: str = "") -> dict[str, Any]:
now = self._timestamp()
payload = prepare_file_payload(name, raw_bytes, mime_type)
node = {
"id": str(uuid.uuid4()),
"parent_id": parent_id,
"type": "file",
"name": name,
**payload,
"created_at": now,
"updated_at": now,
}
self.nodes[node["id"]] = node
return self._serialize(node)
def update_node(self, node_id: str, *, name: str | None | object = BaseDocumentStore._UNSET, parent_id: str | None | object = BaseDocumentStore._UNSET, content: str | None | object = BaseDocumentStore._UNSET) -> dict[str, Any]:
node = self._get(node_id)
if name is not BaseDocumentStore._UNSET:
node["name"] = name
if node["type"] == "file":
node["mime_type"] = infer_mime_type(name, node.get("mime_type") or "")
if parent_id is not BaseDocumentStore._UNSET:
node["parent_id"] = parent_id
if content is not BaseDocumentStore._UNSET:
payload = _text_payload(content)
node.update(payload)
node["updated_at"] = self._timestamp()
return self._serialize(node)
def replace_blob(self, node_id: str, filename: str, raw_bytes: bytes, mime_type: str = "") -> dict[str, Any]:
node = self._get(node_id)
payload = prepare_file_payload(filename, raw_bytes, mime_type)
node.update(payload)
node["name"] = filename
node["mime_type"] = payload["mime_type"]
node["updated_at"] = self._timestamp()
return self._serialize(node)
def delete_node(self, node_id: str) -> None:
descendants = {node_id}
changed = True
while changed:
changed = False
for node in list(self.nodes.values()):
if node.get("parent_id") in descendants and node["id"] not in descendants:
descendants.add(node["id"])
changed = True
for current_id in descendants:
self.nodes.pop(current_id, None)
def get_blob(self, node_id: str) -> BlobPayload:
node = self._get(node_id)
if node["type"] != "file":
raise FileNotFoundError(node_id)
if node.get("blob_data") is not None:
content = node["blob_data"]
else:
content = (node.get("content_text") or "").encode("utf-8")
return BlobPayload(
content=content,
mime_type=node.get("mime_type") or infer_mime_type(node.get("name") or ""),
filename=node["name"],
)
class PostgresDocumentStore(BaseDocumentStore):
def __init__(self, database_url: str) -> None:
if psycopg is None or dict_row is None:
raise RuntimeError("psycopg 未安装，无法使用 PostgreSQL 文档存储")
self.database_url = database_url
self._init_lock = threading.Lock()
self._initialized = False
def _connect(self):
return psycopg.connect(self.database_url, autocommit=True, row_factory=dict_row)
def _ensure_initialized(self) -> None:
if self._initialized:
return
with self._init_lock:
if self._initialized:
return
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
CREATE TABLE IF NOT EXISTS document_nodes (
id TEXT PRIMARY KEY,
parent_id TEXT REFERENCES document_nodes(id) ON DELETE CASCADE,
type TEXT NOT NULL CHECK (type IN ('folder', 'file')),
name TEXT NOT NULL,
mime_type TEXT NOT NULL DEFAULT '',
storage_kind TEXT NOT NULL DEFAULT 'text',
size BIGINT NOT NULL DEFAULT 0,
content_text TEXT NOT NULL DEFAULT '',
preview_text TEXT NOT NULL DEFAULT '',
is_truncated_preview BOOLEAN NOT NULL DEFAULT FALSE,
blob_data BYTEA,
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
)
"""
)
cur.execute(
"CREATE INDEX IF NOT EXISTS document_nodes_parent_idx ON document_nodes(parent_id)"
)
self._initialized = True
def _serialize(self, row: dict[str, Any]) -> dict[str, Any]:
created = row["created_at"]
updated = row["updated_at"]
return {
"id": row["id"],
"parentId": row["parent_id"],
"type": row["type"],
"name": row["name"],
"mimeType": row.get("mime_type") or "",
"storageKind": row.get("storage_kind") or "text",
"size": int(row.get("size") or 0),
"content": row.get("content_text") or "",
"previewText": row.get("preview_text") or "",
"isTruncatedPreview": bool(row.get("is_truncated_preview")),
"createdAt": int(created.timestamp() * 1000),
"updatedAt": int(updated.timestamp() * 1000),
}
def _fetch_node(self, node_id: str) -> dict[str, Any]:
self._ensure_initialized()
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute("SELECT * FROM document_nodes WHERE id = %s", (node_id,))
row = cur.fetchone()
if row is None:
raise KeyError(node_id)
return row
def list_nodes(self) -> list[dict[str, Any]]:
self._ensure_initialized()
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT
id,
parent_id,
type,
name,
mime_type,
storage_kind,
size,
CASE
WHEN storage_kind = 'text' AND size <= %s THEN content_text
ELSE ''
END AS content_text,
preview_text,
is_truncated_preview,
created_at,
updated_at
FROM document_nodes
ORDER BY
CASE WHEN type = 'folder' THEN 0 ELSE 1 END,
LOWER(name),
created_at
""",
(MAX_TEXT_SIZE,),
)
return [self._serialize(row) for row in cur.fetchall()]
def create_folder(self, name: str, parent_id: str | None) -> dict[str, Any]:
self._ensure_initialized()
node_id = str(uuid.uuid4())
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO document_nodes (id, parent_id, type, name, updated_at)
VALUES (%s, %s, 'folder', %s, CURRENT_TIMESTAMP)
RETURNING *
""",
(node_id, parent_id, name),
)
return self._serialize(cur.fetchone())
def create_text_file(self, name: str, parent_id: str | None, content: str = "") -> dict[str, Any]:
self._ensure_initialized()
node_id = str(uuid.uuid4())
payload = _text_payload(content)
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO document_nodes (
id, parent_id, type, name, mime_type, storage_kind, size,
content_text, preview_text, is_truncated_preview, blob_data, updated_at
)
VALUES (%s, %s, 'file', %s, %s, %s, %s, %s, %s, %s, %s, CURRENT_TIMESTAMP)
RETURNING *
""",
(
node_id,
parent_id,
name,
infer_mime_type(name),
payload["storage_kind"],
payload["size"],
payload["content_text"],
payload["preview_text"],
payload["is_truncated_preview"],
None,
),
)
return self._serialize(cur.fetchone())
def upload_file(self, name: str, parent_id: str | None, raw_bytes: bytes, mime_type: str = "") -> dict[str, Any]:
self._ensure_initialized()
node_id = str(uuid.uuid4())
payload = prepare_file_payload(name, raw_bytes, mime_type)
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO document_nodes (
id, parent_id, type, name, mime_type, storage_kind, size,
content_text, preview_text, is_truncated_preview, blob_data, updated_at
)
VALUES (%s, %s, 'file', %s, %s, %s, %s, %s, %s, %s, %s, CURRENT_TIMESTAMP)
RETURNING *
""",
(
node_id,
parent_id,
name,
payload["mime_type"],
payload["storage_kind"],
payload["size"],
payload["content_text"],
payload["preview_text"],
payload["is_truncated_preview"],
payload["blob_data"],
),
)
return self._serialize(cur.fetchone())
def update_node(self, node_id: str, *, name: str | None | object = BaseDocumentStore._UNSET, parent_id: str | None | object = BaseDocumentStore._UNSET, content: str | None | object = BaseDocumentStore._UNSET) -> dict[str, Any]:
self._ensure_initialized()
row = self._fetch_node(node_id)
next_name = row["name"] if name is BaseDocumentStore._UNSET else name
next_parent_id = row["parent_id"] if parent_id is BaseDocumentStore._UNSET else parent_id
next_mime_type = row.get("mime_type") or ""
next_storage_kind = row.get("storage_kind") or "text"
next_size = row.get("size") or 0
next_content_text = row.get("content_text") or ""
next_preview_text = row.get("preview_text") or ""
next_is_truncated = bool(row.get("is_truncated_preview"))
next_blob_data = row.get("blob_data")
if row["type"] == "file" and name is not BaseDocumentStore._UNSET:
next_mime_type = infer_mime_type(next_name, next_mime_type)
if content is not BaseDocumentStore._UNSET:
payload = _text_payload(content)
next_storage_kind = payload["storage_kind"]
next_size = payload["size"]
next_content_text = payload["content_text"]
next_preview_text = payload["preview_text"]
next_is_truncated = payload["is_truncated_preview"]
next_blob_data = None
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
UPDATE document_nodes
SET
name = %s,
parent_id = %s,
mime_type = %s,
storage_kind = %s,
size = %s,
content_text = %s,
preview_text = %s,
is_truncated_preview = %s,
blob_data = %s,
updated_at = CURRENT_TIMESTAMP
WHERE id = %s
RETURNING *
""",
(
next_name,
next_parent_id,
next_mime_type,
next_storage_kind,
next_size,
next_content_text,
next_preview_text,
next_is_truncated,
next_blob_data,
node_id,
),
)
updated = cur.fetchone()
if updated is None:
raise KeyError(node_id)
return self._serialize(updated)
def replace_blob(self, node_id: str, filename: str, raw_bytes: bytes, mime_type: str = "") -> dict[str, Any]:
self._ensure_initialized()
payload = prepare_file_payload(filename, raw_bytes, mime_type)
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
UPDATE document_nodes
SET
name = %s,
mime_type = %s,
storage_kind = %s,
size = %s,
content_text = %s,
preview_text = %s,
is_truncated_preview = %s,
blob_data = %s,
updated_at = CURRENT_TIMESTAMP
WHERE id = %s AND type = 'file'
RETURNING *
""",
(
filename,
payload["mime_type"],
payload["storage_kind"],
payload["size"],
payload["content_text"],
payload["preview_text"],
payload["is_truncated_preview"],
payload["blob_data"],
node_id,
),
)
updated = cur.fetchone()
if updated is None:
raise KeyError(node_id)
return self._serialize(updated)
def delete_node(self, node_id: str) -> None:
self._ensure_initialized()
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
WITH RECURSIVE descendants AS (
SELECT id FROM document_nodes WHERE id = %s
UNION ALL
SELECT child.id
FROM document_nodes child
INNER JOIN descendants parent ON child.parent_id = parent.id
)
DELETE FROM document_nodes
WHERE id IN (SELECT id FROM descendants)
""",
(node_id,),
)
def get_blob(self, node_id: str) -> BlobPayload:
self._ensure_initialized()
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, name, mime_type, content_text, blob_data
FROM document_nodes
WHERE id = %s AND type = 'file'
""",
(node_id,),
)
row = cur.fetchone()
if row is None:
raise FileNotFoundError(node_id)
blob_data = row.get("blob_data")
if blob_data is None:
content = (row.get("content_text") or "").encode("utf-8")
else:
content = bytes(blob_data)
return BlobPayload(
content=content,
mime_type=row.get("mime_type") or infer_mime_type(row.get("name") or ""),
filename=row["name"],
)
_document_store: BaseDocumentStore | None = None
def get_document_store() -> BaseDocumentStore:
global _document_store
if _document_store is not None:
return _document_store
backend = (os.getenv("DOCS_BACKEND") or "postgres").strip().lower()
if backend == "memory":
_document_store = InMemoryDocumentStore()
return _document_store
database_url = os.getenv("DATABASE_URL", "").strip()
if not database_url:
raise RuntimeError("缺少 DATABASE_URL，无法初始化 PostgreSQL 文档存储")
_document_store = PostgresDocumentStore(database_url)
return _document_store
def reset_document_store() -> None:
global _document_store
_document_store = None
backend/docx2pdf_bridge.cjs
-20
View File
@@ -1,20 +0,0 @@
const path = require('path')
const { convert } = require('docx2pdf-converter')
function main() {
const inputPath = process.argv[2]
const outputPath = process.argv[3]
if (!inputPath || !outputPath) {
throw new Error('缺少 DOCX 或 PDF 路径')
}
convert(path.resolve(inputPath), path.resolve(outputPath))
}
try {
main()
} catch (error) {
console.error(error instanceof Error ? error.message : String(error))
process.exit(1)
}
backend/job_handlers.py
-986
View File
@@ -1,986 +0,0 @@
import asyncio
import io
import ipaddress
import json
import os
import re
import socket
import time
import zipfile
from contextlib import suppress
from datetime import datetime
from typing import Any, Callable, Awaitable
from urllib.parse import urlparse
import httpx
import markitdown
from audit_store import get_audit_store
from llm import call_ollama, call_vlm_ocr, stream_ollama_events
from media_utils import extract_audio_wav_bytes, is_video_filename
from prompt import (
build_completion_prompts,
build_pro_completion_prompts,
prepare_prompt_context,
)
from risk_config import load_risk_config
from risk_control import RiskIdentity, estimate_tokens, get_risk_controller
from tts_asr import generate_asr_response, generate_tts_response
IMAGE_MARKDOWN_RE = re.compile(r"!\[[^\]]*]\([^)]+\)")
IMAGE_HTML_RE = re.compile(r"<img\b[^>]*>", re.IGNORECASE)
ALLOWED_CONVERT_EXTENSIONS = {".txt", ".docx", ".pptx", ".pdf"}
SEARXNG_BASE_URL = os.getenv("SEARXNG_BASE_URL", "http://searxng:8080").rstrip("/")
SEARXNG_RESULT_LIMIT = int(os.getenv("SEARXNG_RESULT_LIMIT", "10") or "10")
FIRECRAWL_BASE_URL = os.getenv("FIRECRAWL_BASE_URL", "http://firecrawl:3002").rstrip("/")
FIRECRAWL_API_KEY = os.getenv("FIRECRAWL_API_KEY", "").strip() or ""
WEB_SEARCH_QUERY_COUNT = int(os.getenv("WEB_SEARCH_QUERY_COUNT", "4") or "4")
WEB_SEARCH_SELECTED_URL_LIMIT = int(os.getenv("WEB_SEARCH_SELECTED_URL_LIMIT", "10") or "10")
WEB_SEARCH_CRAWL_CONCURRENCY = max(1, min(6, int(os.getenv("WEB_SEARCH_CRAWL_CONCURRENCY", "3") or "3")))
WEB_SEARCH_CRAWL_TIMEOUT_SECONDS = max(10, min(90, int(os.getenv("WEB_SEARCH_CRAWL_TIMEOUT_SECONDS", "35") or "35")))
_markitdown_instance = None
_risk_config = load_risk_config()
def _get_markitdown():
global _markitdown_instance
if _markitdown_instance is None:
_markitdown_instance = markitdown.MarkItDown()
return _markitdown_instance
def _convert_url_markdown(url: str) -> dict[str, Any]:
result = _get_markitdown().convert_url(url)
markdown = _normalize_multiline_text(str(getattr(result, "markdown", "") or ""))
title = str(getattr(result, "title", "") or "").strip()
return {
"url": url,
"title": title,
"markdown": markdown,
}
def _safe_unlink(path: str | None) -> None:
if not path:
return
with suppress(FileNotFoundError):
os.unlink(path)
def _sanitize_converted_markdown(text: str) -> str:
value = (text or "").replace("\r\n", "\n").replace("\r", "\n")
value = IMAGE_MARKDOWN_RE.sub("", value)
value = IMAGE_HTML_RE.sub("", value)
value = re.sub(r"\n{3,}", "\n\n", value)
return value.strip()
def _normalize_multiline_text(value: str) -> str:
return (value or "").replace("\r\n", "\n").replace("\r", "\n").strip()
def _looks_like_text(raw_bytes: bytes) -> bool:
sample = raw_bytes[:8192]
if not sample or b"\x00" in sample:
return False
try:
text = sample.decode("utf-8")
except UnicodeDecodeError:
return False
if not text.strip():
return False
control_count = sum(
1
for char in text
if (ord(char) < 32 and char not in "\t\n\r") or ord(char) == 127
)
return control_count / max(len(text), 1) < 0.05
def _infer_convert_suffix(raw_bytes: bytes, filename: str) -> str:
sample = raw_bytes[:1024 * 1024]
if sample.startswith(b"%PDF-"):
return ".pdf"
if sample.startswith((b"PK\x03\x04", b"PK\x05\x06")):
try:
with zipfile.ZipFile(io.BytesIO(raw_bytes)) as archive:
names = set(archive.namelist())
if any(name.startswith("ppt/") for name in names):
return ".pptx"
if any(name.startswith("word/") for name in names):
return ".docx"
except Exception:
pass
if _looks_like_text(sample):
return ".txt"
return ""
def _resolve_url_addresses(url: str) -> list[tuple[Any, ...]]:
parsed = urlparse((url or "").strip())
host = (parsed.hostname or "").strip().lower()
if not host:
return []
port = parsed.port or (443 if parsed.scheme == "https" else 80)
return socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
def _is_blocked_public_url(url: str) -> bool:
try:
parsed = urlparse((url or "").strip())
except Exception:
return True
if parsed.scheme not in {"http", "https"}:
return True
host = (parsed.hostname or "").strip().lower()
if not host:
return True
if host in {"localhost", "127.0.0.1", "::1"} or host.endswith((".local", ".localhost")):
return True
try:
ip = ipaddress.ip_address(host)
return not ip.is_global
except ValueError:
pass
try:
addresses = _resolve_url_addresses(url)
except Exception:
return True
for info in addresses:
address = info[4][0]
try:
ip = ipaddress.ip_address(address)
except ValueError:
continue
if not ip.is_global:
return True
return False
def _strip_code_fence(value: str) -> str:
text = _normalize_multiline_text(value)
match = re.match(r"^```(?:json)?\s*([\s\S]*?)\s*```$", text, flags=re.IGNORECASE)
if match:
return match.group(1).strip()
return text
def _extract_json_array(value: str) -> list[Any]:
text = _strip_code_fence(value)
try:
parsed = json.loads(text)
return parsed if isinstance(parsed, list) else []
except Exception:
match = re.search(r"\[[\s\S]*\]", text)
if not match:
return []
try:
parsed = json.loads(match.group(0))
except Exception:
return []
return parsed if isinstance(parsed, list) else []
def _normalize_search_queries(raw: str) -> list[str]:
items = _extract_json_array(raw)
queries: list[str] = []
if items:
for item in items:
text = str(item).strip()
if text and text not in queries:
queries.append(text)
else:
for line in _strip_code_fence(raw).splitlines():
text = re.sub(r"^\s*(?:[-*]|\d+[.)])\s*", "", line).strip()
if text and text not in queries:
queries.append(text)
return queries[:WEB_SEARCH_QUERY_COUNT]
def _clean_search_query_text(value: str) -> str:
text = _normalize_multiline_text(value)
text = re.sub(r"`{1,3}.*?`{1,3}", " ", text)
text = re.sub(r"[*_#>\[\]\(\){}|]+", " ", text)
text = re.sub(r"\s+", " ", text).strip(" -:;,./")
return text
def _build_fallback_search_queries(context: str, primary_queries: list[str]) -> list[str]:
queries: list[str] = []
def add(text: str) -> None:
cleaned = _clean_search_query_text(text)
if len(cleaned) < 2 or cleaned in queries:
return
queries.append(cleaned[:120])
for item in primary_queries:
add(item)
for line in context.splitlines():
cleaned = _clean_search_query_text(line)
if not cleaned:
continue
add(cleaned)
if re.search(r"[A-Za-z]", cleaned):
add(f"{cleaned} official")
add(f"{cleaned} github")
else:
add(f"{cleaned} 官网")
add(f"{cleaned} GitHub")
if context:
compact = _clean_search_query_text(context.replace("\n", " "))
if compact:
add(compact)
if re.search(r"[A-Za-z]", compact):
add(f"{compact} official documentation")
else:
add(f"{compact} 官方文档")
return queries[: max(WEB_SEARCH_QUERY_COUNT + 4, 8)]
def _build_no_result_content(queries: list[str], reason: str) -> str:
lines = [
"未检索到可用公开结果。",
"",
f"原因：{reason}",
]
if queries:
lines.extend(["", "已尝试的检索词："])
lines.extend([f"- {query}" for query in queries])
lines.extend([
"",
"可以尝试缩短主题、补充专有名词，或直接给出官网、产品名、项目名、作者名等更具体的线索。",
])
return "\n".join(lines)
def _normalize_selected_urls(raw: str) -> list[str]:
urls: list[str] = []
for item in _extract_json_array(raw):
text = str(item).strip()
if not text or _is_blocked_public_url(text) or text in urls:
continue
urls.append(text)
return urls[:WEB_SEARCH_SELECTED_URL_LIMIT]
def _summarize_search_results(query: str, results: list[dict[str, Any]]) -> str:
lines = [f"Query: {query}"]
for index, item in enumerate(results, start=1):
lines.append(
f"{index}. title={item.get('title', '')} url={item.get('url', '')} "
f"score={item.get('score', '')} date={item.get('published_date', '')} snippet={item.get('snippet', '')}"
)
return "\n".join(lines)
async def _searxng_search(query: str, *, limit: int) -> list[dict[str, Any]]:
async with httpx.AsyncClient(timeout=httpx.Timeout(20.0, connect=10.0)) as client:
response = await client.get(
f"{SEARXNG_BASE_URL}/search",
params={"q": query, "format": "json"},
headers={
"Accept": "application/json",
"User-Agent": "llm-in-text-websearch/1.0",
"X-Forwarded-For": "127.0.0.1",
"X-Real-IP": "127.0.0.1",
},
)
response.raise_for_status()
payload = response.json()
results: list[dict[str, Any]] = []
for item in payload.get("results") or []:
url = str(item.get("url") or item.get("link") or "").strip()
if not url or await asyncio.to_thread(_is_blocked_public_url, url):
continue
results.append({
"title": str(item.get("title") or "").strip(),
"url": url,
"score": item.get("score"),
"published_date": str(item.get("publishedDate") or item.get("published_date") or item.get("published") or "").strip(),
"snippet": _normalize_multiline_text(str(item.get("content") or item.get("snippet") or "")),
})
if len(results) >= limit:
break
return results
async def _firecrawl_scrape(url: str) -> dict[str, Any]:
headers = {"Content-Type": "application/json"}
if FIRECRAWL_API_KEY:
headers["Authorization"] = f"Bearer {FIRECRAWL_API_KEY}"
headers["X-Api-Key"] = FIRECRAWL_API_KEY
payload = None
async with httpx.AsyncClient(timeout=httpx.Timeout(20.0, connect=5.0, read=20.0)) as client:
try:
response = await client.post(
f"{FIRECRAWL_BASE_URL}/v1/scrape",
json={"url": url, "formats": ["markdown"]},
headers=headers,
)
response.raise_for_status()
payload = response.json()
except Exception:
payload = None
if payload is None:
return await asyncio.to_thread(_convert_url_markdown, url)
data = payload.get("data") or {}
markdown = _normalize_multiline_text(str(data.get("markdown") or data.get("content") or ""))
metadata = data.get("metadata") or {}
return {
"url": url,
"title": str(metadata.get("title") or "").strip(),
"markdown": markdown,
}
def sanitize_inline_completion_content(text: str, prefill: str = "") -> str:
value = (text or "").strip()
if not value:
return ""
fim_middle = value.rfind("<|fim_middle|>")
if fim_middle >= 0:
value = value[fim_middle + len("<|fim_middle|>") :]
end_index = value.find("<|end|>")
if end_index >= 0:
value = value[:end_index]
quoted = re.findall(r'"([^"]+)"', value)
if quoted:
value = quoted[-1]
marker_index = max(value.rfind("|fim_middle|>"), value.rfind("<|start|>assistant"))
if marker_index >= 0:
tail = value.split(">")[-1]
if tail:
value = tail
value = value.strip()
if prefill and value.startswith(prefill):
value = value[len(prefill) :]
return value.strip()
def _payload_identity(payload: dict[str, Any]) -> RiskIdentity:
risk = payload.get("risk") or {}
return RiskIdentity(
request_id=risk.get("request_id") or payload["request_id"],
session_hash=risk.get("session_hash", ""),
ip_hash=risk.get("ip_hash", ""),
route=payload.get("route", payload.get("job_type", payload.get("request_id", ""))),
method="POST",
)
async def _enter_llm_execution(payload: dict[str, Any], emit: Callable[[str, dict[str, Any]], Awaitable[None]]) -> tuple[RiskIdentity, dict[str, Any], list[str]]:
risk = payload.get("risk") or {}
identity = _payload_identity(payload)
delay_ms = int(risk.get("delay_ms", 0) or 0)
policy = risk.get("policy") or {}
if delay_ms > 0:
await emit("resource", {"phase": "delay", "delay_ms": delay_ms})
await asyncio.sleep(delay_ms / 1000.0)
controller = get_risk_controller(_risk_config)
lock_keys = await controller.acquire_execution_slot(identity, model=policy.get("model", ""))
return identity, risk, lock_keys
async def _exit_llm_execution(
payload: dict[str, Any],
identity: RiskIdentity,
risk: dict[str, Any],
lock_keys: list[str],
*,
status: str,
actual_output_text: str = "",
error_code: str = "",
audit_metadata: dict[str, Any] | None = None,
) -> None:
policy = (risk.get("policy") or {})
controller = get_risk_controller(_risk_config)
await controller.release_execution_slot(identity, lock_keys, model=policy.get("model", ""))
await controller.record_model_result(model=policy.get("model", ""), success=(status == "completed"))
store = get_audit_store(os.getenv("DATABASE_URL", "").strip() or None)
estimated_input_tokens = int(risk.get("estimated_input_tokens", 0) or 0)
profile = policy.get("profile", "completion")
pricing_out = {
"completion": _risk_config.completion_output_cost_per_1k,
"pro": _risk_config.pro_output_cost_per_1k,
"vision": _risk_config.vision_output_cost_per_1k,
}.get(profile, _risk_config.completion_output_cost_per_1k)
actual_output_tokens = estimate_tokens(actual_output_text)
extra_metadata = dict(audit_metadata or {})
if profile == "speech_tts":
actual_cost = round(
(int(extra_metadata.get("text_chars", 0) or 0) / 1000.0) * _risk_config.speech_tts_input_cost_per_1k_chars
+ (int(extra_metadata.get("duration_ms", 0) or 0) / 60000.0) * _risk_config.speech_tts_output_cost_per_minute_audio,
8,
)
elif profile == "speech_asr":
actual_cost = round(
(int(extra_metadata.get("audio_bytes", 0) or 0) / (1024.0 * 1024.0)) * _risk_config.speech_asr_input_cost_per_mb,
8,
)
else:
actual_cost = round((estimated_input_tokens / 1000.0) * {
"completion": _risk_config.completion_input_cost_per_1k,
"pro": _risk_config.pro_input_cost_per_1k,
"vision": _risk_config.vision_input_cost_per_1k,
}.get(profile, _risk_config.completion_input_cost_per_1k) + (actual_output_tokens / 1000.0) * pricing_out, 8)
job_context = payload.get("job_context") or {}
now_ms = int(time.time() * 1000)
started_at = int(job_context.get("started_at", 0) or 0)
created_at = int(job_context.get("created_at", 0) or 0)
queue_ms = int(job_context.get("queue_ms", 0) or 0)
run_ms = int(job_context.get("run_ms", 0) or 0)
total_ms = int(job_context.get("total_ms", 0) or 0)
if not run_ms and started_at:
run_ms = max(0, now_ms - started_at)
if not total_ms:
total_ms = max(0, now_ms - created_at) if created_at else run_ms
await asyncio.to_thread(
store.record_llm_call,
{
"request_id": payload["request_id"],
"session_hash": identity.session_hash,
"ip_hash": identity.ip_hash,
"job_type": policy.get("job_type", ""),
"model": policy.get("model", ""),
"estimated_input_tokens": estimated_input_tokens,
"max_output_tokens": int(policy.get("max_output_tokens", 0) or 0),
"estimated_cost": float(risk.get("estimated_cost", 0.0) or 0.0),
"actual_output_chars": len(actual_output_text or ""),
"actual_cost": actual_cost,
"status": status,
"error_code": error_code,
"queue_ms": queue_ms,
"run_ms": run_ms,
"total_ms": total_ms,
"metadata": {"profile": profile, **extra_metadata},
},
)
async def completion_handler(
payload: dict[str, Any],
emit: Callable[[str, dict[str, Any]], Awaitable[None]],
is_cancelled: Callable[[], bool],
) -> dict[str, Any]:
req = payload["request"]
identity, risk, lock_keys = await _enter_llm_execution(payload, emit)
system_prompt, user_prompt, prefill = build_completion_prompts(
req["prefix"],
req["suffix"],
req.get("languageId", "markdown"),
location=payload.get("location", ""),
thinking_level=req.get("model_thinking", "low"),
preferences=req.get("user_preferences"),
)
policy = risk.get("policy") or {}
try:
result = await call_ollama(
user_prompt,
system_prompt=system_prompt,
tag=f'{payload["request_id"][:8]}-completion',
temperature=float(policy.get("temperature", req.get("temperature", 0.7))),
thinking=policy.get("thinking"),
model=policy.get("model"),
prefill=prefill or None,
max_output_tokens=int(policy.get("max_output_tokens", 0) or 0),
)
content = sanitize_inline_completion_content(result.get("content") or "", prefill=prefill or "")
if is_cancelled():
raise asyncio.CancelledError()
await emit("result", {"content": content})
await _exit_llm_execution(payload, identity, risk, lock_keys, status="completed", actual_output_text=content)
return {"content": content, "request_id": payload["request_id"]}
except asyncio.CancelledError:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="cancelled", error_code="cancelled")
raise
except Exception:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="failed", error_code="llm_failed")
raise
async def pro_completion_handler(
payload: dict[str, Any],
emit: Callable[[str, dict[str, Any]], Awaitable[None]],
is_cancelled: Callable[[], bool],
) -> dict[str, Any]:
req = payload["request"]
identity, risk, lock_keys = await _enter_llm_execution(payload, emit)
system_prompt, user_prompt = build_pro_completion_prompts(
prefix=req["prefix"],
suffix=req["suffix"],
instruction=req.get("instruction", ""),
language_id=req.get("languageId", "markdown"),
location=payload.get("location", ""),
pro_thinking_level=req.get("pro_thinking", "medium"),
preferences=req.get("user_preferences"),
)
chunks: list[str] = []
policy = risk.get("policy") or {}
try:
async for event_type, delta in stream_ollama_events(
user_prompt,
system_prompt=system_prompt,
tag=f'{payload["request_id"][:8]}-pro',
temperature=float(policy.get("temperature", 0.7)),
thinking=policy.get("thinking"),
model=policy.get("model"),
enable_thinking=True,
max_output_tokens=int(policy.get("max_output_tokens", 0) or 0),
):
if is_cancelled():
raise asyncio.CancelledError()
if event_type == "thinking":
await emit("progress", {"phase": "thinking"})
continue
if delta:
chunks.append(delta)
await emit("result", {"delta": delta})
content = "".join(chunks)
await _exit_llm_execution(payload, identity, risk, lock_keys, status="completed", actual_output_text=content)
return {"content": content, "request_id": payload["request_id"]}
except asyncio.CancelledError:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="cancelled", error_code="cancelled")
raise
except Exception:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="failed", error_code="llm_failed")
raise
async def web_search_handler(
payload: dict[str, Any],
emit: Callable[[str, dict[str, Any]], Awaitable[None]],
is_cancelled: Callable[[], bool],
) -> dict[str, Any]:
req = payload["request"]
identity, risk, lock_keys = await _enter_llm_execution(payload, emit)
policy = risk.get("policy") or {}
prefix = _normalize_multiline_text(req.get("prefix", ""))
suffix = _normalize_multiline_text(req.get("suffix", ""))
context = "\n\n".join(part for part in [prefix, suffix] if part).strip()
try:
await emit("progress", {"phase": "keywords", "message": "正在生成搜索关键词"})
keyword_prompt = (
"你是联网研究助手。请根据下面的上下文，生成 3 到 5 个适合在搜索引擎中直接使用的检索关键词或短句。\n"
"要求：\n"
"- 只返回 JSON 数组字符串\n"
"- 每个元素是一个简洁检索词\n"
"- 不要解释，不要 Markdown\n\n"
f"上下文：\n{context}"
)
keyword_result = await call_ollama(
keyword_prompt,
system_prompt="Return only a JSON array of search queries.",
tag=f'{payload["request_id"][:8]}-webq',
temperature=float(policy.get("temperature", 0.4)),
thinking=policy.get("thinking"),
model=policy.get("model"),
max_output_tokens=min(int(policy.get("max_output_tokens", 0) or 1024), 1024),
)
queries = _normalize_search_queries(keyword_result.get("content") or "")
if not queries:
queries = [_normalize_multiline_text(prefix or suffix)[:120] or "general research query"]
if is_cancelled():
raise asyncio.CancelledError()
await emit("progress", {"phase": "searching", "message": "正在通过 SearXNG 搜索"})
search_sections: list[str] = []
search_candidates: list[dict[str, Any]] = []
attempted_queries: list[str] = []
for query in queries:
if is_cancelled():
raise asyncio.CancelledError()
attempted_queries.append(query)
results = await _searxng_search(query, limit=SEARXNG_RESULT_LIMIT)
if not results:
continue
search_sections.append(_summarize_search_results(query, results))
search_candidates.extend(results)
if not search_candidates:
fallback_queries = _build_fallback_search_queries(context, queries)
retry_queries = [query for query in fallback_queries if query not in attempted_queries]
if retry_queries:
await emit("progress", {"phase": "searching", "message": "搜索结果较少，正在尝试更宽泛的检索词"})
for query in retry_queries:
if is_cancelled():
raise asyncio.CancelledError()
attempted_queries.append(query)
results = await _searxng_search(query, limit=SEARXNG_RESULT_LIMIT)
if not results:
continue
search_sections.append(_summarize_search_results(query, results))
search_candidates.extend(results)
if not search_candidates:
content = _build_no_result_content(attempted_queries, "SearXNG 未返回可用结果")
created_at = str(req.get("created_at") or payload.get("created_at") or "").strip() or datetime.now().isoformat()
await _exit_llm_execution(payload, identity, risk, lock_keys, status="completed", actual_output_text=content)
return {"content": content, "request_id": payload["request_id"], "created_at": created_at}
deduped_candidates: list[dict[str, Any]] = []
seen_urls: set[str] = set()
for item in search_candidates:
url = item["url"]
if url in seen_urls:
continue
seen_urls.add(url)
deduped_candidates.append(item)
await emit("progress", {"phase": "selecting_urls", "message": "正在筛选可信网址"})
search_results_text = "\n\n".join(search_sections)
selection_prompt = (
"你是研究检索筛选器。下面是多组搜索结果，请从中挑选 5 到 20 个最可信、最相关、最值得进一步抓取的 URL。\n"
"优先选择：官方文档、权威机构、原始来源、信息完整且日期清晰的页面。\n"
"只返回 JSON 数组，元素必须是 URL 字符串。\n\n"
f"原始上下文：\n{context}\n\n"
f"搜索结果：\n{search_results_text}"
)
selection_result = await call_ollama(
selection_prompt,
system_prompt="Return only a JSON array of selected URLs.",
tag=f'{payload["request_id"][:8]}-webu',
temperature=0.2,
thinking=policy.get("thinking"),
model=policy.get("model"),
max_output_tokens=min(int(policy.get("max_output_tokens", 0) or 1024), 1024),
)
selected_urls = _normalize_selected_urls(selection_result.get("content") or "")
if not selected_urls:
selected_urls = [item["url"] for item in deduped_candidates[:WEB_SEARCH_SELECTED_URL_LIMIT]]
if is_cancelled():
raise asyncio.CancelledError()
await emit("progress", {"phase": "crawling", "message": "正在抓取网页内容"})
selected_url_set = set(selected_urls)
selected_candidates = [item for item in deduped_candidates if item["url"] in selected_url_set][:WEB_SEARCH_SELECTED_URL_LIMIT]
crawl_sem = asyncio.Semaphore(WEB_SEARCH_CRAWL_CONCURRENCY)
async def _crawl_candidate(item: dict[str, Any]) -> dict[str, Any] | None:
if is_cancelled():
raise asyncio.CancelledError()
async with crawl_sem:
try:
scraped = await asyncio.wait_for(
_firecrawl_scrape(item["url"]),
timeout=WEB_SEARCH_CRAWL_TIMEOUT_SECONDS,
)
except Exception:
return None
if not scraped.get("markdown"):
return None
return {
"url": item["url"],
"title": item.get("title") or scraped.get("title") or "",
"score": item.get("score"),
"published_date": item.get("published_date") or "",
"content": scraped["markdown"],
}
crawl_results = await asyncio.gather(*[_crawl_candidate(item) for item in selected_candidates])
crawled_pages = [page for page in crawl_results if page]
if not crawled_pages:
content = _build_no_result_content(selected_urls, "已检索到候选网页，但未抓取到可用正文")
created_at = str(req.get("created_at") or payload.get("created_at") or "").strip() or datetime.now().isoformat()
await _exit_llm_execution(payload, identity, risk, lock_keys, status="completed", actual_output_text=content)
return {"content": content, "request_id": payload["request_id"], "created_at": created_at}
await emit("progress", {"phase": "synthesizing", "message": "正在整理搜索结果"})
page_sections = []
for index, page in enumerate(crawled_pages, start=1):
page_sections.append(
f"[Source {index}]\n"
f"URL: {page['url']}\n"
f"Title: {page.get('title', '')}\n"
f"Score: {page.get('score', '')}\n"
f"Date: {page.get('published_date', '')}\n"
f"Content:\n{page['content'][:12000]}"
)
crawled_text = "\n\n".join(page_sections)
synthesis_prompt = (
"你是研究写作助手。请根据原始上下文和抓取到的网页内容，写出一篇长篇、结构完整、信息密集的 Markdown 正文。\n"
"要求：\n"
"- 不要写标题\n"
"- 不要写引用编号、来源表或 URL 列表\n"
"- 直接输出最终正文\n"
"- 如果信息存在不确定性，用审慎措辞表达\n\n"
f"原始上下文：\n{context}\n\n"
f"抓取内容：\n{crawled_text}"
)
# 流式合成 — 逐 delta emit，前端可实时渲染
from llm import stream_ollama_events
accumulated: list[str] = []
async for event_type, text in stream_ollama_events(
synthesis_prompt,
system_prompt="Return only the final markdown body with no title and no citations list.",
tag=f'{payload["request_id"][:8]}-webf',
temperature=float(policy.get("temperature", 0.4)),
thinking=policy.get("thinking"),
model=policy.get("model"),
max_output_tokens=int(policy.get("max_output_tokens", 0) or 0),
):
if is_cancelled():
raise asyncio.CancelledError()
# 只推送 content delta，不展示 thinking
if event_type == "content":
accumulated.append(text)
await emit("delta", {"text": text})
content = _normalize_multiline_text("".join(accumulated))
if not content:
raise RuntimeError("联网搜索生成了空结果")
created_at = str(req.get("created_at") or payload.get("created_at") or "").strip() or datetime.now().isoformat()
await _exit_llm_execution(payload, identity, risk, lock_keys, status="completed", actual_output_text=content)
return {"content": content, "request_id": payload["request_id"], "created_at": created_at}
except asyncio.CancelledError:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="cancelled", error_code="cancelled")
raise
except Exception:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="failed", error_code="web_search_failed")
raise
async def compress_handler(
payload: dict[str, Any],
emit: Callable[[str, dict[str, Any]], Awaitable[None]],
is_cancelled: Callable[[], bool],
) -> dict[str, Any]:
content = payload["content"]
doc_type = payload.get("docType", "txt")
identity, risk, lock_keys = await _enter_llm_execution(payload, emit)
system_prompt = (
f"你是一个专业的文档摘要助手。请将以下 {doc_type} 类型文档内容进行精简压缩，"
"保留核心信息和关键要点，去除冗余和啰嗦的表述。"
"请直接输出压缩后的内容，不要添加任何解释性文字。"
)
policy = risk.get("policy") or {}
try:
result = await call_ollama(
content,
system_prompt=system_prompt,
tag=f'{payload["request_id"][:8]}-compress',
model=policy.get("model"),
temperature=float(policy.get("temperature", 0.2)),
thinking=policy.get("thinking"),
max_output_tokens=int(policy.get("max_output_tokens", 0) or 0),
)
if is_cancelled():
raise asyncio.CancelledError()
compressed = result.get("content") or ""
await emit("result", {"content": compressed})
await _exit_llm_execution(payload, identity, risk, lock_keys, status="completed", actual_output_text=compressed)
return {"content": compressed, "request_id": payload["request_id"]}
except asyncio.CancelledError:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="cancelled", error_code="cancelled")
raise
except Exception:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="failed", error_code="llm_failed")
raise
async def ocr_handler(
payload: dict[str, Any],
emit: Callable[[str, dict[str, Any]], Awaitable[None]],
is_cancelled: Callable[[], bool],
) -> dict[str, Any]:
path = payload["input_path"]
identity, risk, lock_keys = await _enter_llm_execution(payload, emit)
try:
filename = payload.get("filename", "image.jpg")
language = payload.get("language", "auto")
media_type = payload.get("media_type", "image")
mime_type = payload.get("mime_type", "") or ""
with open(path, "rb") as handle:
media_bytes = handle.read()
await emit("progress", {"phase": "ocr", "media_type": media_type})
ocr_text = await call_vlm_ocr(
media_bytes,
language,
mime_type=mime_type or "application/octet-stream",
media_type=media_type,
)
if is_cancelled():
raise asyncio.CancelledError()
result = {
"text": ocr_text,
"ocr_text": ocr_text,
"filename": filename,
"media_type": media_type,
}
if media_type == "video" or is_video_filename(filename, mime_type):
asr_text = ""
try:
await emit("progress", {"phase": "asr", "media_type": media_type})
audio_bytes = await asyncio.to_thread(extract_audio_wav_bytes, path)
asr_response = await generate_asr_response(audio_bytes, language)
asr_text = getattr(asr_response, "text", "") or ""
except Exception as exc:
raise RuntimeError(f"音频解析失败: {exc}") from exc
if ocr_text.strip() or asr_text.strip():
text_parts = []
if ocr_text.strip():
text_parts.append(f"## 视频画面 OCR\n\n{ocr_text.strip()}")
if asr_text.strip():
text_parts.append(f"## 视频音频 ASR\n\n{asr_text.strip()}")
result["text"] = "\n\n".join(text_parts)
result["asr_text"] = asr_text
await emit("result", result)
await _exit_llm_execution(
payload,
identity,
risk,
lock_keys,
status="completed",
actual_output_text=result["text"],
)
return result
except asyncio.CancelledError:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="cancelled", error_code="cancelled")
raise
except Exception:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="failed", error_code="ocr_failed")
raise
finally:
_safe_unlink(path)
async def convert_handler(
payload: dict[str, Any],
emit: Callable[[str, dict[str, Any]], Awaitable[None]],
is_cancelled: Callable[[], bool],
) -> dict[str, Any]:
path = payload["input_path"]
filename = payload.get("filename", "document")
try:
temp_ext = os.path.splitext(path)[1].lower()
except Exception:
temp_ext = ""
if temp_ext not in ALLOWED_CONVERT_EXTENSIONS:
_safe_unlink(path)
raise ValueError("仅支持 txt、docx、pptx、pdf 格式")
try:
if temp_ext == ".txt":
with open(path, "rb") as handle:
markdown = _sanitize_converted_markdown(handle.read().decode("utf-8", errors="ignore"))
else:
md = _get_markitdown()
result = await asyncio.to_thread(md.convert, path)
markdown = _sanitize_converted_markdown(result.text_content)
if is_cancelled():
raise asyncio.CancelledError()
await emit("result", {"markdown": markdown})
return {"markdown": markdown, "filename": filename}
finally:
_safe_unlink(path)
async def tts_handler(
payload: dict[str, Any],
emit: Callable[[str, dict[str, Any]], Awaitable[None]],
is_cancelled: Callable[[], bool],
) -> dict[str, Any]:
text = str(payload.get("text", "") or "").strip()
if not text:
raise ValueError("TTS 文本为空")
identity, risk, lock_keys = await _enter_llm_execution(payload, emit)
try:
response = await generate_tts_response(
text=text,
instruct=str(payload.get("instruct", "") or ""),
speaker=str(payload.get("speaker", "Vivian") or "Vivian"),
output_format=str(payload.get("format", "wav") or "wav"),
)
if is_cancelled():
raise asyncio.CancelledError()
result = dict(response)
await emit("result", result)
await _exit_llm_execution(
payload,
identity,
risk,
lock_keys,
status="completed",
audit_metadata={
"speaker": result.get("speaker", ""),
"format": result.get("format", ""),
"duration_ms": int(result.get("duration_ms", 0) or 0),
"audio_bytes": int(result.get("audio_bytes", 0) or 0),
"text_chars": int(result.get("text_chars", len(text)) or len(text)),
"request_ms": int(result.get("request_ms", 0) or 0),
"upstream_request_id": result.get("upstream_request_id", ""),
},
)
return result
except asyncio.CancelledError:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="cancelled", error_code="cancelled")
raise
except Exception:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="failed", error_code="tts_failed")
raise
async def asr_handler(
payload: dict[str, Any],
emit: Callable[[str, dict[str, Any]], Awaitable[None]],
is_cancelled: Callable[[], bool],
) -> dict[str, Any]:
path = payload["input_path"]
identity, risk, lock_keys = await _enter_llm_execution(payload, emit)
try:
with open(path, "rb") as handle:
audio_bytes = handle.read()
response = await generate_asr_response(audio_bytes, payload.get("language", "zh-CN"))
if is_cancelled():
raise asyncio.CancelledError()
result = dict(response)
await emit("result", result)
await _exit_llm_execution(
payload,
identity,
risk,
lock_keys,
status="completed",
actual_output_text=result.get("text", "") or "",
audit_metadata={
"language": result.get("language", ""),
"audio_bytes": int(result.get("audio_bytes", len(audio_bytes)) or len(audio_bytes)),
"request_ms": int(result.get("request_ms", 0) or 0),
"upstream_request_id": result.get("upstream_request_id", ""),
},
)
return result
except asyncio.CancelledError:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="cancelled", error_code="cancelled")
raise
except Exception:
await _exit_llm_execution(payload, identity, risk, lock_keys, status="failed", error_code="asr_failed")
raise
finally:
_safe_unlink(path)
backend/job_system.py
-872
View File
@@ -1,872 +0,0 @@
import asyncio
import inspect
import json
import logging
import os
import tempfile
import time
import uuid
from contextlib import suppress
from dataclasses import dataclass
from pathlib import Path
from typing import Any, AsyncIterator, Awaitable, Callable, Optional
logger = logging.getLogger("job_system")
try: # pragma: no cover - optional dependency in tests
from redis import asyncio as redis_asyncio # type: ignore
except Exception: # pragma: no cover - optional dependency in tests
redis_asyncio = None
TERMINAL_STATUSES = {"completed", "failed", "cancelled"}
TERMINAL_EVENTS = {"done", "error", "cancelled"}
JOB_TYPES = (
"completion",
"pro_completion",
"web_search",
"compress",
"ocr",
"convert",
"tts",
"asr",
)
def _int_env(name: str, default: int) -> int:
try:
return max(1, int(os.getenv(name, str(default))))
except (TypeError, ValueError):
return default
DEFAULT_CONCURRENCY = {
"completion": 2,
"pro_completion": 1,
"web_search": 1,
"compress": 1,
"ocr": 1,
"convert": 1,
"tts": _int_env("JOB_TTS_CONCURRENCY", 2),
"asr": _int_env("JOB_ASR_CONCURRENCY", 1),
}
DEFAULT_QUEUE_SIZE = {
"completion": 16,
"pro_completion": 8,
"web_search": 4,
"compress": 8,
"ocr": 8,
"convert": 8,
"tts": _int_env("JOB_TTS_MAX_QUEUE", 8),
"asr": _int_env("JOB_ASR_MAX_QUEUE", 8),
}
class JobSystemError(RuntimeError):
"""任务系统内部错误"""
def __init__(self, message: str = "任务系统错误", error_code: str = "job_system_error") -> None:
super().__init__(message)
self.message = message
self.error_code = error_code
def __str__(self) -> str:
return f"{self.error_code}: {self.message}"
class QueueFullError(JobSystemError):
"""任务队列已满"""
def __init__(self, job_type: str, max_queue: int) -> None:
super().__init__(f"{job_type} 队列已满 (当前: {max_queue}/{max_queue})", "queue_full")
self.job_type = job_type
self.max_queue = max_queue
@dataclass(frozen=True)
class QueueConfig:
job_type: str
concurrency: int
max_queue: int
Handler = Callable[[dict[str, Any], Callable[[str, dict[str, Any]], Awaitable[None]], Callable[[], bool]], Awaitable[dict[str, Any]]]
def _bool_env(name: str, default: bool) -> bool:
value = os.getenv(name)
if value is None:
return default
return value.strip().lower() in {"1", "true", "yes", "on"}
def _float_env(name: str, default: float) -> float:
try:
return float(os.getenv(name, str(default)))
except (TypeError, ValueError):
return default
def _now_ms() -> int:
return int(time.time() * 1000)
def _busy_level(ratio: float) -> str:
if ratio >= _float_env("JOB_BUSY_FULL_THRESHOLD", 1.0):
return "full"
if ratio >= _float_env("JOB_BUSY_HIGH_THRESHOLD", 0.75):
return "busy"
if ratio >= _float_env("JOB_BUSY_NORMAL_THRESHOLD", 0.25):
return "normal"
return "idle"
def _json_dumps(value: Any) -> str:
return json.dumps(value, ensure_ascii=False)
def _json_loads(value: str | bytes | None, default: Any = None) -> Any:
if value is None:
return default
if isinstance(value, bytes):
value = value.decode("utf-8")
if not value:
return default
return json.loads(value)
def _queue_config(job_type: str) -> QueueConfig:
upper = job_type.upper()
concurrency = _int_env(f"JOB_{upper}_CONCURRENCY", DEFAULT_CONCURRENCY[job_type])
max_queue = _int_env(f"JOB_{upper}_MAX_QUEUE", DEFAULT_QUEUE_SIZE[job_type])
return QueueConfig(job_type=job_type, concurrency=concurrency, max_queue=max_queue)
def get_job_backend_name() -> str:
value = (os.getenv("JOB_BACKEND") or "").strip().lower()
if value:
return value
if redis_asyncio is not None:
return "redis"
return "memory"
def _shared_temp_dir() -> Path:
path = Path(os.getenv("JOB_SHARED_TEMP_DIR", tempfile.gettempdir()) or tempfile.gettempdir())
path.mkdir(parents=True, exist_ok=True)
return path
def persist_temp_input(raw_bytes: bytes, suffix: str) -> str:
directory = _shared_temp_dir()
fd, path = tempfile.mkstemp(prefix="job-input-", suffix=suffix, dir=directory)
os.close(fd)
with open(path, "wb") as handle:
handle.write(raw_bytes)
return path
async def _maybe_await(value: Any) -> Any:
if inspect.isawaitable(value):
return await value
return value
class BaseJobManager:
def __init__(self) -> None:
self.handlers: dict[str, Handler] = {}
def register_handler(self, job_type: str, handler: Handler) -> None:
self.handlers[job_type] = handler
async def submit(self, job_type: str, payload: dict[str, Any], request_id: str | None = None) -> str:
raise NotImplementedError
async def cancel(self, job_id: str, reason: str = "abort") -> dict[str, Any]:
raise NotImplementedError
async def get_status(self, job_id: str) -> dict[str, Any] | None:
raise NotImplementedError
async def stream_events(self, job_id: str) -> AsyncIterator[dict[str, Any]]:
raise NotImplementedError
async def close(self) -> None:
return None
class InMemoryJobManager(BaseJobManager):
def __init__(self) -> None:
super().__init__()
self.jobs: dict[str, dict[str, Any]] = {}
self.event_history: dict[str, list[dict[str, Any]]] = {}
self.subscribers: dict[str, list[asyncio.Queue]] = {}
self.queues = {job_type: asyncio.Queue() for job_type in JOB_TYPES}
self.semaphores = {job_type: asyncio.Semaphore(_queue_config(job_type).concurrency) for job_type in JOB_TYPES}
self.queue_counts = {job_type: 0 for job_type in JOB_TYPES}
self.running_counts = {job_type: 0 for job_type in JOB_TYPES}
self.running_tasks: dict[str, asyncio.Task] = {}
self.worker_tasks: list[asyncio.Task] = []
self.started = False
self.lock = asyncio.Lock()
async def _ensure_started(self) -> None:
if self.started:
return
self.started = True
for job_type in JOB_TYPES:
self.worker_tasks.append(asyncio.create_task(self._worker_loop(job_type)))
def _metrics(self, job_type: str) -> dict[str, Any]:
config = _queue_config(job_type)
queued = self.queue_counts[job_type]
running = self.running_counts[job_type]
capacity = max(config.max_queue + config.concurrency, 1)
ratio = min((queued + running) / capacity, 1.0)
return {
"queue_position": queued if queued > 0 else 0,
"queued_count": queued,
"running_count": running,
"concurrency_limit": config.concurrency,
"max_queue": config.max_queue,
"busy_ratio": round(ratio, 4),
"busy_level": _busy_level(ratio),
}
async def _publish(self, job_id: str, event: str, data: dict[str, Any]) -> None:
event_payload = {"event": event, **data}
self.event_history.setdefault(job_id, []).append(event_payload)
for queue in self.subscribers.get(job_id, []):
await queue.put(event_payload)
async def submit(self, job_type: str, payload: dict[str, Any], request_id: str | None = None) -> str:
await self._ensure_started()
if job_type not in self.handlers:
raise JobSystemError(f"missing handler for job type: {job_type}")
async with self.lock:
config = _queue_config(job_type)
if self.queue_counts[job_type] >= config.max_queue:
raise QueueFullError(job_type, config.max_queue)
job_id = request_id or str(uuid.uuid4())
self.jobs[job_id] = {
"job_id": job_id,
"request_id": job_id,
"job_type": job_type,
"status": "queued",
"payload": payload,
"result": None,
"error": "",
"cancel_requested": False,
"created_at": _now_ms(),
"updated_at": _now_ms(),
"started_at": 0,
"completed_at": 0,
"queue_ms": 0,
"run_ms": 0,
"total_ms": 0,
}
self.event_history[job_id] = []
self.queue_counts[job_type] += 1
metrics = self._metrics(job_type)
await self._publish(job_id, "queued", {"job_id": job_id, "type": job_type, "status": "queued", **metrics})
await self.queues[job_type].put(job_id)
return job_id
async def cancel(self, job_id: str, reason: str = "abort") -> dict[str, Any]:
async with self.lock:
job = self.jobs.get(job_id)
if not job:
return {"cancelled": False, "status": "not_found"}
if job["status"] in TERMINAL_STATUSES:
return {"cancelled": False, "status": job["status"]}
job["cancel_requested"] = True
job["updated_at"] = _now_ms()
task = self.running_tasks.get(job_id)
if task and not task.done():
task.cancel()
if job["status"] == "queued":
job["status"] = "cancelled"
self.queue_counts[job["job_type"]] = max(0, self.queue_counts[job["job_type"]] - 1)
metrics = self._metrics(job["job_type"])
else:
job["status"] = "cancelled"
metrics = self._metrics(job["job_type"])
await self._publish(job_id, "cancelled", {"job_id": job_id, "type": job["job_type"], "status": "cancelled", "reason": reason, **metrics})
return {"cancelled": True, "status": "ok"}
async def get_status(self, job_id: str) -> dict[str, Any] | None:
job = self.jobs.get(job_id)
if not job:
return None
metrics = self._metrics(job["job_type"])
return {
"job_id": job_id,
"request_id": job["request_id"],
"type": job["job_type"],
"status": job["status"],
"result": job["result"],
"error": job["error"],
"created_at": job.get("created_at", 0),
"started_at": job.get("started_at", 0),
"completed_at": job.get("completed_at", 0),
"queue_ms": job.get("queue_ms", 0),
"run_ms": job.get("run_ms", 0),
"total_ms": job.get("total_ms", 0),
**metrics,
}
async def stream_events(self, job_id: str) -> AsyncIterator[dict[str, Any]]:
queue: asyncio.Queue = asyncio.Queue()
self.subscribers.setdefault(job_id, []).append(queue)
last_index = 0
try:
while True:
history = list(self.event_history.get(job_id, []))
while last_index < len(history):
item = history[last_index]
last_index += 1
yield item
if item.get("event") in TERMINAL_EVENTS:
return
event = await queue.get()
last_index = len(self.event_history.get(job_id, []))
yield event
if event["event"] in TERMINAL_EVENTS:
break
finally:
with suppress(ValueError):
self.subscribers.get(job_id, []).remove(queue)
async def _worker_loop(self, job_type: str) -> None:
queue = self.queues[job_type]
sem = self.semaphores[job_type]
while True:
job_id = await queue.get()
async with self.lock:
job = self.jobs.get(job_id)
if not job or job["status"] == "cancelled":
continue
await sem.acquire()
task = asyncio.create_task(self._run_job(job_id))
self.running_tasks[job_id] = task
async def _run_job(self, job_id: str) -> None:
job = self.jobs[job_id]
job_type = job["job_type"]
try:
async with self.lock:
if job["status"] == "cancelled":
return
self.queue_counts[job_type] = max(0, self.queue_counts[job_type] - 1)
self.running_counts[job_type] += 1
job["status"] = "running"
started_at = _now_ms()
job["updated_at"] = started_at
job["started_at"] = started_at
job["queue_ms"] = max(0, started_at - int(job.get("created_at", started_at)))
metrics = self._metrics(job_type)
await self._publish(job_id, "started", {"job_id": job_id, "type": job_type, "status": "running", **metrics})
async def emit(event: str, data: dict[str, Any]) -> None:
metrics_now = self._metrics(job_type)
await self._publish(job_id, event, {"job_id": job_id, "type": job_type, "status": job["status"], **metrics_now, **data})
def is_cancelled() -> bool:
return bool(job.get("cancel_requested"))
job_payload = dict(job["payload"])
job_payload["job_context"] = {
"job_id": job_id,
"created_at": int(job.get("created_at", 0) or 0),
"started_at": int(job.get("started_at", 0) or 0),
"queue_ms": int(job.get("queue_ms", 0) or 0),
}
result = await self.handlers[job_type](job_payload, emit, is_cancelled)
async with self.lock:
if job["cancel_requested"]:
job["status"] = "cancelled"
metrics = self._metrics(job_type)
await emit("cancelled", {"reason": "abort"})
return
job["status"] = "completed"
job["result"] = result
completed_at = _now_ms()
job["updated_at"] = completed_at
job["completed_at"] = completed_at
job["run_ms"] = max(0, completed_at - int(job.get("started_at", completed_at)))
job["total_ms"] = max(0, completed_at - int(job.get("created_at", completed_at)))
metrics = self._metrics(job_type)
await self._publish(
job_id,
"done",
{
"job_id": job_id,
"type": job_type,
"status": "completed",
"result": result,
"queue_ms": job.get("queue_ms", 0),
"run_ms": job.get("run_ms", 0),
"total_ms": job.get("total_ms", 0),
**metrics,
},
)
except asyncio.CancelledError:
async with self.lock:
job["status"] = "cancelled"
job["cancel_requested"] = True
completed_at = _now_ms()
job["updated_at"] = completed_at
job["completed_at"] = completed_at
job["run_ms"] = max(0, completed_at - int(job.get("started_at", completed_at)))
job["total_ms"] = max(0, completed_at - int(job.get("created_at", completed_at)))
metrics = self._metrics(job_type)
await self._publish(job_id, "cancelled", {"job_id": job_id, "type": job_type, "status": "cancelled", **metrics})
raise
except Exception as exc:
logger.exception("in-memory job failed job_id=%s type=%s", job_id, job_type)
async with self.lock:
job["status"] = "failed"
job["error"] = str(exc)
completed_at = _now_ms()
job["updated_at"] = completed_at
job["completed_at"] = completed_at
job["run_ms"] = max(0, completed_at - int(job.get("started_at", completed_at)))
job["total_ms"] = max(0, completed_at - int(job.get("created_at", completed_at)))
metrics = self._metrics(job_type)
await self._publish(
job_id,
"error",
{
"job_id": job_id,
"type": job_type,
"status": "failed",
"error": str(exc),
"queue_ms": job.get("queue_ms", 0),
"run_ms": job.get("run_ms", 0),
"total_ms": job.get("total_ms", 0),
**metrics,
},
)
finally:
async with self.lock:
self.running_counts[job_type] = max(0, self.running_counts[job_type] - 1)
self.running_tasks.pop(job_id, None)
self.semaphores[job_type].release()
async def close(self) -> None:
for task in self.worker_tasks:
task.cancel()
for task in self.running_tasks.values():
task.cancel()
for task in self.worker_tasks:
with suppress(asyncio.CancelledError):
await task
self.worker_tasks.clear()
self.running_tasks.clear()
self.started = False
class RedisJobManager(BaseJobManager):
def __init__(self) -> None:
super().__init__()
if redis_asyncio is None:
raise JobSystemError("redis package is not installed")
self.redis = redis_asyncio.from_url(
os.getenv("REDIS_URL", "redis://localhost:6379/0"),
encoding="utf-8",
decode_responses=True,
)
self.prefix = (os.getenv("JOB_REDIS_PREFIX") or "llmtext:jobs").strip() or "llmtext:jobs"
self.state_ttl = _int_env("JOB_STATE_TTL_SECONDS", 600)
self.event_ttl = _int_env("JOB_EVENT_TTL_SECONDS", 600)
def _queue_key(self, job_type: str) -> str:
return f"{self.prefix}:queue:{job_type}"
def _event_key(self, job_id: str) -> str:
return f"{self.prefix}:events:{job_id}"
def _state_key(self, job_id: str) -> str:
return f"{self.prefix}:state:{job_id}"
def _metrics_key(self, job_type: str) -> str:
return f"{self.prefix}:metrics:{job_type}"
def _group_name(self, job_type: str) -> str:
return f"{self.prefix}:group:{job_type}"
async def ensure_groups(self) -> None:
for job_type in JOB_TYPES:
stream = self._queue_key(job_type)
group = self._group_name(job_type)
try:
await self.redis.xgroup_create(stream, group, id="0-0", mkstream=True)
except Exception as exc: # pragma: no cover - redis-specific
if "BUSYGROUP" not in str(exc):
raise
async def close(self) -> None:
await self.redis.aclose()
async def _metrics(self, job_type: str) -> dict[str, Any]:
raw = await self.redis.hgetall(self._metrics_key(job_type))
queued = int(raw.get("queued_count", "0") or 0)
running = int(raw.get("running_count", "0") or 0)
config = _queue_config(job_type)
capacity = max(config.max_queue + config.concurrency, 1)
ratio = min((queued + running) / capacity, 1.0)
return {
"queued_count": queued,
"running_count": running,
"concurrency_limit": config.concurrency,
"max_queue": config.max_queue,
"busy_ratio": round(ratio, 4),
"busy_level": _busy_level(ratio),
}
async def _emit_event(self, job_id: str, event: str, data: dict[str, Any]) -> None:
key = self._event_key(job_id)
payload = {k: _json_dumps(v) if not isinstance(v, str) else v for k, v in data.items()}
payload["event"] = event
await self.redis.xadd(key, payload, maxlen=_int_env("JOB_EVENT_STREAM_MAXLEN", 512), approximate=True)
await self.redis.expire(key, self.event_ttl)
async def _set_state(self, job_id: str, state: dict[str, Any]) -> None:
serializable = {k: _json_dumps(v) if isinstance(v, (dict, list)) else str(v) for k, v in state.items()}
await self.redis.hset(self._state_key(job_id), mapping=serializable)
await self.redis.expire(self._state_key(job_id), self.state_ttl)
async def submit(self, job_type: str, payload: dict[str, Any], request_id: str | None = None) -> str:
config = _queue_config(job_type)
metrics = await self._metrics(job_type)
if metrics["queued_count"] >= config.max_queue:
raise QueueFullError(job_type, config.max_queue)
job_id = request_id or str(uuid.uuid4())
created_at = _now_ms()
state = {
"job_id": job_id,
"request_id": job_id,
"type": job_type,
"status": "queued",
"error": "",
"created_at": created_at,
"updated_at": created_at,
"started_at": 0,
"completed_at": 0,
"queue_ms": 0,
"run_ms": 0,
"total_ms": 0,
"cancel_requested": "0",
}
await self._set_state(job_id, state)
await self.redis.hincrby(self._metrics_key(job_type), "queued_count", 1)
await self.redis.expire(self._metrics_key(job_type), self.state_ttl)
metrics = await self._metrics(job_type)
await self._emit_event(job_id, "queued", {"job_id": job_id, "type": job_type, "status": "queued", **metrics})
await self.redis.xadd(self._queue_key(job_type), {"job_id": job_id, "payload": _json_dumps(payload), "request_id": job_id})
return job_id
async def cancel(self, job_id: str, reason: str = "abort") -> dict[str, Any]:
state = await self.get_status(job_id)
if not state:
return {"cancelled": False, "status": "not_found"}
if state["status"] in TERMINAL_STATUSES:
return {"cancelled": False, "status": state["status"]}
await self.redis.hset(self._state_key(job_id), mapping={"cancel_requested": "1", "status": "cancelled", "updated_at": _now_ms(), "cancel_reason": reason})
metrics = await self._metrics(state["type"])
await self._emit_event(job_id, "cancelled", {"job_id": job_id, "type": state["type"], "status": "cancelled", "reason": reason, **metrics})
return {"cancelled": True, "status": "ok"}
async def get_status(self, job_id: str) -> dict[str, Any] | None:
state = await self.redis.hgetall(self._state_key(job_id))
if not state:
return None
job_type = state.get("type", "")
metrics = await self._metrics(job_type) if job_type else {}
result = state.get("result")
error = state.get("error", "")
return {
"job_id": state.get("job_id", job_id),
"request_id": state.get("request_id", job_id),
"type": job_type,
"status": state.get("status", "queued"),
"error": error,
"result": _json_loads(result, result),
"cancel_requested": state.get("cancel_requested") == "1",
"created_at": int(state.get("created_at", "0") or 0),
"started_at": int(state.get("started_at", "0") or 0),
"completed_at": int(state.get("completed_at", "0") or 0),
"queue_ms": int(state.get("queue_ms", "0") or 0),
"run_ms": int(state.get("run_ms", "0") or 0),
"total_ms": int(state.get("total_ms", "0") or 0),
**metrics,
}
async def stream_events(self, job_id: str) -> AsyncIterator[dict[str, Any]]:
stream = self._event_key(job_id)
last_id = "0-0"
while True:
events = await self.redis.xread({stream: last_id}, block=1000, count=20)
if not events:
state = await self.get_status(job_id)
if state and state["status"] in TERMINAL_STATUSES:
break
continue
for _, entries in events:
for entry_id, fields in entries:
last_id = entry_id
event_payload: dict[str, Any] = {}
for key, value in fields.items():
if key == "event":
event_payload[key] = value
continue
try:
event_payload[key] = json.loads(value)
except Exception:
event_payload[key] = value
yield event_payload
if event_payload.get("event") in TERMINAL_EVENTS:
return
class RedisWorker:
def __init__(self, manager: RedisJobManager) -> None:
self.manager = manager
self.running_tasks: dict[str, asyncio.Task] = {}
self.queue_semaphores = {job_type: asyncio.Semaphore(_queue_config(job_type).concurrency) for job_type in JOB_TYPES}
self.poll_interval = _float_env("JOB_CANCEL_POLL_SECONDS", 0.5)
self.consumer_name = (os.getenv("JOB_CONSUMER_NAME") or f"worker-{uuid.uuid4().hex[:8]}").strip()
async def run_forever(self) -> None:
await self.manager.ensure_groups()
cancel_task = asyncio.create_task(self._cancel_watch_loop())
consumers = [asyncio.create_task(self._consume_loop(job_type)) for job_type in JOB_TYPES]
try:
await asyncio.gather(*consumers)
finally:
cancel_task.cancel()
with suppress(asyncio.CancelledError):
await cancel_task
async def _cancel_watch_loop(self) -> None:
while True:
await asyncio.sleep(self.poll_interval)
for job_id, task in list(self.running_tasks.items()):
state = await self.manager.get_status(job_id)
if state and state.get("cancel_requested") and not task.done():
task.cancel()
async def _consume_loop(self, job_type: str) -> None:
queue_key = self.manager._queue_key(job_type)
group = self.manager._group_name(job_type)
semaphore = self.queue_semaphores[job_type]
while True:
try:
streams = await self.manager.redis.xreadgroup(group, self.consumer_name, {queue_key: ">"}, count=1, block=1000)
except asyncio.CancelledError:
raise
except Exception as exc:
logger.warning("redis worker consume loop retrying type=%s error=%s", job_type, exc)
await asyncio.sleep(1)
continue
if not streams:
continue
for _, messages in streams:
for message_id, fields in messages:
await semaphore.acquire()
task = asyncio.create_task(self._run_message(job_type, queue_key, group, message_id, fields, semaphore))
self.running_tasks[fields["job_id"]] = task
async def _run_message(
self,
job_type: str,
queue_key: str,
group: str,
message_id: str,
fields: dict[str, str],
semaphore: asyncio.Semaphore,
) -> None:
job_id = fields["job_id"]
started_at = 0
created_at = 0
queue_ms = 0
try:
state = await self.manager.get_status(job_id)
if not state or state["status"] == "cancelled":
await self.manager.redis.xack(queue_key, group, message_id)
return
await self.manager.redis.hincrby(self.manager._metrics_key(job_type), "queued_count", -1)
await self.manager.redis.hincrby(self.manager._metrics_key(job_type), "running_count", 1)
started_at = _now_ms()
created_at = int(state.get("created_at", 0) or 0)
queue_ms = max(0, started_at - created_at)
await self.manager._set_state(job_id, {
"job_id": job_id,
"request_id": state["request_id"],
"type": job_type,
"status": "running",
"updated_at": started_at,
"created_at": created_at or started_at,
"started_at": started_at,
"completed_at": 0,
"queue_ms": queue_ms,
"run_ms": 0,
"total_ms": 0,
"cancel_requested": "1" if state.get("cancel_requested") else "0",
"error": "",
})
metrics = await self.manager._metrics(job_type)
await self.manager._emit_event(job_id, "started", {"job_id": job_id, "type": job_type, "status": "running", **metrics})
payload = _json_loads(fields["payload"], {})
payload["job_context"] = {
"job_id": job_id,
"created_at": created_at,
"started_at": started_at,
"queue_ms": queue_ms,
}
async def emit(event: str, data: dict[str, Any]) -> None:
live_state = await self.manager.get_status(job_id) or {"status": "running"}
live_metrics = await self.manager._metrics(job_type)
await self.manager._emit_event(job_id, event, {"job_id": job_id, "type": job_type, "status": live_state["status"], **live_metrics, **data})
def is_cancelled() -> bool:
task = self.running_tasks.get(job_id)
return bool(task and task.cancelled())
result = await self.manager.handlers[job_type](payload, emit, is_cancelled)
current = await self.manager.get_status(job_id)
if current and current["status"] == "cancelled":
await self.manager.redis.xack(queue_key, group, message_id)
return
await self.manager._set_state(job_id, {
"job_id": job_id,
"request_id": state["request_id"],
"type": job_type,
"status": "completed",
"updated_at": _now_ms(),
"created_at": created_at or started_at,
"started_at": started_at,
"completed_at": _now_ms(),
"queue_ms": queue_ms,
"run_ms": max(0, _now_ms() - started_at),
"total_ms": max(0, _now_ms() - (created_at or started_at)),
"cancel_requested": "0",
"error": "",
"result": _json_dumps(result),
})
metrics = await self.manager._metrics(job_type)
final_state = await self.manager.get_status(job_id) or {}
await self.manager._emit_event(
job_id,
"done",
{
"job_id": job_id,
"type": job_type,
"status": "completed",
"result": result,
"queue_ms": final_state.get("queue_ms", queue_ms),
"run_ms": final_state.get("run_ms", 0),
"total_ms": final_state.get("total_ms", 0),
**metrics,
},
)
await self.manager.redis.xack(queue_key, group, message_id)
except asyncio.CancelledError:
cancelled_at = _now_ms()
await self.manager.redis.hset(
self.manager._state_key(job_id),
mapping={
"status": "cancelled",
"cancel_requested": "1",
"updated_at": cancelled_at,
"completed_at": cancelled_at,
"run_ms": max(0, cancelled_at - started_at),
"total_ms": max(0, cancelled_at - (created_at or started_at)),
},
)
metrics = await self.manager._metrics(job_type)
await self.manager._emit_event(job_id, "cancelled", {"job_id": job_id, "type": job_type, "status": "cancelled", **metrics})
await self.manager.redis.xack(queue_key, group, message_id)
raise
except Exception as exc:
logger.exception("redis worker failed job_id=%s type=%s", job_id, job_type)
state = await self.manager.get_status(job_id)
request_id = state["request_id"] if state else job_id
await self.manager._set_state(job_id, {
"job_id": job_id,
"request_id": request_id,
"type": job_type,
"status": "failed",
"updated_at": _now_ms(),
"created_at": created_at or (_now_ms() if state else _now_ms()),
"started_at": started_at,
"completed_at": _now_ms(),
"queue_ms": queue_ms,
"run_ms": max(0, _now_ms() - started_at),
"total_ms": max(0, _now_ms() - (created_at or started_at)),
"cancel_requested": "0",
"error": str(exc),
})
metrics = await self.manager._metrics(job_type)
final_state = await self.manager.get_status(job_id) or {}
await self.manager._emit_event(
job_id,
"error",
{
"job_id": job_id,
"type": job_type,
"status": "failed",
"error": str(exc),
"queue_ms": final_state.get("queue_ms", queue_ms),
"run_ms": final_state.get("run_ms", 0),
"total_ms": final_state.get("total_ms", 0),
**metrics,
},
)
await self.manager.redis.xack(queue_key, group, message_id)
finally:
self.running_tasks.pop(job_id, None)
await self.manager.redis.hincrby(self.manager._metrics_key(job_type), "running_count", -1)
semaphore.release()
_job_manager: BaseJobManager | None = None
def get_job_manager() -> BaseJobManager:
global _job_manager
if _job_manager is None:
backend = get_job_backend_name()
if backend == "redis":
_job_manager = RedisJobManager()
else:
_job_manager = InMemoryJobManager()
return _job_manager
def reset_job_manager() -> None:
global _job_manager
manager = _job_manager
if manager is not None:
close = getattr(manager, "close", None)
if close is not None:
try:
loop = asyncio.get_running_loop()
except RuntimeError:
try:
asyncio.run(close())
except Exception:
pass
else:
loop.create_task(close())
_job_manager = None
backend/llm.py
+122 -542
View File
@@ -2,626 +2,206 @@ import os
import time
import logging
import asyncio
import inspect
import json
import base64
from datetime import datetime
from typing import AsyncIterator, Literal
import httpx
import ollama
from dotenv import load_dotenv
from prompts import get_vlm_ocr_prompt
load_dotenv()
# OpenAI-compatible endpoint config
LLM_BASE_URL = os.getenv('LLM_BASE_URL', 'http://localhost:11434/v1/')
LLM_API_KEY = os.getenv('LLM_API_KEY', 'ollama')
OLLAMA_MODEL = os.getenv('OLLAMA_MODEL', 'gpt-oss:20b')
OLLAMA_HOST = os.getenv('OLLAMA_HOST', 'http://localhost:11434')
VLM_MODEL = os.getenv('VLM_MODEL', 'qwen3-vl:30b')
# Auth headers for upstream LLM service (OpenAI-compatible Bearer token)
LLM_HEADERS = {'Authorization': f'Bearer {LLM_API_KEY}'}
# Timeouts in seconds
COMPLETION_TIMEOUT = 30
OCR_TIMEOUT = 60
CONVERT_TIMEOUT = 30
# Model names
DEFAULT_LLM_MODEL = 'Nex-N2-mini-mlx-OptiQ-8bit-MTP'
_raw_model = os.getenv('LLM_MODEL', DEFAULT_LLM_MODEL)
LLM_MODEL = _raw_model.strip() if _raw_model else DEFAULT_LLM_MODEL
PRO_LLM_MODEL = os.getenv('PRO_LLM_MODEL', LLM_MODEL)
client = ollama.AsyncClient(host=OLLAMA_HOST)
logger = logging.getLogger("llm")
# VLM for OCR (vision models)
VLM_MODEL = os.getenv('VLM_MODEL', DEFAULT_LLM_MODEL)
VLM_OCR_CONTEXT_PROMPT = """You are an OCR and visual-context extractor for markdown writing assistance.
# Normalize trailing slash for base URL
LLM_BASE_URL = LLM_BASE_URL.rstrip('/') + '/'
Your output will be embedded inside an HTML comment as hidden context for a text-completion model.
# Timeouts in seconds (10 minutes for large model loading)
COMPLETION_TIMEOUT = int(os.getenv("LLM_COMPLETION_TIMEOUT", "600"))
OCR_TIMEOUT = int(os.getenv("LLM_OCR_TIMEOUT", "600"))
Requirements:
- Keep output compact: maximum 120 words.
- Use plain text only (no markdown code fences).
- Never output <!-- or -->.
- Do not invent unreadable text; mark uncertain characters with ?.
- Preserve original script for recognized text (do not forcibly translate).
Return exactly this format:
async def _maybe_await(value):
if inspect.isawaitable(value):
return await value
return value
TEXT:
<exact transcription of visible text; use " | " for line breaks; write "(none)" if no readable text>
KEY_DETAILS:
- <3-5 short factual bullets about relevant objects/layout>
class _AsyncClientContext:
def __init__(self, client):
self.client = client
LANGUAGE:
<dominant language(s) in visible text, e.g. English / Chinese / Mixed>
async def __aenter__(self):
return self.client
SUMMARY:
<one short sentence, <= 20 words>"""
async def __aexit__(self, *args):
close = getattr(self.client, "aclose", None) or getattr(self.client, "close", None)
if close:
await _maybe_await(close())
def _extract_message(response) -> tuple[str, str]:
content = ""
thinking = ""
if hasattr(response, 'message') and response.message:
content = response.message.content or ""
thinking = getattr(response.message, 'thinking', '') or ""
elif isinstance(response, dict):
msg = response.get('message', {})
content = msg.get('content', '') or ""
thinking = msg.get('thinking', '') or ""
async def _create_async_client(timeout: httpx.Timeout):
client = await _maybe_await(
httpx.AsyncClient(base_url=LLM_BASE_URL, headers=LLM_HEADERS, timeout=timeout)
)
if hasattr(client, "__aenter__"):
return client
return _AsyncClientContext(client)
async def _client_post(client, url: str, payload: dict):
try:
return await client.post(url, json=payload)
except TypeError as exc:
raw_post = getattr(type(client), "__dict__", {}).get("post")
if raw_post is None or "multiple values for argument" not in str(exc):
raise
return await raw_post(url, json=payload)
async def _stream_line_iterator(response):
lines = await _maybe_await(response.aiter_lines())
if hasattr(lines, "__aiter__"):
return lines.__aiter__()
return lines
logger = logging.getLogger('llm')
def _extract_message(response: dict) -> tuple[str, str]:
"""Extract content and thinking from an OpenAI-compatible response dict."""
choices = response.get('choices', []) if isinstance(response, dict) else []
msg = (choices[0].get('message', {}) if choices and isinstance(choices, list) else {}).copy()
content = msg.get('content', '') or ''
thinking = (msg.get('reasoning_content') or msg.get('thinking', '') or '').strip()
return content, thinking
def _resolve_system_prompt(system_prompt: str | None) -> str:
if system_prompt and system_prompt.strip():
return system_prompt.strip()
return ''
def _resolve_model_name(model: str | None = None, *, use_pro_model: bool = False) -> str:
candidate = (model or '').strip()
if candidate:
return candidate
return PRO_LLM_MODEL if use_pro_model else LLM_MODEL
def _build_chat_payload(
prompt: str,
*,
system_prompt: str | None = None,
temperature: float = 0.7,
thinking: str | None = None,
model: str | None = None,
use_pro_model: bool = False,
prefill: str | None = None,
max_output_tokens: int | None = None,
) -> dict:
messages = []
sys_prompt = _resolve_system_prompt(system_prompt)
if sys_prompt:
messages.append({'role': 'system', 'content': sys_prompt})
messages.append({'role': 'user', 'content': prompt})
if prefill:
messages.append({'role': 'assistant', 'content': prefill})
options = {'temperature': temperature}
if thinking:
options['think'] = thinking
payload = {
'model': _resolve_model_name(model, use_pro_model=use_pro_model),
'messages': messages,
'stream': False,
'options': options,
}
if max_output_tokens and max_output_tokens > 0:
payload['max_tokens'] = int(max_output_tokens)
return payload
def _build_chat_stream_payload(
prompt: str,
*,
system_prompt: str | None = None,
temperature: float = 0.7,
thinking: str | None = None,
model: str | None = None,
use_pro_model: bool = False,
prefill: str | None = None,
max_output_tokens: int | None = None,
) -> dict:
messages = []
sys_prompt = _resolve_system_prompt(system_prompt)
if sys_prompt:
messages.append({'role': 'system', 'content': sys_prompt})
messages.append({'role': 'user', 'content': prompt})
if prefill:
messages.append({'role': 'assistant', 'content': prefill})
options = {'temperature': temperature}
if thinking:
options['think'] = thinking
payload = {
'model': _resolve_model_name(model, use_pro_model=use_pro_model),
'messages': messages,
'stream': True,
'options': options,
}
if max_output_tokens and max_output_tokens > 0:
payload['max_tokens'] = int(max_output_tokens)
return payload
def _extract_delta_text(chunk: dict) -> str:
"""Extract text delta from an OpenAI-compatible SSE chunk."""
choices = chunk.get('choices', []) if isinstance(chunk, dict) else []
delta = (choices[0].get('delta', {}) if choices and isinstance(choices, list) else {}).copy()
content = delta.get('content', '') or ''
return content
def _extract_delta_thinking(chunk: dict) -> str:
"""Extract thinking/reasoning delta from an SSE chunk."""
choices = chunk.get('choices', []) if isinstance(chunk, dict) else []
delta = (choices[0].get('delta', {}) if choices and isinstance(choices, list) else {}).copy()
return (delta.get('reasoning_content') or delta.get('thinking', '') or '').strip()
async def call_ollama(
prompt: str,
*,
system_prompt: str | None = None,
tag: str = 'default',
system_prompt: str = None,
tag: str = "default",
temperature: float = 0.7,
thinking: str | None = None,
model: str | None = None,
use_pro_model: bool = False,
prefill: str | None = None,
max_output_tokens: int | None = None,
thinking: str = None,
) -> dict:
"""Call OpenAI-compatible chat completions (non-streaming) and return content/thinking."""
"""
调用 Ollama API 并返回 content 和 thinking。
"""
start = time.perf_counter()
start_dt = datetime.now()
model_name = _resolve_model_name(model, use_pro_model=use_pro_model)
log_model_name = 'pro' if (model is None and use_pro_model) else model_name
logger.info(
'[LLM][%s] request model=%s base_url=%s prompt_chars=%d system_chars=%d temp=%.2f thinking=%s',
tag, log_model_name, LLM_BASE_URL, len(prompt),
len(system_prompt or ''), temperature, thinking,
"[LLM][%s] request model=%s host=%s prompt_chars=%d system_chars=%d temp=%.2f thinking=%s",
tag,
OLLAMA_MODEL,
OLLAMA_HOST,
len(prompt),
len(system_prompt or ""),
temperature,
thinking,
)
payload = _build_chat_payload(
prompt=prompt, system_prompt=system_prompt, temperature=temperature,
thinking=thinking, model=model, use_pro_model=use_pro_model, prefill=prefill,
max_output_tokens=max_output_tokens,
)
http_timeout = httpx.Timeout(connect=10.0, read=None, write=30.0, pool=30.0)
try:
async with await _create_async_client(http_timeout) as client:
resp = await asyncio.wait_for(
_client_post(client, '/chat/completions', payload), timeout=COMPLETION_TIMEOUT,
)
messages = []
if system_prompt and system_prompt.strip():
messages.append({"role": "system", "content": system_prompt})
messages.append({"role": "user", "content": prompt})
resp.raise_for_status()
response = resp.json()
kwargs = {
"model": OLLAMA_MODEL,
"messages": messages,
"stream": False,
"options": {
'temperature': temperature,
'repeat_penalty': 1.1,
},
}
if thinking:
kwargs["think"] = thinking
response = await asyncio.wait_for(client.chat(**kwargs), timeout=COMPLETION_TIMEOUT)
except asyncio.CancelledError:
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] call_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
"[LLM][%s] call_time [%s --> %s]",
tag,
start_dt.strftime("%H:%M:%S"),
end_dt.strftime("%H:%M:%S"),
)
logger.warning('[LLM][%s] request cancelled after %.1fms', tag, elapsed_ms)
logger.warning("[LLM][%s] request cancelled after %.1fms", tag, elapsed_ms)
raise
except Exception:
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] call_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
"[LLM][%s] call_time [%s --> %s]",
tag,
start_dt.strftime("%H:%M:%S"),
end_dt.strftime("%H:%M:%S"),
)
logger.exception('[LLM][%s] request failed after %.1fms', tag, elapsed_ms)
logger.exception("[LLM][%s] request failed after %.1fms", tag, elapsed_ms)
raise
content, thinking_out = _extract_message(response)
content, thinking = _extract_message(response)
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] call_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
"[LLM][%s] call_time [%s --> %s]",
tag,
start_dt.strftime("%H:%M:%S"),
end_dt.strftime("%H:%M:%S"),
)
logger.info(
'[LLM][%s] response in %.1fms content_chars=%d thinking_chars=%d',
tag, elapsed_ms, len(content), len(thinking_out or ''),
"[LLM][%s] response in %.1fms response_type=%s content_chars=%d thinking_chars=%d",
tag,
elapsed_ms,
type(response).__name__,
len(content),
len(thinking),
)
if not content.strip():
logger.warning('[LLM][%s] empty content returned by model', tag)
logger.warning("[LLM][%s] empty content returned by model", tag)
return {'content': content, 'think': thinking_out or ''}
return {"content": content, "think": thinking}
async def stream_ollama(
prompt: str,
*,
system_prompt: str | None = None,
tag: str = 'default-stream',
temperature: float = 0.7,
thinking: str | None = None,
model: str | None = None,
use_pro_model: bool = False,
prefill: str | None = None,
max_output_tokens: int | None = None,
) -> AsyncIterator[str]:
"""Stream text deltas from OpenAI-compatible chat completions."""
async def call_vlm_ocr(image_bytes: bytes, language: str = 'auto') -> str:
start = time.perf_counter()
start_dt = datetime.now()
model_name = _resolve_model_name(model, use_pro_model=use_pro_model)
log_model_name = 'pro' if (model is None and use_pro_model) else model_name
yielded_chars = 0
logger.info(
'[LLM][%s] stream request model=%s base_url=%s prompt_chars=%d system_chars=%d temp=%.2f thinking=%s',
tag, log_model_name, LLM_BASE_URL, len(prompt),
len(system_prompt or ''), temperature, thinking,
"[VLM][ocr] request model=%s host=%s image_bytes=%d language=%s",
VLM_MODEL,
OLLAMA_HOST,
len(image_bytes),
language,
)
payload = _build_chat_stream_payload(
prompt=prompt, system_prompt=system_prompt, temperature=temperature,
thinking=thinking, model=model, use_pro_model=use_pro_model, prefill=prefill,
max_output_tokens=max_output_tokens,
)
http_timeout = httpx.Timeout(connect=10.0, read=None, write=30.0, pool=30.0)
try:
async with await _create_async_client(http_timeout) as client:
try:
async with client.stream('POST', '/chat/completions', json=payload) as response:
await _maybe_await(response.raise_for_status())
deadline = time.perf_counter() + COMPLETION_TIMEOUT
line_iterator = await _stream_line_iterator(response)
while True:
remaining = deadline - time.perf_counter()
if remaining <= 0:
raise TimeoutError('LLM stream timed out')
try:
line = await asyncio.wait_for(line_iterator.__anext__(), timeout=remaining)
except StopAsyncIteration:
break
if not line or line.startswith(':'):
continue
# SSE data lines: "data: {json}" or "data: [DONE]"
if line.startswith('data: '):
data_str = line[6:] # strip "data: " prefix
else:
data_str = line.strip()
if not data_str or data_str == '[DONE]':
continue
try:
chunk = json.loads(data_str)
except json.JSONDecodeError:
logger.warning('[LLM][%s] ignored invalid stream line', tag)
continue
if not isinstance(chunk, dict):
continue
text = _extract_delta_text(chunk)
if not text:
continue
yielded_chars += len(text)
yield text
except asyncio.CancelledError:
if response is not None:
await response.aclose()
raise
except asyncio.CancelledError:
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] stream_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
response = await asyncio.wait_for(
client.chat(
model=VLM_MODEL,
messages=[{
'role': 'user',
'content': VLM_OCR_CONTEXT_PROMPT,
'images': [image_bytes]
}],
stream=False,
options={'temperature': 0.3}
),
timeout=OCR_TIMEOUT
)
logger.warning('[LLM][%s] stream cancelled after %.1fms', tag, elapsed_ms)
raise
except Exception:
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] stream_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
"[VLM][ocr] call_time [%s --> %s]",
start_dt.strftime("%H:%M:%S"),
end_dt.strftime("%H:%M:%S"),
)
logger.exception('[LLM][%s] stream failed after %.1fms', tag, elapsed_ms)
logger.exception("[VLM][ocr] request failed after %.1fms", elapsed_ms)
raise
content, thinking = _extract_message(response)
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] stream_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
"[VLM][ocr] call_time [%s --> %s]",
start_dt.strftime("%H:%M:%S"),
end_dt.strftime("%H:%M:%S"),
)
logger.info(
'[LLM][%s] stream finished in %.1fms yielded_chars=%d',
tag, elapsed_ms, yielded_chars,
)
async def stream_ollama_events(
prompt: str,
*,
system_prompt: str | None = None,
tag: str = 'default-events',
temperature: float = 0.7,
thinking: str | None = None,
model: str | None = None,
use_pro_model: bool = False,
enable_thinking: bool = True,
prefill: str | None = None,
timeout: float | None = None,
max_output_tokens: int | None = None,
) -> AsyncIterator[tuple[Literal['thinking', 'content'], str]]:
"""Stream (event_type, payload) tuples from OpenAI-compatible chat completions."""
start = time.perf_counter()
start_dt = datetime.now()
model_name = _resolve_model_name(model, use_pro_model=use_pro_model)
log_model_name = 'pro' if (model is None and use_pro_model) else model_name
yielded_chars = 0
logger.info(
'[LLM][%s] event_stream request model=%s base_url=%s prompt_chars=%d system_chars=%d temp=%.2f thinking=%s',
tag, log_model_name, LLM_BASE_URL, len(prompt),
len(system_prompt or ''), temperature, thinking,
)
payload = _build_chat_stream_payload(
prompt=prompt, system_prompt=system_prompt, temperature=temperature,
thinking=thinking if enable_thinking else None, model=model, use_pro_model=use_pro_model, prefill=prefill,
max_output_tokens=max_output_tokens,
)
effective_timeout = timeout if timeout is not None else COMPLETION_TIMEOUT
http_timeout = httpx.Timeout(connect=10.0, read=None, write=30.0, pool=30.0)
sent_thinking = False
try:
async with await _create_async_client(http_timeout) as client:
try:
async with client.stream('POST', '/chat/completions', json=payload) as response:
await _maybe_await(response.raise_for_status())
deadline = time.perf_counter() + effective_timeout
line_iterator = await _stream_line_iterator(response)
while True:
remaining = deadline - time.perf_counter()
if remaining <= 0:
raise TimeoutError('LLM event stream timed out')
try:
line = await asyncio.wait_for(line_iterator.__anext__(), timeout=remaining)
except StopAsyncIteration:
break
if not line or line.startswith(':'):
continue
# SSE data lines: "data: {json}" or "data: [DONE]"
if line.startswith('data: '):
data_str = line[6:] # strip "data: " prefix
else:
data_str = line.strip()
if not data_str or data_str == '[DONE]':
continue
try:
chunk = json.loads(data_str)
except json.JSONDecodeError:
logger.warning('[LLM][%s] ignored invalid Ollama stream line', tag)
continue
if not isinstance(chunk, dict):
continue
error = chunk.get('error')
if error:
raise RuntimeError(str(error))
thinking_delta = _extract_delta_thinking(chunk)
if thinking_delta and not sent_thinking:
sent_thinking = True
yield 'thinking', ''
text = _extract_delta_text(chunk)
if not text:
continue
yielded_chars += len(text)
yield 'content', text
except asyncio.CancelledError:
if response is not None:
await response.aclose()
raise
except asyncio.CancelledError:
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] event_stream_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
)
logger.warning('[LLM][%s] event stream cancelled after %.1fms', tag, elapsed_ms)
raise
except Exception:
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] event_stream_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
)
logger.exception('[LLM][%s] event stream failed after %.1fms', tag, elapsed_ms)
raise
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[LLM][%s] event_stream_time [%s --> %s]', tag,
start_dt.strftime('%H:%M:%S'), end_dt.strftime('%H:%M:%S'),
)
logger.info(
'[LLM][%s] event stream finished in %.1fms yielded_chars=%d thinking_seen=%s',
tag, elapsed_ms, yielded_chars, sent_thinking,
)
async def call_vlm_ocr(
media_bytes: bytes,
language: str = 'auto',
*,
mime_type: str = 'image/png',
media_type: str = 'image',
) -> str:
"""OCR via VLM using OpenAI-compatible multimodal API."""
start = time.perf_counter()
start_dt = datetime.now()
logger.info(
'[VLM][ocr] request model=%s base_url=%s media_type=%s media_bytes=%d language=%s mime=%s',
VLM_MODEL, LLM_BASE_URL, media_type, len(media_bytes), language, mime_type,
)
media_b64 = base64.b64encode(media_bytes).decode('ascii')
content_part_type = 'video_url' if media_type == 'video' else 'image_url'
url_key = 'video_url' if media_type == 'video' else 'image_url'
payload = {
'model': VLM_MODEL,
'messages': [{
'role': 'user',
'content': [
{'type': 'text', 'text': f"{get_vlm_ocr_prompt()}\n\nTarget language hint: {language or 'auto'}"},
{
'type': content_part_type,
url_key: {'url': f'data:{mime_type or "application/octet-stream"};base64,{media_b64}'},
},
],
}],
'stream': False,
'options': {
'temperature': 0,
'think': False,
},
}
http_timeout = httpx.Timeout(connect=10.0, read=None, write=30.0, pool=30.0)
try:
async with await _create_async_client(http_timeout) as client:
resp = await asyncio.wait_for(
_client_post(client, '/chat/completions', payload), timeout=OCR_TIMEOUT,
)
resp.raise_for_status()
response = resp.json()
except Exception:
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[VLM][ocr] call_time [%s --> %s]', start_dt.strftime('%H:%M:%S'),
end_dt.strftime('%H:%M:%S'),
)
logger.exception('[VLM][ocr] request failed after %.1fms', elapsed_ms)
raise
content, _ = _extract_message(response)
elapsed_ms = (time.perf_counter() - start) * 1000
end_dt = datetime.now()
logger.info(
'[VLM][ocr] call_time [%s --> %s]', start_dt.strftime('%H:%M:%S'),
end_dt.strftime('%H:%M:%S'),
)
logger.info(
'[VLM][ocr] response in %.1fms content_chars=%d', elapsed_ms, len(content),
"[VLM][ocr] response in %.1fms response_type=%s content_chars=%d thinking_chars=%d",
elapsed_ms,
type(response).__name__,
len(content),
len(thinking),
)
if not content.strip():
logger.warning('[VLM][ocr] empty content returned by model')
logger.warning("[VLM][ocr] empty content returned by model")
return content
backend/llm_policy.py
-100
View File
@@ -1,100 +0,0 @@
from dataclasses import dataclass
from typing import Any
from risk_config import RiskConfig
@dataclass(frozen=True)
class LLMPolicy:
job_type: str
model: str
profile: str
max_input_chars: int
max_output_tokens: int
temperature: float
thinking: str | None
def _normalize_thinking(value: str | None, *, allow_high: bool) -> str | None:
candidate = (value or "").strip().lower()
if candidate in {"", "none", "off"}:
return None
if candidate not in {"low", "medium", "high"}:
return "low"
if candidate == "high" and not allow_high:
return "medium"
return candidate
def resolve_llm_policy(job_type: str, request_payload: dict[str, Any], config: RiskConfig) -> LLMPolicy:
if job_type == "completion":
return LLMPolicy(
job_type=job_type,
model=config.completion_model,
profile="completion",
max_input_chars=config.completion_max_input_chars,
max_output_tokens=config.completion_max_output_tokens,
temperature=config.completion_temperature,
thinking=_normalize_thinking(request_payload.get("model_thinking"), allow_high=False),
)
if job_type == "pro_completion":
return LLMPolicy(
job_type=job_type,
model=config.pro_model,
profile="pro",
max_input_chars=config.pro_max_input_chars,
max_output_tokens=config.pro_max_output_tokens,
temperature=config.pro_temperature,
thinking=_normalize_thinking(request_payload.get("pro_thinking"), allow_high=True) or "medium",
)
if job_type == "web_search":
return LLMPolicy(
job_type=job_type,
model=config.web_search_model,
profile="completion",
max_input_chars=config.web_search_max_input_chars,
max_output_tokens=config.web_search_max_output_tokens,
temperature=config.web_search_temperature,
thinking="low",
)
if job_type == "compress":
return LLMPolicy(
job_type=job_type,
model=config.completion_model,
profile="completion",
max_input_chars=config.compress_max_input_chars,
max_output_tokens=config.compress_max_output_tokens,
temperature=0.2,
thinking="low",
)
if job_type == "ocr":
return LLMPolicy(
job_type=job_type,
model=config.vision_model,
profile="vision",
max_input_chars=config.ocr_max_input_bytes,
max_output_tokens=config.completion_max_output_tokens,
temperature=0.0,
thinking=None,
)
if job_type == "tts":
return LLMPolicy(
job_type=job_type,
model=config.speech_tts_model,
profile="speech_tts",
max_input_chars=config.speech_tts_max_input_chars,
max_output_tokens=0,
temperature=0.0,
thinking=None,
)
if job_type == "asr":
return LLMPolicy(
job_type=job_type,
model=config.speech_asr_model,
profile="speech_asr",
max_input_chars=config.speech_asr_max_input_bytes,
max_output_tokens=0,
temperature=0.0,
thinking=None,
)
raise ValueError(f"unsupported llm policy job type: {job_type}")
backend/main.py
+250 -921
View File
File diff suppressed because it is too large Load Diff
backend/media_utils.py
-47
View File
@@ -1,47 +0,0 @@
from __future__ import annotations
import os
import subprocess
import tempfile
VIDEO_EXTENSIONS = {".mp4", ".webm", ".mov", ".avi", ".mkv", ".m4v", ".ogv"}
def is_video_filename(filename: str = "", mime_type: str = "") -> bool:
ext = os.path.splitext(filename or "")[1].lower()
mime = (mime_type or "").strip().lower()
return ext in VIDEO_EXTENSIONS or mime.startswith("video/")
def extract_audio_wav_bytes(input_path: str) -> bytes:
if not input_path or not os.path.exists(input_path):
raise FileNotFoundError("输入媒体文件不存在")
fd, output_path = tempfile.mkstemp(suffix=".wav")
os.close(fd)
try:
subprocess.run(
[
"ffmpeg",
"-y",
"-i",
input_path,
"-vn",
"-acodec",
"pcm_s16le",
"-ar",
"16000",
"-ac",
"1",
output_path,
],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
with open(output_path, "rb") as handle:
return handle.read()
finally:
if os.path.exists(output_path):
os.unlink(output_path)
backend/models.py
-9
View File
@@ -1,9 +0,0 @@
"""共享的 Pydantic 模型定义"""
from pydantic import BaseModel
class UserPreferences(BaseModel):
"""用户偏好设置"""
language: str = "auto"
country: str = "auto"
timezone: str = "auto"
backend/pro_completions.py
-377
View File
@@ -1,377 +0,0 @@
import asyncio
import contextlib
import json
import logging
import os
import time
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional
from fastapi import FastAPI, HTTPException, Request, Security
from fastapi.responses import JSONResponse, StreamingResponse
from pydantic import BaseModel
from geoip import get_ip_location_text
from llm import stream_ollama_events
from models import UserPreferences
from prompt import build_pro_completion_prompts
logger = logging.getLogger("api.pro")
PRO_COMPLETION_TIMEOUT = float(os.getenv("PRO_COMPLETION_TIMEOUT", "3600"))
PRO_QUEUE_TIMEOUT = float(os.getenv("PRO_QUEUE_TIMEOUT", "600"))
PRO_MAX_CONCURRENCY = max(1, int(os.getenv("PRO_MAX_CONCURRENCY", "1")))
PRO_QUEUE_MAX_SIZE = max(0, int(os.getenv("PRO_QUEUE_MAX_SIZE", "5")))
PRO_STATUS_RETENTION_SECONDS = float(os.getenv("PRO_STATUS_RETENTION_SECONDS", "600"))
PRO_CANCEL_ACK_TIMEOUT = 5.0
STREAM_HEARTBEAT_SECONDS = float(os.getenv("STREAM_HEARTBEAT_SECONDS", "2"))
PUBLIC_PRO_ERROR = "PRO generation failed. Please retry or adjust the instruction."
class ProCompletionRequest(BaseModel):
prefix: str
suffix: str
languageId: str = "markdown"
instruction: str = ""
pro_thinking: str = "medium"
privacy_mode: bool = False
user_preferences: Optional[UserPreferences] = None
class ProCancelRequest(BaseModel):
request_id: str
reason: str = "abort"
@dataclass
class ProRequestState:
request_id: str
status: str = "queued"
created_at: float = field(default_factory=time.time)
updated_at: float = field(default_factory=time.time)
error: str = ""
task: asyncio.Task | None = None
cancel_requested: bool = False
done_event: asyncio.Event = field(default_factory=asyncio.Event)
def touch(self, status: str | None = None, error: str = "") -> None:
if status:
self.status = status
if error:
self.error = error
self.updated_at = time.time()
def request_cancel(self) -> None:
self.cancel_requested = True
self.touch("cancelled")
PRO_STATES: dict[str, ProRequestState] = {}
PRO_STATES_LOCK = asyncio.Lock()
PRO_SEMAPHORE = asyncio.Semaphore(PRO_MAX_CONCURRENCY)
def _iso_timestamp(value: float) -> str:
return datetime.fromtimestamp(value, tz=timezone.utc).isoformat()
def _clamp_thinking(value: str | None) -> str | None:
normalized = (value or "medium").strip().lower()
if normalized in {"none", "off", "false"}:
return None
if normalized in {"low", "medium", "high"}:
return normalized
return "medium"
def _queued_states() -> list[ProRequestState]:
return [state for state in PRO_STATES.values() if state.status == "queued"]
def _queue_position(request_id: str) -> int | None:
queued = sorted(_queued_states(), key=lambda item: item.created_at)
for index, state in enumerate(queued, start=1):
if state.request_id == request_id:
return index
return None
async def _cleanup_states() -> None:
now = time.time()
expired = [
request_id
for request_id, state in PRO_STATES.items()
if state.status in {"done", "error", "cancelled"}
and now - state.updated_at > PRO_STATUS_RETENTION_SECONDS
]
for request_id in expired:
PRO_STATES.pop(request_id, None)
def _state_payload(state: ProRequestState) -> dict:
return {
"request_id": state.request_id,
"status": state.status,
"queue_position": _queue_position(state.request_id),
"created_at": _iso_timestamp(state.created_at),
"updated_at": _iso_timestamp(state.updated_at),
"error": state.error,
}
def _build_pro_prompts(
*,
prefix: str,
suffix: str,
language_id: str,
instruction: str,
pro_thinking: str = "medium",
location: str = "",
preferences: UserPreferences | None = None,
) -> tuple[str, str]:
return build_pro_completion_prompts(
prefix=prefix,
suffix=suffix,
instruction=instruction,
language_id=language_id,
location=location,
pro_thinking_level=pro_thinking,
preferences=preferences,
)
def _get_client_ip(request: Request) -> str:
if request.client:
return request.headers.get("X-Client-IP") or request.client.host
return request.headers.get("X-Client-IP") or "unknown"
async def _send_sse_event(queue: asyncio.Queue, event_name: str, data: dict) -> None:
await queue.put((event_name, json.dumps(data, ensure_ascii=False)))
async def _wait_for_cancel_cleanup(state: ProRequestState, request_tag: str, reason: str) -> None:
if state.done_event.is_set():
return
try:
await asyncio.wait_for(state.done_event.wait(), timeout=PRO_CANCEL_ACK_TIMEOUT)
except asyncio.TimeoutError:
logger.warning(
"[%s] /v1/pro/completions cancel cleanup not confirmed request_id=%s reason=%s",
request_tag,
state.request_id,
reason,
)
def register_pro_completion_routes(app: FastAPI, get_api_key):
@app.post("/v1/pro/completions")
async def create_pro_completion(
request: Request,
req: ProCompletionRequest,
api_key: str = Security(get_api_key),
):
request_id = request.headers.get("X-Request-Id") or str(uuid.uuid4())
request_tag = request_id[:8]
event_queue: asyncio.Queue[tuple[str, str] | None] = asyncio.Queue()
previous_state: ProRequestState | None = None
async with PRO_STATES_LOCK:
await _cleanup_states()
queued_count = len(_queued_states())
if queued_count >= PRO_QUEUE_MAX_SIZE:
logger.info("[%s] /v1/pro/completions rejected queue_full request_id=%s", request_tag, request_id)
return JSONResponse(
content={"error": "PRO queue is full", "request_id": request_id},
status_code=429,
)
existing = PRO_STATES.get(request_id)
if existing and existing.task and not existing.task.done():
existing.request_cancel()
existing.task.cancel()
previous_state = existing
state = ProRequestState(request_id=request_id)
PRO_STATES[request_id] = state
if previous_state:
await _wait_for_cancel_cleanup(previous_state, request_tag, "replace")
client_ip = "hidden"
location = ""
if not req.privacy_mode: # pragma: no cover
client_ip = _get_client_ip(request)
location = get_ip_location_text(client_ip)
prefix = req.prefix or ""
suffix = req.suffix or ""
system_prompt, user_prompt = _build_pro_prompts(
prefix=prefix,
suffix=suffix,
language_id=req.languageId,
instruction=req.instruction,
pro_thinking=req.pro_thinking,
location=location,
preferences=req.user_preferences,
)
logger.info(
"[%s] /v1/pro/completions request_id=%s client_ip=%s prefix_chars=%d suffix_chars=%d instruction_chars=%d lang=%s thinking=%s",
request_tag,
request_id,
client_ip,
len(prefix),
len(suffix),
len(req.instruction or ""),
req.languageId,
req.pro_thinking,
)
async def producer() -> None:
acquired = False
chunks: list[str] = []
try:
async with PRO_STATES_LOCK:
if state.cancel_requested:
raise asyncio.CancelledError()
state.touch("queued")
queue_position = _queue_position(request_id)
await _send_sse_event(event_queue, "queued", {"request_id": request_id, "queue_position": queue_position})
await asyncio.wait_for(PRO_SEMAPHORE.acquire(), timeout=PRO_QUEUE_TIMEOUT)
acquired = True
async with PRO_STATES_LOCK:
if state.cancel_requested:
raise asyncio.CancelledError()
state.touch("started")
await _send_sse_event(event_queue, "started", {"request_id": request_id})
async for event_type, payload in stream_ollama_events(
user_prompt,
system_prompt=system_prompt,
tag=f"{request_tag}-pro",
temperature=0.7,
thinking=_clamp_thinking(req.pro_thinking),
use_pro_model=True,
enable_thinking=True,
timeout=PRO_COMPLETION_TIMEOUT,
):
# Handle 'thinking' event - just update state, don't accumulate
if event_type == "thinking":
await _send_sse_event(event_queue, "thinking", {"request_id": request_id})
continue
# Handle 'chunk' event - accumulate content
if not payload:
continue
chunks.append(payload)
await _send_sse_event(event_queue, "chunk", {"delta": payload, "request_id": request_id})
# Handle 'done' event - return full content
content = "".join(chunks)
async with PRO_STATES_LOCK:
if state.cancel_requested:
raise asyncio.CancelledError()
if not content:
raise ValueError("PRO returned empty content")
async with PRO_STATES_LOCK:
state.touch("done")
logger.info("[%s] /v1/pro/completions done request_id=%s content_chars=%d", request_tag, request_id, len(content))
await _send_sse_event(event_queue, "done", {"content": content, "request_id": request_id})
except asyncio.CancelledError:
async with PRO_STATES_LOCK:
state.request_cancel()
logger.info("[%s] /v1/pro/completions cancelled request_id=%s", request_tag, request_id)
await _send_sse_event(event_queue, "cancelled", {"cancelled": True, "request_id": request_id})
raise
except Exception as exc:
async with PRO_STATES_LOCK:
state.touch("error", PUBLIC_PRO_ERROR)
logger.exception("[%s] /v1/pro/completions failed request_id=%s", request_tag, request_id)
await _send_sse_event(event_queue, "error", {"error": PUBLIC_PRO_ERROR, "request_id": request_id})
finally:
if acquired:
PRO_SEMAPHORE.release()
state.done_event.set()
await event_queue.put(None)
producer_task = asyncio.create_task(producer())
async with PRO_STATES_LOCK:
state.task = producer_task
async def event_stream():
try:
while True:
try:
item = await asyncio.wait_for(event_queue.get(), timeout=STREAM_HEARTBEAT_SECONDS)
except asyncio.TimeoutError:
yield ": keepalive\n\n"
continue
if item is None:
break
event_name, data = item
yield f"event: {event_name}\ndata: {data}\n\n"
if event_name in {"done", "error", "cancelled"}:
break
except asyncio.CancelledError:
async with PRO_STATES_LOCK:
state.request_cancel()
producer_task.cancel()
raise
finally:
if not producer_task.done() and not state.done_event.is_set():
async with PRO_STATES_LOCK:
state.request_cancel()
producer_task.cancel()
with contextlib.suppress(asyncio.TimeoutError):
await asyncio.wait_for(state.done_event.wait(), timeout=PRO_CANCEL_ACK_TIMEOUT)
return StreamingResponse(
event_stream(),
media_type="text/event-stream; charset=utf-8",
headers={
"Cache-Control": "no-cache, no-transform",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
},
)
@app.post("/v1/pro/completions/cancel")
async def cancel_pro_completion(req: ProCancelRequest, api_key: str = Security(get_api_key)):
request_id = req.request_id or ""
request_tag = request_id[:8]
state_to_wait: ProRequestState | None = None
async with PRO_STATES_LOCK:
await _cleanup_states()
state = PRO_STATES.get(request_id)
if not state:
return {"cancelled": False, "status": "not_found"}
if state.task and not state.task.done():
state.request_cancel()
state.task.cancel()
state_to_wait = state
if state.status in {"done", "error", "cancelled"}:
if not state_to_wait:
return {"cancelled": False, "status": state.status}
else:
state.request_cancel()
if state_to_wait:
await _wait_for_cancel_cleanup(state_to_wait, request_tag, req.reason)
return {"cancelled": True, "status": "ok"}
@app.get("/v1/pro/completions/status/{request_id}")
async def get_pro_completion_status(request_id: str, api_key: str = Security(get_api_key)):
async with PRO_STATES_LOCK:
await _cleanup_states()
state = PRO_STATES.get(request_id)
if not state:
raise HTTPException(status_code=404, detail="PRO request not found")
return _state_payload(state)
backend/prompt.py
+326 -221
View File
@@ -1,17 +1,7 @@
from collections.abc import Mapping
from datetime import datetime, timedelta, timezone
import re
from typing import Tuple
from models import UserPreferences
from prompts import (
get_inline_examples,
get_inline_examples_pro,
get_language_guidance_map,
get_system_prompt_pro_template,
get_system_prompt_template,
)
def _get_current_datetime(timezone_pref: str = "auto") -> str:
# Default to UTC+8 if auto or not specified.
@@ -46,16 +36,6 @@ def _get_current_datetime(timezone_pref: str = "auto") -> str:
)
def _normalize_preferences(preferences: UserPreferences | Mapping | None) -> UserPreferences | None:
if preferences is None:
return None
if isinstance(preferences, UserPreferences):
return preferences
if isinstance(preferences, Mapping):
return UserPreferences(**preferences)
return preferences
def _sanitize_language_id(language_id: str) -> str:
if not language_id:
return "markdown"
@@ -82,53 +62,6 @@ def _prepare_context(prefix: str, suffix: str) -> Tuple[str, str]:
return clean_prefix, clean_suffix
def _strip_hidden_tail_context(text: str) -> str:
"""
Return the likely visible tail segment used for prefill.
The frontend prepends hidden OCR/doc context before the visible markdown and
joins those blocks with blank lines. For prefill we only want the active
visible segment near the cursor, not earlier hidden context.
"""
value = _normalize_newlines(text or "")
if not value:
return ""
tail = re.split(r"\n{2,}", value)[-1]
tail = re.sub(r"<!--[\s\S]*?-->", "", tail)
tail = re.sub(r"<OCR:[^>\n]*>", "", tail)
return tail.split("\n")[-1]
def _build_completion_prefill(prefix: str) -> str:
"""
Build a short tail prefill after <|fim_middle|> so completion models keep
writing from the existing text instead of explaining the boundary rules.
"""
normalized = _normalize_newlines(prefix or "")
if not normalized or normalized[-1].isspace():
return ""
tail = _strip_hidden_tail_context(normalized).strip()
if len(tail) < 2:
return ""
cjk_match = re.search(r"[\u3400-\u9fff]{2,6}$", tail)
if cjk_match:
value = cjk_match.group(0)
return value[-2:] if len(value) > 2 else value
token_match = re.search(r"[A-Za-z0-9_+\-.]{2,12}$", tail)
if token_match:
value = token_match.group(0)
return value[-12:]
compact_match = re.search(r"\S{2,12}$", tail)
if compact_match:
return compact_match.group(0)[-12:]
return ""
FENCE_LINE_RE = re.compile(r"^[ \t]*```.*$")
FENCE_INFO_RE = re.compile(r"^[ \t]*```[ \t]*(.*)$")
MERMAID_CONTEXT_RE = re.compile(
@@ -281,47 +214,304 @@ def _canonical_language_id(language_id: str) -> str:
return LANGUAGE_SYNONYMS.get(safe, safe)
_JS_LANGS = {"javascript", "typescript"}
_CODE_LANGS = {"python", "go", "rust", "java", "kotlin", "swift", "ruby", "php", "lua", "c", "cpp", "csharp", "r", "matlab", "dart"}
def _language_guidance(language_id: str) -> str:
canonical = _canonical_language_id(language_id)
if canonical == "markdown":
return ""
guidance_map = get_language_guidance_map()
guidance = guidance_map.get(canonical)
if guidance:
return guidance
if canonical in _JS_LANGS:
return guidance_map.get("_js_code", "").replace("{lang}", canonical)
if canonical in _CODE_LANGS:
return guidance_map.get("_generic_code", "").replace("{lang}", canonical)
return guidance_map.get("_generic_code", "").replace("{lang}", canonical)
if canonical == "mermaid":
return """
Language-specific guidance (mermaid):
- Output valid Mermaid syntax only.
- Prefer concise, syntactically correct diagram statements.
- Avoid prose unless the user prompt explicitly requires it."""
if canonical == "latex":
return """
Language-specific guidance (latex):
- Output LaTeX math content only when completing LaTeX.
- If CURSOR_IN_FENCED_CODE_BLOCK=true and CURSOR_FENCE_LANGUAGE is latex/tex/katex:
- Output raw LaTeX lines only.
- Do not wrap with $ or $$."""
if canonical == "json":
return """
Language-specific guidance (json):
- Output strict JSON only (no comments, no trailing commas).
- Ensure valid quotes and braces."""
if canonical == "yaml":
return """
Language-specific guidance (yaml):
- Output valid YAML only.
- Use consistent indentation and avoid tabs."""
if canonical == "toml":
return """
Language-specific guidance (toml):
- Output valid TOML only.
- Keep key types consistent."""
if canonical == "ini":
return """
Language-specific guidance (ini):
- Output valid INI only.
- Keep section headers and key=value pairs consistent."""
if canonical == "sql":
return """
Language-specific guidance (sql):
- Output a single, valid SQL statement unless context requires multiple.
- Prefer ANSI SQL when dialect is unclear."""
if canonical == "bash":
return """
Language-specific guidance (bash):
- Output POSIX-compatible shell when possible.
- Avoid interactive prompts or destructive commands unless requested."""
if canonical == "powershell":
return """
Language-specific guidance (powershell):
- Output valid PowerShell commands.
- Avoid destructive commands unless explicitly requested."""
if canonical == "html":
return """
Language-specific guidance (html):
- Output valid HTML only.
- Keep markup minimal and well-formed."""
if canonical == "css":
return """
Language-specific guidance (css):
- Output valid CSS only.
- Use concise, readable selectors."""
if canonical == "diff":
return """
Language-specific guidance (diff):
- Output a unified diff only.
- Ensure @@ hunk headers and +/- lines are consistent."""
if canonical == "regex":
return """
Language-specific guidance (regex):
- Output the regex pattern only.
- Avoid delimiters unless explicitly requested."""
if canonical in {"javascript", "typescript"}:
return f"""
Language-specific guidance ({canonical}):
- Output valid {canonical} code.
- Prefer modern syntax and avoid prose unless comments are needed."""
if canonical in {"python", "go", "rust", "java", "kotlin", "swift", "ruby", "php", "lua", "c", "cpp", "csharp", "r", "matlab", "dart"}:
return f"""
Language-specific guidance ({canonical}):
- Output valid {canonical} code.
- Avoid prose unless context clearly expects comments or docstrings."""
if canonical == "text":
return """
Language-specific guidance (text):
- Output plain text only.
- Avoid markdown formatting unless explicitly asked."""
if canonical == "xml":
return """
Language-specific guidance (xml):
- Output well-formed XML only.
- Ensure matching tags and proper escaping."""
if canonical == "dockerfile":
return """
Language-specific guidance (dockerfile):
- Output valid Dockerfile instructions only.
- Keep layers minimal and ordered logically."""
if canonical == "makefile":
return """
Language-specific guidance (makefile):
- Output valid Makefile syntax only.
- Use tabs for recipe lines."""
return f"""
Language-specific guidance ({canonical}):
- Output valid {canonical} code.
- Avoid prose unless context clearly expects comments or docstrings."""
def build_inline_system_prompt(language_id: str = "markdown") -> str:
safe_language_id = _canonical_language_id(language_id)
language_guidance = _language_guidance(safe_language_id)
template = get_system_prompt_template()
system_prompt = template.replace("{language_id}", safe_language_id)
system_prompt = f"""You are an inline completion engine for a {safe_language_id} editor with ghost-text suggestions.
Return only the insertion text that should be placed between PREFIX and SUFFIX.
Hard constraints you must follow:
1) Output-only contract:
- Output insertion text only.
- No explanations, no meta labels, no wrapper quotes around the whole answer.
2) Strict math formatting (KaTeX):
- If you output any math expression, it must be strict KaTeX-compatible math.
- Every formula must be wrapped with either $...$ (inline) or $$...$$ (block).
- Never output bare formulas without $ or $$ wrappers.
- Exception: If CURSOR_IN_FENCED_CODE_BLOCK=true and CURSOR_FENCE_LANGUAGE is latex/tex/katex,
output raw LaTeX without $ or $$ wrappers.
3) Strict code formatting:
- Read CURSOR_IN_FENCED_CODE_BLOCK from the user prompt.
- If CURSOR_IN_FENCED_CODE_BLOCK=true:
- You are already inside a fenced code block.
- Never output triple backticks.
- Output code lines only.
- If CURSOR_IN_FENCED_CODE_BLOCK=false:
- Any code output must be in a fenced code block with a language tag:
```{{language}}
...
```
- Do not output code snippets as inline backticks.
- Choose the language tag from context (no default fallback tag instruction).
4) Mermaid-specific completion rules:
- Read CURSOR_FENCE_LANGUAGE and MERMAID_CONTEXT from the user prompt.
- If CURSOR_FENCE_LANGUAGE=mermaid:
- Output Mermaid statements only.
- Never output triple backticks.
- Never output prose explanations.
- If CURSOR_IN_FENCED_CODE_BLOCK=false and MERMAID_CONTEXT=true:
- Output a complete Mermaid fenced block:
```mermaid
...
```
- Keep Mermaid syntax valid and concise.
- Never mix Mermaid code and explanatory narration in one output.
5) Boundary newline repair:
- Read PREFIX_ENDS_WITH_NEWLINE and SUFFIX_STARTS_WITH_NEWLINE from the user prompt.
- Carefully reason about whether OUTPUT should start or end with a newline.
- If PREFIX lacks a required boundary newline, add it at OUTPUT start.
- If SUFFIX lacks a required boundary newline, add it at OUTPUT end.
- Ensure PREFIX + OUTPUT + SUFFIX is structurally natural.
6) Context stitching:
- Do not repeat text that already appears at the start of SUFFIX.
- Preserve nearby language, tone, punctuation, indentation, and markdown structure.
- Continue existing structures naturally (lists, tables, block quotes, headings).
7) OCR safety:
- PREFIX may include hidden OCR metadata tags like <OCR:...>.
- Never output any OCR tag.
- Never output OCR tag fragments such as <OCR:...>."""
if language_guidance:
system_prompt = f"{system_prompt.rstrip()}\n{language_guidance.strip()}"
return system_prompt.strip()
_INLINE_EXAMPLES = get_inline_examples()
_PRO_INLINE_EXAMPLES = get_inline_examples_pro()
INLINE_EXAMPLES = """[EX01] Prose continuation
<PREFIX>The quick brown fox </PREFIX>
<SUFFIX>jumps over the lazy dog.</SUFFIX>
Expected OUTPUT:
moved quietly and then
[EX02] Avoid repeating suffix beginning
<PREFIX>Our launch plan starts with </PREFIX>
<SUFFIX>phase one, followed by phase two.</SUFFIX>
Expected OUTPUT:
careful internal testing before
[EX03] Continue markdown checklist
<PREFIX>## TODO
- [ ] Buy milk
- [ ] </PREFIX>
<SUFFIX></SUFFIX>
Expected OUTPUT:
Write release notes and share draft with team
[EX04] Cursor outside code block, code must use fenced block
CURSOR_IN_FENCED_CODE_BLOCK=false
<PREFIX>Parse this JSON payload in Python:</PREFIX>
<SUFFIX></SUFFIX>
Expected OUTPUT:
```python
import json
data = json.loads(payload)
```
[EX05] Cursor inside fenced code block, do not output fences
CURSOR_IN_FENCED_CODE_BLOCK=true
<PREFIX>```python
def add(a, b):
return </PREFIX>
<SUFFIX>
```</SUFFIX>
Expected OUTPUT:
a + b
[EX06] Inline math must use $...$
<PREFIX>The derivative of x^2 is </PREFIX>
<SUFFIX>.</SUFFIX>
Expected OUTPUT:
$2x$
[EX07] Block math must use $$...$$
<PREFIX>We can write the Gaussian integral as:</PREFIX>
<SUFFIX></SUFFIX>
Expected OUTPUT:
$$
\\int_{-\\infty}^{\\infty} e^{-x^2}\\,dx = \\sqrt{\\pi}
$$
[EX08] Prefix misses boundary newline; add newline at output start
PREFIX_ENDS_WITH_NEWLINE=false
<PREFIX>Deployment steps:</PREFIX>
<SUFFIX></SUFFIX>
Expected OUTPUT:
- Build artifact
- Deploy service
[EX09] Suffix misses boundary newline; add newline at output end
SUFFIX_STARTS_WITH_NEWLINE=false
<PREFIX>Summary paragraph complete.</PREFIX>
<SUFFIX>## Next Section</SUFFIX>
Expected OUTPUT:
def build_pro_system_prompt(language_id: str = "markdown") -> str:
safe_language_id = _canonical_language_id(language_id)
language_guidance = _language_guidance(safe_language_id)
template = get_system_prompt_pro_template()
system_prompt = template.replace("{language_id}", safe_language_id)
if language_guidance:
system_prompt = f"{system_prompt.rstrip()}\n{language_guidance.strip()}"
return system_prompt.strip()
[EX10] OCR metadata exists but must never be emitted
<PREFIX>![whiteboard](img.png) <OCR:equation y = mx + b>
The relationship is </PREFIX>
<SUFFIX>.</SUFFIX>
Expected OUTPUT:
$y = mx + b$
[EX11] Continue markdown table with correct row shape
<PREFIX>| Name | Score |
| --- | --- |
| Alice | 92 |
| Bob | </PREFIX>
<SUFFIX></SUFFIX>
Expected OUTPUT:
88 |
[EX12] Mixed text + math + code in one insertion
CURSOR_IN_FENCED_CODE_BLOCK=false
<PREFIX>Use the area formula and provide a tiny JS helper.</PREFIX>
<SUFFIX></SUFFIX>
Expected OUTPUT:
The area is $A = \\pi r^2$.
```javascript
const area = (r) => Math.PI * r * r;
```
[EX13] Cursor inside mermaid fence: no backticks, mermaid lines only
CURSOR_IN_FENCED_CODE_BLOCK=true
CURSOR_FENCE_LANGUAGE=mermaid
<PREFIX>```mermaid
flowchart TD
A[Start] --> </PREFIX>
<SUFFIX>
```</SUFFIX>
Expected OUTPUT:
B{Valid?}
B -->|Yes| C[Done]
[EX14] Mermaid context outside fence: return full mermaid block
CURSOR_IN_FENCED_CODE_BLOCK=false
MERMAID_CONTEXT=true
<PREFIX>Please provide a simple release pipeline diagram.</PREFIX>
<SUFFIX></SUFFIX>
Expected OUTPUT:
```mermaid
flowchart LR
Build --> Test --> Deploy
```"""
def build_completion_prompts(
@@ -330,109 +520,8 @@ def build_completion_prompts(
language_id: str = "markdown",
location: str = "",
thinking_level: str = "low",
preferences: UserPreferences | None = None,
) -> Tuple[str, str, str]:
preferences = _normalize_preferences(preferences)
safe_language_id = _canonical_language_id(language_id)
recent_prefix, recent_suffix = _prepare_context(prefix, suffix)
recent_prefix = _normalize_newlines(recent_prefix)
recent_suffix = _normalize_newlines(recent_suffix)
cursor_fence_language = _active_fence_language(recent_prefix)
cursor_in_fenced_code_block = cursor_fence_language != "none"
mermaid_context = _is_mermaid_context(
recent_prefix, recent_suffix, cursor_fence_language
)
prefix_ends_with_newline = recent_prefix.endswith("\n")
suffix_starts_with_newline = recent_suffix.startswith("\n")
prefill = _build_completion_prefill(recent_prefix)
tz_pref = preferences.timezone if preferences else "auto"
current_time = _get_current_datetime(tz_pref)
location_info = f"\nUser location: {location}" if location else ""
pref_info = []
if preferences:
if preferences.language and preferences.language != "auto":
pref_info.append(f"Preferred language: {preferences.language}")
if preferences.country and preferences.country != "auto":
pref_info.append(f"Preferred country: {preferences.country}")
preferences_instruction = "\n".join(pref_info)
if preferences_instruction:
preferences_instruction = f"\nUser Preferences:\n{preferences_instruction}"
user_prompt = f"""Current time: {current_time}{location_info}{preferences_instruction}
Reasoning level: {thinking_level}
Editor language: {safe_language_id}
=== STATE FLAGS ===
- CURSOR_IN_FENCED_CODE_BLOCK: {"true" if cursor_in_fenced_code_block else "false"}
- CURSOR_FENCE_LANGUAGE: {cursor_fence_language}
- MERMAID_CONTEXT: {"true" if mermaid_context else "false"}
- PREFIX_ENDS_WITH_NEWLINE: {"true" if prefix_ends_with_newline else "false"}
- SUFFIX_STARTS_WITH_NEWLINE: {"true" if suffix_starts_with_newline else "false"}
=== TASK ===
Produce the best insertion text between PREFIX and SUFFIX.
Requirements:
- Non-empty and meaningful
- Concise unless structure needs more
- Follows markdown rules in system prompt
- Use real line breaks instead of spelled-out escape sequences unless PREFIX or SUFFIX clearly requires that text
- If a boundary needs separation, put the real newline directly in OUTPUT
- Do not explain newline or boundary choices
- Continue after the PREFILL text already placed after <|fim_middle|>
=== CONTEXT NOTES ===
- OCR metadata (e.g., <OCR:description>) is hidden context, never copy to output
- Match PREFIX tone, style, and indentation
- Do not repeat text from SUFFIX beginning
- <|fim_prefix|>, <|fim_suffix|>, <|fim_middle|>, and PREFILL are control context only; never output these markers
=== EXAMPLES BY CATEGORY ===
{_INLINE_EXAMPLES}
=== NOW COMPLETE THE TASK ===
<|fim_prefix|>{recent_prefix}<|fim_suffix|>{recent_suffix}<|fim_middle|>{prefill}"""
system_prompt = build_inline_system_prompt(safe_language_id)
return system_prompt.strip(), user_prompt.strip(), prefill
def build_prompt(
prefix: str,
suffix: str,
language_id: str = "markdown",
location: str = "",
thinking_level: str = "low",
preferences: UserPreferences | None = None,
) -> str:
"""
Backward-compatible helper. Returns only the user prompt body.
"""
_, user_prompt, _ = build_completion_prompts(
prefix=prefix,
suffix=suffix,
language_id=language_id,
location=location,
thinking_level=thinking_level,
preferences=preferences,
)
return user_prompt
def build_pro_completion_prompts(
prefix: str,
suffix: str,
instruction: str = "",
language_id: str = "markdown",
location: str = "",
pro_thinking_level: str = "medium",
preferences: UserPreferences | None = None,
preferences: object = None,
) -> Tuple[str, str]:
preferences = _normalize_preferences(preferences)
safe_language_id = _canonical_language_id(language_id)
recent_prefix, recent_suffix = _prepare_context(prefix, suffix)
recent_prefix = _normalize_newlines(recent_prefix)
@@ -454,49 +543,43 @@ def build_pro_completion_prompts(
if preferences:
if preferences.language and preferences.language != "auto":
pref_info.append(f"Preferred language: {preferences.language}")
if preferences.country and preferences.country != "auto":
pref_info.append(f"Preferred country: {preferences.country}")
if preferences.timezone and preferences.timezone != "auto":
pref_info.append(f"Preferred timezone: {preferences.timezone}")
if preferences.currency and preferences.currency != "auto":
pref_info.append(f"Preferred currency: {preferences.currency}")
preferences_instruction = "\n".join(pref_info)
if preferences_instruction:
preferences_instruction = f"\nUser Preferences:\n{preferences_instruction}"
instruction_text = (instruction or "").strip() or "Continue the Markdown naturally."
user_prompt = f"""Current time: {current_time}{location_info}{preferences_instruction}
PRO_MODE: true
PRO_THINKING_LEVEL: {pro_thinking_level}
Editor language: {safe_language_id}
Reasoning hint: {thinking_level}
Editor language id: {safe_language_id}
=== STATE FLAGS ===
Completion state flags:
- CURSOR_IN_FENCED_CODE_BLOCK: {"true" if cursor_in_fenced_code_block else "false"}
- CURSOR_FENCE_LANGUAGE: {cursor_fence_language}
- MERMAID_CONTEXT: {"true" if mermaid_context else "false"}
- PREFIX_ENDS_WITH_NEWLINE: {"true" if prefix_ends_with_newline else "false"}
- SUFFIX_STARTS_WITH_NEWLINE: {"true" if suffix_starts_with_newline else "false"}
=== PRO INSTRUCTION (HIGHEST PRIORITY) ===
{instruction_text}
Task:
- Produce the best insertion text at the cursor between PREFIX and SUFFIX.
- Keep insertion meaningful and non-empty.
- Keep insertion concise unless structure requires more content.
=== PRO TASK ===
Produce the best insertion text between PREFIX and SUFFIX for [PRO] mode.
Requirements:
- Output only the markdown insertion text
- Long paragraphs or section-level output are allowed when instruction asks for it
- Be precise, concrete, and structurally coherent
- Never output hidden tags, control tokens, or boundary-analysis commentary
- Never repeat text from SUFFIX beginning
Context notes:
- PREFIX may include OCR metadata after image markdown, e.g. ![alt](url) <OCR:description>.
- OCR metadata is hidden context and must never be copied into output.
- Preserve local style and formatting.
=== CONTEXT NOTES ===
- OCR metadata and document-side snippets are hidden context; never copy tags to output
- Match PREFIX style, language, terminology, and markdown conventions
- Keep boundaries safe with minimal required newlines
Decision policy:
- Prioritize seamless join: PREFIX + OUTPUT + SUFFIX must read naturally.
- Do not repeat SUFFIX-leading text.
- If uncertain, prefer a complete short phrase/sentence with clear meaning.
=== PRO EXAMPLES BY CATEGORY ===
{_PRO_INLINE_EXAMPLES}
Comprehensive examples:
{INLINE_EXAMPLES}
=== NOW COMPLETE THE TASK ===
Now produce the insertion.
<PREFIX>
{recent_prefix}
@@ -508,5 +591,27 @@ Requirements:
Output:"""
system_prompt = build_pro_system_prompt(safe_language_id)
system_prompt = build_inline_system_prompt(safe_language_id)
return system_prompt.strip(), user_prompt.strip()
def build_prompt(
prefix: str,
suffix: str,
language_id: str = "markdown",
location: str = "",
thinking_level: str = "low",
preferences: object = None,
) -> str:
"""
Backward-compatible helper. Returns only the user prompt body.
"""
_, user_prompt = build_completion_prompts(
prefix=prefix,
suffix=suffix,
language_id=language_id,
location=location,
thinking_level=thinking_level,
preferences=preferences,
)
return user_prompt
backend/prompts/__init__.py
-52
View File
@@ -1,52 +0,0 @@
import json
from pathlib import Path
from typing import Any
_PROMPTS_DIR = Path(__file__).parent
class PromptManager:
_instance = None
_data: dict[str, Any] = {}
def __new__(cls):
if cls._instance is None:
cls._instance = super().__new__(cls)
cls._instance._load_all()
return cls._instance
def _load_all(self):
for json_file in _PROMPTS_DIR.glob("*.json"):
key = json_file.stem
with open(json_file, "r", encoding="utf-8") as f:
self._data[key] = json.load(f)
def get(self, key: str, default: Any = None) -> Any:
return self._data.get(key, default)
_prompts = PromptManager()
def get_system_prompt_template() -> str:
return _prompts.get("system_prompt", {}).get("template", "")
def get_language_guidance_map() -> dict[str, str]:
return _prompts.get("language_guidance", {})
def get_inline_examples() -> str:
return _prompts.get("inline_examples", {}).get("content", "")
def get_system_prompt_pro_template() -> str:
return _prompts.get("system_prompt_pro", {}).get("template", "")
def get_inline_examples_pro() -> str:
return _prompts.get("inline_examples_pro", {}).get("content", "")
def get_vlm_ocr_prompt() -> str:
return _prompts.get("vlm_ocr", {}).get("prompt", "")
backend/prompts/inline_examples.json
-3
View File
@@ -1,3 +0,0 @@
{
"content": "=== CATEGORY A: PROSE CONTINUATION ===\n\n[EX01] Simple prose continuation\n<PREFIX>The quick brown fox </PREFIX>\n<SUFFIX>jumps over the lazy dog.</SUFFIX>\nExpected OUTPUT:\nmoved quietly and then\n\n[EX02] Avoid repeating suffix\n<PREFIX>Our launch plan starts with </PREFIX>\n<SUFFIX>phase one, followed by phase two.</SUFFIX>\nExpected OUTPUT:\ncareful internal testing before\nWRONG: phase one starts with (repeats suffix)\n\n=== CATEGORY B: MARKDOWN STRUCTURES ===\n\n[EX03] Continue checklist\n<PREFIX>## TODO\n- [ ] Buy milk\n- [ ] </PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\nWrite release notes and share draft with team\n\n[EX04] Start list after header (PREFIX lacks newline)\nPREFIX_ENDS_WITH_NEWLINE=false\n<PREFIX>Deployment steps:</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\n- Build artifact\n- Deploy service\n\n[EX05] Continue table row\n<PREFIX>| Name | Score |\n| --- | --- |\n| Alice | 92 |\n| Bob | </PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n88 |\n\n[EX06] Start new paragraph\n<PREFIX>First paragraph ends.</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\nSecond paragraph starts.\nWRONG: Second paragraph starts. (missing leading \\n\\n)\n\n[EX07] Add newline before heading\nPREFIX_ENDS_WITH_NEWLINE=false\n<PREFIX>End of previous section.</PREFIX>\n<SUFFIX>## Next Heading</SUFFIX>\nExpected OUTPUT:\n\nWRONG: (would join with heading without separation)\n\n=== CATEGORY C: CODE BLOCKS ===\n\n[EX08] Outside fence: wrap code in fence\nCURSOR_IN_FENCED_CODE_BLOCK=false\n<PREFIX>Parse this JSON payload in Python:</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n```python\nimport json\ndata = json.loads(payload)\n```\nWRONG: import json\\ndata = json.loads(payload) (no fence)\n\n[EX09] Inside fence: output code only\nCURSOR_IN_FENCED_CODE_BLOCK=true\n<PREFIX>```python\ndef add(a, b):\n return </PREFIX>\n<SUFFIX>\n```</SUFFIX>\nExpected OUTPUT:\na + b\nWRONG: ```python\\nreturn a + b\\n``` (duplicate fences)\n\n[EX10] Code inside fence uses single newline\nCURSOR_IN_FENCED_CODE_BLOCK=true\n<PREFIX>```python\ndef hello():</PREFIX>\n<SUFFIX>\n```</SUFFIX>\nExpected OUTPUT:\n print(\"Hello\")\n return True\n(Note: single \\n between code lines, no markdown rules)\n\n=== CATEGORY D: MATH ===\n\n[EX11] Inline math\n<PREFIX>The derivative of x^2 is </PREFIX>\n<SUFFIX>.</SUFFIX>\nExpected OUTPUT:\n$2x$\nWRONG: 2x (bare formula)\n\n[EX12] Block math\n<PREFIX>We can write the Gaussian integral as:</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n$$\n\\int_{-\\infty}^{\\infty} e^{-x^2}\\,dx = \\sqrt{\\pi}\n$$\nWRONG: \\int... (bare formula without $$)\n\n=== CATEGORY E: MERMAID ===\n\n[EX13] Inside mermaid fence\nCURSOR_FENCE_LANGUAGE=mermaid\nCURSOR_IN_FENCED_CODE_BLOCK=true\n<PREFIX>```mermaid\nflowchart TD\n A[Start] --> </PREFIX>\n<SUFFIX>\n```</SUFFIX>\nExpected OUTPUT:\nB{Valid?}\nB -->|Yes| C[Done]\nWRONG: ```mermaid\\nB{Valid?}... (duplicate fence)\n\n[EX14] Outside fence with mermaid context\nCURSOR_IN_FENCED_CODE_BLOCK=false\nMERMAID_CONTEXT=true\n<PREFIX>Please provide a simple release pipeline diagram.</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n```mermaid\nflowchart LR\n Build --> Test --> Deploy\n```\n\n=== CATEGORY F: OCR METADATA ===\n\n[EX15] Use OCR as context, never output\n<PREFIX>![whiteboard](img.png) <OCR:equation y = mx + b>\nThe relationship is </PREFIX>\n<SUFFIX>.</SUFFIX>\nExpected OUTPUT:\n$y = mx + b$\nWRONG: <OCR:equation y = mx + b> (OCR tag in output)"
}
backend/prompts/inline_examples_pro.json
-3
View File
@@ -1,3 +0,0 @@
{
"content": "=== PRO CATEGORY A: CONTINUATION AND EXPANSION ===\n\n[PRO-EX01] Continue naturally without suffix repetition\n<PREFIX>Project update: This week we completed </PREFIX>\n<SUFFIX>and started preparing the release checklist.</SUFFIX>\nExpected OUTPUT:\nbackend integration tests\nWRONG: and started preparing (repeats suffix start)\n\n[PRO-EX02] Long paragraph expansion allowed\nINSTRUCTION: Expand into a fuller paragraph with concrete details.\n<PREFIX>Our migration reduced incidents.</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\nIt also improved deployment confidence by cutting rollback frequency and clarifying ownership for each service boundary, which made post-release diagnosis significantly faster.\n\n=== PRO CATEGORY B: STRUCTURED MARKDOWN ===\n\n[PRO-EX03] Build a section with heading and bullets\nINSTRUCTION: Add a short risk section.\n<PREFIX>## Launch Plan\nCurrent status is green.</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\n### Risks\n- Third-party API latency may delay webhook retries.\n- Data backfill window could overlap with peak traffic.\n\n[PRO-EX04] Preserve list numbering continuity\n<PREFIX>1. Prepare schema\n2. Run dry-run\n3. </PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\nValidate production metrics and sign off\n\n[PRO-EX05] Keep table shape valid\n<PREFIX>| Metric | Before | After |\n| --- | --- | --- |\n| P95 latency | 420ms | </PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n260ms |\n\n=== PRO CATEGORY C: CODE AND TECHNICAL CONTEXT ===\n\n[PRO-EX06] Inside code fence: output code only\nCURSOR_IN_FENCED_CODE_BLOCK=true\n<PREFIX>```python\ndef build_payload(user_id):\n return </PREFIX>\n<SUFFIX>\n```</SUFFIX>\nExpected OUTPUT:\n{\"id\": user_id, \"active\": True}\n\n[PRO-EX07] Outside code fence: include fenced block when instruction asks code\nCURSOR_IN_FENCED_CODE_BLOCK=false\nINSTRUCTION: Show a minimal SQL query.\n<PREFIX>Fetch active users:</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\n```sql\nSELECT id, email\nFROM users\nWHERE active = TRUE;\n```\n\n=== PRO CATEGORY D: MATH AND MERMAID ===\n\n[PRO-EX08] Inline math remains inline\n<PREFIX>The expected value is </PREFIX>\n<SUFFIX> under this distribution.</SUFFIX>\nExpected OUTPUT:\n$\\mu$\n\n[PRO-EX09] Mermaid inside mermaid fence\nCURSOR_FENCE_LANGUAGE=mermaid\nCURSOR_IN_FENCED_CODE_BLOCK=true\n<PREFIX>```mermaid\nflowchart TD\nA[Input] --> </PREFIX>\n<SUFFIX>\n```</SUFFIX>\nExpected OUTPUT:\nB{Validated?}\nB -->|Yes| C[Persist]\n\n[PRO-EX10] Mermaid outside fence with context\nMERMAID_CONTEXT=true\nCURSOR_IN_FENCED_CODE_BLOCK=false\n<PREFIX>Show the pipeline as a diagram.</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\n```mermaid\nflowchart LR\nQueue --> Worker --> Storage\n```\n\n=== PRO CATEGORY E: INSTRUCTION-FIRST REWRITE ===\n\n[PRO-EX11] Rewrite style per instruction\nINSTRUCTION: Rewrite as concise executive tone in Chinese.\n<PREFIX>这个方案看起来不错，但是细节很多，可能会拖慢推进。</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n该方案方向正确，但需聚焦关键路径并压缩实现范围，以保障交付节奏。\n\n[PRO-EX12] Add constrained output length\nINSTRUCTION: Add one sentence under 25 words.\n<PREFIX>结论：</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\n先完成最小可用版本，再按风险优先级迭代。\n\n=== PRO CATEGORY F: HIDDEN CONTEXT SAFETY ===\n\n[PRO-EX13] Never leak OCR tags\n<PREFIX>![board](a.png) <OCR:roadmap Q3 launch>\nTimeline:</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\nQ3 launch with weekly checkpoints and ownership tracking.\nWRONG: <OCR:roadmap Q3 launch>\n\n=== PRO CATEGORY G: BOUNDARY PRECISION ===\n\n[PRO-EX14] Add leading newline when needed\nPREFIX_ENDS_WITH_NEWLINE=false\n<PREFIX>Action items:</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\n- Confirm rollout window\n- Notify on-call rotation\n\n[PRO-EX15] Do not break upcoming heading\nPREFIX_ENDS_WITH_NEWLINE=false\nSUFFIX_STARTS_WITH_NEWLINE=false\n<PREFIX>Summary complete.</PREFIX>\n<SUFFIX>## Next Steps</SUFFIX>\nExpected OUTPUT:\n\n\n=== PRO CATEGORY H: MIXED STRUCTURE ===\n\n[PRO-EX16] Combine short prose + list + code block\nINSTRUCTION: Add a short explanation, then checklist, then shell command.\n<PREFIX>Deployment guide draft:</PREFIX>\n<SUFFIX></SUFFIX>\nExpected OUTPUT:\n\nUse the following sequence to reduce release risk:\n- Verify migrations on staging\n- Freeze non-critical merges\n- Capture rollback snapshot\n\n```bash\n./scripts/deploy.sh --env prod\n```"
}
backend/prompts/language_guidance.json
-21
View File
@@ -1,21 +0,0 @@
{
"mermaid": "\nLanguage-specific guidance (mermaid):\n- Output valid Mermaid syntax only.\n- Prefer concise, syntactically correct diagram statements.\n- Avoid prose unless the user prompt explicitly requires it.",
"latex": "\nLanguage-specific guidance (latex):\n- Output LaTeX math content only when completing LaTeX.\n- If CURSOR_IN_FENCED_CODE_BLOCK=true and CURSOR_FENCE_LANGUAGE is latex/tex/katex:\n- Output raw LaTeX lines only.\n- Do not wrap with $ or $$.",
"json": "\nLanguage-specific guidance (json):\n- Output strict JSON only (no comments, no trailing commas).\n- Ensure valid quotes and braces.",
"yaml": "\nLanguage-specific guidance (yaml):\n- Output valid YAML only.\n- Use consistent indentation and avoid tabs.",
"toml": "\nLanguage-specific guidance (toml):\n- Output valid TOML only.\n- Keep key types consistent.",
"ini": "\nLanguage-specific guidance (ini):\n- Output valid INI only.\n- Keep section headers and key=value pairs consistent.",
"sql": "\nLanguage-specific guidance (sql):\n- Output a single, valid SQL statement unless context requires multiple.\n- Prefer ANSI SQL when dialect is unclear.",
"bash": "\nLanguage-specific guidance (bash):\n- Output POSIX-compatible shell when possible.\n- Avoid interactive prompts or destructive commands unless requested.",
"powershell": "\nLanguage-specific guidance (powershell):\n- Output valid PowerShell commands.\n- Avoid destructive commands unless explicitly requested.",
"html": "\nLanguage-specific guidance (html):\n- Output valid HTML only.\n- Keep markup minimal and well-formed.",
"css": "\nLanguage-specific guidance (css):\n- Output valid CSS only.\n- Use concise, readable selectors.",
"diff": "\nLanguage-specific guidance (diff):\n- Output a unified diff only.\n- Ensure @@ hunk headers and +/- lines are consistent.",
"regex": "\nLanguage-specific guidance (regex):\n- Output the regex pattern only.\n- Avoid delimiters unless explicitly requested.",
"text": "\nLanguage-specific guidance (text):\n- Output plain text only.\n- Avoid markdown formatting unless explicitly asked.",
"xml": "\nLanguage-specific guidance (xml):\n- Output well-formed XML only.\n- Ensure matching tags and proper escaping.",
"dockerfile": "\nLanguage-specific guidance (dockerfile):\n- Output valid Dockerfile instructions only.\n- Keep layers minimal and ordered logically.",
"makefile": "\nLanguage-specific guidance (makefile):\n- Output valid Makefile syntax only.\n- Use tabs for recipe lines.",
"_generic_code": "\nLanguage-specific guidance ({lang}):\n- Output valid {lang} code.\n- Avoid prose unless context clearly expects comments or docstrings.",
"_js_code": "\nLanguage-specific guidance ({lang}):\n- Output valid {lang} code.\n- Prefer modern syntax and avoid prose unless comments are needed."
}
backend/prompts/system_prompt.json
-3
View File
@@ -1,3 +0,0 @@
{
"template": "You are an inline completion engine for a {language_id} editor with ghost-text suggestions.\n\nReturn only the insertion text that should be placed between PREFIX and SUFFIX.\n\nCORE PRINCIPLE: Output insertion text only. No explanations, no meta labels, no wrapper quotes.\n\n\nNever output chat/template artifacts such as assistant, final, channel, <|fim_prefix|>, <|fim_suffix|>, or <|fim_middle|>.\n\nPRIORITY 1: CONTEXT AWARENESS (Read these flags from user prompt)\n- CURSOR_IN_FENCED_CODE_BLOCK: Are you inside a code fence?\n- CURSOR_FENCE_LANGUAGE: What language is the current fence?\n- PREFIX_ENDS_WITH_NEWLINE: Does prefix end with newline?\n- SUFFIX_STARTS_WITH_NEWLINE: Does suffix start with newline?\n- MERMAID_CONTEXT: Is this a Mermaid diagram context?\n\nPRIORITY 2: SPECIALIZED CONTENT RULES\n\n2.1 Code Block Handling:\nIf CURSOR_IN_FENCED_CODE_BLOCK=true:\n- You are inside a code fence\n- Output code lines ONLY (no triple backticks)\n- Separate code lines with actual newline characters\n\nIf CURSOR_IN_FENCED_CODE_BLOCK=false and code needed:\n- Wrap code in fenced block with language tag:\n```{language}\ncode here\n```\n- Never use inline backticks for code snippets\n\n2.2 Math Formatting (KaTeX):\n- Inline math: wrap with $...$\n- Block math: wrap with $$...$$\n- Never output bare formulas\n- Exception: inside latex/tex/katex fence, output raw LaTeX\n\n2.3 Mermaid Diagrams:\nIf CURSOR_FENCE_LANGUAGE=mermaid:\n- Output Mermaid syntax ONLY\n- No backticks, no explanations\n\nIf MERMAID_CONTEXT=true and outside fence:\n- Output complete fenced block:\n```mermaid\ndiagram syntax\n```\n\nPRIORITY 3: MARKDOWN STRUCTURE\n\n3.1 Newline Semantics:\n- Use actual line breaks in output, not spelled-out escape sequences, unless the surrounding content explicitly needs that text\n- A single line break usually continues the current block\n- A blank line starts a new paragraph or block\n- Use blank lines for: new paragraphs, before headings, starting lists/tables\n- Use single line breaks for: continuation within blocks (list items, table cells)\n- Exception: inside code blocks, use actual newline characters freely for code lines\n\n3.2 Boundary Management:\nCheck PREFIX_ENDS_WITH_NEWLINE and SUFFIX_STARTS_WITH_NEWLINE:\n- If PREFIX lacks needed newline: start OUTPUT on a new line\n\n- If SUFFIX lacks needed newline: end OUTPUT with a trailing line break\n\n- Common cases requiring a leading line break:\n* Starting a list after \"Steps:\"\n* Creating new paragraph after text\n* Adding heading after paragraph\n- Common cases requiring a trailing line break:\n* Before new heading\n* End of section\n\n3.3 Context Stitching:\n- Never repeat text from SUFFIX beginning\n- Match PREFIX tone, style, indentation\n- Continue structures: lists, tables, quotes, headings\n\nPRIORITY 4: HIDDEN CONTEXT\n- OCR metadata like <OCR:...> is hidden context\n- Never copy OCR tags to output\n- Use OCR content as semantic hint only"
}
backend/prompts/system_prompt_pro.json
-3
View File
@@ -1,3 +0,0 @@
{
"template": "You are the [PRO] model for LLM-IN-TEXT, specializing in high-precision markdown insertion for a {language_id} editor.\n\nReturn only the insertion text that should be placed between PREFIX and SUFFIX.\n\nPRO CORE PRINCIPLE:\n- Output insertion text only. No explanations, no analysis, no labels, no wrapper quotes.\n- Never wrap the entire answer in an outer ```markdown code fence; only use fenced code blocks when the inserted content itself requires code.\n- Never output chain-of-thought or internal reasoning.\n- Never output control markers like <|fim_prefix|>, <|fim_suffix|>, <|fim_middle|>, assistant, final, channel.\n\nPRO MODE INTENT:\n- This is PRO_MODE=true. You may produce longer, structured markdown when instruction requires it.\n- Prioritize instruction fidelity first, then boundary safety, then style continuity.\n- If instruction is vague, continue naturally with concrete and useful content.\n\nBOUNDARY AND CONTEXT RULES:\n- Respect CURSOR_IN_FENCED_CODE_BLOCK, CURSOR_FENCE_LANGUAGE, MERMAID_CONTEXT, PREFIX_ENDS_WITH_NEWLINE, and SUFFIX_STARTS_WITH_NEWLINE.\n- Never repeat text from the beginning of SUFFIX.\n- Use minimum necessary newlines to avoid boundary collision.\n- Match PREFIX tone, language, and formatting conventions.\n\nSYNTAX PRIORITY:\n- Code block contexts must keep valid syntax and indentation.\n- Math must use $...$ for inline and $$...$$ for blocks unless inside latex fences.\n- Mermaid contexts must output valid mermaid statements; do not duplicate fences when already inside one.\n\nHIDDEN CONTEXT SAFETY:\n- OCR metadata and document-side context are hidden hints only.\n- Never copy hidden tags (e.g., <OCR:...>) into output.\n\nQUALITY BAR FOR PRO:\n- Prefer specific, information-dense output over generic filler.\n- For structured requests, preserve headings/list hierarchy and produce coherent section flow.\n- Keep output directly insertable without post-edit cleanups."
}
backend/prompts/vlm_ocr.json
-3
View File
@@ -1,3 +0,0 @@
{
"prompt": "You are an OCR and visual-context extractor for markdown writing assistance.\n\nYour output will be embedded inside an HTML comment as hidden context for a text-completion model.\n\nRequirements:\n- Keep output compact: maximum 120 words.\n- Use plain text only (no markdown code fences).\n- Never output <!-- or -->.\n- Do not invent unreadable text; mark uncertain characters with ?.\n- Preserve original script for recognized text (do not forcibly translate).\n\nReturn exactly this format:\n\nTEXT:\n<exact transcription of visible text; use \" | \" for line breaks; write \"(none)\" if no readable text>\n\nKEY_DETAILS:\n- <3-5 short factual bullets about relevant objects/layout>\n\nLANGUAGE:\n<dominant language(s) in visible text, e.g. English / Chinese / Mixed>\n\nSUMMARY:\n<one short sentence, <= 20 words>"
}
backend/requirements.docker.txt
-10
View File
@@ -1,10 +0,0 @@
fastapi>=0.95.0
uvicorn[standard]>=0.23.0
pydantic>=1.10.0
httpx>=0.24.0
redis>=5.0.0
psycopg[binary]>=3.2.0
python-multipart>=0.0.9
python-dotenv>=1.0.0
markitdown>=0.1.1
geoip2>=4.8.0
backend/requirements.txt
+12 -14
View File
@@ -1,14 +1,12 @@
fastapi>=0.95.0
uvicorn[standard]>=0.23.0
pydantic>=1.10.0
httpx>=0.24.0
redis>=5.0.0
psycopg[binary]>=3.2.0
python-multipart>=0.0.9
python-dotenv>=1.0.0
markitdown>=0.1.1
geoip2>=4.8.0
# testing
pytest>=7.0.0
pytest-cov>=4.1.0
fastapi
uvicorn
ollama
pydantic
python-dotenv
httpx
geoip2
markitdown[all]
python-docx
python-pptx
openpyxl
pypdf
backend/risk_config.py
-162
View File
@@ -1,162 +0,0 @@
import os
from dataclasses import dataclass
def _bool_env(name: str, default: bool) -> bool:
value = os.getenv(name)
if value is None:
return default
return value.strip().lower() in {"1", "true", "yes", "on"}
def _int_env(name: str, default: int) -> int:
try:
return int(os.getenv(name, str(default)))
except (TypeError, ValueError):
return default
def _float_env(name: str, default: float) -> float:
try:
return float(os.getenv(name, str(default)))
except (TypeError, ValueError):
return default
def _str_env(name: str, default: str) -> str:
value = os.getenv(name)
if value is None:
return default
return value.strip() or default
def _optional_env(name: str) -> str | None:
value = os.getenv(name)
if value is None:
return None
value = value.strip()
return value or None
@dataclass(frozen=True)
class RiskConfig:
cors_allow_origins: tuple[str, ...]
session_cookie_name: str
session_cookie_secure: bool
session_cookie_samesite: str
session_cookie_domain: str | None
session_cookie_max_age: int
session_cookie_path: str
session_rotation_seconds: int
api_window_seconds: int
api_soft_limit_per_window: int
api_hard_limit_per_window: int
llm_window_seconds: int
llm_soft_limit_per_window: int
llm_hard_limit_per_window: int
session_concurrency_limit: int
global_concurrency_limit: int
daily_budget_global_usd: float
daily_budget_session_usd: float
daily_budget_ip_usd: float
single_request_max_cost_usd: float
delay_step_ms: int
delay_cap_ms: int
model_circuit_breaker_failures: int
model_circuit_ttl_seconds: int
enforce_redis_fail_closed: bool
completion_model: str
pro_model: str
vision_model: str
completion_max_input_chars: int
completion_max_output_tokens: int
completion_temperature: float
pro_max_input_chars: int
pro_max_output_tokens: int
pro_temperature: float
web_search_model: str
speech_tts_model: str
speech_asr_model: str
web_search_max_input_chars: int
web_search_max_output_tokens: int
web_search_temperature: float
compress_max_input_chars: int
compress_max_output_tokens: int
ocr_max_input_bytes: int
speech_tts_max_input_chars: int
speech_asr_max_input_bytes: int
completion_input_cost_per_1k: float
completion_output_cost_per_1k: float
pro_input_cost_per_1k: float
pro_output_cost_per_1k: float
vision_input_cost_per_1k: float
vision_output_cost_per_1k: float
speech_tts_input_cost_per_1k_chars: float
speech_tts_output_cost_per_minute_audio: float
speech_asr_input_cost_per_mb: float
def load_risk_config() -> RiskConfig:
raw_origins = _str_env(
"CORS_ALLOW_ORIGINS",
"https://chat.imageteach.tech,http://localhost:8080,http://127.0.0.1:8080,http://localhost:5173,http://127.0.0.1:5173",
)
cors_allow_origins = tuple(
origin.strip() for origin in raw_origins.split(",") if origin.strip()
)
return RiskConfig(
cors_allow_origins=cors_allow_origins,
session_cookie_name=_str_env("SESSION_COOKIE_NAME", "llm_anonymous_session"),
session_cookie_secure=_bool_env("SESSION_COOKIE_SECURE", True),
session_cookie_samesite=_str_env("SESSION_COOKIE_SAMESITE", "none"),
session_cookie_domain=_optional_env("SESSION_COOKIE_DOMAIN"),
session_cookie_max_age=_int_env("SESSION_COOKIE_MAX_AGE_SECONDS", 60 * 60 * 24 * 30),
session_cookie_path=_str_env("SESSION_COOKIE_PATH", "/"),
session_rotation_seconds=_int_env("SESSION_ROTATION_SECONDS", 60 * 60 * 24),
api_window_seconds=_int_env("RISK_API_WINDOW_SECONDS", 60),
api_soft_limit_per_window=_int_env("RISK_API_SOFT_LIMIT", 90),
api_hard_limit_per_window=_int_env("RISK_API_HARD_LIMIT", 180),
llm_window_seconds=_int_env("RISK_LLM_WINDOW_SECONDS", 600),
llm_soft_limit_per_window=_int_env("RISK_LLM_SOFT_LIMIT", 8),
llm_hard_limit_per_window=_int_env("RISK_LLM_HARD_LIMIT", 16),
session_concurrency_limit=_int_env("RISK_SESSION_CONCURRENCY_LIMIT", 2),
global_concurrency_limit=_int_env("RISK_GLOBAL_CONCURRENCY_LIMIT", 12),
daily_budget_global_usd=_float_env("RISK_DAILY_BUDGET_GLOBAL_USD", 20.0),
daily_budget_session_usd=_float_env("RISK_DAILY_BUDGET_SESSION_USD", 2.0),
daily_budget_ip_usd=_float_env("RISK_DAILY_BUDGET_IP_USD", 5.0),
single_request_max_cost_usd=_float_env("RISK_SINGLE_REQUEST_MAX_COST_USD", 0.8),
delay_step_ms=_int_env("RISK_DELAY_STEP_MS", 2500),
delay_cap_ms=_int_env("RISK_DELAY_CAP_MS", 30000),
model_circuit_breaker_failures=_int_env("RISK_MODEL_CIRCUIT_FAILURES", 8),
model_circuit_ttl_seconds=_int_env("RISK_MODEL_CIRCUIT_TTL_SECONDS", 300),
enforce_redis_fail_closed=_bool_env("RISK_ENFORCE_REDIS_FAIL_CLOSED", False),
completion_model=_str_env("RISK_COMPLETION_MODEL", os.getenv("LLM_MODEL", "Nex-N2-mini-mlx-OptiQ-8bit-MTP")),
pro_model=_str_env("RISK_PRO_MODEL", os.getenv("PRO_LLM_MODEL", os.getenv("LLM_MODEL", "Nex-N2-mini-mlx-OptiQ-8bit-MTP"))),
vision_model=_str_env("RISK_VISION_MODEL", os.getenv("VLM_MODEL", "Nex-N2-mini-mlx-OptiQ-8bit-MTP")),
completion_max_input_chars=_int_env("RISK_COMPLETION_MAX_INPUT_CHARS", 24000),
completion_max_output_tokens=_int_env("RISK_COMPLETION_MAX_OUTPUT_TOKENS", 768),
completion_temperature=_float_env("RISK_COMPLETION_TEMPERATURE", 0.4),
pro_max_input_chars=_int_env("RISK_PRO_MAX_INPUT_CHARS", 48000),
pro_max_output_tokens=_int_env("RISK_PRO_MAX_OUTPUT_TOKENS", 2048),
pro_temperature=_float_env("RISK_PRO_TEMPERATURE", 0.6),
web_search_model=_str_env("RISK_WEB_SEARCH_MODEL", os.getenv("LLM_MODEL", "Nex-N2-mini-mlx-OptiQ-8bit-MTP")),
speech_tts_model=_str_env("RISK_SPEECH_TTS_MODEL", "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit"),
speech_asr_model=_str_env("RISK_SPEECH_ASR_MODEL", "Qwen3-ASR-0.6B-8bit"),
web_search_max_input_chars=_int_env("RISK_WEB_SEARCH_MAX_INPUT_CHARS", 128000),
web_search_max_output_tokens=_int_env("RISK_WEB_SEARCH_MAX_OUTPUT_TOKENS", 4096),
web_search_temperature=_float_env("RISK_WEB_SEARCH_TEMPERATURE", 0.4),
compress_max_input_chars=_int_env("RISK_COMPRESS_MAX_INPUT_CHARS", 128000),
compress_max_output_tokens=_int_env("RISK_COMPRESS_MAX_OUTPUT_TOKENS", 1536),
ocr_max_input_bytes=_int_env("RISK_OCR_MAX_INPUT_BYTES", 100 * 1024 * 1024),
speech_tts_max_input_chars=_int_env("RISK_SPEECH_TTS_MAX_INPUT_CHARS", 4096),
speech_asr_max_input_bytes=_int_env("RISK_SPEECH_ASR_MAX_INPUT_BYTES", 100 * 1024 * 1024),
completion_input_cost_per_1k=_float_env("RISK_COMPLETION_INPUT_COST_PER_1K", 0.0004),
completion_output_cost_per_1k=_float_env("RISK_COMPLETION_OUTPUT_COST_PER_1K", 0.0016),
pro_input_cost_per_1k=_float_env("RISK_PRO_INPUT_COST_PER_1K", 0.003),
pro_output_cost_per_1k=_float_env("RISK_PRO_OUTPUT_COST_PER_1K", 0.012),
vision_input_cost_per_1k=_float_env("RISK_VISION_INPUT_COST_PER_1K", 0.0008),
vision_output_cost_per_1k=_float_env("RISK_VISION_OUTPUT_COST_PER_1K", 0.0024),
speech_tts_input_cost_per_1k_chars=_float_env("RISK_SPEECH_TTS_INPUT_COST_PER_1K_CHARS", 0.0),
speech_tts_output_cost_per_minute_audio=_float_env("RISK_SPEECH_TTS_OUTPUT_COST_PER_MINUTE_AUDIO", 0.0),
speech_asr_input_cost_per_mb=_float_env("RISK_SPEECH_ASR_INPUT_COST_PER_MB", 0.0),
)
backend/risk_control.py
-357
View File
@@ -1,357 +0,0 @@
import asyncio
import hashlib
import math
import os
import time
from dataclasses import dataclass
from datetime import date
from typing import Any
from risk_config import RiskConfig
try: # pragma: no cover
from redis import asyncio as redis_asyncio
except Exception: # pragma: no cover
redis_asyncio = None
def _now_ms() -> int:
return int(time.time() * 1000)
def _utc_day() -> str:
return date.today().isoformat()
def stable_hash(value: str) -> str:
return hashlib.sha256(value.encode("utf-8")).hexdigest()
def estimate_tokens(text: str) -> int:
if not text:
return 0
ascii_chars = sum(1 for ch in text if ord(ch) < 128)
non_ascii = len(text) - ascii_chars
ascii_tokens = math.ceil(ascii_chars / 4)
non_ascii_tokens = math.ceil(non_ascii * 1.5)
return max(ascii_tokens + non_ascii_tokens, 1)
@dataclass(frozen=True)
class RiskIdentity:
request_id: str
session_hash: str
ip_hash: str
route: str
method: str
@dataclass(frozen=True)
class RiskDecision:
allowed: bool
status_code: int = 200
reason: str = ""
error_code: str = ""
retry_after_seconds: int = 0
delay_ms: int = 0
class RiskRejected(RuntimeError):
def __init__(self, decision: RiskDecision) -> None:
super().__init__(decision.reason or decision.error_code or "request rejected")
self.decision = decision
class BaseRiskBackend:
async def incr_window(self, key: str, ttl_seconds: int) -> int:
raise NotImplementedError
async def get_float(self, key: str) -> float:
raise NotImplementedError
async def add_float(self, key: str, value: float, ttl_seconds: int) -> float:
raise NotImplementedError
async def get_int(self, key: str) -> int:
raise NotImplementedError
async def set_int(self, key: str, value: int, ttl_seconds: int) -> None:
raise NotImplementedError
async def set_float(self, key: str, value: float, ttl_seconds: int) -> None:
raise NotImplementedError
async def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
raise NotImplementedError
async def release_lock(self, key: str) -> None:
raise NotImplementedError
class InMemoryRiskBackend(BaseRiskBackend):
def __init__(self) -> None:
self.values: dict[str, tuple[float, float]] = {}
self.locks: dict[str, float] = {}
self.guard = asyncio.Lock()
def _purge(self) -> None:
now = time.time()
for key, (_, expires_at) in list(self.values.items()):
if expires_at and expires_at <= now:
self.values.pop(key, None)
for key, expires_at in list(self.locks.items()):
if expires_at <= now:
self.locks.pop(key, None)
async def incr_window(self, key: str, ttl_seconds: int) -> int:
async with self.guard:
self._purge()
value, _ = self.values.get(key, (0.0, 0.0))
next_value = int(value) + 1
self.values[key] = (float(next_value), time.time() + ttl_seconds)
return next_value
async def get_float(self, key: str) -> float:
async with self.guard:
self._purge()
return float(self.values.get(key, (0.0, 0.0))[0])
async def add_float(self, key: str, value: float, ttl_seconds: int) -> float:
async with self.guard:
self._purge()
current, _ = self.values.get(key, (0.0, 0.0))
next_value = current + value
self.values[key] = (next_value, time.time() + ttl_seconds)
return next_value
async def get_int(self, key: str) -> int:
return int(await self.get_float(key))
async def set_int(self, key: str, value: int, ttl_seconds: int) -> None:
async with self.guard:
self._purge()
self.values[key] = (float(value), time.time() + ttl_seconds)
async def set_float(self, key: str, value: float, ttl_seconds: int) -> None:
async with self.guard:
self._purge()
self.values[key] = (float(value), time.time() + ttl_seconds)
async def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
async with self.guard:
self._purge()
if key in self.locks:
return False
self.locks[key] = time.time() + ttl_seconds
return True
async def release_lock(self, key: str) -> None:
async with self.guard:
self.locks.pop(key, None)
class RedisRiskBackend(BaseRiskBackend):
def __init__(self, redis_url: str) -> None:
if redis_asyncio is None:
raise RuntimeError("redis package is not installed")
self.redis = redis_asyncio.from_url(redis_url, encoding="utf-8", decode_responses=True)
async def incr_window(self, key: str, ttl_seconds: int) -> int:
value = await self.redis.incr(key)
if value == 1:
await self.redis.expire(key, ttl_seconds)
return int(value)
async def get_float(self, key: str) -> float:
value = await self.redis.get(key)
if value is None:
return 0.0
return float(value)
async def add_float(self, key: str, value: float, ttl_seconds: int) -> float:
current = await self.get_float(key)
next_value = current + value
await self.redis.set(key, next_value, ex=ttl_seconds)
return next_value
async def get_int(self, key: str) -> int:
value = await self.redis.get(key)
if value is None:
return 0
try:
return int(value)
except (TypeError, ValueError):
return int(float(value))
async def set_int(self, key: str, value: int, ttl_seconds: int) -> None:
await self.redis.set(key, value, ex=ttl_seconds)
async def set_float(self, key: str, value: float, ttl_seconds: int) -> None:
await self.redis.set(key, value, ex=ttl_seconds)
async def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
return bool(await self.redis.set(key, "1", ex=ttl_seconds, nx=True))
async def release_lock(self, key: str) -> None:
await self.redis.delete(key)
class RiskController:
def __init__(self, config: RiskConfig) -> None:
self.config = config
self.prefix = "llmtext:risk"
redis_url = os.getenv("REDIS_URL", "").strip()
if redis_url and redis_asyncio is not None:
self.backend: BaseRiskBackend = RedisRiskBackend(redis_url)
else:
self.backend = InMemoryRiskBackend()
def _api_key(self, identity: RiskIdentity, scope: str) -> str:
return f"{self.prefix}:api:{scope}:{identity.session_hash}:{identity.ip_hash}"
def _llm_key(self, identity: RiskIdentity, scope: str) -> str:
return f"{self.prefix}:llm:{scope}:{identity.session_hash}:{identity.ip_hash}"
def _budget_key(self, scope: str, scope_hash: str, current_day: str) -> str:
return f"{self.prefix}:budget:{scope}:{scope_hash}:{current_day}"
def _lock_key(self, scope: str, scope_hash: str) -> str:
return f"{self.prefix}:lock:{scope}:{scope_hash}"
def _circuit_key(self, scope: str) -> str:
return f"{self.prefix}:circuit:{scope}"
def _failure_key(self, model: str) -> str:
return f"{self.prefix}:failure:{model}"
async def check_api(self, identity: RiskIdentity, *, scope: str = "default") -> RiskDecision:
key = self._api_key(identity, scope)
count = await self.backend.incr_window(key, self.config.api_window_seconds)
if count > self.config.api_hard_limit_per_window:
return RiskDecision(
allowed=False,
status_code=429,
reason="请求过于频繁，请稍后再试",
error_code="api_rate_limited",
retry_after_seconds=self.config.api_window_seconds,
)
if count > self.config.api_soft_limit_per_window:
overflow = count - self.config.api_soft_limit_per_window
delay_ms = min(self.config.delay_cap_ms, overflow * self.config.delay_step_ms)
return RiskDecision(allowed=True, delay_ms=delay_ms)
return RiskDecision(allowed=True)
async def check_llm(
self,
identity: RiskIdentity,
*,
scope: str,
estimated_cost: float,
) -> RiskDecision:
global_circuit = await self.backend.get_int(self._circuit_key("global"))
model_circuit = await self.backend.get_int(self._circuit_key(scope))
if global_circuit > 0 or model_circuit > 0:
return RiskDecision(
allowed=False,
status_code=503,
reason="当前推理服务繁忙，请稍后再试",
error_code="llm_circuit_open",
retry_after_seconds=self.config.model_circuit_ttl_seconds,
)
if estimated_cost > self.config.single_request_max_cost_usd:
return RiskDecision(
allowed=False,
status_code=429,
reason="单次请求成本过高，已被拒绝",
error_code="llm_cost_too_high",
)
day = _utc_day()
session_budget = await self.backend.get_float(self._budget_key("session", identity.session_hash, day))
ip_budget = await self.backend.get_float(self._budget_key("ip", identity.ip_hash, day))
global_budget = await self.backend.get_float(self._budget_key("global", "global", day))
if session_budget + estimated_cost > self.config.daily_budget_session_usd:
return RiskDecision(False, 429, "当前匿名会话今日额度已用尽", "session_budget_exhausted", 3600)
if ip_budget + estimated_cost > self.config.daily_budget_ip_usd:
return RiskDecision(False, 429, "当前网络环境今日额度已用尽", "ip_budget_exhausted", 3600)
if global_budget + estimated_cost > self.config.daily_budget_global_usd:
await self.backend.set_int(self._circuit_key("global"), 1, self.config.model_circuit_ttl_seconds)
return RiskDecision(False, 503, "今日全局推理预算已耗尽", "global_budget_exhausted", 3600)
key = self._llm_key(identity, scope)
count = await self.backend.incr_window(key, self.config.llm_window_seconds)
if count > self.config.llm_hard_limit_per_window:
return RiskDecision(False, 429, "推理请求过于频繁，请稍后重试", "llm_rate_limited", self.config.llm_window_seconds)
if count > self.config.llm_soft_limit_per_window:
overflow = count - self.config.llm_soft_limit_per_window
delay_ms = min(self.config.delay_cap_ms, overflow * self.config.delay_step_ms)
return RiskDecision(True, delay_ms=delay_ms)
return RiskDecision(True)
async def reserve_budget(self, identity: RiskIdentity, estimated_cost: float) -> None:
day = _utc_day()
ttl_seconds = 60 * 60 * 24
await self.backend.add_float(self._budget_key("session", identity.session_hash, day), estimated_cost, ttl_seconds)
await self.backend.add_float(self._budget_key("ip", identity.ip_hash, day), estimated_cost, ttl_seconds)
await self.backend.add_float(self._budget_key("global", "global", day), estimated_cost, ttl_seconds)
async def acquire_execution_slot(self, identity: RiskIdentity, *, model: str) -> list[str]:
ttl_seconds = 60 * 15
keys = [
self._lock_key("session", f"{identity.session_hash}:{identity.request_id}"),
self._lock_key("global", identity.request_id),
]
session_running = await self.backend.get_int(self._lock_key("session-count", identity.session_hash))
global_running = await self.backend.get_int(self._lock_key("global-count", "global"))
if session_running >= self.config.session_concurrency_limit:
raise RiskRejected(
RiskDecision(False, 429, "当前会话并发推理过多，请稍后重试", "session_concurrency_limited", 30)
)
if global_running >= self.config.global_concurrency_limit:
raise RiskRejected(
RiskDecision(False, 503, "当前全局推理负载过高，请稍后重试", "global_concurrency_limited", 30)
)
acquired: list[str] = []
for key in keys:
ok = await self.backend.acquire_lock(key, ttl_seconds)
if not ok:
for acquired_key in acquired:
await self.backend.release_lock(acquired_key)
raise RiskRejected(
RiskDecision(False, 429, "当前请求正在执行，请勿重复提交", "duplicate_request", 10)
)
acquired.append(key)
await self.backend.add_float(self._lock_key("session-count", identity.session_hash), 1.0, ttl_seconds)
await self.backend.add_float(self._lock_key("global-count", "global"), 1.0, ttl_seconds)
return acquired
async def release_execution_slot(self, identity: RiskIdentity, lock_keys: list[str], *, model: str) -> None:
for key in lock_keys:
await self.backend.release_lock(key)
session_count_key = self._lock_key("session-count", identity.session_hash)
global_count_key = self._lock_key("global-count", "global")
session_count = max(0.0, await self.backend.get_float(session_count_key) - 1.0)
global_count = max(0.0, await self.backend.get_float(global_count_key) - 1.0)
await self.backend.set_float(session_count_key, session_count, 60 * 15)
await self.backend.set_float(global_count_key, global_count, 60 * 15)
async def record_model_result(self, *, model: str, success: bool) -> None:
if success:
await self.backend.set_int(self._failure_key(model), 0, self.config.model_circuit_ttl_seconds)
return
failures = await self.backend.incr_window(self._failure_key(model), self.config.model_circuit_ttl_seconds)
if failures >= self.config.model_circuit_breaker_failures:
await self.backend.set_int(self._circuit_key(model), 1, self.config.model_circuit_ttl_seconds)
_risk_controller: RiskController | None = None
def get_risk_controller(config: RiskConfig) -> RiskController:
global _risk_controller
if _risk_controller is None:
_risk_controller = RiskController(config)
return _risk_controller
def reset_risk_controller() -> None:
global _risk_controller
_risk_controller = None
backend/session_store.py
-192
View File
@@ -1,192 +0,0 @@
import hashlib
import secrets
import threading
import time
from dataclasses import dataclass
from typing import Any
try:
import psycopg
from psycopg.rows import dict_row
except Exception: # pragma: no cover
psycopg = None
dict_row = None
def _now_ms() -> int:
return int(time.time() * 1000)
def hash_value(value: str) -> str:
return hashlib.sha256(value.encode("utf-8")).hexdigest()
def new_session_id() -> str:
return secrets.token_urlsafe(32)
@dataclass
class SessionRecord:
session_id: str
session_hash: str
created_at_ms: int
last_seen_at_ms: int
first_ip_hash: str
last_ip_hash: str
user_agent_hash: str
risk_score: int = 0
blocked_until_ms: int = 0
is_new: bool = False
class BaseSessionStore:
def get_or_create(self, session_id: str | None, *, client_ip_hash: str, user_agent_hash: str) -> SessionRecord:
raise NotImplementedError
class InMemorySessionStore(BaseSessionStore):
def __init__(self) -> None:
self.sessions: dict[str, SessionRecord] = {}
self.lock = threading.Lock()
def get_or_create(self, session_id: str | None, *, client_ip_hash: str, user_agent_hash: str) -> SessionRecord:
now = _now_ms()
with self.lock:
if session_id:
session_hash = hash_value(session_id)
record = self.sessions.get(session_hash)
if record is not None:
record.last_seen_at_ms = now
record.last_ip_hash = client_ip_hash
record.user_agent_hash = user_agent_hash
record.is_new = False
return record
next_session_id = new_session_id()
next_hash = hash_value(next_session_id)
record = SessionRecord(
session_id=next_session_id,
session_hash=next_hash,
created_at_ms=now,
last_seen_at_ms=now,
first_ip_hash=client_ip_hash,
last_ip_hash=client_ip_hash,
user_agent_hash=user_agent_hash,
is_new=True,
)
self.sessions[next_hash] = record
return record
class PostgresSessionStore(BaseSessionStore):
def __init__(self, database_url: str) -> None:
if psycopg is None or dict_row is None:
raise RuntimeError("psycopg 未安装，无法使用 PostgreSQL session 存储")
self.database_url = database_url
self._init_lock = threading.Lock()
self._initialized = False
def _connect(self):
return psycopg.connect(self.database_url, autocommit=True, row_factory=dict_row)
def _ensure_initialized(self) -> None:
if self._initialized:
return
with self._init_lock:
if self._initialized:
return
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
CREATE TABLE IF NOT EXISTS anonymous_sessions (
session_hash TEXT PRIMARY KEY,
created_at_ms BIGINT NOT NULL,
last_seen_at_ms BIGINT NOT NULL,
first_ip_hash TEXT NOT NULL,
last_ip_hash TEXT NOT NULL,
user_agent_hash TEXT NOT NULL,
risk_score INTEGER NOT NULL DEFAULT 0,
blocked_until_ms BIGINT NOT NULL DEFAULT 0,
metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb
)
"""
)
cur.execute(
"CREATE INDEX IF NOT EXISTS anonymous_sessions_last_seen_idx ON anonymous_sessions(last_seen_at_ms)"
)
self._initialized = True
def get_or_create(self, session_id: str | None, *, client_ip_hash: str, user_agent_hash: str) -> SessionRecord:
self._ensure_initialized()
now = _now_ms()
if session_id:
session_hash = hash_value(session_id)
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
UPDATE anonymous_sessions
SET last_seen_at_ms = %s,
last_ip_hash = %s,
user_agent_hash = %s
WHERE session_hash = %s
RETURNING session_hash, created_at_ms, last_seen_at_ms, first_ip_hash, last_ip_hash, user_agent_hash, risk_score, blocked_until_ms
""",
(now, client_ip_hash, user_agent_hash, session_hash),
)
row = cur.fetchone()
if row is not None:
return SessionRecord(
session_id=session_id,
session_hash=row["session_hash"],
created_at_ms=int(row["created_at_ms"]),
last_seen_at_ms=int(row["last_seen_at_ms"]),
first_ip_hash=row["first_ip_hash"],
last_ip_hash=row["last_ip_hash"],
user_agent_hash=row["user_agent_hash"],
risk_score=int(row["risk_score"] or 0),
blocked_until_ms=int(row["blocked_until_ms"] or 0),
is_new=False,
)
next_session_id = new_session_id()
next_hash = hash_value(next_session_id)
with self._connect() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO anonymous_sessions (
session_hash, created_at_ms, last_seen_at_ms, first_ip_hash, last_ip_hash, user_agent_hash
)
VALUES (%s, %s, %s, %s, %s, %s)
""",
(next_hash, now, now, client_ip_hash, client_ip_hash, user_agent_hash),
)
return SessionRecord(
session_id=next_session_id,
session_hash=next_hash,
created_at_ms=now,
last_seen_at_ms=now,
first_ip_hash=client_ip_hash,
last_ip_hash=client_ip_hash,
user_agent_hash=user_agent_hash,
is_new=True,
)
_session_store: BaseSessionStore | None = None
def get_session_store(database_url: str | None = None) -> BaseSessionStore:
global _session_store
if _session_store is not None:
return _session_store
if database_url:
_session_store = PostgresSessionStore(database_url)
else:
_session_store = InMemorySessionStore()
return _session_store
def reset_session_store() -> None:
global _session_store
_session_store = None
backend/test_geoip.py
+80
View File
@@ -0,0 +1,80 @@
"""
GeoIP2 IP归属地查询测试脚本
使用方法：
1. 安装依赖：pip install geoip2
2. 下载数据库：https://dev.maxmind.com/geoip/geoip2/geolite2/
3. 运行测试：python test_geoip.py
"""
import os
import sys
try:
import geoip2.database
except ImportError:
print("请先安装 geoip2: pip install geoip2")
sys.exit(1)
DB_PATH = os.path.join(os.path.dirname(__file__), "GeoLite2-City.mmdb")
TEST_IPS = [
"8.8.8.8", # Google DNS (美国)
"114.114.114.114", # 114 DNS (中国南京)
"223.5.5.5", # 阿里DNS (中国杭州)
"1.1.1.1", # Cloudflare DNS (澳大利亚)
"119.29.29.29", # 腾讯DNS (中国)
]
def get_location(reader, ip: str) -> dict:
try:
response = reader.city(ip)
return {
"ip": ip,
"country": response.country.name,
"country_code": response.country.iso_code,
"region": response.subdivisions.most_specific.name if response.subdivisions else None,
"city": response.city.name,
"latitude": response.location.latitude,
"longitude": response.location.longitude,
"timezone": response.location.time_zone,
}
except geoip2.errors.AddressNotFoundError:
return {"ip": ip, "error": "IP未在数据库中找到"}
except Exception as e:
return {"ip": ip, "error": str(e)}
def main():
if not os.path.exists(DB_PATH):
print(f"数据库文件不存在: {DB_PATH}")
print("请从 https://dev.maxmind.com/geoip/geoip2/geolite2/ 下载 GeoLite2-City.mmdb")
return
print(f"加载数据库: {DB_PATH}")
reader = geoip2.database.Reader(DB_PATH)
print("\n" + "=" * 60)
print("IP归属地查询测试")
print("=" * 60)
for ip in TEST_IPS:
result = get_location(reader, ip)
if "error" in result:
print(f"\n{ip}: {result['error']}")
else:
print(f"\n{ip}:")
print(f" 国家: {result['country']} ({result['country_code']})")
print(f" 地区: {result['region'] or '未知'}")
print(f" 城市: {result['city'] or '未知'}")
print(f" 坐标: {result['latitude']}, {result['longitude']}")
print(f" 时区: {result['timezone']}")
reader.close()
print("\n" + "=" * 60)
print("测试完成")
if __name__ == "__main__":
main()
backend/tests/AGENTS.md
-62
View File
@@ -1,62 +0,0 @@
# Backend Tests 测试指引
本文件适用于 backend/tests/ 下的测试和测试脚本。
## 测试入口
- pytest.ini 指定默认测试目录为 backend/tests，并设置后端覆盖率门槛为 90%。
- run_tests.py 提供 unit、integration、simulate、all 几种快捷入口。
- 默认优先使用 pytest 跑窄测试；只有在需要脚本封装参数时再用 run_tests.py。
## 测试分布
- test_main_endpoints.py：主 API 路由行为
- test_main_cancel.py：补全取消和任务生命周期
- test_prompt.py、test_prompt_extended.py：Prompt 上下文与规则
- test_llm.py、test_llm_extended.py：LLM 包装层
- test_geoip.py：GeoIP 逻辑
- test_tts_asr_*.py：TTS 相关与历史 TTS/ASR 面
- simulate_macos.py：历史模拟脚本
- quick_verify.py、verify_cross.py、play_audio.py：人工验证或辅助脚本
## 常用命令
- pytest
- pytest backend/tests/test_main_endpoints.py -v
- pytest backend/tests/test_main_cancel.py -v
- pytest backend/tests/test_prompt.py -v
- pytest backend/tests/test_llm.py -v
- python backend/tests/run_tests.py unit
- python backend/tests/run_tests.py integration --url http://localhost:8001 --key your-secret-key-here
## 测试原则
- 优先跑与改动直接对应的窄测试，不要动不动全量跑。
- 单元测试尽量 mock 掉外部依赖，不要直连真实 Ollama。
- 涉及 main.py 时，优先用 monkeypatch 或 fake 对象替代：
- call_ollama
- call_vlm_ocr
- MarkItDown
- GeoIP 查询
- TTS 模型加载
- 测试要保持确定性，不依赖全局状态、环境顺序或人工输入。
## 容易误判的点
- 覆盖率门槛是针对多个 backend 模块一起算的，改核心文件时，即使单测通过也可能因为覆盖率不够失败。
- htmlcov、.pytest_cache、api_performance_report.md 属于生成产物，不是需要维护的源码。
- 这一目录里有一批 TTS/ASR 测试和说明明显继承自旧实现；当它们与当前 backend/tts_asr.py 冲突时，不要默认代码错了，先确认目标产品面。
- integration 脚本通常假设本地服务在 http://localhost:8001，且默认 API Key 还是占位值。
## 改动定位建议
- 路由返回值不对：先看 test_main_endpoints.py 和 test_main_cancel.py
- Prompt 规则不对：先看 test_prompt.py 和 test_prompt_extended.py
- Ollama 调用包装不对：先看 test_llm.py 和 test_llm_extended.py
- TTS 面变化：先确认当前 backend/tts_asr.py 是不是仍然以旧文档描述为目标，再决定修测试还是修实现
## 维护原则
- 新增后端行为时，优先给对应模块补测试，不要只依赖全量回归。
- 如果变更的是历史 TTS/ASR 面，先把“当前规范是什么”确定下来，再批量修测试。
- 如果覆盖率策略变化，记得同步这个文件，而不是只改 pytest.ini。
backend/tests/benchmark_tts_asr.py
-209
View File
@@ -1,209 +0,0 @@
"""Lightweight benchmark for TTS/ASR queueing and API throughput.
This benchmark uses the FastAPI app with a mocked upstream speech API so it
measures this project's queueing, request handling, and SSE delivery cost
without requiring a real external model endpoint.
"""
from __future__ import annotations
import argparse
import asyncio
import base64
import json
import os
import statistics
import time
from pathlib import Path
import httpx
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in __import__("sys").path:
__import__("sys").path.insert(0, str(BACKEND_DIR))
import main # noqa: E402
import tts_asr # noqa: E402
from job_system import reset_job_manager # noqa: E402
def _wav_bytes(duration_ms: int = 320) -> bytes:
sample_rate = 16000
frames = max(1, int(sample_rate * duration_ms / 1000))
data = b"".join((i % 32768).to_bytes(2, "little", signed=False) for i in range(frames))
data_size = len(data)
return (
b"RIFF" + (36 + data_size).to_bytes(4, "little")
+ b"WAVE"
+ b"fmt " + (16).to_bytes(4, "little")
+ (1).to_bytes(2, "little")
+ (1).to_bytes(2, "little")
+ sample_rate.to_bytes(4, "little")
+ sample_rate.to_bytes(4, "little")
+ (2).to_bytes(2, "little")
+ (16).to_bytes(2, "little")
+ b"data" + data_size.to_bytes(4, "little")
+ data
)
def _parse_sse_done(text: str) -> dict:
for chunk in reversed([item for item in text.split("\n\n") if item.strip()]):
event = ""
data = ""
for line in chunk.splitlines():
if line.startswith("event:"):
event = line.split(":", 1)[1].strip()
elif line.startswith("data:"):
data = line.split(":", 1)[1].strip()
if event == "done" and data:
payload = json.loads(data)
result = dict(payload.get("result") or {})
for key in ("queue_ms", "run_ms", "total_ms", "queued_count", "running_count", "busy_level", "busy_ratio"):
if key in payload:
result[key] = payload[key]
return result
raise RuntimeError("done event not found")
def _percentile(values: list[float], q: float) -> float:
if not values:
return 0.0
if len(values) == 1:
return values[0]
index = (len(values) - 1) * q
lower = int(index)
upper = min(lower + 1, len(values) - 1)
if lower == upper:
return values[lower]
weight = index - lower
return values[lower] * (1 - weight) + values[upper] * weight
async def _build_mock_client(tts_delay_ms: int, asr_delay_ms: int) -> httpx.AsyncClient:
async def transport(request: httpx.Request):
if request.url.path.endswith("/audio/speech"):
await asyncio.sleep(tts_delay_ms / 1000.0)
return httpx.Response(200, content=_wav_bytes(420), headers={"x-request-id": "bench-tts"}, request=request)
await asyncio.sleep(asr_delay_ms / 1000.0)
return httpx.Response(200, json={"text": "benchmark transcript", "language": "zh"}, headers={"x-request-id": "bench-asr"}, request=request)
return httpx.AsyncClient(
base_url="https://benchmark.example/v1/",
transport=httpx.MockTransport(transport),
)
async def _run_case(case_name: str, concurrency: int, request_count: int, audio_b64: str | None = None) -> dict:
results: list[dict] = []
latencies: list[float] = []
async with httpx.AsyncClient(
transport=httpx.ASGITransport(app=main.app),
base_url="http://testserver",
timeout=120.0,
) as client:
semaphore = asyncio.Semaphore(concurrency)
async def fire(index: int) -> None:
async with semaphore:
started = time.perf_counter()
if case_name == "tts":
response = await client.post(
"/v1/tts-asr/tts",
json={"text": f"第 {index} 条基准文本", "speaker": "Vivian", "format": "wav"},
)
else:
response = await client.post(
"/v1/tts-asr/asr",
json={"audio_base64": audio_b64, "language": "zh-CN"},
)
response.raise_for_status()
payload = _parse_sse_done(response.text)
latencies.append((time.perf_counter() - started) * 1000.0)
results.append(payload)
await asyncio.gather(*(fire(index) for index in range(request_count)))
queue_values = sorted(float(item.get("queue_ms", 0) or 0) for item in results)
run_values = sorted(float(item.get("run_ms", 0) or 0) for item in results)
total_values = sorted(float(item.get("total_ms", 0) or 0) for item in results)
latency_values = sorted(latencies)
elapsed_sum_ms = sum(latency_values)
return {
"case": case_name,
"requests": request_count,
"concurrency": concurrency,
"avg_latency_ms": round(statistics.fmean(latency_values), 2),
"p95_latency_ms": round(_percentile(latency_values, 0.95), 2),
"avg_queue_ms": round(statistics.fmean(queue_values), 2),
"p95_queue_ms": round(_percentile(queue_values, 0.95), 2),
"avg_run_ms": round(statistics.fmean(run_values), 2),
"p95_run_ms": round(_percentile(run_values, 0.95), 2),
"avg_total_ms": round(statistics.fmean(total_values), 2),
"p95_total_ms": round(_percentile(total_values, 0.95), 2),
"throughput_rps_estimate": round((request_count * 1000.0) / max(latency_values[-1], elapsed_sum_ms / max(request_count, 1)), 2),
}
async def main_async(args) -> None:
os.environ["JOB_BACKEND"] = "memory"
os.environ["JOB_TTS_CONCURRENCY"] = str(args.tts_workers)
os.environ["JOB_TTS_MAX_QUEUE"] = str(max(args.tts_requests, args.tts_workers))
os.environ["JOB_ASR_CONCURRENCY"] = str(args.asr_workers)
os.environ["JOB_ASR_MAX_QUEUE"] = str(max(args.asr_requests, args.asr_workers))
reset_job_manager()
mock_client = await _build_mock_client(args.tts_delay_ms, args.asr_delay_ms)
tts_asr._httpx_client = mock_client
try:
audio_b64 = base64.b64encode(_wav_bytes(args.audio_duration_ms)).decode("utf-8")
tts_stats = await _run_case("tts", args.tts_concurrency, args.tts_requests)
asr_stats = await _run_case("asr", args.asr_concurrency, args.asr_requests, audio_b64=audio_b64)
finally:
await mock_client.aclose()
tts_asr._httpx_client = None
reset_job_manager()
print(
json.dumps(
{
"benchmark_date": time.strftime("%Y-%m-%d %H:%M:%S"),
"assumptions": {
"upstream_tts_delay_ms": args.tts_delay_ms,
"upstream_asr_delay_ms": args.asr_delay_ms,
"job_backend": "memory",
},
"tts": tts_stats,
"asr": asr_stats,
"recommended_defaults": {
"JOB_TTS_CONCURRENCY": args.tts_workers,
"JOB_TTS_MAX_QUEUE": max(16, args.tts_workers * 4),
"JOB_ASR_CONCURRENCY": args.asr_workers,
"JOB_ASR_MAX_QUEUE": max(8, args.asr_workers * 4),
"TTS_ASR_MAX_CONNECTIONS": max(24, (args.tts_workers + args.asr_workers) * 4),
"TTS_ASR_MAX_KEEPALIVE_CONNECTIONS": max(12, (args.tts_workers + args.asr_workers) * 2),
},
},
ensure_ascii=False,
indent=2,
)
)
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--tts-delay-ms", type=int, default=120)
parser.add_argument("--asr-delay-ms", type=int, default=280)
parser.add_argument("--tts-workers", type=int, default=4)
parser.add_argument("--asr-workers", type=int, default=2)
parser.add_argument("--tts-concurrency", type=int, default=8)
parser.add_argument("--asr-concurrency", type=int, default=4)
parser.add_argument("--tts-requests", type=int, default=32)
parser.add_argument("--asr-requests", type=int, default=16)
parser.add_argument("--audio-duration-ms", type=int, default=320)
return parser.parse_args()
if __name__ == "__main__":
asyncio.run(main_async(parse_args()))
backend/tests/run_tests.py
-81
View File
@@ -1,81 +0,0 @@
#!/usr/bin/env python3
"""Speech test runner for the current API-based TTS/ASR stack."""
from __future__ import annotations
import argparse
import os
import subprocess
import sys
from pathlib import Path
def run_command(cmd: list[str], cwd: str | None = None) -> int:
print(f"\n执行: {' '.join(cmd)}")
print("-" * 70)
return subprocess.run(cmd, cwd=cwd).returncode
def run_unit_tests(verbose: bool = False) -> int:
cmd = ["pytest", "backend/tests/test_tts_asr.py"]
if verbose:
cmd.append("-v")
return run_command(cmd)
def run_benchmark(extra_args: list[str] | None = None) -> int:
cmd = ["python", "backend/tests/benchmark_tts_asr.py"]
if extra_args:
cmd.extend(extra_args)
return run_command(cmd)
def run_all(verbose: bool = False) -> int:
results = [
("单元测试", run_unit_tests(verbose=verbose)),
("基准测试", run_benchmark()),
]
print("\n" + "=" * 70)
print("测试结果汇总")
print("=" * 70)
passed = 0
for name, code in results:
ok = code == 0
passed += int(ok)
print(f"{name}: {'✓ 通过' if ok else '✗ 失败'}")
print("-" * 70)
print(f"总计: {passed}/{len(results)} 通过")
return 0 if passed == len(results) else 1
def main() -> int:
parser = argparse.ArgumentParser(description="当前 API 化 TTS/ASR 测试运行器")
subparsers = parser.add_subparsers(dest="command", help="测试类型")
unit_parser = subparsers.add_parser("unit", help="运行当前 TTS/ASR 单元测试")
unit_parser.add_argument("-v", "--verbose", action="store_true", help="详细输出")
benchmark_parser = subparsers.add_parser("benchmark", help="运行当前 TTS/ASR benchmark")
benchmark_parser.add_argument("benchmark_args", nargs="*", help="透传给 benchmark_tts_asr.py")
all_parser = subparsers.add_parser("all", help="运行当前 TTS/ASR 单元测试和 benchmark")
all_parser.add_argument("-v", "--verbose", action="store_true", help="详细输出")
args = parser.parse_args()
project_root = Path(__file__).parent.parent.parent
os.chdir(project_root)
if args.command == "unit":
return run_unit_tests(verbose=args.verbose)
if args.command == "benchmark":
return run_benchmark(extra_args=args.benchmark_args)
if args.command == "all":
return run_all(verbose=args.verbose)
parser.print_help()
return 0
if __name__ == "__main__":
sys.exit(main())
backend/tests/test_audit_store.py
-75
View File
@@ -1,75 +0,0 @@
"""Regression tests for PostgreSQL audit persistence."""
from __future__ import annotations
from pathlib import Path
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in __import__("sys").path:
__import__("sys").path.insert(0, str(BACKEND_DIR))
import audit_store # noqa: E402
from audit_store import PostgresAuditStore # noqa: E402
class _RecordingCursor:
def __init__(self) -> None:
self.query = ""
self.params = ()
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def execute(self, query: str, params=()) -> None:
self.query = query
self.params = params or ()
assert query.count("%s") == len(self.params)
class _RecordingConnection:
def __init__(self, cursor: _RecordingCursor) -> None:
self._cursor = cursor
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def cursor(self) -> _RecordingCursor:
return self._cursor
def test_record_llm_call_keeps_columns_placeholders_and_params_aligned(monkeypatch):
cursor = _RecordingCursor()
monkeypatch.setattr(audit_store, "psycopg", object())
store = PostgresAuditStore("postgresql://unused")
store._initialized = True
monkeypatch.setattr(store, "_connect", lambda: _RecordingConnection(cursor))
store.record_llm_call({
"request_id": "request-1",
"session_hash": "session",
"ip_hash": "ip",
"job_type": "ocr",
"model": "vision-model",
"estimated_input_tokens": 12,
"max_output_tokens": 256,
"estimated_cost": 0.01,
"actual_output_chars": 42,
"actual_cost": 0.02,
"queue_ms": 10,
"run_ms": 20,
"total_ms": 30,
"status": "completed",
"error_code": "",
"metadata": {"source": "test"},
})
assert "INSERT INTO llm_call_audit" in cursor.query
assert cursor.query.count("%s") == 16
assert len(cursor.params) == 16
backend/tests/test_compress.py
-79
View File
@@ -1,79 +0,0 @@
import importlib
import os
import sys
from pathlib import Path
from fastapi.testclient import TestClient
os.environ["JOB_BACKEND"] = "memory"
CURRENT_DIR = Path(__file__).resolve().parent
BACKEND_DIR = CURRENT_DIR.parent
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
import job_handlers # type: ignore
import job_system # type: ignore
main = importlib.import_module("main")
API_KEY = main.API_KEY
HEADERS = {"X-API-Key": API_KEY}
def setup_function():
job_system.reset_job_manager()
main._handlers_registered = False
def _submit(client, content="test document", doc_type="txt", headers=None):
return client.post("/v1/compress/submit", headers=headers if headers is not None else HEADERS, json={
"content": content,
"docType": doc_type,
})
def _status(client, task_id, headers=None):
return client.get(f"/v1/compress/status?task_id={task_id}", headers=headers if headers is not None else HEADERS)
def test_submit_empty_content_returns_400():
with TestClient(main.app) as client:
resp = _submit(client, "")
assert resp.status_code == 400
def test_submit_too_long_returns_400(monkeypatch):
monkeypatch.setattr(main, "DOC_COMPRESS_CONTEXT_LIMIT", 10)
with TestClient(main.app) as client:
resp = _submit(client, "a" * 100)
assert resp.status_code == 400
def test_submit_success_returns_task_id():
with TestClient(main.app) as client:
resp = _submit(client, "hello world")
assert resp.status_code == 200
data = resp.json()
assert "task_id" in data
assert data["status"] == "queued"
def test_status_not_found_returns_404():
with TestClient(main.app) as client:
resp = _status(client, "nonexistent-id")
assert resp.status_code == 404
def test_status_completed(monkeypatch):
async def fake_call_ollama(prompt, system_prompt=None, **kwargs): # noqa: ARG001
return {"content": f"[compressed] {prompt[:20]}"}
monkeypatch.setattr(job_handlers, "call_ollama", fake_call_ollama)
with TestClient(main.app) as client:
resp = _submit(client, "important document text")
task_id = resp.json()["task_id"]
status_resp = _status(client, task_id)
data = status_resp.json()
assert status_resp.status_code == 200
assert data["status"] in {"queued", "processing", "completed"}
backend/tests/test_geoip.py
-167
View File
@@ -1,167 +0,0 @@
import sys
import types
import pathlib
import pytest
# Ensure the backend directory is on sys.path so we can import the geoip module directly
BACKEND_DIR = pathlib.Path(__file__).resolve().parents[1] # backend/ folder
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
import geoip as geoip
@pytest.fixture(autouse=True)
def reset_geoip_reader():
# Ensure each test starts with a clean cache
geoip._geoip_reader = None
yield
geoip._geoip_reader = None
def test_get_reader_import_error(monkeypatch):
import builtins
real_import = getattr(builtins, "__import__")
def fake_import(name, globals=None, locals=None, fromlist=(), level=0):
if name == "geoip2.database":
raise ImportError("simulate missing geoip2")
return real_import(name, globals, locals, fromlist, level)
monkeypatch.setattr(builtins, "__import__", fake_import)
geoip._geoip_reader = None
assert geoip._get_reader() is None
def test_get_reader_db_missing(monkeypatch):
# Provide a fake geoip2 module, but force the database file to be considered missing
fake_db_module = types.ModuleType("geoip2.database")
class FakeReader:
def __init__(self, path):
self.path = path
fake_db_module.Reader = FakeReader
fake_geoip2 = types.ModuleType("geoip2")
fake_geoip2.database = fake_db_module
sys.modules["geoip2"] = fake_geoip2
sys.modules["geoip2.database"] = fake_db_module
# Ensure path existence check returns False
monkeypatch.setattr(geoip.os.path, "exists", lambda p: False)
geoip._geoip_reader = None
assert geoip._get_reader() is None
# Clean up injected modules
del sys.modules["geoip2"]
del sys.modules["geoip2.database"]
def test_get_reader_loads_and_caches(monkeypatch):
fake_db_module = types.ModuleType("geoip2.database")
class FakeReader:
def __init__(self, path):
self.path = path
fake_db_module.Reader = FakeReader
fake_geoip2 = types.ModuleType("geoip2")
fake_geoip2.database = fake_db_module
sys.modules["geoip2"] = fake_geoip2
sys.modules["geoip2.database"] = fake_db_module
# Simulate that the database file exists
monkeypatch.setattr(geoip.os.path, "exists", lambda p: True)
geoip._geoip_reader = None
r1 = geoip._get_reader()
assert isinstance(r1, FakeReader)
# Second call should return the same cached instance
r2 = geoip._get_reader()
assert r1 is r2
# Clean up injected modules
del sys.modules["geoip2"]
del sys.modules["geoip2.database"]
@pytest.mark.parametrize("ip", [None, "", "127.0.0.1", "localhost", "::1"])
def test_get_ip_location_none_inputs(ip):
assert geoip.get_ip_location(ip) is None
def test_get_ip_location_reader_none(monkeypatch):
# When there is no reader (no database), return None
monkeypatch.setattr(geoip, "_get_reader", lambda: None)
assert geoip.get_ip_location("1.2.3.4") is None
def test_get_ip_location_successful_lookup(monkeypatch):
from types import SimpleNamespace
country = SimpleNamespace(name="United States")
region = SimpleNamespace(name="California")
resp = SimpleNamespace(
country=country,
subdivisions=SimpleNamespace(most_specific=region),
city=SimpleNamespace(name="Mountain View"),
)
class FakeReader:
def city(self, ip):
return resp
monkeypatch.setattr(geoip, "_get_reader", lambda: FakeReader())
loc = geoip.get_ip_location("1.2.3.4")
assert loc == {
"country": "United States",
"region": "California",
"city": "Mountain View",
"display": "United States California Mountain View",
}
def test_get_ip_location_reader_exception(monkeypatch):
class FakeReader:
def city(self, ip):
raise Exception("boom")
monkeypatch.setattr(geoip, "_get_reader", lambda: FakeReader())
assert geoip.get_ip_location("1.2.3.4") is None
def test_get_ip_location_no_location_parts(monkeypatch):
from types import SimpleNamespace
resp = SimpleNamespace(country=SimpleNamespace(name=None), subdivisions=None, city=None)
class FakeReader:
def city(self, ip):
return resp
monkeypatch.setattr(geoip, "_get_reader", lambda: FakeReader())
assert geoip.get_ip_location("1.2.3.4") is None
def test_get_ip_location_text_valid(monkeypatch):
from types import SimpleNamespace
country = SimpleNamespace(name="United States")
region = SimpleNamespace(name="California")
resp = SimpleNamespace(
country=country,
subdivisions=SimpleNamespace(most_specific=region),
city=SimpleNamespace(name="Mountain View"),
)
class FakeReader:
def city(self, ip):
return resp
monkeypatch.setattr(geoip, "_get_reader", lambda: FakeReader())
assert geoip.get_ip_location_text("1.2.3.4") == "United States California Mountain View"
def test_get_ip_location_text_none_when_no_location(monkeypatch):
# Force get_ip_location to return None
monkeypatch.setattr(geoip, "get_ip_location", lambda ip: None)
assert geoip.get_ip_location_text("1.2.3.4") == ""
backend/tests/test_llm.py
+33 -270
View File
@@ -1,6 +1,5 @@
import asyncio
import importlib
import json
import sys
from pathlib import Path
@@ -17,286 +16,50 @@ except ModuleNotFoundError:
pytest.skip("llm module dependencies are not available", allow_module_level=True)
def test_extract_message_openai_format():
resp = {"choices": [{"message": {"content": "hello world", "thinking": "reasoning"}}]}
content, thinking = llm._extract_message(resp)
assert content == "hello world"
assert thinking == "reasoning"
def test_extract_message_openai_reasoning_content():
resp = {"choices": [{"message": {"content": "answer", "reasoning_content": "deep thought"}}]}
content, thinking = llm._extract_message(resp)
assert content == "answer"
assert thinking == "deep thought"
def test_extract_message_empty_choices():
resp = {"choices": []}
content, thinking = llm._extract_message(resp)
assert content == ""
assert thinking == ""
def test_extract_message_no_choices_key():
resp = {}
content, thinking = llm._extract_message(resp)
assert content == ""
assert thinking == ""
def test_extract_message_none_content():
resp = {"choices": [{"message": {"content": None, "thinking": None}}]}
content, thinking = llm._extract_message(resp)
assert content == ""
assert thinking == ""
def test_extract_delta_text():
chunk = {"choices": [{"delta": {"content": "hello"}}]}
assert llm._extract_delta_text(chunk) == "hello"
def test_extract_delta_text_empty():
chunk = {"choices": [{"delta": {}}]}
assert llm._extract_delta_text(chunk) == ""
def test_extract_delta_thinking():
chunk = {"choices": [{"delta": {"thinking": "reasoning step"}}]}
assert llm._extract_delta_thinking(chunk) == "reasoning step"
def test_extract_delta_reasoning_content():
chunk = {"choices": [{"delta": {"reasoning_content": "deep thought"}}]}
assert llm._extract_delta_thinking(chunk) == "deep thought"
def test_resolve_model_name_explicit():
assert llm._resolve_model_name("custom-model") == "custom-model"
def test_resolve_model_name_default():
assert llm._resolve_model_name() == llm.LLM_MODEL
def test_resolve_model_name_pro():
assert llm._resolve_model_name(use_pro_model=True) == llm.PRO_LLM_MODEL
def test_resolve_system_prompt():
assert llm._resolve_system_prompt(" system prompt ") == "system prompt"
assert llm._resolve_system_prompt("") == ""
assert llm._resolve_system_prompt(None) == ""
def test_build_chat_payload_with_system():
payload = llm._build_chat_payload(
"user prompt", system_prompt="sys prompt", temperature=0.5, model="test-model"
)
assert payload["model"] == "test-model"
assert len(payload["messages"]) == 2
assert payload["messages"][0]["role"] == "system"
assert payload["messages"][1]["role"] == "user"
assert payload["stream"] is False
def test_build_chat_payload_no_system():
payload = llm._build_chat_payload("user prompt", system_prompt=None)
assert len(payload["messages"]) == 1
assert payload["stream"] is False
def test_build_chat_payload_with_thinking():
payload = llm._build_chat_payload("prompt", thinking="low")
assert "options" in payload
assert payload["options"]["think"] == "low"
def test_build_chat_stream_payload():
payload = llm._build_chat_stream_payload("prompt", system_prompt="sys")
assert payload["stream"] is True
assert len(payload["messages"]) == 2
def test_build_chat_stream_payload_with_thinking():
payload = llm._build_chat_stream_payload("prompt", thinking="high")
assert "options" in payload
assert payload["options"]["think"] == "high"
def test_call_ollama_non_streaming(monkeypatch):
def test_call_ollama_messages_roles_with_system(monkeypatch):
captured = {}
async def fake_post(*args, **kwargs):
captured["url"] = args[1] if len(args) > 1 else kwargs.get("url", "")
captured["json"] = kwargs.get("json")
async def fake_chat(**kwargs):
captured["messages"] = kwargs["messages"]
return {"message": {"content": "ok", "thinking": ""}}
class FakeResp:
def raise_for_status(self): pass
def json(self): return {"choices": [{"message": {"content": "done"}}]}
return FakeResp()
def fake_client(*args, **kwargs):
class Ctx:
async def __aenter__(self2): return self2
async def __aexit__(*a): pass
post = fake_post
return Ctx()
monkeypatch.setattr(llm.httpx, "AsyncClient", fake_client)
monkeypatch.setattr(llm.asyncio, "wait_for", lambda coro, **kw: coro)
monkeypatch.setattr(llm.client, "chat", fake_chat)
result = asyncio.run(
llm.call_ollama("test prompt", system_prompt="sys", tag="t1")
llm.call_ollama(
"user prompt body",
system_prompt="system prompt body",
tag="test",
temperature=0.1,
)
)
assert result["content"] == "done"
assert captured["url"] == "/chat/completions"
assert captured["json"]["stream"] is False
assert result["content"] == "ok"
assert captured["messages"][0]["role"] == "system"
assert captured["messages"][0]["content"] == "system prompt body"
assert captured["messages"][1]["role"] == "user"
assert captured["messages"][1]["content"] == "user prompt body"
def test_stream_ollama_text_deltas(monkeypatch):
def test_call_ollama_messages_roles_without_system(monkeypatch):
captured = {}
def make_lines():
lines_iter = iter([
'data: {"choices": [{"delta": {"content": "hel"}}]}',
'data: {"choices": [{"delta": {"content": "lo"}}]}',
"data: [DONE]",
])
async def fake_chat(**kwargs):
captured["messages"] = kwargs["messages"]
return {"message": {"content": "ok", "thinking": ""}}
class LineIterator:
def __aiter__(self2): return self2
async def __anext__(self2):
try:
return next(lines_iter)
except StopIteration:
raise StopAsyncIteration()
monkeypatch.setattr(llm.client, "chat", fake_chat)
class Response:
def __init__(self2): self2._lines = LineIterator()
result = asyncio.run(
llm.call_ollama(
"user prompt only",
system_prompt="",
tag="test-no-system",
temperature=0.1,
)
)
def raise_for_status(self2): pass
def aiter_lines(self2): return self2._lines
class StreamCtx:
async def __aenter__(self2): return Response()
async def __aexit__(*a): pass
class Client:
stream = lambda self2, *args, **kw: StreamCtx()
async def __aenter__(self2): return self2
async def __aexit__(*a): pass
return Client()
def fake_client(*args, **kwargs):
captured["called"] = True
return make_lines()
monkeypatch.setattr(llm.httpx, "AsyncClient", fake_client)
monkeypatch.setattr(llm.asyncio, "wait_for", lambda coro, **kw: coro)
results = []
async def collect():
async for delta in llm.stream_ollama("prompt", tag="t1"):
results.append(delta)
asyncio.run(collect())
assert captured.get("called") is True
assert results == ["hel", "lo"]
def test_stream_ollama_events_thinking_and_content(monkeypatch):
captured = {}
def make_lines():
lines_iter = iter([
'data: {"choices": [{"delta": {"thinking": "reasoning"}}]}',
'data: {"choices": [{"delta": {"content": "answer"}}]}',
"data: [DONE]",
])
class LineIterator:
def __aiter__(self2): return self2
async def __anext__(self2):
try:
return next(lines_iter)
except StopIteration:
raise StopAsyncIteration()
class Response:
def __init__(self2): self2._lines = LineIterator()
def raise_for_status(self2): pass
def aiter_lines(self2): return self2._lines
class StreamCtx:
async def __aenter__(self2): return Response()
async def __aexit__(*a): pass
class Client:
stream = lambda self2, *args, **kw: StreamCtx()
async def __aenter__(self2): return self2
async def __aexit__(*a): pass
return Client()
def fake_client(*args, **kwargs):
captured["called"] = True
return make_lines()
monkeypatch.setattr(llm.httpx, "AsyncClient", fake_client)
monkeypatch.setattr(llm.asyncio, "wait_for", lambda coro, **kw: coro)
results = []
async def collect():
async for event_type, payload in llm.stream_ollama_events("prompt", tag="t1"):
results.append((event_type, payload))
asyncio.run(collect())
assert captured.get("called") is True
# First event should be thinking, then content
assert results[0] == ("thinking", "")
assert results[1][0] == "content"
def test_call_vlm_ocr(monkeypatch):
captured = {}
async def fake_post(*args, **kwargs):
captured["url"] = args[1] if len(args) > 1 else kwargs.get("url", "")
captured["json"] = kwargs.get("json")
class FakeResp:
def raise_for_status(self): pass
def json(self): return {"choices": [{"message": {"content": "ocr text"}}]}
return FakeResp()
def fake_client(*args, **kwargs):
class Ctx:
async def __aenter__(self2): return self2
async def __aexit__(*a): pass
post = fake_post
return Ctx()
monkeypatch.setattr(llm.httpx, "AsyncClient", fake_client)
monkeypatch.setattr(llm.asyncio, "wait_for", lambda coro, **kw: coro)
result = asyncio.run(llm.call_vlm_ocr(b"fake image bytes"))
assert result == "ocr text"
# Verify the payload uses OpenAI vision format (image_url)
assert captured["url"] == "/chat/completions"
messages = captured["json"]["messages"]
assert len(messages) == 1
content_parts = messages[0]["content"]
# Should have text part and image_url part
assert any(p.get("type") == "text" for p in content_parts)
image_part = [p for p in content_parts if p.get("type") == "image_url"]
assert len(image_part) == 1
assert image_part[0]["image_url"]["url"].startswith("data:image/png;base64,")
assert captured["json"]["options"]["think"] is False
assert result["content"] == "ok"
assert len(captured["messages"]) == 1
assert captured["messages"][0]["role"] == "user"
assert captured["messages"][0]["content"] == "user prompt only"
backend/tests/test_main_cancel.py
+41 -90
View File
@@ -1,37 +1,26 @@
import asyncio
import importlib
import os
import sys
import threading
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
os.environ["JOB_BACKEND"] = "memory"
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
import job_handlers # type: ignore
import job_system # type: ignore
import risk_control # type: ignore
import session_store # type: ignore
import audit_store # type: ignore
try:
main = importlib.import_module("main")
except ModuleNotFoundError:
pytest.skip("main module dependencies are not available", allow_module_level=True)
main = importlib.import_module("main")
API_KEY_HEADERS = {"X-API-Key": "your-secret-key-here"}
def setup_function():
job_system.reset_job_manager()
risk_control.reset_risk_controller()
session_store.reset_session_store()
audit_store.reset_audit_store()
main._handlers_registered = False
def _completion_payload():
return {
"prefix": "hello",
@@ -43,6 +32,7 @@ def _completion_payload():
def test_cancel_endpoint_cancels_running_task(monkeypatch):
main.ACTIVE_COMPLETIONS.clear()
started = threading.Event()
cancelled = threading.Event()
@@ -55,21 +45,21 @@ def test_cancel_endpoint_cancels_running_task(monkeypatch):
cancelled.set()
raise
monkeypatch.setattr(job_handlers, "call_ollama", fake_call_ollama)
request_id = "req-cancel-1"
monkeypatch.setattr(main, "call_ollama", fake_call_ollama)
monkeypatch.setattr(main, "build_completion_prompts", lambda *a, **k: ("system", "user"))
monkeypatch.setattr(main, "prepare_prompt_context", lambda *a, **k: ("prefix", "suffix"))
with TestClient(main.app) as client:
request_id = "req-cancel-1"
completion_headers = {**API_KEY_HEADERS, "X-Request-Id": request_id}
response_box = {}
def send_completion():
with client.stream(
"POST",
response_box["response"] = client.post(
"/v1/completions",
headers={**API_KEY_HEADERS, "X-Request-Id": request_id},
headers=completion_headers,
json=_completion_payload(),
) as response:
response_box["status_code"] = response.status_code
response_box["body"] = "".join(response.iter_text())
)
completion_thread = threading.Thread(target=send_completion, daemon=True)
completion_thread.start()
@@ -87,76 +77,14 @@ def test_cancel_endpoint_cancels_running_task(monkeypatch):
completion_thread.join(timeout=5.0)
assert not completion_thread.is_alive()
assert cancelled.wait(timeout=2.0)
assert "event: cancelled" in response_box["body"]
class FakeRedis:
def __init__(self):
self.acks = []
async def xack(self, *args):
self.acks.append(args)
async def hincrby(self, key, field, amount):
return 0
class FakeManager:
def __init__(self):
self.redis = FakeRedis()
self.statuses = {}
async def get_status(self, job_id):
return self.statuses.get(job_id)
async def _set_state(self, job_id, state):
self.statuses[job_id] = state
async def _metrics(self, job_type):
return {"queued_count": 0, "running_count": 0}
async def _emit_event(self, job_id, event, data):
self.statuses[job_id]["event"] = event
def _metrics_key(self, job_type):
return f"metrics:{job_type}"
def _state_key(self, job_id):
return f"state:{job_id}"
async def _run_cancelled_after_handler(manager, job_type):
worker = job_system.RedisWorker(manager)
await worker._run_message(
job_type,
"queue",
"group",
"msg-1",
{"job_id": "job-1"},
asyncio.Semaphore(1),
)
def test_redis_worker_acks_when_handler_returns_cancelled_state():
async def handler(payload, emit, is_cancelled):
return {"ok": True}
async def coro():
manager = FakeManager()
manager.handlers = {"completion": handler}
manager.statuses["job-1"] = {
"request_id": "req-1",
"type": "completion",
"status": "running",
"created_at": 1,
}
await _run_cancelled_after_handler(manager, "completion")
assert manager.redis.acks == [("queue", "group", "msg-1")]
asyncio.run(coro())
completion_response = response_box["response"]
assert completion_response.status_code == 200
assert '"cancelled": true' in completion_response.text
def test_cancel_not_found():
main.ACTIVE_COMPLETIONS.clear()
with TestClient(main.app) as client:
response = client.post(
"/v1/completions/cancel",
@@ -165,3 +93,26 @@ def test_cancel_not_found():
)
assert response.status_code == 200
assert response.json() == {"cancelled": False, "status": "not_found"}
def test_completion_normal_flow(monkeypatch):
main.ACTIVE_COMPLETIONS.clear()
async def fake_call_ollama(*args, **kwargs):
return {"content": "completion text", "think": ""}
monkeypatch.setattr(main, "call_ollama", fake_call_ollama)
monkeypatch.setattr(main, "build_completion_prompts", lambda *a, **k: ("system", "user"))
monkeypatch.setattr(main, "prepare_prompt_context", lambda *a, **k: ("prefix", "suffix"))
with TestClient(main.app) as client:
response = client.post(
"/v1/completions",
headers=API_KEY_HEADERS,
json=_completion_payload(),
)
assert response.status_code == 200
assert '"content": "completion text"' in response.text
assert '"done": true' in response.text
assert main.ACTIVE_COMPLETIONS == {}
backend/tests/test_main_endpoints.py
-286
View File
@@ -1,286 +0,0 @@
import base64
import asyncio
import base64
import importlib
import os
import sys
from pathlib import Path
from types import SimpleNamespace
from fastapi.testclient import TestClient
os.environ["JOB_BACKEND"] = "memory"
os.environ["DOCS_BACKEND"] = "memory"
CURRENT_DIR = Path(__file__).resolve().parent
BACKEND_DIR = CURRENT_DIR.parent
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
import job_handlers # type: ignore
import job_system # type: ignore
import docs_store # type: ignore
import risk_control # type: ignore
import session_store # type: ignore
import audit_store # type: ignore
main = importlib.import_module("main")
HEADERS = {"X-API-Key": main.API_KEY}
def setup_function():
job_system.reset_job_manager()
docs_store.reset_document_store()
risk_control.reset_risk_controller()
session_store.reset_session_store()
audit_store.reset_audit_store()
main._handlers_registered = False
class DummyRequest:
def __init__(self, host=None, headers=None):
class Client:
pass
self.client = Client() if host is not None else None
if self.client is not None:
self.client.host = host
self.headers = headers or {}
def test_preview_short_text():
assert main._preview("Hello") == "Hello"
def test_preview_long_text_truncated():
long_text = "a" * 100
assert main._preview(long_text) == long_text[:80] + "..."
def test_sanitize_markdown_strips_image_markdown():
assert "![alt](image.png)" not in main._sanitize_converted_markdown("text ![alt](image.png)")
def test_sanitize_inline_completion_strips_prefill():
assert main.sanitize_inline_completion_content("系统非常适合写作", prefill="系统") == "非常适合写作"
def test_get_client_ip_header_overrides_host():
req = DummyRequest(host="1.2.3.4", headers={"X-Client-IP": "5.6.7.8"})
assert main.get_client_ip(req) == "5.6.7.8"
def test_post_completions_without_api_key_uses_anonymous_session(monkeypatch):
async def fake_call(*args, **kwargs):
return {"content": "系统done", "think": ""}
monkeypatch.setattr(job_handlers, "call_ollama", fake_call)
with TestClient(main.app) as client:
with client.stream("POST", "/v1/completions", json={
"prefix": "hello", "suffix": "", "languageId": "markdown",
"model_thinking": "low", "privacy_mode": True,
}) as resp:
assert resp.status_code == 200
assert main.config.session_cookie_name in resp.cookies
def test_post_completions_invalid_api_key_returns_403():
with TestClient(main.app) as client:
resp = client.post(
"/v1/completions",
headers={"X-API-Key": "invalid-key"},
json={
"prefix": "hello", "suffix": "", "languageId": "markdown",
"model_thinking": "low", "privacy_mode": True,
},
)
assert resp.status_code == 403
def test_post_completions_returns_sse_done(monkeypatch):
async def fake_call(*args, **kwargs):
return {"content": "系统done", "think": ""}
monkeypatch.setattr(job_handlers, "call_ollama", fake_call)
with TestClient(main.app) as client:
with client.stream("POST", "/v1/completions", headers=HEADERS, json={
"prefix": "hello", "suffix": "", "languageId": "markdown",
"model_thinking": "low", "privacy_mode": True,
}) as resp:
assert resp.status_code == 200
body = "".join(resp.iter_text())
assert "event: queued" in body
assert "event: started" in body
assert "event: result" in body
assert "event: done" in body
def test_stream_job_emits_keepalive_during_idle(monkeypatch):
async def fake_queue_job(*_args, **_kwargs):
return "job-keepalive"
class FakeManager:
def register_handler(self, *_args, **_kwargs):
return None
async def stream_events(self, job_id):
yield {"event": "queued", "job_id": job_id}
yield {"event": "started", "job_id": job_id}
await asyncio.sleep(0.03)
yield {"event": "done", "job_id": job_id, "result": {"content": "ok"}}
monkeypatch.setattr(main, "STREAM_HEARTBEAT_SECONDS", 0.01)
monkeypatch.setattr(main, "_queue_job", fake_queue_job)
monkeypatch.setattr(main, "get_job_manager", lambda: FakeManager())
with TestClient(main.app) as client:
with client.stream("POST", "/v1/pro/completions", headers=HEADERS, json={
"prefix": "hello", "suffix": "", "languageId": "markdown",
"instruction": "expand", "pro_thinking": "medium", "privacy_mode": True,
}) as resp:
assert resp.status_code == 200
body = "".join(resp.iter_text())
assert ": keepalive" in body
assert "event: done" in body
def test_post_ocr_mocked(monkeypatch):
async def fake_ocr(*args, **kwargs):
return "OCR result text"
monkeypatch.setattr(job_handlers, "call_vlm_ocr", fake_ocr)
img_b64 = base64.b64encode(b"pretend image data").decode()
with TestClient(main.app) as client:
with client.stream("POST", "/v1/ocr", headers=HEADERS, json={
"image": img_b64, "filename": "test.jpg", "language": "auto",
}) as resp:
assert resp.status_code == 200
body = "".join(resp.iter_text())
assert "OCR result text" in body
def test_post_video_ocr_merges_ocr_and_asr(monkeypatch):
async def fake_ocr(*args, **kwargs):
return "画面文字"
async def fake_asr(*args, **kwargs):
return SimpleNamespace(text="音频转写")
monkeypatch.setattr(job_handlers, "call_vlm_ocr", fake_ocr)
monkeypatch.setattr(job_handlers, "generate_asr_response", fake_asr)
monkeypatch.setattr(job_handlers, "extract_audio_wav_bytes", lambda _path: b"fake wav")
video_b64 = base64.b64encode(b"pretend video data").decode()
with TestClient(main.app) as client:
with client.stream("POST", "/v1/ocr", headers=HEADERS, json={
"image": video_b64,
"filename": "sample.mp4",
"language": "auto",
"media_type": "video",
"mime_type": "video/mp4",
}) as resp:
assert resp.status_code == 200
body = "".join(resp.iter_text())
assert "视频画面 OCR" in body
assert "视频音频 ASR" in body
assert "画面文字" in body
assert "音频转写" in body
def test_post_convert_txt_returns_markdown():
content = base64.b64encode(b"hello world").decode()
with TestClient(main.app) as client:
with client.stream("POST", "/v1/convert", headers=HEADERS, json={
"file": content, "filename": "sample.txt",
}) as resp:
assert resp.status_code == 200
body = "".join(resp.iter_text())
assert "hello world" in body
def test_post_convert_unsupported_extension_returns_500():
content = base64.b64encode(b"data").decode()
with TestClient(main.app) as client:
resp = client.post("/v1/convert", headers=HEADERS, json={
"file": content, "filename": "sample.xlsx",
})
assert resp.status_code == 500
assert "仅支持" in resp.json()["error"]
def test_post_convert_rejects_mismatched_content_suffix():
content = base64.b64encode(b"%PDF-1.4\n%%EOF").decode()
with TestClient(main.app) as client:
resp = client.post("/v1/convert", headers=HEADERS, json={
"file": content, "filename": "sample.txt",
})
assert resp.status_code == 500
assert "仅支持" in resp.json()["error"]
def test_docs_nodes_crud_round_trip():
with TestClient(main.app) as client:
folder_resp = client.post("/v1/docs/folders", headers=HEADERS, json={
"name": "项目资料",
"parentId": None,
})
assert folder_resp.status_code == 200
folder = folder_resp.json()["node"]
file_resp = client.post("/v1/docs/files/text", headers=HEADERS, json={
"name": "notes.md",
"parentId": folder["id"],
"content": "# hello",
})
assert file_resp.status_code == 200
file_node = file_resp.json()["node"]
assert file_node["previewText"] == "# hello"
list_resp = client.get("/v1/docs/nodes", headers=HEADERS)
assert list_resp.status_code == 200
nodes = list_resp.json()["nodes"]
assert len(nodes) == 2
rename_resp = client.patch(f"/v1/docs/nodes/{file_node['id']}", headers=HEADERS, json={
"name": "renamed.md",
})
assert rename_resp.status_code == 200
assert rename_resp.json()["node"]["name"] == "renamed.md"
blob_resp = client.get(f"/v1/docs/files/{file_node['id']}/blob", headers=HEADERS)
assert blob_resp.status_code == 200
assert blob_resp.content == b"# hello"
delete_resp = client.delete(f"/v1/docs/nodes/{folder['id']}", headers=HEADERS)
assert delete_resp.status_code == 200
final_list = client.get("/v1/docs/nodes", headers=HEADERS)
assert final_list.status_code == 200
assert final_list.json()["nodes"] == []
def test_docs_file_upload_and_blob_replace():
with TestClient(main.app) as client:
upload_resp = client.post(
"/v1/docs/files/upload",
headers=HEADERS,
files={"file": ("image.png", b"png-bytes", "image/png")},
data={"parent_id": ""},
)
assert upload_resp.status_code == 200
node = upload_resp.json()["node"]
assert node["storageKind"] == "blob"
replace_resp = client.put(
f"/v1/docs/files/{node['id']}/blob",
headers=HEADERS,
files={"file": ("photo.jpg", b"jpeg-bytes", "image/jpeg")},
)
assert replace_resp.status_code == 200
assert replace_resp.json()["node"]["name"] == "photo.jpg"
blob_resp = client.get(f"/v1/docs/files/{node['id']}/blob", headers=HEADERS)
assert blob_resp.status_code == 200
assert blob_resp.content == b"jpeg-bytes"
backend/tests/test_pro_completions.py
-110
View File
@@ -1,110 +0,0 @@
import importlib
import os
import sys
from pathlib import Path
from fastapi.testclient import TestClient
os.environ["JOB_BACKEND"] = "memory"
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
import job_handlers # type: ignore
import job_system # type: ignore
import prompt # type: ignore
main = importlib.import_module("main")
HEADERS = {"X-API-Key": main.API_KEY}
def setup_function():
job_system.reset_job_manager()
main._handlers_registered = False
def _payload():
return {
"prefix": "Before",
"suffix": "After",
"languageId": "markdown",
"instruction": "expand",
"pro_thinking": "medium",
"privacy_mode": True,
"user_preferences": {
"language": "zh",
"country": "CN",
"timezone": "Asia/Shanghai",
},
}
def test_pro_queue_full_returns_429(monkeypatch):
async def fake_queue_job(*args, **kwargs):
raise job_system.QueueFullError("pro_completion", 8)
monkeypatch.setattr(main, "_queue_job", fake_queue_job)
with TestClient(main.app) as client:
response = client.post("/v1/pro/completions", headers=HEADERS, json=_payload())
assert response.status_code == 429
def test_pro_status_missing_returns_404():
with TestClient(main.app) as client:
response = client.get("/v1/pro/completions/status/missing", headers=HEADERS)
assert response.status_code == 404
def test_pro_prompt_uses_pro_specific_instruction():
system_prompt, user_prompt = prompt.build_pro_completion_prompts(
prefix="欢迎使用 LLM-IN-TEXT\n\n即时可用的 LLM 系统",
suffix="",
language_id="markdown",
instruction="",
pro_thinking_level="high",
)
combined = f"{system_prompt}\n{user_prompt}".lower()
assert "[pro] model for llm-in-text" in combined
assert "pro_mode: true" in combined
assert "pro_thinking_level: high" in combined
def test_pro_prompt_accepts_serialized_preferences():
_, user_prompt = prompt.build_pro_completion_prompts(
prefix="Before",
suffix="After",
language_id="markdown",
instruction="expand",
preferences={
"language": "zh",
"country": "CN",
"timezone": "Asia/Shanghai",
},
)
assert "Preferred language: zh" in user_prompt
assert "Preferred country: CN" in user_prompt
assert "Preferred timezone: Asia/Shanghai" in user_prompt
def test_pro_stream_returns_standard_events(monkeypatch):
async def fake_stream_events(*args, **kwargs):
yield "thinking", ""
yield "content", "深度"
yield "content", "回答"
monkeypatch.setattr(job_handlers, "stream_ollama_events", fake_stream_events)
with TestClient(main.app) as client:
with client.stream("POST", "/v1/pro/completions", headers=HEADERS, json=_payload()) as resp:
assert resp.status_code == 200
body = "".join(resp.iter_text())
assert "event: queued" in body
assert "event: started" in body
assert "event: progress" in body
assert "event: result" in body
assert "event: done" in body
assert "深度" in body
assert "回答" in body
backend/tests/test_prompt.py
+10 -37
View File
@@ -10,53 +10,26 @@ import prompt # noqa: E402
def test_prompt_builds_system_and_user():
system_prompt, user_prompt, prefill = prompt.build_completion_prompts(
system_prompt, user_prompt = prompt.build_completion_prompts(
prefix="The result is ",
suffix="for this dataset.",
language_id="markdown",
)
assert "inline completion engine" in system_prompt
assert "Hard constraints you must follow" in system_prompt
assert "strict KaTeX-compatible math" in system_prompt
assert "$...$" in system_prompt
assert "$$...$$" in system_prompt
assert "```{language}" in system_prompt
assert "Mermaid" in system_prompt
assert "Mermaid-specific completion rules" in system_prompt
assert "CURSOR_FENCE_LANGUAGE" in system_prompt
assert "MERMAID_CONTEXT" in system_prompt
assert "Output Mermaid statements only." in system_prompt
assert "CURSOR_IN_FENCED_CODE_BLOCK" in user_prompt
assert "CURSOR_FENCE_LANGUAGE" in user_prompt
assert "MERMAID_CONTEXT" in user_prompt
assert "PREFIX_ENDS_WITH_NEWLINE" in user_prompt
assert "SUFFIX_STARTS_WITH_NEWLINE" in user_prompt
assert "actual line breaks" in system_prompt
assert "Use real line breaks instead of spelled-out escape sequences" in user_prompt
assert "Do not explain newline or boundary choices" in user_prompt
assert "Continue after the PREFILL text" in user_prompt
assert "Step 1" not in user_prompt
assert "Does output need" not in user_prompt
assert "assistant" in system_prompt
assert "fim_middle" in system_prompt
assert prefill == ""
assert "start output with \\n" not in user_prompt
assert "Use single \\n" not in system_prompt
def test_completion_prefill_appended_to_fim_middle():
_, user_prompt, prefill = prompt.build_completion_prompts(
prefix="即时可用的 LLM 系统",
suffix="",
)
assert prefill == "系统"
assert user_prompt.endswith("<|fim_middle|>系统")
def test_completion_prefill_empty_after_newline():
_, user_prompt, prefill = prompt.build_completion_prompts(
prefix="即时可用的 LLM 系统\n",
suffix="",
)
assert prefill == ""
assert user_prompt.endswith("<|fim_middle|>")
def test_cursor_in_fence_detection():
@@ -75,7 +48,7 @@ def test_active_fence_language_detection():
def test_newline_flags():
_, user_prompt_a, _ = prompt.build_completion_prompts(
_, user_prompt_a = prompt.build_completion_prompts(
prefix="Hello",
suffix="World",
)
@@ -85,7 +58,7 @@ def test_newline_flags():
assert "PREFIX_ENDS_WITH_NEWLINE: false" in user_prompt_a
assert "SUFFIX_STARTS_WITH_NEWLINE: false" in user_prompt_a
_, user_prompt_b, _ = prompt.build_completion_prompts(
_, user_prompt_b = prompt.build_completion_prompts(
prefix="Hello\n",
suffix="\nWorld",
)
@@ -95,7 +68,7 @@ def test_newline_flags():
def test_mermaid_context_flags():
_, prompt_in_mermaid, _ = prompt.build_completion_prompts(
_, prompt_in_mermaid = prompt.build_completion_prompts(
prefix="```mermaid\nflowchart TD\nA --> ",
suffix="\n```",
)
@@ -103,7 +76,7 @@ def test_mermaid_context_flags():
assert "CURSOR_FENCE_LANGUAGE: mermaid" in prompt_in_mermaid
assert "MERMAID_CONTEXT: true" in prompt_in_mermaid
_, prompt_mermaid_keyword, _ = prompt.build_completion_prompts(
_, prompt_mermaid_keyword = prompt.build_completion_prompts(
prefix="Please draw a mermaid flowchart for deploy pipeline.",
suffix="",
)
@@ -113,6 +86,6 @@ def test_mermaid_context_flags():
def test_examples_coverage():
_, user_prompt, _ = prompt.build_completion_prompts(prefix="", suffix="")
_, user_prompt = prompt.build_completion_prompts(prefix="", suffix="")
for ex in range(1, 15):
assert f"[EX{ex:02d}]" in user_prompt
backend/tests/test_tts_asr.py
-334
View File
@@ -1,334 +0,0 @@
"""Tests for the shared LLM speech adapter and speech job handlers."""
from __future__ import annotations
import asyncio
import base64
import json
import tempfile
from pathlib import Path
import httpx
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in __import__("sys").path:
__import__("sys").path.insert(0, str(BACKEND_DIR))
import job_handlers # noqa: E402
import tts_asr # noqa: E402
from audit_store import BaseAuditStore # noqa: E402
def _wav_bytes(duration_ms: int = 100) -> bytes:
sample_rate = 16000
frames = max(1, int(sample_rate * duration_ms / 1000))
data = b"".join((i % 32768).to_bytes(2, "little", signed=False) for i in range(frames))
data_size = len(data)
return (
b"RIFF" + (36 + data_size).to_bytes(4, "little")
+ b"WAVE"
+ b"fmt " + (16).to_bytes(4, "little")
+ (1).to_bytes(2, "little")
+ (1).to_bytes(2, "little")
+ sample_rate.to_bytes(4, "little")
+ sample_rate.to_bytes(4, "little")
+ (2).to_bytes(2, "little")
+ (16).to_bytes(2, "little")
+ b"data" + data_size.to_bytes(4, "little")
+ data
)
def _run_async(coro):
return asyncio.run(coro)
class _CaptureAuditStore(BaseAuditStore):
def __init__(self) -> None:
self.llm_calls: list[dict] = []
def record_llm_call(self, payload: dict) -> None:
self.llm_calls.append(payload)
def test_tts_calls_shared_llm_speech_endpoint(monkeypatch):
captured: dict[str, object] = {}
monkeypatch.setattr(tts_asr, "LLM_API_KEY", "test-api-key")
monkeypatch.setattr(tts_asr, "TTS_MODEL_ID", "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit")
def transport(request: httpx.Request):
captured["url"] = str(request.url)
captured["headers"] = dict(request.headers)
captured["json"] = json.loads(request.read().decode("utf-8"))
return httpx.Response(200, content=b"speech-ok", headers={"x-request-id": "tts-req-1"})
async def run():
client = httpx.AsyncClient(
base_url="https://speech.example/v1",
transport=httpx.MockTransport(transport),
)
try:
tts_asr._httpx_client = client
return await tts_asr.generate_tts_response(
"你好世界",
instruct="A warm Mandarin voice.",
speaker="Vivian",
output_format="wav",
)
finally:
await client.aclose()
tts_asr._httpx_client = None
result = _run_async(run())
assert result["format"] == "wav"
assert result["speaker"] == "Vivian"
assert result["model"] == "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit"
assert result["upstream_request_id"] == "tts-req-1"
assert base64.b64decode(result["audio_base64"]) == b"speech-ok"
assert captured["url"] == "https://speech.example/v1/audio/speech"
assert captured["headers"]["authorization"] == "Bearer test-api-key"
payload = captured["json"]
assert payload["model"] == "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit"
assert payload["voice"] == "Vivian"
assert payload["input"] == "你好世界"
assert payload["instructions"] == "A warm Mandarin voice."
assert "instruction" not in payload
def test_tts_uses_nonempty_default_instructions(monkeypatch):
captured: dict[str, object] = {}
def transport(request: httpx.Request):
captured["json"] = json.loads(request.read().decode("utf-8"))
return httpx.Response(200, content=b"speech-ok")
async def run():
client = httpx.AsyncClient(
base_url="https://speech.example/v1",
transport=httpx.MockTransport(transport),
)
try:
tts_asr._httpx_client = client
return await tts_asr.generate_tts_response("你好世界")
finally:
await client.aclose()
tts_asr._httpx_client = None
_run_async(run())
payload = captured["json"]
assert payload["instructions"] == tts_asr.DEFAULT_TTS_INSTRUCTIONS
assert payload["instructions"].strip()
def test_asr_calls_shared_llm_transcriptions_endpoint(monkeypatch):
captured: dict[str, object] = {}
monkeypatch.setattr(tts_asr, "LLM_API_KEY", "test-api-key")
monkeypatch.setattr(tts_asr, "ASR_MODEL_ID", "Qwen3-ASR-0.6B-8bit")
def transport(request: httpx.Request):
captured["url"] = str(request.url)
captured["headers"] = dict(request.headers)
captured["content"] = request.read()
return httpx.Response(200, json={"text": "hello world", "language": "zh"}, headers={"x-request-id": "asr-req-1"})
async def run():
client = httpx.AsyncClient(
base_url="https://speech.example/v1",
transport=httpx.MockTransport(transport),
)
try:
tts_asr._httpx_client = client
return await tts_asr.generate_asr_response(_wav_bytes(), language="zh-CN")
finally:
await client.aclose()
tts_asr._httpx_client = None
result = _run_async(run())
assert result["text"] == "hello world"
assert result["language"] == "zh"
assert result["model"] == "Qwen3-ASR-0.6B-8bit"
assert result["upstream_request_id"] == "asr-req-1"
assert captured["url"] == "https://speech.example/v1/audio/transcriptions"
assert captured["headers"]["authorization"] == "Bearer test-api-key"
content = captured["content"]
assert b'name="model"' in content
assert b"Qwen3-ASR-0.6B-8bit" in content
assert b'name="language"' in content
assert b"zh" in content
def test_invalid_tts_text_returns_http_exception():
with pytest.raises(tts_asr.HTTPException) as exc:
_run_async(tts_asr._call_tts_api("", speaker="Vivian"))
assert exc.value.status_code == 400
def test_invalid_asr_audio_returns_http_exception():
with pytest.raises(tts_asr.HTTPException) as exc:
_run_async(tts_asr._call_asr_api(b"", language="zh-CN"))
assert exc.value.status_code == 400
def test_status_config_routes(monkeypatch):
app = FastAPI()
app.include_router(tts_asr.meta_router)
monkeypatch.setattr(tts_asr, "LLM_BASE_URL", "https://speech.example/v1")
monkeypatch.setattr(tts_asr, "LLM_API_KEY", "")
monkeypatch.setattr(tts_asr, "TTS_MODEL_ID", "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit")
monkeypatch.setattr(tts_asr, "ASR_MODEL_ID", "Qwen3-ASR-0.6B-8bit")
with TestClient(app) as client:
status = client.get("/status")
config = client.get("/config")
assert status.status_code == 200
assert config.status_code == 200
assert status.json()["llm_url"] == "https://speech.example/v1"
assert status.json()["tts_model"] == "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit"
assert status.json()["asr_model"] == "Qwen3-ASR-0.6B-8bit"
assert status.json()["status"]["api_key_configured"] is False
assert status.json()["status"]["max_connections"] == tts_asr.SPEECH_MAX_CONNECTIONS
def test_tts_concurrent_requests_respect_connection_limit(monkeypatch):
monkeypatch.setattr(tts_asr, "SPEECH_MAX_CONNECTIONS", 4)
monkeypatch.setattr(tts_asr, "SPEECH_MAX_KEEPALIVE_CONNECTIONS", 1)
class LimitedClient:
def __init__(self):
self.semaphore = asyncio.Semaphore(4)
self.active = 0
self.max_active = 0
async def post(self, url: str, **kwargs):
async with self.semaphore:
self.active += 1
self.max_active = max(self.max_active, self.active)
await asyncio.sleep(0.01)
self.active -= 1
return httpx.Response(200, content=b"speech-ok", request=httpx.Request("POST", f"https://speech.example{url}"))
async def run():
client = LimitedClient()
async def get_client():
return client
monkeypatch.setattr(tts_asr, "_get_speech_client", get_client)
await asyncio.gather(*(tts_asr.generate_tts_response(f"文本 {index}") for index in range(20)))
return client
client = _run_async(run())
assert client.max_active <= 4
def test_tts_asr_handlers_record_audit(monkeypatch):
audit_store = _CaptureAuditStore()
async def fake_tts(*args, **kwargs):
return {
"audio_base64": base64.b64encode(b"ok").decode("utf-8"),
"format": "wav",
"duration_ms": 1200,
"audio_bytes": 2,
"text_chars": 2,
"speaker": "Vivian",
"model": "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit",
"request_ms": 45,
"upstream_request_id": "tts-upstream",
}
async def fake_asr(*args, **kwargs):
return {
"text": "hello world",
"language": "zh",
"audio_bytes": len(_wav_bytes()),
"model": "Qwen3-ASR-0.6B-8bit",
"request_ms": 80,
"upstream_request_id": "asr-upstream",
}
monkeypatch.setattr(job_handlers, "generate_tts_response", fake_tts)
monkeypatch.setattr(job_handlers, "generate_asr_response", fake_asr)
monkeypatch.setattr(job_handlers, "get_audit_store", lambda *_args, **_kwargs: audit_store)
base_payload = {
"request_id": "req-1",
"risk": {
"request_id": "req-1",
"session_hash": "session",
"ip_hash": "ip",
"estimated_input_tokens": 12,
"estimated_cost": 0.0,
"policy": {
"job_type": "tts",
"model": "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit",
"profile": "speech_tts",
"max_output_tokens": 0,
},
},
"job_context": {
"created_at": 1000,
"started_at": 1200,
"queue_ms": 200,
},
}
async def run():
events = []
async def emit(event: str, data: dict):
events.append((event, data))
tts_payload = {
**base_payload,
"text": "你好",
"speaker": "Vivian",
"format": "wav",
}
await job_handlers.tts_handler(tts_payload, emit, lambda: False)
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as handle:
handle.write(_wav_bytes())
audio_path = handle.name
try:
asr_payload = {
**base_payload,
"risk": {
**base_payload["risk"],
"policy": {
"job_type": "asr",
"model": "Qwen3-ASR-0.6B-8bit",
"profile": "speech_asr",
"max_output_tokens": 0,
},
},
"input_path": audio_path,
"language": "zh-CN",
}
await job_handlers.asr_handler(asr_payload, emit, lambda: False)
finally:
job_handlers._safe_unlink(audio_path)
return events
events = _run_async(run())
assert any(event == "result" for event, _data in events)
assert len(audit_store.llm_calls) == 2
tts_audit = audit_store.llm_calls[0]
asr_audit = audit_store.llm_calls[1]
assert tts_audit["job_type"] == "tts"
assert tts_audit["queue_ms"] == 200
assert tts_audit["metadata"]["duration_ms"] == 1200
assert tts_audit["metadata"]["upstream_request_id"] == "tts-upstream"
assert asr_audit["job_type"] == "asr"
assert asr_audit["metadata"]["language"] == "zh"
assert asr_audit["metadata"]["upstream_request_id"] == "asr-upstream"
backend/tests/test_web_search.py
-171
View File
@@ -1,171 +0,0 @@
import asyncio
import importlib
import socket
import os
import sys
import threading
from pathlib import Path
from fastapi.testclient import TestClient
os.environ["JOB_BACKEND"] = "memory"
os.environ["DOCS_BACKEND"] = "memory"
CURRENT_DIR = Path(__file__).resolve().parent
BACKEND_DIR = CURRENT_DIR.parent
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
import job_handlers # type: ignore
import job_system # type: ignore
import llm # type: ignore
import risk_control # type: ignore
import session_store # type: ignore
import audit_store # type: ignore
main = importlib.import_module("main")
HEADERS = {"X-API-Key": main.API_KEY}
def setup_function():
job_system.reset_job_manager()
risk_control.reset_risk_controller()
session_store.reset_session_store()
audit_store.reset_audit_store()
main._handlers_registered = False
def _payload():
return {
"prefix": "比较当前主流向量数据库的设计差异",
"suffix": "",
"languageId": "markdown",
"privacy_mode": True,
}
def test_is_blocked_public_url(monkeypatch):
def fake_getaddrinfo(host, port, type=0, flags=0): # noqa: ARG001
del host, flags
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", port, 0, 0))]
monkeypatch.setattr(job_handlers.socket, "getaddrinfo", fake_getaddrinfo)
assert job_handlers._is_blocked_public_url("http://127.0.0.1/test") is True
assert job_handlers._is_blocked_public_url("file:///tmp/test") is True
assert job_handlers._is_blocked_public_url("https://example.com/docs") is False
def test_is_blocked_public_url_resolves_private_hostname(monkeypatch):
def fake_getaddrinfo(host, port, type=0, flags=0): # noqa: ARG001
del host, flags
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("127.0.0.1", port, 0, 0))]
monkeypatch.setattr(job_handlers.socket, "getaddrinfo", fake_getaddrinfo)
assert job_handlers._is_blocked_public_url("https://private.example.com/docs") is True
def test_web_search_route_returns_done(monkeypatch):
async def fake_call_ollama(prompt, system_prompt=None, tag="", **kwargs): # noqa: ARG001
if tag.endswith("-webq"):
return {"content": '["vector database comparison", "pinecone weaviate qdrant"]'}
if tag.endswith("-webu"):
return {"content": '["https://example.com/a", "https://example.com/b"]'}
raise AssertionError(f"unexpected tag: {tag}")
async def fake_stream_ollama_events(prompt, system_prompt=None, tag="", **kwargs): # noqa: ARG001
if not tag.endswith("-webf"):
raise AssertionError(f"unexpected stream tag: {tag}")
yield "content", "第一段\n\n"
yield "content", "第二段"
async def fake_searxng_search(query, *, limit): # noqa: ARG001
return [
{
"title": "Doc A",
"url": "https://example.com/a",
"score": 9.1,
"published_date": "2026-06-08",
"snippet": "snippet a",
},
{
"title": "Doc B",
"url": "https://example.com/b",
"score": 8.8,
"published_date": "2026-06-07",
"snippet": "snippet b",
},
]
async def fake_firecrawl_scrape(url):
return {"url": url, "title": f"title for {url}", "markdown": f"content for {url}"}
monkeypatch.setattr(job_handlers, "call_ollama", fake_call_ollama)
monkeypatch.setattr(job_handlers, "_searxng_search", fake_searxng_search)
monkeypatch.setattr(job_handlers, "_firecrawl_scrape", fake_firecrawl_scrape)
monkeypatch.setattr(llm, "stream_ollama_events", fake_stream_ollama_events)
with TestClient(main.app) as client:
with client.stream("POST", "/v1/web-search", headers=HEADERS, json=_payload()) as resp:
assert resp.status_code == 200
body = "".join(resp.iter_text())
assert "event: progress" in body
assert "keywords" in body
assert "searching" in body
assert "selecting_urls" in body
assert "crawling" in body
assert "synthesizing" in body
assert "event: done" in body
assert "第一段" in body
def test_cancel_web_search(monkeypatch):
started = threading.Event()
cancelled = threading.Event()
async def fake_call_ollama(*args, **kwargs):
tag = kwargs.get("tag", "")
if tag.endswith("-webq"):
started.set()
try:
while True:
await asyncio.sleep(0.05)
except asyncio.CancelledError:
cancelled.set()
raise
return {"content": "[]"}
monkeypatch.setattr(job_handlers, "call_ollama", fake_call_ollama)
request_id = "req-web-search-cancel"
with TestClient(main.app) as client:
response_box = {}
def send_request():
with client.stream(
"POST",
"/v1/web-search",
headers={**HEADERS, "X-Request-Id": request_id},
json=_payload(),
) as response:
response_box["status_code"] = response.status_code
response_box["body"] = "".join(response.iter_text())
search_thread = threading.Thread(target=send_request, daemon=True)
search_thread.start()
assert started.wait(timeout=2.0)
cancel_response = client.post(
"/v1/web-search/cancel",
headers=HEADERS,
json={"request_id": request_id, "reason": "abort"},
)
assert cancel_response.status_code == 200
assert cancel_response.json() == {"cancelled": True, "status": "ok"}
search_thread.join(timeout=5.0)
assert not search_thread.is_alive()
assert cancelled.wait(timeout=2.0)
assert "event: cancelled" in response_box["body"]
backend/tts_asr.py
-371
View File
@@ -1,371 +0,0 @@
"""OpenAI-compatible TTS/ASR adapter bound to the shared LLM API."""
from __future__ import annotations
import asyncio
import base64
import logging
import os
import time
from typing import Any, Optional
import httpx
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
logger = logging.getLogger(__name__)
def _int_env(name: str, default: int) -> int:
try:
return max(1, int(os.getenv(name, str(default))))
except (TypeError, ValueError):
return default
meta_router = APIRouter()
LLM_BASE_URL = (os.getenv("LLM_BASE_URL", "https://api.openai.com/v1/") or "").strip().rstrip("/")
LLM_API_KEY = (os.getenv("LLM_API_KEY", "") or "").strip()
DEFAULT_TTS_MODEL_ID = "Qwen3-TTS-12Hz-1.7B-VoiceDesign-8bit"
DEFAULT_ASR_MODEL_ID = "Qwen3-ASR-0.6B-8bit"
DEFAULT_TTS_INSTRUCTIONS = (
os.getenv("TTS_DEFAULT_INSTRUCTIONS", "A clear, natural voice speaking Mandarin Chinese.")
or "A clear, natural voice speaking Mandarin Chinese."
).strip()
TTS_MODEL_ID = (os.getenv("TTS_MODEL_ID", DEFAULT_TTS_MODEL_ID) or DEFAULT_TTS_MODEL_ID).strip()
ASR_MODEL_ID = (os.getenv("ASR_MODEL_ID", DEFAULT_ASR_MODEL_ID) or DEFAULT_ASR_MODEL_ID).strip()
TTS_MAX_TEXT_CHARS = _int_env("TTS_ASR_MAX_TEXT_CHARS", 4096)
ASR_MAX_AUDIO_BYTES = _int_env("ASR_MAX_AUDIO_BYTES", 100 * 1024 * 1024)
TTS_TIMEOUT_SECONDS = _int_env("TTS_ASR_TTS_TIMEOUT_SECONDS", 180)
ASR_TIMEOUT_SECONDS = _int_env("TTS_ASR_ASR_TIMEOUT_SECONDS", 300)
HEALTHCHECK_TIMEOUT_SECONDS = _int_env("TTS_ASR_HEALTHCHECK_TIMEOUT_SECONDS", 5)
SPEECH_MAX_CONNECTIONS = _int_env("TTS_ASR_MAX_CONNECTIONS", 16)
SPEECH_MAX_KEEPALIVE_CONNECTIONS = _int_env("TTS_ASR_MAX_KEEPALIVE_CONNECTIONS", 8)
_httpx_client: Optional[httpx.AsyncClient] = None
_httpx_client_lock = asyncio.Lock()
def _read_uint16(data: bytes, offset: int) -> Optional[int]:
if len(data) < offset + 2:
return None
return int.from_bytes(data[offset : offset + 2], "little", signed=False)
def _read_uint32(data: bytes, offset: int) -> Optional[int]:
if len(data) < offset + 4:
return None
return int.from_bytes(data[offset : offset + 4], "little", signed=False)
def _parse_wav_duration_ms(audio_bytes: bytes) -> int:
if len(audio_bytes) < 44 or audio_bytes[:4] != b"RIFF" or audio_bytes[8:12] != b"WAVE":
return 0
data_size = 0
byte_rate = 0
offset = 12
while offset + 8 <= len(audio_bytes):
chunk_id = audio_bytes[offset : offset + 4]
chunk_size = _read_uint32(audio_bytes, offset + 4)
if chunk_size is None:
break
chunk_start = offset + 8
chunk_end = min(chunk_start + chunk_size, len(audio_bytes))
if chunk_id == b"fmt ":
audio_format = _read_uint16(audio_bytes, chunk_start)
channels = _read_uint16(audio_bytes, chunk_start + 2)
sample_rate = _read_uint32(audio_bytes, chunk_start + 4)
bits_per_sample = _read_uint16(audio_bytes, chunk_start + 14)
if audio_format == 1 and channels and sample_rate and bits_per_sample:
byte_rate = int(sample_rate * channels * bits_per_sample // 8)
if chunk_id == b"data":
data_size = chunk_size
offset = chunk_end + (chunk_end - chunk_start) % 2
if data_size and byte_rate:
return max(0, int(data_size * 1000 / byte_rate))
return 0
def _duration_from_audio_bytes(audio_bytes: bytes) -> int:
return _parse_wav_duration_ms(audio_bytes)
def _audio_bytes_to_base64(audio_bytes: bytes) -> str:
return base64.b64encode(audio_bytes).decode("utf-8")
def _normalize_tts_text(text: str) -> str:
value = (text or "").strip()
if not value:
raise HTTPException(status_code=400, detail="TTS 文本为空")
if len(value) > TTS_MAX_TEXT_CHARS:
raise HTTPException(status_code=400, detail=f"TTS 文本过长，超过限制 {TTS_MAX_TEXT_CHARS} 个字符")
return value
def _normalize_output_format(output_format: str) -> str:
value = (output_format or "wav").strip().lower()
if value not in {"wav", "mp3"}:
raise HTTPException(status_code=400, detail="不支持的 TTS 输出格式")
return value
def _normalize_asr_language(language: Optional[str]) -> Optional[str]:
if not language:
return None
value = str(language).strip().lower()
if value in {"", "auto"}:
return None
mapping = {
"zh-cn": "zh",
"zh-hans": "zh",
"en-us": "en",
"ja-jp": "ja",
"ko-kr": "ko",
}
return mapping.get(value, value.split("-")[0])
def _speech_headers() -> dict[str, str]:
headers = {"Accept": "*/*"}
if LLM_API_KEY:
headers["Authorization"] = f"Bearer {LLM_API_KEY}"
headers["X-API-Key"] = LLM_API_KEY
return headers
def _raise_http_error(response: httpx.Response, operation: str) -> None:
try:
response.raise_for_status()
except httpx.HTTPStatusError as exc:
body = (exc.response.text or "").strip()[:1000]
detail = f"{operation} 请求失败 HTTP {exc.response.status_code}"
if body:
detail = f"{detail}: {body}"
raise HTTPException(status_code=exc.response.status_code, detail=detail) from exc
except Exception as exc:
raise HTTPException(status_code=502, detail=f"{operation} 请求失败: {exc}") from exc
def _tts_timeout() -> httpx.Timeout:
return httpx.Timeout(TTS_TIMEOUT_SECONDS, connect=5.0)
def _asr_timeout() -> httpx.Timeout:
return httpx.Timeout(ASR_TIMEOUT_SECONDS, connect=5.0)
def _extract_upstream_request_id(response: httpx.Response) -> str:
for header_name in ("x-request-id", "request-id", "openai-request-id"):
value = (response.headers.get(header_name) or "").strip()
if value:
return value
return ""
async def _get_speech_client() -> httpx.AsyncClient:
global _httpx_client
if _httpx_client is None or getattr(_httpx_client, "is_closed", False):
limits = httpx.Limits(
max_connections=SPEECH_MAX_CONNECTIONS,
max_keepalive_connections=max(1, SPEECH_MAX_KEEPALIVE_CONNECTIONS),
)
async with _httpx_client_lock:
if _httpx_client is None or getattr(_httpx_client, "is_closed", False):
_httpx_client = httpx.AsyncClient(
base_url=LLM_BASE_URL,
timeout=_tts_timeout(),
headers=_speech_headers(),
follow_redirects=True,
limits=limits,
)
return _httpx_client
async def close_speech_client() -> None:
global _httpx_client
if _httpx_client is not None and not getattr(_httpx_client, "is_closed", False):
await _httpx_client.aclose()
_httpx_client = None
async def _call_tts_api(text: str, instruct: str = "", speaker: str = "Vivian", output_format: str = "wav") -> dict[str, Any]:
normalized_text = _normalize_tts_text(text)
normalized_format = _normalize_output_format(output_format)
client = await _get_speech_client()
payload: dict[str, Any] = {
"model": TTS_MODEL_ID,
"input": normalized_text,
"response_format": normalized_format,
"voice": speaker or "Vivian",
}
payload["instructions"] = (instruct or "").strip() or DEFAULT_TTS_INSTRUCTIONS
started_at = time.perf_counter()
response = await client.post("audio/speech", json=payload, timeout=_tts_timeout(), headers=_speech_headers())
elapsed_ms = int((time.perf_counter() - started_at) * 1000)
_raise_http_error(response, "TTS")
audio_bytes = response.content
if not audio_bytes:
raise HTTPException(status_code=502, detail="TTS API 返回音频为空")
return {
"audio_bytes": audio_bytes,
"request_ms": elapsed_ms,
"upstream_request_id": _extract_upstream_request_id(response),
}
async def _call_asr_api(audio_bytes: bytes, language: Optional[str] = "zh-CN") -> dict[str, Any]:
if not audio_bytes:
raise HTTPException(status_code=400, detail="ASR 音频内容为空")
if len(audio_bytes) > ASR_MAX_AUDIO_BYTES:
raise HTTPException(status_code=400, detail=f"ASR 音频过大，超过限制 {ASR_MAX_AUDIO_BYTES} 字节")
normalized_language = _normalize_asr_language(language)
client = await _get_speech_client()
files = {"file": ("audio.wav", audio_bytes, "audio/wav")}
data = {"model": ASR_MODEL_ID}
if normalized_language:
data["language"] = normalized_language
started_at = time.perf_counter()
response = await client.post(
"audio/transcriptions",
files=files,
data=data,
timeout=_asr_timeout(),
headers=_speech_headers(),
)
elapsed_ms = int((time.perf_counter() - started_at) * 1000)
_raise_http_error(response, "ASR")
try:
result = response.json()
except ValueError as exc:
raise HTTPException(status_code=502, detail="ASR API 返回非 JSON 数据") from exc
if not isinstance(result, dict):
raise HTTPException(status_code=502, detail="ASR API 返回结构异常")
text = str(result.get("text", "") or "").strip()
if not text:
raise HTTPException(status_code=422, detail="ASR API 返回结果为空")
detected_language = result.get("language") or normalized_language or "auto"
return {
"text": text,
"language": str(detected_language),
"request_ms": elapsed_ms,
"upstream_request_id": _extract_upstream_request_id(response),
}
async def generate_tts_response(
text: str,
instruct: str = "",
speaker: str = "Vivian",
output_format: str = "wav",
) -> dict[str, Any]:
result = await _call_tts_api(
text=text,
instruct=instruct or "",
speaker=speaker or "Vivian",
output_format=output_format or "wav",
)
audio_bytes = bytes(result["audio_bytes"])
return {
"audio_base64": _audio_bytes_to_base64(audio_bytes),
"format": _normalize_output_format(output_format or "wav"),
"duration_ms": _duration_from_audio_bytes(audio_bytes),
"audio_bytes": len(audio_bytes),
"text_chars": len(_normalize_tts_text(text)),
"speaker": speaker or "Vivian",
"model": TTS_MODEL_ID,
"request_ms": int(result.get("request_ms", 0) or 0),
"upstream_request_id": str(result.get("upstream_request_id", "") or ""),
}
async def generate_asr_response(audio_bytes: bytes, language: Optional[str] = "zh-CN") -> dict[str, Any]:
result = await _call_asr_api(bytes(audio_bytes or b""), language or "zh-CN")
return {
"text": str(result["text"]),
"language": str(result["language"]),
"audio_bytes": len(audio_bytes or b""),
"model": ASR_MODEL_ID,
"request_ms": int(result.get("request_ms", 0) or 0),
"upstream_request_id": str(result.get("upstream_request_id", "") or ""),
}
class TTSResponse(BaseModel):
audio_base64: str = ""
format: str = "wav"
duration_ms: int = 0
audio_bytes: int = 0
text_chars: int = 0
speaker: str = "Vivian"
model: str = TTS_MODEL_ID
request_ms: int = 0
upstream_request_id: str = ""
class ASRResponse(BaseModel):
text: str = ""
language: Optional[str] = None
audio_bytes: int = 0
model: str = ASR_MODEL_ID
request_ms: int = 0
upstream_request_id: str = ""
class ModelStatus(BaseModel):
llm_url: str
tts_model: str
asr_model: str
status: dict[str, Any]
def _status_payload() -> dict[str, Any]:
return {
"llm_url": LLM_BASE_URL or "",
"tts_model": TTS_MODEL_ID,
"asr_model": ASR_MODEL_ID,
"status": {
"api_configured": bool(LLM_BASE_URL),
"api_key_configured": bool(LLM_API_KEY),
"tts_model": TTS_MODEL_ID,
"asr_model": ASR_MODEL_ID,
"tts_timeout_seconds": TTS_TIMEOUT_SECONDS,
"asr_timeout_seconds": ASR_TIMEOUT_SECONDS,
"healthcheck_timeout_seconds": HEALTHCHECK_TIMEOUT_SECONDS,
"max_connections": SPEECH_MAX_CONNECTIONS,
"keepalive_connections": max(1, SPEECH_MAX_KEEPALIVE_CONNECTIONS),
"max_tts_text_chars": TTS_MAX_TEXT_CHARS,
"max_asr_audio_bytes": ASR_MAX_AUDIO_BYTES,
},
}
@meta_router.get("/status", response_model=ModelStatus)
async def get_status():
return _status_payload()
@meta_router.get("/config")
async def get_config():
return _status_payload()
def register_tts_asr_routes(app) -> None:
app.include_router(meta_router, prefix="/v1/tts-asr")
backend/worker.py
-41
View File
@@ -1,41 +0,0 @@
import asyncio
import logging
from job_handlers import (
asr_handler,
completion_handler,
compress_handler,
convert_handler,
ocr_handler,
pro_completion_handler,
tts_handler,
web_search_handler,
)
from job_system import RedisJobManager, RedisWorker
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s - %(message)s",
)
logger = logging.getLogger("worker")
async def main() -> None:
manager = RedisJobManager()
manager.register_handler("completion", completion_handler)
manager.register_handler("pro_completion", pro_completion_handler)
manager.register_handler("web_search", web_search_handler)
manager.register_handler("compress", compress_handler)
manager.register_handler("ocr", ocr_handler)
manager.register_handler("convert", convert_handler)
manager.register_handler("tts", tts_handler)
manager.register_handler("asr", asr_handler)
worker = RedisWorker(manager)
try:
await worker.run_forever()
finally:
await manager.close()
if __name__ == "__main__":
asyncio.run(main())
docker-compose.yml
-185
View File
@@ -1,185 +0,0 @@
services:
frontend:
build:
context: .
dockerfile: Dockerfile.frontend
args:
DOCKER_REGISTRY_PREFIX: ${DOCKER_REGISTRY_PREFIX:-}
cache_from:
- type=local,src=./docker-data/build-cache/frontend
cache_to:
- type=local,dest=./docker-data/build-cache/frontend,mode=max
depends_on:
api:
condition: service_started
restart: unless-stopped
ports:
- "8080:80"
healthcheck:
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1/ || exit 1"]
interval: 30s
timeout: 5s
retries: 3
postgres:
image: ${DOCKER_REGISTRY_PREFIX:-}postgres:16-alpine
restart: unless-stopped
environment:
POSTGRES_DB: ${POSTGRES_DB:-llm_in_text}
POSTGRES_USER: ${POSTGRES_USER:-llm_in_text}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-llm_in_text_change_me}
volumes:
- ./docker-data/postgres:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-llm_in_text} -d ${POSTGRES_DB:-llm_in_text}"]
interval: 15s
timeout: 5s
retries: 5
redis:
image: ${DOCKER_REGISTRY_PREFIX:-}redis:7-alpine
restart: unless-stopped
command: ["redis-server", "--appendonly", "yes"]
volumes:
- ./docker-data/redis:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 15s
timeout: 5s
retries: 5
searxng:
image: ${DOCKER_REGISTRY_PREFIX:-}searxng/searxng:latest
restart: unless-stopped
environment:
BASE_URL: http://searxng:8080/
INSTANCE_NAME: llm-in-text-search
volumes:
- ./docker-data/searxng:/etc/searxng
firecrawl-postgres:
image: ${DOCKER_REGISTRY_PREFIX:-}postgres:16-alpine
restart: unless-stopped
environment:
POSTGRES_DB: firecrawl
POSTGRES_USER: firecrawl
POSTGRES_PASSWORD: firecrawl
volumes:
- ./docker-data/firecrawl-postgres:/var/lib/postgresql/data
firecrawl-rabbitmq:
image: ${DOCKER_REGISTRY_PREFIX:-}rabbitmq:3-management-alpine
restart: unless-stopped
volumes:
- ./docker-data/firecrawl-rabbitmq:/var/lib/rabbitmq
firecrawl-playwright:
image: ghcr.io/firecrawl/playwright-service:latest
restart: unless-stopped
environment:
PORT: 3000
firecrawl:
image: ghcr.io/firecrawl/firecrawl:latest
restart: unless-stopped
environment:
HOST: 0.0.0.0
PORT: 3002
POSTGRES_HOST: firecrawl-postgres
POSTGRES_PORT: 5432
POSTGRES_DB: firecrawl
POSTGRES_USER: firecrawl
POSTGRES_PASSWORD: firecrawl
REDIS_URL: redis://redis:6379/1
REDIS_RATE_LIMIT_URL: redis://redis:6379/1
NUQ_RABBITMQ_URL: amqp://guest:guest@firecrawl-rabbitmq:5672/
PLAYWRIGHT_MICROSERVICE_URL: http://firecrawl-playwright:3000/scrape
SEARXNG_ENDPOINT: http://searxng:8080
USE_DB_AUTHENTICATION: "false"
FIRECRAWL_API_KEY: ${FIRECRAWL_API_KEY:-change-me}
depends_on:
- redis
- firecrawl-postgres
- firecrawl-rabbitmq
- firecrawl-playwright
api:
build:
context: .
dockerfile: backend/Dockerfile
args:
DOCKER_REGISTRY_PREFIX: ${DOCKER_REGISTRY_PREFIX:-}
cache_from:
- type=local,src=./docker-data/build-cache/api
cache_to:
- type=local,dest=./docker-data/build-cache/api,mode=max
env_file:
- backend/.env
environment:
JOB_BACKEND: redis
REDIS_URL: redis://redis:6379/0
DATABASE_URL: ${DATABASE_URL:-postgresql://llm_in_text:llm_in_text_change_me@postgres:5432/llm_in_text}
DOCS_BACKEND: postgres
JOB_SHARED_TEMP_DIR: /shared-jobs
SEARXNG_BASE_URL: http://searxng:8080
FIRECRAWL_BASE_URL: http://firecrawl:3002
restart: unless-stopped
init: true
extra_hosts:
- "host.docker.internal:host-gateway"
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
searxng:
condition: service_started
firecrawl:
condition: service_started
ports:
- "8001:8001"
volumes:
- ./docker-data/jobs:/shared-jobs
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8001/v1/tts-asr/status', timeout=5)"]
interval: 30s
timeout: 10s
retries: 5
start_period: 20s
worker:
build:
context: .
dockerfile: backend/Dockerfile
args:
DOCKER_REGISTRY_PREFIX: ${DOCKER_REGISTRY_PREFIX:-}
cache_from:
- type=local,src=./docker-data/build-cache/worker
cache_to:
- type=local,dest=./docker-data/build-cache/worker,mode=max
command: ["python", "worker.py"]
env_file:
- backend/.env
environment:
JOB_BACKEND: redis
REDIS_URL: redis://redis:6379/0
DATABASE_URL: ${DATABASE_URL:-postgresql://llm_in_text:llm_in_text_change_me@postgres:5432/llm_in_text}
DOCS_BACKEND: postgres
JOB_SHARED_TEMP_DIR: /shared-jobs
SEARXNG_BASE_URL: http://searxng:8080
FIRECRAWL_BASE_URL: http://firecrawl:3002
restart: unless-stopped
init: true
extra_hosts:
- "host.docker.internal:host-gateway"
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
searxng:
condition: service_started
firecrawl:
condition: service_started
volumes:
- ./docker-data/jobs:/shared-jobs
docker/nginx.conf
-11
View File
@@ -1,11 +0,0 @@
server {
listen 80;
server_name _;
root /usr/share/nginx/html;
index index.html;
location / {
try_files $uri $uri/ /index.html;
}
}
milkdown-docs/blogs/announcing-telemetry-inspector.md
+142
View File
@@ -0,0 +1,142 @@
# Announcing Telemetry Inspector
There's a lot of questions from community asking that how can they know what plugins are enabled.
From Milkdown@7.2, we've added telemetries for milkdown, it can be available by inspectors.
With this API, you can inspect editor inner status.
You can even use visualizer to visualize the data. We create a simple example on [our playground](/playground).
![Milkdown Inspector](/blogs/announcing-telemetry-inspector/milkdown-inspector.gif)
## Get Started
Inspector will be a top-level API in Milkdown. You can use it like this:
```ts
import { Editor } from "@milkdown/core";
import { Telemetry } from "@milkdown/ctx";
const editor = await Editor.make()
// Inspector is disabled by default considering performance. You need to enable it manually.
.enableInspector()
// ...
.create();
const telemetry: Telemetry[] = editor.inspect();
```
The `Telemetry` interface will have the following fields:
```ts
interface Telemetry {
// User defined information for the plugin.
metadata: Meta;
// The slices and their current value defined by the plugin.
injectedSlices: { name: string; value: unknown }[];
// The slices and their current value consumed by the plugin.
consumedSlices: { name: string; value: unknown }[];
// The timers and their duration defined by the plugin.
recordedTimers: { name: string; duration: number; status: TimerStatus }[];
// The timers and their duration consumed by the plugin.
// Generally, the plugin will wait for them.
waitTimers: { name: string; duration: number; status: TimerStatus }[];
}
type TimerStatus = "pending" | "resolved" | "rejected";
interface Meta {
displayName: string;
description?: string;
package: string;
group?: string;
additional?: Record<string, any>;
}
```
For every plugin, it'll have a telemetry if it has metadata declared.
With the data, you'll know the sequence of the plugins loaded, the slices and timers they defined and consumed.
For example:
```ts
[
{
metadata: {
displayName: "Config",
package: "@milkdown/core",
group: "System",
},
injectedSlices: [],
consumedSlices: [
/* ... */
],
recordedTimers: [
{
name: "ConfigReady",
duration: 3,
status: "resolved",
},
],
waitTimers: [],
},
{
metadata: {
displayName: "Init",
package: "@milkdown/core",
group: "System",
},
injectedSlices: [],
consumedSlices: [
/* ... */
],
recordedTimers: [
{
name: "InitReady",
duration: 5,
status: "resolved",
},
],
waitTimers: [
{
name: "ConfigReady",
duration: 5,
status: "resolved",
},
],
},
];
```
From above information, we can know that the `Init` plugin wait for `Config` plugin to be ready.
We can build a sequence diagram from the data.
![Timer Sequence](/blogs/announcing-telemetry-inspector/timer-sequence.gif)
## Add Metadata for Plugin
For plugin maintainers, you can add metadata to your plugin to make it more friendly to the inspector.
```ts
import { MilkdownPlugin } from "@milkdown/ctx";
const yourMilkdownPlugin: MilkdownPlugin = () => {
/* your implementation */
};
yourMilkdownPlugin.metadata = {
displayName: "Your Plugin",
package: "your-plugin-package",
description: "Your plugin description",
group: "If you have a lot of plugins in your package, you can group them.",
addtitional: {
/* You can add any additional information here. */
version: "1.0.0",
authror: "Mike",
},
};
```
With metadata, your plugin will report telemetry correctly to the inspector.
milkdown-docs/blogs/build-your-own-milkdown-copilot.md
+247
View File
@@ -0,0 +1,247 @@
# Build Your Own Milkdown Copilot
OpenAI introduced ChatGPT in 2020, which is a chatbot that can generate natural language responses to user input.
Which brings us a new way to interact with devices and applications.
Nowadays, there are more and more tools that are powered by AI. Such as Notion, GitHub and even Microsoft 365.
Since OpenAI also released the [API](https://openai.com/blog/openai-api) of it. And Milkdown is composed by plugins.
I think it's possible to build a Milkdown Copilot Plugin that can help you write documents. So I did it.
Let's see the result.
![Milkdown Copilot](/blogs/build-your-own-milkdown-copilot/milkdown-copilot.gif)
Looks cool, right? But how does it work? I'll explain it in the following sections.
## Prepare a Backend
**Before we start, you need to have a OpenAI API Key.** You'll need to get one [here](https://platform.openai.com/account/api-keys).
I'll not explain how to get it. You can find the details in their [official docs](https://platform.openai.com/).
I'll use Node.js to build the backend. You can use any language you like.
The backend is very simple. It just calls the OpenAI API and returns the result.
```ts
import { Configuration, OpenAIApi } from "openai";
const configuration = new Configuration({
// Get your API key from env variable
apiKey: process.env.OPENAPI_KEY,
});
const openai = new OpenAIApi(configuration);
export const handler = async (req, res, next) => {
if (req.path === "/api/copilot" && req.method === "POST") {
const buffers = [];
// Get the body of the request.
const body = JSON.parse(req.body);
// Get prompt from the body.
const { prompt } = body;
const completion = await openai.createCompletion({
// Pick a model you like
model: "text-davinci-003",
prompt,
});
const hint = completion.data.choices[0].text;
return res.end(JSON.stringify({ hint }));
}
next();
return;
};
```
We watch the `/api/copilot` route and call the OpenAI API when we receive a POST request.
The post request should contain a `prompt` field which is the text that we want to complete.
To call our API, we just need one single helper in browser environment:
```ts
async function fetchAIHint(prompt: string) {
const data: Record<string, string> = { prompt };
const response = await fetch("/api/copilot", {
method: "POST",
body: JSON.stringify(data),
});
const res = (await response.json()) as { hint: string };
return res.hint;
}
```
## Build a Milkdown Plugin
Now let's focus on the Milkdown Copilot Plugin.
Basically I want to implement two things:
1. When the user types `<Enter>` or `<Space>`, they will get a hint from the copilot.
2. When the user types `<Tab>`, they will apply the content from the hint to the editor.
### Overview
To build a bridge between the copilot and the editor,
we can build a prosemirror plugin and use the `onKeyDown` hook to listen to the keydown event.
```ts
function keyDownHandler(ctx: Ctx, event: Event) {
if (event.key === "Enter" || event.code === "Space") {
getHint(ctx);
return;
}
if (event.key === "Tab") {
// prevent the browser from focusing on the next element.
event.preventDefault();
applyHint(ctx);
return;
}
hideHint(ctx);
}
```
When the user types `<Enter>` or `<Space>`, we will call the `getHint` function to get a hint from the copilot.
And when the user types `<Tab>`, we will call the `applyHint` function to apply the hint to the editor.
If user types other keys, we will hide the hint.
And we also need a component to render the hint. Here I choose to use a simple [widget decoration in prosemirror](https://prosemirror.net/docs/ref/#view.Decoration^widget).
```ts
function renderHint(message: string) {
const dom = document.createElement("pre");
dom.className = "copilot-hint";
dom.innerHTML = message;
return dom;
}
```
So our component looks like:
```ts
import { Plugin, PluginKey } from "@milkdown/prose/state";
import { Decoration, DecorationSet } from "@milkdown/prose/view";
import { $prose } from "@milkdown/utils";
const initialState = {
deco: DecorationSet.empty,
message: "",
};
export const copilotPluginKey = new PluginKey("milkdown-copilot");
export const copilotPlugin = $prose(
(ctx) =>
new Plugin({
key: copilotPluginKey,
props: {
handleKeyDwon(view, event) {
keydownHandler(ctx, event);
},
decorations(state) {
return copilotPluginKey.getState(state).deco;
},
},
state: {
init() {
return { ...initialState };
},
apply(tr, value, _prevState, state) {
const message = tr.getMeta(copilotPluginKey);
if (typeof message !== "string") return value;
if (message.length === 0) {
return { ...initialState };
}
const { to } = tr.selection;
const widget = Decoration.widget(to + 1, () => renderHint(message));
return {
deco: DecorationSet.create(state.doc, [widget]),
message,
};
},
},
}),
);
```
### Get Hint
To get a hint from the copilot, we need to get the text before the cursor.
```ts
function getHint(ctx: Ctx) {
const view = ctx.get(editorViewCtx);
const { state } = view;
const { tr, schema } = state;
const { from } = tr.selection;
const slice = tr.doc.slice(0, from);
const serializer = ctx.get(serializerCtx);
const doc = schema.topNodeType.createAndFill(undefined, slice.content);
if (!doc) return;
const markdown = serializer(doc);
fetchAIHint(markdown).then((hint) => {
const tr = view.state.tr;
view.dispatch(tr.setMeta(copilotPluginKey, hint));
});
}
```
1. First of all, we get the `selection` from the `state` of the editor.
2. Then we get a `slice` of the document from the start to the cursor.
3. Then we use the `serializer` to convert the slice to markdown.
4. After that, we call the `fetchAIHint` function to get a hint from the copilot.
5. Finally, we dispatch a transaction with the hint message we get to update the state of the editor.
### Hide Hint
To hide the hint, we just need to dispatch a transaction with an empty message.
```ts
function hideHint(ctx: Ctx) {
const view = ctx.get(editorViewCtx);
const { state } = view;
const { tr } = state;
view.dispatch(tr.setMeta(copilotPluginKey, ""));
}
```
### Apply Hint
Since we pass markdown to the OpenAI API. It may return a markdown snippet.
So, before we apply the hint to the editor, we need to convert the markdown snippet to prosemirror node.
```ts
function applyHint(ctx: Ctx) {
const view = ctx.get(editorViewCtx);
const { state } = view;
const { tr, schema } = state;
const { message } = copilotPluginKey.getState(state);
const parser = ctx.get(parserCtx);
const slice = parser(message);
const dom = DOMSerializer.fromSchema(schema).serializeFragment(slice.content);
const node = DOMParser.fromSchema(schema).parseSlice(dom);
// Reset the hint since it's applied
tr.setMeta(copilotPluginKey, "")
// Replace the selection with the hint
.replaceSelection(node);
view.dispatch(tr);
}
```
1. First of all, we get the hint message from the state of the editor.
2. Then we use the `parser` to convert the markdown snippet to prosemirror node.
3. Finally, we dispatch a transaction to replace the selection with the hint.
## Conclusion
In this article, we have built a really simple Copilot plugin for Milkdown.
The plugin is not perfect, but it's a good start to help you build your own.
The source code is available on [Milkdown/examples/vanilla-openapi](https://github.com/Milkdown/examples/tree/main/vanilla-openai).
I hope it can give you some inspiration.
milkdown-docs/blogs/introducing-milkdown@7.md
+151
View File
@@ -0,0 +1,151 @@
# Introducing Milkdown@7
It's been almost one year since the release of [milkdown](https://milkdown.dev) V6.
It helped a lot of users to build their own markdown based applications.
It has 13k downloads per month and I feel so grateful that users like that.
However, we noticed that there're some problems cannot be resolved if we don't make a new major version.
What big changes did we made? I'll introduce them to you in this blog.
## TL;DR
- The editor becomes a first-class headless component.
- Factory plugins are fully replaced by **composable plugins**.
- Runtime plugin toggling is supported.
- Universal widget plugins.
- Better Vue and React support.
- API documentation is provided.
## Why Headless?
In the past, milkdown had a lot of internal styles to make sure the editor can work out of box and the themes are easy to create.
However, I found it limits the users to design their own editor.
Even worse, if you have an well designed application,
it is really hard to keep the style of the milkdown editor same with the rest of the application.
You'll need to override lots of styles everywhere.
It stops a log of users from using milkdown.
If we think about why users need an editor,
the most important thing is always the functionality of the editor.
Users just want a component that can provide smooth editing experience.
Style will always be the second thing.
So, why not remove all the internal styles and make the editor a headless component?
The users can easily integrate the editor into their own application.
They can use their own styles and even use their own components to render the editor.
We just care about the functionality of the editor. Make sure it works well.
## Composable Plugins
Although the composable plugins have been existed in milkdown for a long time,
we use factory plugins to create most of the official plugins in V6.
But, the problem is that factory plugins limit the possibility of the plugins.
The factory plugins handle a bunch of complex logic and it is hard to extend.
So for users who want to create a plugin in a easy way, they must follow the factory plugin's way.
```ts
const nodePlugin = createPlugin(() => ({
id: 'node',
schema: someSchema,
inputRules: someInputRules
commands: someCommands
}))
```
See? You can define a lot of things inside the factory plugin.
But if you want to use some part of them in another plugin, it's really hard to do that.
However, the milkdown's plugin system is designed to be flexible and composable.
We want to let users to control the data flow entirely.
So, we decided to remove all the factory plugins and use composable plugins to replace them.
The composable plugins can keep the atomicity of the plugins and make the plugin system more flexible.
They also make the plugin system easier to maintain.
```ts
const nodeSchema = $node("node", someSchema);
const nodeInputRules = $inputRules(someInputRules);
const nodeCommands = $commands(someCommands);
```
If you want to reuse them, it also will be very easy.
```ts
const anotherCommand = $commands(() => {
return setBlockType(nodeSchema.type());
});
```
## Runtime Plugin Toggling
In the past, once you register a plugin, you cannot remove it.
In V7, we support runtime plugin toggling by providing two new API: `editor.remove` and `editor.removeConfig`.
They can let users remove the plugins and configs at runtime.
```ts
import { Editor } from "@milkdown/core";
import { someMilkdownPlugin } from "some-milkdown-plugin";
const editor = await Editor.config(configForPlugin)
.use(someMilkdownPlugin)
.create();
// remove plugin
await editor.remove(someMilkdownPlugin);
// remove config
editor.removeConfig(configForPlugin);
// add another plugin
editor.use(anotherMilkdownPlugin);
// Recreate the editor to apply changes.
await editor.create();
```
Also, if you call the `editor.create` method after the editor is created,
it will recreate the editor and apply all the changes.
## Universal Widget Plugins
We have 4 official widget plugins in V6: _slash_, _tooltip_, _block_ and _menu_.
They are all well designed and easy to use.
But if you want to customize them, what you can do is really limited.
Also, it's hard to reuse their logic even if you want to create something similar to them.
For example, if you want to create a mention plugin which will show a list of users when you type `@`,
you need to create a new plugin from scratch.
So, in V7, we make _slash_, _tooltip_ and _block_ plugins universal.
You can use them to build you features easily.
For example, if you want to create a mention plugin, you can use the new slash plugin to do that.
Another example is that you can also create tooltips for different types of nodes.
Display a tooltip with input when you focus on an image node, or display a tooltip with buttons when you select some text.
What about the _menu_ plugin? We removed it because we think it's easy to create a menu plugin by yourself.
We've already done that in the [official playground](https://milkdown.dev/playground).
And, trust me, [it won't need much code](https://github.com/milkdown/website/blob/main/src/component/Playground/Milkdown/index.tsx#L57).
## Better Vue and React Support
Thanks to the [Saul-Mirone/prosemirror-adapter project](https://github.com/Saul-Mirone/prosemirror-adapter).
In milkdown V7. We allow users to use vue and react to render lots of parts of the editor.
For example, you can use them to render your own code block, drag handle or even small icons.
- React Example: [![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/Milkdown/examples/tree/main/react-custom-component)
- Vue Example: [![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/Milkdown/examples/tree/main/vue-custom-component)
## API Documentation
What's the hardest thing to do when maintaining an open source project?
Keep the documentation up to date.
Thanks to the [marijnh/builddocs project](https://github.com/marijnh/builddocs),
we can generate the API documentation automatically from the source code.
We also redesigned the documentation website, provide a more powerful playground and lots of examples.
milkdown-docs/blogs/understanding-headless-slash-plugin.md
+172
View File
@@ -0,0 +1,172 @@
# Understanding Headless Slash Plugin
In the old Milkdown versions. The slash plugin can be used to display a list of commands when users type `/` in the editor.
It provides a way to insert nodes and commands into the editor, and it's really easy to use.
![legacy slash plugin](/blogs/understanding-headless-slash-plugin/legacy-slash-plugin.png)
However, it's hard to extend the slash plugin to support more commands, or if you want to change the UI of the slash plugin, you have to rewrite the whole plugin.
But, write a new plugin is always a hard work. You have to understand a lot of context and APIs of both ProseMirror and Milkdown.
## User Story
So, why don't we provide the slash plugin as a headless plugin?
In most cases, developers just want to make sure that when users type a special character, a dropdown menu will be displayed.
But the trigger character and the UI of the dropdown menu are different in different cases.
For example:
- When user type `/`, the menu contains a list of **commands**.
- When user type `:`, the menu contains a list of **emoji**.
- When user type `@`, the menu contains a list of **users**.
That's the story behind the headless slash plugin. We provide the plugin to solve a single problem: **display a dropdown menu when users input satisfy a condition**.
## How to use
In the new slash plugin, you'll need to control when to display the dropdown menu by yourself.
And you'll also need to provide the UI of the dropdown menu.
So, you'll need to create a `SlashProvider` instance.
```ts
import { slashPlugin, SlashProvider } from "@milkdown/plugin-slash";
const slashProvider = new SlashProvider({
content: YourDropdownUI,
shouldShow(this: SlashProvider, view: EditorView) {
const currentText = this.getContent(view);
if (currentText === "") {
return false;
}
// Display the menu if the last character is `/`.
if (currentText.endsWith("/")) {
return true;
}
return false;
},
});
```
Then, you can use the slash provider in your plugin view.
```ts
import { EditorState } from "@milkdown/prose/state";
import { EditorView, PluginView } from "@milkdown/prose/view";
function yourSlashView(): PluginView {
return {
update: (view: EditorView, prevState: EditorState) => {
slashProvider.update(view, prevState);
},
destroy: () => {
slashProvider.destroy();
},
};
}
```
Last, you'll need to add the slash plugin to your editor.
```ts
import { Editor } from "@milkdown/core";
import { slashFactory } from "@milkdown/plugin-slash";
const slash = slashFactory("my-slash");
Editor.make()
.config((ctx) => {
ctx.set(slash.key, {
view: slashPluginView,
});
})
.use(slash)
.create();
```
## Use with Prosemirror Adapter
If you're using milkdown with UI frameworks like React,
I recommend you to use the [Prosemirror Adapter](https://github.com/Saul-Mirone/prosemirror-adapter).
It can help you build prosemirror UI components with your favorite UI framework.
For example, if you're using React:
```tsx
import { SlashProvider } from "@milkdown/plugin-slash";
import { useInstance } from "@milkdown/react";
import { usePluginViewContext } from "@prosemirror-adapter/react";
export const DropdownMenu = () => {
const { view, prevState } = usePluginViewContext();
const slashProvider = useRef<SlashProvider>();
const divRef = useRef<HTMLDivElement>(null);
const [loading] = useInstance();
useEffect(() => {
if (!ref.current || loading) return;
slashProvider.current ??= new SlashProvider({
content: divRef.current,
// ...
});
return () => {
slashProvider.current?.destroy();
slashProvider.current = undefined;
};
}, [loading, root, setOpened, setSearch, setSelected]);
useEffect(() => {
slashProvider.current?.update(view, prevState);
});
// Add a wrapper `div` to hide the dropdown menu when initializing.
return (
<div className="hidden">
<div role="tooltip" ref={divRef}>
<h1>Hi! I'm a dropdown menu.</h1>
</div>
</div>
);
};
```
And in your editor component:
```ts
import { usePluginViewFactory } from "@prosemirror-adapter/react";
export const YourEditor = () => {
const pluginViewFactory = usePluginViewFactory();
useEditor((editor) => {
return Editor.make()
.config((ctx) => {
ctx.set(slash.key, {
view: pluginViewFactory({
component: DopdownMenu,
}),
});
})
.use(slash);
});
// ...
};
```
## Real World Example
In [milkdown playground](/playground), you can type `/` to display a dropdown menu.
![command dropdown](/blogs/understanding-headless-slash-plugin/command-dropdown.png)
You can also type `:(\S)+` (for example: `:mil`) to display a list of emojis.
![emoji dropdown](/blogs/understanding-headless-slash-plugin/emoji-dropdown.png)
You can find the source code of them in [Milkdown website](https://github.com/Milkdown/website).
I hope you enjoy the new slash plugin.
milkdown-docs/guide/architecture-overview.md
+199
View File
@@ -0,0 +1,199 @@
# Architecture Overview
Milkdown is built with a modular, layered architecture that provides flexibility and extensibility. This document explains the core architectural concepts and how they work together.
![0.75](/guide/milkdown-architecture.png "Milkdown Architecture")
## Core Architecture Layers
Milkdown's architecture is built upon four distinct layers, each providing specific functionality and extensibility:
### 🥛 Core Layer
The foundation of Milkdown that provides:
- Plugin loading and management system
- Core editor concepts and interfaces
- Base document model integration
- Essential utilities and helpers
### 🧇 Plugin Layer
A comprehensive collection of modular plugins that extend the editor's functionality:
- Syntax plugins (Markdown parsing, GFM, etc.)
- UI plugins (toolbar, menu, etc.)
- Feature plugins (image upload, table, etc.)
- Utility plugins (history, clipboard, etc.)
### 🍮 Component Layer
Headless UI components that serve as building blocks:
- Toolbar components
- Slash menu components
- Table components
### 🍰 Editor Layer
Ready-to-use, user-friendly editors:
- Crepe editor
- Custom editor implementations
## Architecture Benefits
This layered approach provides several key benefits:
1. **Modularity**: Each layer can be used independently
2. **Flexibility**: Mix and match components as needed
3. **Extensibility**: Create custom implementations at any layer
4. **Maintainability**: Clear separation of concerns
5. **Reusability**: Components can be shared across implementations
## Markdown Transformation
![0.75](/guide/transformer.png "Transformer")
Milkdown's transformation system handles the conversion between Markdown and the editor's internal document model:
### Parsing Process
1. Markdown text → Remark AST
2. Remark AST → ProseMirror Schema
3. Schema → ProseMirror Document
### Serialization Process
1. ProseMirror Document → ProseMirror Schema
2. Schema → Remark AST
3. Remark AST → Markdown text
This transformation system ensures:
- Accurate Markdown parsing
- Consistent document structure
- Reliable serialization
- Extensible transformation pipeline
## Context System
The Context System is a powerful state management and dependency coordination system that enables plugins to work together seamlessly.
![1.00](/guide/plugin-sequence.png "Plugin Sequence")
### Core Concepts
#### 1. Context (Ctx)
The main interface for plugins to interact with the system:
```typescript
interface Ctx {
get: <T>(slice: Slice<T>) => T;
set: <T>(slice: Slice<T>, value: T) => void;
wait: (timer: Timer) => Promise<void>;
done: (timer: Timer) => void;
inject: <T>(slice: Slice<T>, value: T) => void;
remove: <T>(slice: Slice<T>) => void;
}
```
#### 2. Slices
State containers that can be shared between plugins:
```typescript
// Create a slice with initial value and name
const themeSlice = createSlice("light", "theme");
// Use in a plugin
const themePlugin: MilkdownPlugin = (ctx) => {
return () => {
// Read current theme
const theme = ctx.get(themeSlice);
// Update theme
ctx.set(themeSlice, "dark");
// React to theme changes
ctx.watch(themeSlice, (newTheme) => {
// Handle theme change
});
};
};
```
#### 3. Timers
Dependency management system for plugin coordination:
```typescript
// Define a timer
const dataReady = createTimer("DataReady");
// Use in a plugin
const dataPlugin: MilkdownPlugin = (ctx) => {
ctx.record(dataReady);
return async () => {
// Wait for dependencies
await ctx.wait(SchemaReady);
// Do work
// ...
// Mark as ready
ctx.done(dataReady);
};
};
```
### Plugin Lifecycle
Plugins follow a consistent lifecycle pattern:
```typescript
const examplePlugin: MilkdownPlugin = (ctx) => {
// 1. Setup Phase
ctx.inject(mySlice, defaultValue);
ctx.record(myTimer);
return async () => {
// 2. Initialization Phase
await ctx.wait(RequiredTimer);
// 3. Runtime Phase
const value = ctx.get(mySlice);
ctx.set(mySlice, newValue);
// 4. Cleanup Phase
return () => {
ctx.remove(mySlice);
};
};
};
```
### Best Practices
1. **State Management**
- Use slices for shared state
- Keep state minimal and focused
- Watch for state changes when needed
2. **Dependency Management**
- Use timers for coordination
- Wait for required dependencies
- Mark completion appropriately
3. **Plugin Organization**
- Follow the lifecycle pattern
- Clean up resources properly
- Document dependencies clearly
## Next Steps
- Start to [use Crepe editor](/docs/guide/using-crepe)
- Learn more about [writing plugins](/docs/plugin/plugins-101)
- Explore [available plugins](/docs/plugin/using-plugins)
milkdown-docs/guide/code-highlighting.md
+160
View File
@@ -0,0 +1,160 @@
# Code Highlighting
Milkdown supports syntax highlighting for code blocks through the `@milkdown/plugin-highlight` plugin. This plugin provides several options for highlighting code with different syntax highlighters.
## Installation
```bash
npm install @milkdown/plugin-highlight
```
## Basic Usage
The highlight plugin requires a parser to be configured. Here's a basic example using the Shiki parser:
```typescript
import { Editor } from "@milkdown/core";
import { commonmark } from "@milkdown/preset-commonmark";
import { highlight, highlightPluginConfig } from "@milkdown/plugin-highlight";
import { createParser } from "@milkdown/plugin-highlight/shiki";
const editor = Editor.make()
.config(async (ctx) => {
const parser = await createParser({
theme: "github-light",
langs: ["javascript", "typescript", "python", "html", "css"],
});
ctx.set(highlightPluginConfig.key, { parser });
})
.use(commonmark)
.use(highlight)
.create();
```
## Available Parsers
The plugin supports multiple syntax highlighting libraries:
### Shiki
Provides high-quality syntax highlighting with VS Code themes. Learn more at [Shiki](https://shiki.style/):
```typescript
import { createParser } from "@milkdown/plugin-highlight/shiki";
const parser = await createParser({
theme: "github-light",
langs: ["javascript", "typescript", "python"],
});
ctx.set(highlightPluginConfig.key, { parser });
```
### Lowlight
Based on [highlight.js](https://highlightjs.org/), supports many languages:
```typescript
import { createParser } from "@milkdown/plugin-highlight/lowlight";
import { common } from "lowlight";
const parser = createParser({ common });
ctx.set(highlightPluginConfig.key, { parser });
```
Learn more about Lowlight at [lowlight](https://github.com/wooorm/lowlight).
### Refractor
Based on [Prism.js](https://prismjs.com/):
```typescript
import { createParser } from "@milkdown/plugin-highlight/refractor";
import { refractor } from "refractor";
const parser = createParser({ refractor });
ctx.set(highlightPluginConfig.key, { parser });
```
Learn more about Refractor at [refractor](https://github.com/wooorm/refractor).
### Sugar High
A lightweight and fast syntax highlighter. Learn more at [Sugar High](https://github.com/huozhi/sugar-high):
```typescript
import { createParser } from "@milkdown/plugin-highlight/sugar-high";
const parser = createParser();
ctx.set(highlightPluginConfig.key, { parser });
```
## Styling
The highlighted code will have CSS classes applied based on the chosen parser. You'll need to include appropriate CSS to style the highlighted tokens.
### Sugar High Classes
Sugar High uses classes like:
- `sh__token--identifier`
- `sh__token--string`
- `sh__token--keyword`
- `sh__token--sign`
- `sh__token--property`
You can style these using CSS variables:
```css
.sh__token--identifier {
color: var(--sh-identifier);
}
.sh__token--string {
color: var(--sh-string);
}
.sh__token--keyword {
color: var(--sh-keyword);
}
```
### Other Parsers
For Lowlight, Refractor, and Shiki, refer to their respective documentation for styling information.
## Example
Here's a complete example with Shiki:
```typescript
import { Editor } from "@milkdown/core";
import { commonmark } from "@milkdown/preset-commonmark";
import { highlight, highlightPluginConfig } from "@milkdown/plugin-highlight";
import { createParser } from "@milkdown/plugin-highlight/shiki";
async function createHighlightedEditor() {
const parser = await createParser({
theme: "github-light",
langs: ["javascript", "typescript", "python", "html", "css", "json"],
});
const editor = Editor.make()
.config((ctx) => {
ctx.set(highlightPluginConfig.key, { parser });
})
.use(commonmark)
.use(highlight);
await editor.create();
return editor;
}
```
With this setup, your code blocks will be automatically highlighted:
````markdown
```javascript
console.log("Hello, world!");
const greeting = (name) => `Hello, ${name}!`;
```
````
The code above will render with syntax highlighting applied to keywords, strings, and other language constructs.
milkdown-docs/guide/collaborative-editing.md
+122
View File
@@ -0,0 +1,122 @@
# Collaborative Editing
Milkdown supports collaborative editing powered by [Y.js](https://docs.yjs.dev/).
We provide the [@milkdown/plugin-collab](/docs/api/plugin-collab) plugin to help you use milkdown with yjs easily.
This plugin includes basic collaborative editing features like:
- Sync between clients.
- Remote cursor support.
- Undo/Redo support.
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/vanilla-collab"}
## Configure Plugin
First you need to install the plugin and yjs through npm:
```bash
npm install @milkdown/plugin-collab
npm install yjs y-protocols y-prosemirror
```
And you also need to choose a [provider for yjs](https://docs.yjs.dev/ecosystem/connection-provider), here we use [y-websocket](https://docs.yjs.dev/ecosystem/connection-provider/y-websocket) as an example.
After the installation, you can configure your editor:
```typescript
// ...import other plugins
import { collab, collabServiceCtx } from "@milkdown/plugin-collab";
async function setup() {
const editor = await Editor.make()
.config(nord)
.use(commonmark)
.use(collab)
.create();
const doc = new Doc();
const wsProvider = new WebsocketProvider("<YOUR_WS_HOST>", "milkdown", doc);
editor.action((ctx) => {
const collabService = ctx.get(collabServiceCtx);
collabService
// bind doc and awareness
.bindDoc(doc)
.setAwareness(wsProvider.awareness)
// connect yjs with milkdown
.connect();
});
}
```
Now your editor can support collaborative editing. Isn't it easy?
## Connect and Disconnect
You may want to control the connect status of the editor manually.
```typescript
editor.action((ctx) => {
const collabService = ctx.get(collabServiceCtx);
const doc = new Doc();
const wsProvider = new WebsocketProvider("<YOUR_WS_HOST>", "milkdown", doc);
collabService.bindDoc(doc).setAwareness(wsProvider.awareness);
document.getElementById("connect").onclick = () => {
wsProvider.connect();
collabService.connect();
};
document.getElementById("disconnect").onclick = () => {
wsProvider.disconnect();
collabService.disconnect();
};
});
```
## Default Template
By default, the editor will show a empty document. You may want to use a template to show a document.
```typescript
const template = `# Heading`;
editor.action((ctx) => {
const collabService = ctx.get(collabServiceCtx);
const doc = new Doc();
const wsProvider = new WebsocketProvider("<YOUR_WS_HOST>", "milkdown", doc);
collabService.bindDoc(doc).setAwareness(wsProvider.awareness);
wsProvider.once("synced", async (isSynced: boolean) => {
if (isSynced) {
collabService
// apply your template
.applyTemplate(markdown)
// don't forget connect
.connect();
}
});
});
```
Keep in mind that applying a template multiple times may cause some unexpected behavior, such as duplicate content.
Because of this you need to make sure **the template is applied only once**.
By default, the template will only be applied if _document get from remote server is empty_.
You can control this behavior through passing second parameter to `applyTemplate`:
```typescript
collabService
.applyTemplate(markdown, (remoteNode, templateNode) => {
// return true to apply template
})
// don't forget connect
.connect();
```
Here the nodes we get are [prosemirror nodes](https://prosemirror.net/docs/ref/#model.Node).
You should return `true` if the template should be applied, and `false` if not.
milkdown-docs/guide/commands.md
+250
View File
@@ -0,0 +1,250 @@
# Commands
Commands are a powerful way to programmatically modify editor content. The command system in Milkdown provides a flexible and type-safe way to create, manage, and execute commands.
## Command Manager
---
The command manager is the central place for handling all editor commands. It provides methods to:
- Register new commands
- Execute commands
- Chain multiple commands together
- Handle command arguments
## Run a Command
---
You can execute commands using the command manager through the editor's action system:
```typescript
import { Editor, commandsCtx } from "@milkdown/kit/core";
import {
commonmark,
toggleEmphasisCommand,
} from "@milkdown/kit/preset/commonmark";
async function setup() {
const editor = await Editor.make().use(commonmark).create();
const toggleItalic = () =>
editor.action((ctx) => {
// get command manager
const commandManager = ctx.get(commandsCtx);
// call command
commandManager.call(toggleEmphasisCommand.key);
});
// get markdown string:
$button.onClick = toggleItalic;
}
```
## Command Chaining
---
You can chain multiple commands together using the command manager's `chain` method. Commands in the chain will be executed in order until one of them returns `true`:
```typescript
import { Editor, commandsCtx } from "@milkdown/kit/core";
import {
commonmark,
toggleEmphasisCommand,
toggleStrongCommand,
} from "@milkdown/kit/preset/commonmark";
const editor = await Editor.make().use(commonmark).create();
editor.action((ctx) => {
const commandManager = ctx.get(commandsCtx);
// Chain multiple commands
commandManager
.chain()
.pipe(toggleEmphasisCommand.key) // Try to toggle emphasis
.pipe(toggleStrongCommand.key) // If emphasis fails, try to toggle strong
.run();
});
```
You can also mix inline commands with registered commands:
```typescript
import { chainCommands } from "@milkdown/prose/commands";
editor.action((ctx) => {
const commandManager = ctx.get(commandsCtx);
commandManager
.chain()
.inline(someInlineCommand) // Add an inline command
.pipe(toggleEmphasisCommand.key) // Add a registered command
.run();
});
```
## Create a Command
---
To create a command, use the `$command` utility from `@milkdown/utils`. Commands should be [prosemirror commands](https://prosemirror.net/docs/guide/#commands).
### Example: Command without argument
```typescript
import { Editor } from "@milkdown/kit/core";
import { blockquoteSchema } from "@milkdown/kit/preset/commonmark";
import { wrapIn } from "@milkdown/kit/prose/commands";
import { $command, callCommand } from "@milkdown/kit/utils";
const wrapInBlockquoteCommand = $command(
"WrapInBlockquote",
(ctx) => () => wrapIn(blockquoteSchema.type(ctx)),
);
// register the command when creating the editor
const editor = Editor().make().use(wrapInBlockquoteCommand).create();
// call command
editor.action(callCommand(wrapInBlockquoteCommand.key));
```
### Example: Command with argument
Commands can accept arguments of any type:
```typescript
import { headingSchema } from "@milkdown/kit/preset/commonmark";
import { setBlockType } from "@milkdown/kit/prose/commands";
import { $command, callCommand } from "@milkdown/kit/utils";
// use number as the type of argument
export const WrapInHeading = createCmdKey<number>();
const wrapInHeadingCommand = $command(
"WrapInHeading",
(ctx) =>
(level = 1) =>
setBlockType(headingSchema.type(ctx), { level }),
);
// call command
editor.action(callCommand(wrapInHeadingCommand.key)); // turn to h1 by default
editor.action(callCommand(wrapInHeadingCommand.key, 2)); // turn to h2
```
### Example: Command with Multiple Arguments
```typescript
interface TableConfig {
rows: number;
cols: number;
withHeader: boolean;
}
const insertTableCommand = $command(
"InsertTable",
(ctx) => (config: TableConfig) => {
// Implementation for inserting a table
return (state, dispatch) => {
// ... table insertion logic
return true;
};
},
);
// Usage
editor.action(
callCommand(insertTableCommand.key, {
rows: 3,
cols: 3,
withHeader: true,
}),
);
```
## Best Practices
---
1. **Command Naming**
- Use clear, descriptive names
- Follow the pattern: `[Action][Target]Command`
- Example: `toggleEmphasisCommand`, `insertTableCommand`
2. **Command Organization**
- Group related commands together
- Use namespaces for command keys
- Keep commands focused and single-purpose
3. **Error Handling**
- Always check if the command can be executed
- Return `false` if the command cannot be executed
- Handle edge cases gracefully
4. **Performance**
- Keep commands lightweight
- Avoid unnecessary state updates
- Use command chaining for complex operations
5. **Type Safety**
- Use TypeScript for command arguments
- Define clear interfaces for command payloads
- Use generics for type-safe command keys
## Common Patterns
---
### Toggle Commands
```typescript
const toggleCommand = $command(
"ToggleFeature",
(ctx) => () => (state, dispatch) => {
const isActive = checkIfActive(state);
return isActive
? removeFeature(state, dispatch)
: addFeature(state, dispatch);
},
);
```
### Insert Commands
```typescript
const insertCommand = $command(
"InsertContent",
(ctx) => (content: string) => (state, dispatch) => {
const { selection } = state;
if (!selection) return false;
const tr = state.tr.insertText(content, selection.from);
dispatch?.(tr);
return true;
},
);
```
### Transform Commands
```typescript
const transformCommand = $command(
"TransformContent",
(ctx) => (transform: (node: ProseNode) => ProseNode) => (state, dispatch) => {
const { selection } = state;
if (!selection) return false;
const tr = state.tr.replaceWith(
selection.from,
selection.to,
transform(state.doc.nodeAt(selection.from)!),
);
dispatch?.(tr);
return true;
},
);
```
milkdown-docs/guide/faq.md
+39
View File
@@ -0,0 +1,39 @@
# FAQ
This page lists answers of FAQ.
---
### How can I change contents programmatically?
You should use `editor.action` to change the contents.
We provide two macros for that allow you to change content in milkdown, `insert` and `replaceAll`.
```typescript
import { insert, replaceAll } from "@milkdown/kit/utils";
const editor = await Editor.make()
// .use(<All Your Plugins>)
.create();
editor.action(insert("# New Heading"));
editor.action(replaceAll("# New Document"));
```
---
### How to configure remark?
```typescript
import { remarkStringifyOptionsCtx } from "@milkdown/kit/core";
editor.config((ctx) => {
ctx.set(remarkStringifyOptionsCtx, {
// some options, for example:
bullet: "*",
fences: true,
incrementListMarker: false,
});
});
```
milkdown-docs/guide/getting-started.md
+165
View File
@@ -0,0 +1,165 @@
# Getting Started with Milkdown
Milkdown is a powerful WYSIWYG markdown editor that combines the simplicity of markdown with the flexibility of a modern editor. It's designed to be lightweight yet extensible, making it perfect for both simple and complex editing needs.
## Quick Start
The fastest way to get started is using `@milkdown/crepe`:
```bash
npm install @milkdown/crepe
```
```typescript
import { Crepe } from "@milkdown/crepe";
import "@milkdown/crepe/theme/common/style.css";
import "@milkdown/crepe/theme/frame.css";
const crepe = new Crepe({
root: "#app",
defaultValue: "Hello, Milkdown!",
});
crepe.create();
```
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/editor-crepe"}
## Core Concepts
Milkdown consists of two main parts:
1. **Core Package** (`@milkdown/core`)
- Plugin loader
- Internal plugins
2. **Additional Plugins**
- Syntax support
- Commands
- UI components
- Custom features
This modular architecture allows you to enable or disable features as needed, from basic markdown support to advanced features like tables, LaTeX equations, and collaborative editing.
## Key Features
- 📝 **WYSIWYG Markdown** - Write markdown in an elegant way
- 🎨 **Themable** - Create your own theme and publish it as an npm package
- 🎮 **Hackable** - Create your own plugin to support your awesome idea
- 🦾 **Reliable** - Built on top of [prosemirror](https://prosemirror.net/) and [remark](https://github.com/remarkjs/remark)
- ⚡ **Slash & Tooltip** - Write faster than ever, enabled by a plugin
- 🧮 **Math** - LaTeX math equations support via math plugin
- 📊 **Table** - Table support with fluent ui, via table plugin
- 🍻 **Collaborate** - Shared editing support with [yjs](https://docs.yjs.dev/)
- 💾 **Clipboard** - Support copy and paste markdown, via clipboard plugin
- 👍 **Emoji** - Support emoji shortcut and picker, via emoji plugin
## Tech Stack
Milkdown is built on top of these powerful libraries:
- [Prosemirror](https://prosemirror.net/) - A toolkit for building rich-text editors on the web
- [Remark](https://github.com/remarkjs/remark) - Markdown parser done right
- [TypeScript](https://www.typescriptlang.org/) - For type safety and better developer experience
## Creating Your First Editor
Milkdown provides two distinct approaches to create an editor, each suited for different needs:
### 1. 🍼 Using `@milkdown/kit` (Build from Scratch)
This approach gives you complete control over your editor. Use this if you want to:
- Build a custom editor from the ground up
- Have full control over which features to include
- Create a highly customized editing experience
- Integrate with specific frameworks or requirements
First, install the required packages:
```bash
npm install @milkdown/kit
```
Create a basic editor with commonmark syntax:
```typescript
import { Editor } from "@milkdown/kit/core";
import { commonmark } from "@milkdown/kit/preset/commonmark";
// This is the must have css for prosemirror
import "@milkdown/kit/prose/view/style/prosemirror.css";
Editor.make().use(commonmark).create();
```
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/vanilla-commonmark"}
Add undo & redo support:
```typescript
import { Editor } from "@milkdown/kit/core";
import { history } from "@milkdown/kit/plugin/history";
import { commonmark } from "@milkdown/kit/preset/commonmark";
import { nord } from "@milkdown/theme-nord";
import "@milkdown/theme-nord/style.css";
const milkdown = Editor.make()
.config(nord)
.use(commonmark)
.use(history)
.create()
.then(() => {
console.log("Editor created");
});
// To destroy the editor
milkdown.destroy();
```
> **Note**: `<Mod>` is `<Cmd>` for macOS and `<Ctrl>` for other platforms.
### 2. 🥞 Using `@milkdown/crepe` (Ready to Use)
This is the quickest way to get started with a fully-featured editor. Use this if you want to:
- Get up and running quickly
- Have a well-designed editor out of the box
- Focus on content rather than configuration
- Have a production-ready solution with minimal setup
```bash
npm install @milkdown/crepe
```
```typescript
import { Crepe } from "@milkdown/crepe";
import "@milkdown/crepe/theme/common/style.css";
/**
* Available themes:
* frame, classic, nord
* frame-dark, classic-dark, nord-dark
*/
import "@milkdown/crepe/theme/frame.css";
const crepe = new Crepe({
root: "#app",
defaultValue: "Hello, Milkdown!",
});
crepe.create().then(() => {
console.log("Editor created");
});
// To destroy the editor
crepe.destroy();
```
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/editor-crepe"}
## Next Steps
- Learn more about [overview](/guide/architecture-overview)
- Explore [available plugins](/plugins/using-plugins)
- Check out [theming](/guide/theming)
> 🍼 Fun fact: This documentation is rendered by Milkdown itself!
milkdown-docs/guide/interacting-with-editor.md
+398
View File
@@ -0,0 +1,398 @@
# Interacting with Editor
This guide covers the essential ways to interact with the Milkdown editor, including initialization, content management, and editor lifecycle.
## Using Crepe Editor
---
Crepe is a high-level wrapper around Milkdown that provides a simpler API for common editor operations. Here's how to use it:
```typescript
import { Crepe } from "@milkdown/crepe";
// Create a new editor instance
const editor = new Crepe({
// Optional: specify root element (DOM node or selector)
root: "#editor",
// Optional: set default content, supports markdown, json and dom.
defaultValue: "# Hello Crepe!",
});
// Create the editor
await editor.create();
// Get markdown content
const markdown = editor.getMarkdown();
// Set readonly mode
editor.setReadonly(true);
// Register event listeners
editor.on((listener) => {
listener.markdownUpdated((ctx, markdown) => {
console.log("Content updated:", markdown);
});
listener.focus((ctx) => {
console.log("Editor focused");
});
listener.blur((ctx) => {
console.log("Editor blurred");
});
listener.selectionUpdated((ctx, selection, prevSelection) => {
console.log("Selection updated:", selection);
});
listener.updated((ctx, doc, prevDoc) => {
console.log("Document updated:", doc);
});
});
// Destroy the editor when done
await editor.destroy();
```
## Register to DOM
---
By default, milkdown will create editor on the `document.body`. Alternatively, you can also point out which dom node you want it to load into:
```typescript
import { rootCtx } from "@milkdown/kit/core";
Editor.make().config((ctx) => {
ctx.set(rootCtx, document.querySelector("#editor"));
});
```
It's also possible to just pass a selector to `rootCtx`:
> The selector will be passed to `document.querySelector` to get the dom.
```typescript
import { rootCtx } from "@milkdown/kit/core";
Editor.make().config((ctx) => {
ctx.set(rootCtx, "#editor");
});
```
## Setting Default Value
---
We support three types of default values:
- Markdown strings
- HTML DOM
- Prosemirror documentation JSON
### Markdown
You can set a markdown string as the default value of the editor.
```typescript
import { defaultValueCtx } from "@milkdown/kit/core";
const defaultValue = "# Hello milkdown";
Editor.make().config((ctx) => {
ctx.set(defaultValueCtx, defaultValue);
});
```
### Dom
You can also use HTML as default value.
Let's assume that we have the following html snippets:
```html
<div id="pre">
<h1>Hello milkdown!</h1>
</div>
```
Then we can use it as a defaultValue with a `type` specification:
```typescript
import { defaultValueCtx } from "@milkdown/kit/core";
const defaultValue = {
type: "html",
dom: document.querySelector("#pre"),
};
Editor.make().config((ctx) => {
ctx.set(defaultValueCtx, defaultValue);
});
```
### JSON
We can also use a JSON object as a default value.
This JSON object can be obtained by a listener through the [listener-plugin](https://www.npmjs.com/package/@milkdown/plugin-listener), for example:
```typescript
import { listener, listenerCtx } from "@milkdown/kit/plugin/listener";
let jsonOutput;
Editor.make()
.config((ctx) => {
ctx.get(listenerCtx).updated((ctx, doc, prevDoc) => {
jsonOutput = doc.toJSON();
});
})
.use(listener);
```
Then we can use this `jsonOutput` as default Value:
```typescript
import { defaultValueCtx } from "@milkdown/kit/core";
const defaultValue = {
type: "json",
value: jsonOutput,
};
Editor.make().config((ctx) => {
ctx.set(defaultValueCtx, defaultValue);
});
```
## Inspecting Editor Status
---
You can inspect the editor's status through the `status` property.
```typescript
import { Editor, EditorStatus } from "@milkdown/kit/core";
const editor = Editor.make().use(/* some plugins */);
assert(editor.status === EditorStatus.Idle);
editor.create().then(() => {
assert(editor.status === EditorStatus.Created);
});
assert(editor.status === EditorStatus.OnCreate);
editor.destroy().then(() => {
assert(editor.status === EditorStatus.Destroyed);
});
assert(editor.status === EditorStatus.OnDestroyed);
```
You can also listen to the status changes:
```typescript
import { Editor, EditorStatus } from "@milkdown/kit/core";
const editor = Editor.make().use(/* some plugins */);
editor.onStatusChange((status: EditorStatus) => {
console.log(status);
});
```
### Status Lifecycle
1. `Idle`: Initial state
2. `OnCreate`: During creation
3. `Created`: Successfully created
4. `OnDestroyed`: During destruction
5. `Destroyed`: Successfully destroyed
## Adding Listeners
---
As mentioned above, you can add a listener to the editor, in order to get its value when needed.
You can add as many listeners as you want, all the listeners will be triggered at once.
### Markdown Listener
You can add markdown listener to get the editor's contents as a markdown string.
> ⚠️ Markdown listener will influence the performance for large documents, please use it carefully.
> If you have a large document, I suggest you to only `parse` and `serialize` the document when needed.
```typescript
import { listener, listenerCtx } from "@milkdown/kit/plugin/listener";
let output = "";
Editor.make()
.config((ctx) => {
ctx.get(listenerCtx).markdownUpdated((ctx, markdown, prevMarkdown) => {
output = markdown;
});
})
.use(listener);
```
### Doc Listener
You can also listen to the [raw prosemirror document node](https://prosemirror.net/docs/ref/#model.Node), and do things you want from there.
```typescript
import { listener, listenerCtx } from "@milkdown/kit/plugin/listener";
let jsonOutput;
Editor.make()
.config((ctx) => {
ctx.get(listenerCtx).updated((ctx, doc, prevDoc) => {
jsonOutput = doc.toJSON();
});
})
.use(listener);
```
### Selection Listener
You can track changes to the editor's selection using the `selectionUpdated` event. This is useful for implementing features like:
- Custom toolbars that update based on selection
- Context menus
- Selection-based formatting controls
```typescript
import { listener, listenerCtx } from "@milkdown/kit/plugin/listener";
import { Selection, TextSelection } from "@milkdown/prose/state";
Editor.make()
.config((ctx) => {
ctx.get(listenerCtx).selectionUpdated((ctx, selection, prevSelection) => {
if (selection instanceof TextSelection) {
// Get selection range
const { from, to } = selection;
// Example: Update toolbar based on selection
updateToolbar({
hasSelection: from !== to,
selectionStart: from,
selectionEnd: to,
});
}
});
})
.use(listener);
```
The selection listener will be triggered when the selection is changed.
So you don't need to compare them manually.
For more details about listeners, please check [Using Listeners](/docs/api/plugin-listener).
## Readonly Mode
---
You can set the editor to readonly mode by setting the `editable` property.
```typescript
import { editorViewOptionsCtx } from "@milkdown/kit/core";
let readonly = false;
const editable = () => !readonly;
Editor.make().config((ctx) => {
ctx.update(editorViewOptionsCtx, (prev) => ({
...prev,
editable,
}));
});
// set to readonly after 5 secs.
setTimeout(() => {
readonly = true;
}, 5000);
```
### Use Cases for Readonly Mode
- Preview mode
- Document review
- Print-friendly views
- Mobile device optimization
## Using Actions
---
You can use an action to get the context value in a running editor on demand.
For example, to get the markdown string by running an action:
```typescript
import { Editor, editorViewCtx, serializerCtx } from "@milkdown/kit/core";
async function playWithEditor() {
const editor = await Editor.make().use(commonmark).create();
const getMarkdown = () =>
editor.action((ctx) => {
const editorView = ctx.get(editorViewCtx);
const serializer = ctx.get(serializerCtx);
return serializer(editorView.state.doc);
});
// get markdown string:
getMarkdown();
}
```
We provide some macros out of the box, you can use them as actions:
```typescript
import { insert } from "@milkdown/kit/utils";
editor.action(insert("# Hello milkdown"));
```
### Common Actions
- Insert content
- Get current selection
- Apply formatting
- Execute commands
For more details about macros, please check [macros](/docs/guide/macros).
## Destroying
---
You can call `editor.destroy` to destroy an existing editor. You can create a new editor again with `editor.create`.
```typescript
await editor.destroy();
// Then create again
await editor.create();
```
If you just want to recreate the editor, you can use `editor.create`, it will **destroy the old editor and create a new one**.
```typescript
await editor.create();
// This equals to call `editor.destroy` and `editor.create` again.
await editor.create();
```
If you want to **clear the plugins and configs for the editor** when calling `editor.destroy`, you can pass `true` to `editor.destroy`.
```typescript
await editor.destroy(true);
```
milkdown-docs/guide/keyboard-shortcuts.md
+252
View File
@@ -0,0 +1,252 @@
# Keyboard Shortcuts
Keyboard shortcuts are a crucial part of the editor's user experience. Milkdown provides a flexible system for configuring keyboard shortcuts through presets and plugins.
## Default Shortcuts
---
Milkdown comes with a set of default keyboard shortcuts from both presets and plugins. Here's a comprehensive list of all internal shortcuts:
> #### 💡 Note
>
> `Mod` represents the platform-specific modifier key:
>
> - Windows/Linux: `Ctrl`
> - macOS: `Command`
### Commonmark Preset Shortcuts
#### Headings
| Shortcut | Description |
| -------------------- | ----------------------- |
| `Mod-Alt-1` | Turn block into h1 |
| `Mod-Alt-2` | Turn block into h2 |
| `Mod-Alt-3` | Turn block into h3 |
| `Mod-Alt-4` | Turn block into h4 |
| `Mod-Alt-5` | Turn block into h5 |
| `Mod-Alt-6` | Turn block into h6 |
| `Delete`/`Backspace` | Downgrade heading level |
#### Block Elements
| Shortcut | Description |
| ------------- | ---------------------------- |
| `Mod-Shift-b` | Wrap selection in blockquote |
| `Mod-Shift-8` | Wrap in bullet list |
| `Mod-Shift-7` | Wrap in ordered list |
| `Mod-Shift-c` | Wrap in code block |
| `Shift-Enter` | Insert hard break |
| `Mod-Alt-0` | Wrap in paragraph |
#### Text Formatting
| Shortcut | Description |
| -------- | ------------------ |
| `Mod-b` | Toggle bold |
| `Mod-i` | Toggle italic |
| `Mod-e` | Toggle inline code |
### GFM Preset Shortcuts
#### Text Formatting
| Shortcut | Description |
| ----------- | -------------------- |
| `Mod-Alt-x` | Toggle strikethrough |
#### Tables
| Shortcut | Description |
| ------------------- | -------------------------------- |
| `Mod-]` | Move to next cell |
| `Mod-[` | Move to previous cell |
| `Mod-Enter`/`Enter` | Exit table and break if possible |
## Configuring Shortcuts
---
You can customize keyboard shortcuts by configuring the keymap in the editor setup:
```typescript
import { blockquoteKeymap, commonmark } from "@milkdown/kit/preset/commonmark";
Editor.make()
.config((ctx) => {
ctx.set(blockquoteKeymap.key, {
WrapInBlockquote: "Mod-Shift-b",
// or you may want to bind multiple keys:
WrapInBlockquote: ["Mod-Shift-b", "Mod-b"],
});
})
.use(commonmark);
```
## Defining Keymaps
---
Keymaps in Milkdown are defined using the `$useKeymap` utility. Here's how to define keymaps for different features:
### Heading Keymap Example
```typescript
import { $useKeymap } from "@milkdown/utils";
import { commandsCtx } from "@milkdown/core";
export const headingKeymap = $useKeymap("headingKeymap", {
TurnIntoH1: {
shortcuts: "Mod-Alt-1",
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => commands.call(wrapInHeadingCommand.key, 1);
},
},
TurnIntoH2: {
shortcuts: "Mod-Alt-2",
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => commands.call(wrapInHeadingCommand.key, 2);
},
},
// ... more heading levels
DowngradeHeading: {
shortcuts: ["Delete", "Backspace"],
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => commands.call(downgradeHeadingCommand.key);
},
},
});
```
### Strong (Bold) Keymap Example
```typescript
import { $useKeymap } from "@milkdown/utils";
import { commandsCtx } from "@milkdown/core";
export const strongKeymap = $useKeymap("strongKeymap", {
ToggleBold: {
shortcuts: ["Mod-b"],
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => commands.call(toggleStrongCommand.key);
},
},
});
```
### Keymap Structure
Each keymap definition follows this structure:
```typescript
$useKeymap('keymapName', {
CommandName: {
shortcuts: string | string[], // Single shortcut or array of shortcuts
priority?: number, // (Optional) Priority of the shortcut
command: (ctx) => () => { // Command to execute
const commands = ctx.get(commandsCtx);
return () => commands.call(commandKey, ...args);
},
},
});
```
## Creating Custom Shortcuts
---
If you need to add custom shortcuts, you can create a keymap plugin:
```typescript
import { $useKeymap } from "@milkdown/utils";
import { commandsCtx } from "@milkdown/core";
const customKeymap = $useKeymap("customKeymap", {
CustomCommand: {
shortcuts: "F1",
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => commands.call(someCommand.key);
},
},
});
// Usage
Editor.make().use(customKeymap).use(commonmark);
```
### Example: Custom Command with Shortcut
```typescript
import { $command, $useKeymap } from "@milkdown/utils";
import { commandsCtx } from "@milkdown/core";
// Create a custom command
const customCommand = $command("CustomCommand", (ctx) => () => {
return (state, dispatch) => {
// Command implementation
return true;
};
});
// Create a keymap
const customKeymap = $useKeymap("customKeymap", {
CustomCommand: {
shortcuts: ["F1", "Mod-F1"], // Multiple shortcuts
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => commands.call(customCommand.key);
},
},
});
// Usage
Editor.make().use(customCommand).use(customKeymap);
```
## Shortcut Priority
You can control the order in which shortcuts are handled by specifying a `priority` property. Shortcuts with higher priority values are handled before those with lower values. This is useful if you want your custom shortcut to override or take precedence over other shortcuts that use the same key combination.
When multiple shortcuts are registered for the same key, they are executed in order of priority. If a shortcut command returns `false`, the next shortcut with the same key will be tried. If it returns `true`, no further commands for that key will be run. This allows you to chain or override shortcut behaviors as needed.
- The default priority is **50**.
- Normal priority values should be between **1** and **100**.
- Use higher numbers to ensure your shortcut is registered before others with the same key.
#### Example: Using Priority
```typescript
import { $useKeymap } from "@milkdown/utils";
import { commandsCtx } from "@milkdown/core";
export const customKeymap = $useKeymap("customKeymap", {
CustomBold: {
shortcuts: "Mod-b",
priority: 100, // Highest in the normal range, so this runs first
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => {
// Custom bold logic
return true;
};
},
},
CustomAnotherBold: {
shortcuts: "Mod-b",
priority: 75, // Lower priority, will run only if CustomBold returns false
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => {
// Custom italic logic
return true;
};
},
},
});
```
milkdown-docs/guide/macros.md
+278
View File
@@ -0,0 +1,278 @@
# Macros
Macros are helper functions that provide a convenient way to interact with the editor. They take a payload (or nothing) as parameters and return a callback function that takes the `ctx` of milkdown as a parameter. When called with `ctx`, they apply the specified action to the editor.
## Usage
There are two main ways to use macros:
```typescript
import { insert } from "@milkdown/kit/utils";
import { listenerCtx } from "@milkdown/plugin-listener";
// Method 1: Using editor.action()
editor.action(insert("# Hello Macro"));
// Method 2: Using listener
editor.config((ctx) => {
ctx.get(listenerCtx).mounted(insert("# Default Title"));
});
```
## Available Macros
### Content Manipulation
#### `insert`
Inserts content at the current cursor position. The macro accepts two parameters:
- `markdown`: The markdown string to insert
- `inline`: Optional boolean flag (default: false) that determines how the content is inserted
```typescript
import { insert } from "@milkdown/kit/utils";
// Insert as block content (default)
editor.action(insert("# Hello World"));
// Insert as inline content
editor.action(insert("inline text", true));
```
The behavior differs based on the `inline` parameter:
- When `inline` is `false` (default):
- Replaces the current selection with the parsed markdown content
- Maintains the selection's open start/end positions
- Scrolls the view to show the inserted content
- When `inline` is `true`:
- Attempts to insert the content as inline text
- If the content is text-only, replaces the selection with a text node
- Otherwise, replaces the selection with the parsed content
#### `insertPos`
Inserts markdown at a given position. The macro accepts two parameters:
- `markdown`: The markdown string to insert
- `pos`: The position to insert the content at
```typescript
import { insertPos } from "@milkdown/kit/utils";
// Insert "Hello" at the beginning of the document
editor.action(insertPos("Hello", 0));
```
#### `replaceAll`
Replaces all content in the editor. The macro accepts two parameters:
- `markdown`: The markdown string to replace the current content with
- `flush`: Optional boolean flag (default: false) that determines how the replacement is performed
```typescript
import { replaceAll } from "@milkdown/kit/utils";
// Replace content without flushing state
editor.action(replaceAll("# New Content"));
// Replace content and flush editor state
editor.action(replaceAll("# New Content", true));
```
The behavior differs based on the `flush` parameter:
- When `flush` is `false` (default):
- Replaces the entire document content with the new markdown
- Maintains the current editor state
- More efficient for simple content replacements
- When `flush` is `true`:
- Creates a new editor state with the new content
- Reinitializes all plugins
- Useful when you need a completely fresh editor state
#### `replaceRange`
Replaces the content of the given range with a markdown string.
```typescript
import { replaceRange } from "@milkdown/kit/utils";
// Replace content from position 0 to 5 with "Hello"
editor.action(replaceRange("Hello", { from: 0, to: 5 }));
```
### Content Retrieval
#### `getMarkdown`
Gets the current content as markdown. If a range is provided, it will return the markdown for that range; otherwise, it will return the markdown for the entire document.
```typescript
import { getMarkdown } from "@milkdown/kit/utils";
// Get markdown for the entire document
const markdown = editor.action(getMarkdown());
// Get markdown for a specific range
const selectionMarkdown = editor.action(getMarkdown({ from: 0, to: 5 }));
```
#### `getHTML`
Gets the current content as HTML.
```typescript
import { getHTML } from "@milkdown/kit/utils";
const html = editor.action(getHTML());
```
### Editor State
#### `forceUpdate`
Forces the editor to update its state.
```typescript
import { forceUpdate } from "@milkdown/kit/utils";
editor.action(forceUpdate());
```
#### `setAttr`
Sets attributes for a node at a specific position. The macro accepts two parameters:
- `pos`: The position of the node to update
- `update`: A function that takes the previous attributes and returns the new attributes
```typescript
import { setAttr } from "@milkdown/kit/utils";
// Update node attributes at position 10
editor.action(
setAttr(10, (prevAttrs) => ({
...prevAttrs,
class: "custom-class",
})),
);
// Example: Update heading level
editor.action(
setAttr(10, (prevAttrs) => ({
...prevAttrs,
level: 2,
})),
);
```
The macro:
- Takes a specific position in the document
- Retrieves the node at that position
- Applies the update function to modify the node's attributes
- Dispatches the changes to update the editor state
Note: The position must be valid and contain a node, otherwise the operation will be ignored.
### Navigation
#### `outline`
Gets the outline of the document.
```typescript
import { outline } from "@milkdown/kit/utils";
const docOutline = editor.action(outline());
```
### Command Execution
#### `callCommand`
Calls a registered command with optional payload. The macro has two overloads:
Examples:
```typescript
import { callCommand } from "@milkdown/kit/utils";
import { wrapInHeadingCommand } from "@milkdown/plugin-heading";
// Using command key
editor.action(callCommand(wrapInHeadingCommand.key, 1));
// With complex payload
editor.action(
callCommand("CustomCommand", {
type: "heading",
level: 1,
content: "New Heading",
}),
);
```
The macro:
- Takes a command key
- Optionally accepts a payload parameter
- Returns a boolean indicating whether the command was successful
Note: The command must be registered in the editor's command context before it can be called.
### Utility Macros
#### `markdownToSlice`
Converts a markdown string to a [slice](https://prosemirror.net/docs/ref/#model.Slice). This is useful when you need to manipulate the content before inserting it into the editor.
```typescript
import { markdownToSlice } from "@milkdown/kit/utils";
const slice = editor.action(markdownToSlice("# Hello Slice"));
```
## Examples
### Adding Content
```typescript
import { insert } from "@milkdown/kit/utils";
import { listenerCtx } from "@milkdown/plugin-listener";
editor.config((ctx) => {
ctx.get(listenerCtx).mounted(insert("# Welcome\nStart editing..."));
});
```
### Saving Content
```typescript
import { getMarkdown } from "@milkdown/kit/utils";
editor.config((ctx) => {
ctx.get(listenerCtx).updated(() => {
const content = getMarkdown()(ctx);
localStorage.setItem("editor-content", content);
});
});
```
### Custom Command with Macro
```typescript
import { callCommand } from "@milkdown/kit/utils";
editor.action(
callCommand("customCommand", {
type: "heading",
level: 1,
content: "New Heading",
}),
);
```
For more details about each macro's parameters and return types, check the [API Reference](/docs/api/utils#macros).
milkdown-docs/guide/prosemirror-api.md
+38
View File
@@ -0,0 +1,38 @@
# Prosemirror API
Milkdown is built on top of prosemirror. Which means you can use the entire prosemirror API in Milkdown.
To access the prosemirror API, you can use the `@milkdown/prose` package. It re-exports all of the prosemirror API.
Using this package you can make sure that you are using the same version of prosemirror as Milkdown.
## Installation
To access a certain API in the `prosemirror-x` package, you need to import them from `@milkdown/kit/prose/x`.
For example:
```ts
// Originally in prosemirror-state
import { EditorState } from "@milkdown/kit/prose/state";
// Originally in prosemirror-view
import { EditorView } from "@milkdown/kit/prose/view";
```
## List of packages
The following is a list of all the re-exported prosemirror API.
- `@milkdown/kit/prose/changeset`
- `@milkdown/kit/prose/commands`
- `@milkdown/kit/prose/dropcursor`
- `@milkdown/kit/prose/gapcursor`
- `@milkdown/kit/prose/history`
- `@milkdown/kit/prose/inputrules`
- `@milkdown/kit/prose/keymap`
- `@milkdown/kit/prose/model`
- `@milkdown/kit/prose/schema-list`
- `@milkdown/kit/prose/state`
- `@milkdown/kit/prose/transform`
- `@milkdown/kit/prose/view`
- `@milkdown/kit/prose/tables`
You can find the documentation of the prosemirror API [here](https://prosemirror.net/docs/ref/).
milkdown-docs/guide/styling.md
+215
View File
@@ -0,0 +1,215 @@
# Styling Guide
Milkdown is a headless editor, which means it doesn't come with any default styles. This gives you complete control over the appearance of your editor. You can either use existing themes or create your own custom styling solution.
# Styling Crepe Theme
---
Crepe is a collection of themes for Milkdown that provides both light and dark variants. The theme structure is organized as follows:
```
theme/
├── common/ # Shared styles and utilities
├── crepe/ # Light theme variant
├── crepe-dark/ # Dark theme variant
├── frame/ # Frame theme (light)
├── frame-dark/ # Frame theme (dark)
├── nord/ # Nord theme (light)
└── nord-dark/ # Nord theme (dark)
```
## Using Crepe Theme
To use the Crepe theme in your project:
```ts
// Import base styles first
import "@milkdown/crepe/theme/common/style.css";
// Choose the theme you want to use
import "@milkdown/crepe/theme/crepe.css";
```
## Theme Variables
Crepe theme uses CSS variables for consistent styling. Here are all the available variables:
### Colors
```css
.milkdown {
/* Background Colors */
--crepe-color-background: #fffdfb; /* Main background color */
--crepe-color-surface: #fff8f4; /* Surface color for cards/panels */
--crepe-color-surface-low: #fff1e5; /* Lower surface color for depth */
/* Text Colors */
--crepe-color-on-background: #1f1b16; /* Text color on background */
--crepe-color-on-surface: #201b13; /* Text color on surface */
--crepe-color-on-surface-variant: #4f4539; /* Secondary text color */
/* Accent Colors */
--crepe-color-primary: #805610; /* Primary brand color */
--crepe-color-secondary: #fbdebc; /* Secondary accent color */
--crepe-color-on-secondary: #271904; /* Text color on secondary */
/* UI Colors */
--crepe-color-outline: #817567; /* Border/outline color */
--crepe-color-inverse: #362f27; /* Inverse color for contrast */
--crepe-color-on-inverse: #fcefe2; /* Text color on inverse */
--crepe-color-inline-code: #ba1a1a; /* Inline code color */
--crepe-color-error: #ba1a1a; /* Error state color */
/* Interactive Colors */
--crepe-color-hover: #f9ecdf; /* Hover state color */
--crepe-color-selected: #ede0d4; /* Selected state color */
--crepe-color-inline-area: #e4d8cc; /* Inline editing area color */
}
```
### Typography
```css
.milkdown {
/* Font Families */
--crepe-font-title: Georgia, Cambria, "Times New Roman", Times, serif;
--crepe-font-default: "Open Sans", Arial, Helvetica, sans-serif;
--crepe-font-code:
Fira Code, Menlo, Monaco, "Courier New", Courier, monospace;
}
```
### Shadows
```css
.milkdown {
/* Small Shadow */
--crepe-shadow-1:
0px 1px 3px 1px rgba(0, 0, 0, 0.15), 0px 1px 2px 0px rgba(0, 0, 0, 0.3);
/* Large Shadow */
--crepe-shadow-2:
0px 2px 6px 2px rgba(0, 0, 0, 0.15), 0px 1px 2px 0px rgba(0, 0, 0, 0.3);
}
```
## Customizing Crepe Theme
You can customize the Crepe theme by overriding its variables:
```css
/* custom-overrides.css */
.crepe .milkdown {
/* Override colors */
--crepe-color-primary: #your-primary-color;
--crepe-color-background: #your-background-color;
/* Override typography */
--crepe-font-default: "Your Font", sans-serif;
/* Override shadows */
--crepe-shadow-1: your-shadow-value;
}
```
# Styling Milkdown
---
## Basic Styling
The editor is rendered within a container that has the class `.milkdown`, and the editable content area is wrapped in a container with the class `.editor`. You can use these classes to scope your styles:
```css
/* Basic styling example */
.milkdown .editor {
max-width: 800px;
margin: 0 auto;
padding: 1rem;
}
.milkdown .editor p {
margin: 1rem 0;
line-height: 1.6;
}
```
## Node and Mark Classes
Milkdown provides default class names for each node and mark. Here are some common examples:
```css
/* Paragraph styling */
.milkdown .editor .paragraph {
margin: 1rem 0;
}
/* Heading styling */
.milkdown .editor .heading {
font-weight: 600;
margin: 1.5rem 0 1rem;
}
/* List styling */
.milkdown .editor .bullet-list {
padding-left: 1.5rem;
}
.milkdown .editor .ordered-list {
padding-left: 1.5rem;
}
```
## Custom Attributes
You can add custom attributes to nodes and marks, which is particularly useful when working with CSS frameworks like Tailwind CSS.
```typescript
import { Editor, editorViewOptionsCtx } from "@milkdown/kit/core";
import {
commonmark,
headingAttr,
paragraphAttr,
} from "@milkdown/kit/preset/commonmark";
Editor.make()
.config((ctx) => {
// Add attributes to the editor container
ctx.update(editorViewOptionsCtx, (prev) => ({
...prev,
attributes: {
class: "milkdown-editor mx-auto outline-hidden",
spellcheck: "false",
},
}));
// Add attributes to nodes and marks
ctx.set(headingAttr.key, (node) => {
const level = node.attrs.level;
return {
class: `heading-${level} font-bold`,
"data-level": level,
};
});
ctx.set(paragraphAttr.key, () => ({
class: "text-base leading-relaxed",
}));
})
.use(commonmark);
```
# Best Practices
---
1. **Use CSS Variables**: Define your theme's colors and spacing using CSS variables for easy customization.
2. **Responsive Design**: Ensure your editor styles work well on different screen sizes.
3. **Dark Mode Support**: Consider adding dark mode support using CSS variables and media queries.
4. **Accessibility**: Maintain good contrast ratios and readable font sizes.
5. **Performance**: Keep your CSS selectors specific and avoid overly complex rules.
For more examples and inspiration, check out:
- [@milkdown/theme-nord](https://github.com/Milkdown/milkdown/tree/main/packages/theme-nord)
- [@milkdown/crepe/theme](https://github.com/Milkdown/milkdown/tree/main/packages/crepe/src/theme)
milkdown-docs/guide/using-crepe.md
+234
View File
@@ -0,0 +1,234 @@
# Using Crepe Editor
Crepe is a powerful, feature-rich Markdown editor built on top of Milkdown. It provides a complete editing experience with a beautiful UI and extensive customization options.
## Why Choose Crepe?
---
- 🚀 **Ready to Use**: Works out of the box with sensible defaults
- 🎨 **Beautiful UI**: Modern design with multiple theme options
- 🔧 **Highly Customizable**: Extensive configuration options
- 📦 **Feature Complete**: Includes all essential Markdown editing features
- 🛠️ **Extensible**: Built on Milkdown's plugin system
## Quick Start
---
### Installation
```bash
# Using npm
npm install @milkdown/crepe
# Using yarn
yarn add @milkdown/crepe
# Using pnpm
pnpm add @milkdown/crepe
```
### Basic Usage
```typescript
import { Crepe } from "@milkdown/crepe";
import "@milkdown/crepe/theme/common/style.css";
import "@milkdown/crepe/theme/frame.css";
// Choose your preferred theme
// Create editor instance
const crepe = new Crepe({
root: document.getElementById("app"),
defaultValue: "# Hello, Crepe!\n\nStart writing your markdown...",
});
// Initialize the editor
await crepe.create();
// Clean up when done
crepe.destroy();
```
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/editor-crepe"}
## Themes
---
Crepe comes with several beautiful themes out of the box:
### Light Themes
- `frame` - Modern frame-based design
- `classic` - Traditional editor look
- `nord` - Clean, minimal Nord color scheme
### Dark Themes
- `frame-dark` - Dark version of frame theme
- `classic-dark` - Dark version of classic theme
- `nord-dark` - Dark version of nord theme
To use a theme:
```typescript
// Import base styles first
import "@milkdown/crepe/theme/common/style.css";
// Then import your chosen theme
import "@milkdown/crepe/theme/frame.css";
```
### Custom Themes
You can create your own theme by extending the base styles. Check out the [existing themes](https://github.com/Milkdown/milkdown/tree/main/packages/crepe/src/theme) for reference.
## Features
---
Crepe includes a comprehensive set of features that can be enabled or disabled as needed.
### Feature Configuration
> **Note**: For any configuration that ends with `Icon` (like `boldIcon`, `linkIcon`, etc.), you can use a HTML string or a simply string. This applies to all icon configurations throughout Crepe's features.
```typescript
const crepe = new Crepe({
features: {
// Disable specific features
[Crepe.Feature.CodeMirror]: false,
[Crepe.Feature.Table]: false,
},
featureConfigs: {
// Configure feature behavior
[Crepe.Feature.LinkTooltip]: {
inputPlaceholder: "Enter URL...",
},
},
});
```
### Available Features
#### 1. Code Editor (`CodeMirror`)
Syntax highlighting and editing for code blocks with language support, theme customization, and preview capabilities.
#### 2. List Management (`ListItem`)
Support for bullet lists, ordered lists, and todo lists with customizable icons and formatting.
#### 3. Link Management (`LinkTooltip`)
Enhanced link editing and preview with customizable tooltips, edit/remove actions, and copy functionality.
#### 4. Image Handling (`ImageBlock`)
Image upload and management with resizing, captions, and support for both inline and block images.
#### 5. Block Editing (`BlockEdit`)
Drag-and-drop block management and slash commands for quick content insertion and organization.
#### 6. Table Support (`Table`)
Full-featured table editing with row/column management, alignment options, and drag-and-drop functionality.
#### 7. Toolbar (`Toolbar`)
Formatting toolbar for selected text with customizable icons and actions.
#### 8. Cursor (`Cursor`)
Enhanced cursor experience with drop cursor and gap cursor for better content placement.
#### 9. Placeholder (`Placeholder`)
Document or block level placeholders to guide users when content is empty.
#### 10. Latex (`Latex`)
Mathematical formula support with both inline and block math rendering using KaTeX.
For detailed configuration options of each feature, please refer to the [API documentation](/docs/api/crepe).
## Editor Instance Methods
---
#### `crepe.editor`
Access the underlying Milkdown editor instance.
```typescript
const editor = crepe.editor;
editor.use(customPlugin);
editor.action(insert("Hello"));
```
#### `crepe.create()`
Initialize the editor.
```typescript
await crepe.create();
```
#### `crepe.destroy()`
Clean up the editor instance.
```typescript
crepe.destroy();
```
#### `crepe.setReadonly(value: boolean)`
Toggle readonly mode.
```typescript
crepe.setReadonly(true); // Make editor read-only
crepe.setReadonly(false); // Make editor editable
```
#### `crepe.on`
Add event listeners.
```typescript
crepe.on((listener) => {
listener.markdownUpdated((markdown) => {
console.log("Markdown updated:", markdown);
});
listener.updated((doc) => {
console.log("Document updated");
});
listener.focus(() => {
console.log("Editor focused");
});
listener.blur(() => {
console.log("Editor blurred");
});
});
```
#### `crepe.getMarkdown()`
Get current markdown content.
```typescript
const markdown = crepe.getMarkdown();
```
## Next Steps
---
- Learn about [Milkdown's architecture](/docs/guide/architecture-overview)
- Explore [available plugins](/docs/plugin/using-plugins)
- Read the [API reference](/docs/api/crepe)
milkdown-docs/guide/using-milkdown-kit.md
+37
View File
@@ -0,0 +1,37 @@
# Using @milkdown/kit
Milkdown provides a set of utilities to help you build your editor.
These utilities are re-exported from the `@milkdown/kit` package.
Thus, you don't need to install the common dependencies manually like `@milkdown/prose`, `@milkdown/core` or `@milkdown/preset-common` in your project.
## What's included
`@milkdown/kit` re-exports the following packages:
| Package | Import path | Scope |
| ---------------------------------------------------------- | ----------------------------------------- | --------- |
| [@milkdown/core](/docs/api/core) | `@milkdown/kit/core` | Framework |
| [@milkdown/ctx](/docs/api/ctx) | `@milkdown/kit/ctx` | Framework |
| [@milkdown/prose](/docs/guide/prosemirror-api) | `@milkdown/kit/prose` | Framework |
| [@milkdown/prose/\*](/docs/guide/prosemirror-api) | `@milkdown/kit/prose/*` | Framework |
| [@milkdown/transformer](/docs/api/transformer) | `@milkdown/kit/transformer` | Framework |
| [@milkdown/utils](/docs/api/utils) | `@milkdown/kit/utils` | Framework |
| [@milkdown/preset-commonmark](/docs/api/preset-commonmark) | `@milkdown/kit/preset/commonmark` | Preset |
| [@milkdown/preset-gfm](/docs/api/preset-gfm) | `@milkdown/kit/preset/gfm` | Preset |
| [@milkdown/plugin-block](/docs/api/plugin-block) | `@milkdown/kit/plugin/block` | Plugin |
| [@milkdown/plugin-clipboard](/docs/api/plugin-clipboard) | `@milkdown/kit/plugin/clipboard` | Plugin |
| [@milkdown/plugin-cursor](/docs/api/plugin-cursor) | `@milkdown/kit/plugin/cursor` | Plugin |
| [@milkdown/plugin-history](/docs/api/plugin-history) | `@milkdown/kit/plugin/history` | Plugin |
| [@milkdown/plugin-indent](/docs/api/plugin-indent) | `@milkdown/kit/plugin/indent` | Plugin |
| [@milkdown/plugin-listener](/docs/api/plugin-listener) | `@milkdown/kit/plugin/listener` | Plugin |
| [@milkdown/plugin-slash](/docs/api/plugin-slash) | `@milkdown/kit/plugin/slash` | Plugin |
| [@milkdown/plugin-tooltip](/docs/api/plugin-tooltip) | `@milkdown/kit/plugin/tooltip` | Plugin |
| [@milkdown/plugin-trailing](/docs/api/plugin-trailing) | `@milkdown/kit/plugin/trailing` | Plugin |
| [@milkdown/plugin-upload](/docs/api/plugin-upload) | `@milkdown/kit/plugin/upload` | Plugin |
| @milkdown/component | `@milkdown/kit/component` | Component |
| @milkdown/component/code-block | `@milkdown/kit/component/code-block` | Component |
| @milkdown/component/image-block | `@milkdown/kit/component/image-block` | Component |
| @milkdown/component/image-inline | `@milkdown/kit/component/image-inline` | Component |
| @milkdown/component/link-tooltip | `@milkdown/kit/component/link-tooltip` | Component |
| @milkdown/component/list-item-block | `@milkdown/kit/component/list-item-block` | Component |
| @milkdown/component/table-block | `@milkdown/kit/component/table-block` | Component |
milkdown-docs/guide/why-milkdown.md
+30
View File
@@ -0,0 +1,30 @@
# Why Milkdown
There are different kinds of markdown editors, such as [Typora](https://typora.io/), [tui](https://github.com/nhn/tui.editor) and [Bear](https://bear.app/).
They work pretty well for writing notes in markdown on different platforms. So why bother making Milkdown?
Milkdown aims to provide an **open source solution** for developers to make their editors more powerful, and attractive, it also ensures it runs everywhere.
---
## Open Source & Easy to Integrate
Different from industrial apps such as [Notion](https://notion.so) and [Typora](https://typora.io/),
Milkdown is open source and fully free. You can integrate it everywhere legally.
> If you like milkdown, please consider to fund me in order to help with the maintenance.
## Plugin Driven
Milkdown treats every feature as a plugin.
With this pattern, developers can choose what they need in an editor instead of bundling all features even they won't need.
Developers can extend their plugins to satisfy their habits such as defining a vim keymap via a custom plugin.
## Reliable
Milkdown is powered by [Prosemirror](https://prosemirror.net/) and [Remark](https://github.com/remarkjs/remark), which has a large community and stands the test of the industry.
What's more, plugins from the prosemirror and remark community can be easily reused in order to build a Milkdown plugin.
## Themable & Hackable
Themes and plugins for Milkdown can be shared and installed using npm packages. Milkdown is a headless component, which means you can fully control its style.
milkdown-docs/playground/template.md
+90
View File
@@ -0,0 +1,90 @@
# Milkdown
👋 Welcome to Milkdown. We are so glad to see you here!
💭 You may wonder, what is Milkdown? Please write something here.
> ⚠️ **Not the right side!**
>
> Please try something on the left side.
![1.00](/polar.jpeg "Hello by a polar bear")
You're seeing this editor called **🥞Crepe**, which is an editor built on top of Milkdown.
If you want to install this editor, you can run `npm install @milkdown/crepe`. Then you can use it like this:
```js
import { Crepe } from "@milkdown/crepe";
import "@milkdown/crepe/theme/common/style.css";
// We have some themes for you to choose, ex.
import "@milkdown/crepe/theme/frame.css";
// Or you can create your own theme
import "./your-theme.css";
const crepe = new Crepe({
root: "#app",
defaultValue: "# Hello, Milkdown!",
});
crepe.create().then(() => {
console.log("Milkdown is ready!");
});
// Before unmount
crepe.destroy();
```
---
## Structure
> 🍼 [Milkdown][repo] is a WYSIWYG markdown editor framework.
>
> Which means you can build your own markdown editor with Milkdown.
In the real world, a typical milkdown editor is built on top of 3 layers:
- [x] 🥛 Core: The core of Milkdown, which provides the plugin loading system with the editor concepts.
- [x] 🧇 Plugins: A set of plugins that can be used to extend the functionalities of the editor.
- [x] 🍮 Components: Some headless components that can be used to build your own editor.
At the start, you may find it hard to understand all these concepts.
But don't worry, we have this `@milkdown/crepe` editor for you to get started quickly.
---
## You can do more with Milkdown
In Milkdown, you can extend the editor in many ways:
| Feature | Description | Example |
| ------------ | ---------------------------------------------------- | ------------------------- |
| 🎨 Theme | Create your own theme with CSS | Nord, Dracula |
| 🧩 Plugin | Create your own plugin to extend the editor | Search, Collab |
| 📦 Component | Create your own component to build your own editor | Slash Menu, Toolbar |
| 📚 Syntax | Create your own syntax to extend the markdown parser | Image with Caption, LaTex |
We have provided a lot of plugins and components, with an out-of-the-box crepe editor for you to use and learn.
---
## Open Source
- Milkdown is an open-source project under the MIT license.
- Everyone is welcome to contribute to the project, and you can use it in your own project for free.
- Please let me know what you are building with Milkdown, I would be so glad to see that!
Maintaining Milkdown is a lot of work, and we are working on it in our spare time.
If you like Milkdown, please consider supporting us by [sponsoring][sponsor] the project.
We'll be so grateful for your support.
## Who built Milkdown?
Milkdown is built by [Mirone][mirone] and designed by [Meo][meo].
[repo]: https://github.com/Milkdown/milkdown
[mirone]: https://github.com/Saul-Mirone
[meo]: https://meo.cool
[sponsor]: https://github.com/sponsors/Saul-Mirone
milkdown-docs/plugin/composable-plugins.md
+85
View File
@@ -0,0 +1,85 @@
# Composable Plugins
In the previous section, we showed you how to create a plugin from scratch. Luckily, you don't need to do that in most cases. Milkdown provides a lot of helpers in [@milkdown/utils](/docs/api/utils) to make it easier to create plugins. The **composable** here means that you can use the plugin in other plugins. For example, you can use a command plugin in a keymap plugin. This is a very common pattern in Milkdown.
I'll show you some examples of how to use composable plugins. But I won't go into detail about the options and the usage of each plugin. You can find the details in the [API reference](/docs/api/utils#composable).
## Schema
The schema plugin is the most important plugin in Milkdown. It defines the structure of the document. A schema plugin in milkdown is a super set of the [node schema spec](https://prosemirror.net/docs/ref/#model.NodeSpec) or [mark schema spec](https://prosemirror.net/docs/ref/#model.MarkSpec) in ProseMirror.
Let's create a simple blockquote node plugin as an example:
```typescript
import { $node } from "@milkdown/kit/utils";
const blockquote = $node("blockquote", () => ({
content: "block+",
group: "block",
defining: true,
parseDOM: [{ tag: "blockquote" }],
toDOM: (node) => ["blockquote", ctx.get(blockquoteAttr.key)(node), 0],
parseMarkdown: {
match: ({ type }) => type === "blockquote",
runner: (state, node, type) => {
state.openNode(type).next(node.children).closeNode();
},
},
toMarkdown: {
match: (node) => node.type.name === "blockquote",
runner: (state, node) => {
state.openNode("blockquote").next(node.content).closeNode();
},
},
}));
```
## Input Rule
Since we have a blockquote node, we can create an input rule plugin to make it easier to create a blockquote node.
We expect that when we type `> ` at the beginning of a line, the blockquote node will be created.
```typescript
import { wrappingInputRule } from "@milkdown/kit/prose/inputrules";
import { $inputRule } from "@milkdown/kit/utils";
export const wrapInBlockquoteInputRule = $inputRule(() =>
wrappingInputRule(/^\s*>\s$/, blockquoteSchema.type()),
);
```
## Command
We can also create a command plugin to create a blockquote node.
The command is useful when we want to create a button to create a blockquote node.
```typescript
import { wrapIn } from "@milkdown/kit/prose/commands";
import { $command } from "@milkdown/kit/utils";
export const wrapInBlockquoteCommand = $command(
"WrapInBlockquote",
() => () => wrapIn(blockquoteSchema.type()),
);
```
## Shortcut
We can also create a shortcut plugin for blockquote.
Here we use `Ctrl + Shift + B` as the shortcut. When we press this shortcut, the blockquote node will be created.
And we can also use the command we created in the previous section.
```typescript
import { commandsCtx } from "@milkdown/kit/core";
import { $useKeymap } from "@milkdown/kit/utils";
export const blockquoteKeymap = $useKeymap("blockquoteKeymap", {
WrapInBlockquote: {
shortcuts: "Mod-Shift-b",
command: (ctx) => {
const commands = ctx.get(commandsCtx);
return () => commands.call(wrapInBlockquoteCommand.key);
},
},
});
```
milkdown-docs/plugin/example-block-plugin.md
+125
View File
@@ -0,0 +1,125 @@
# Example: Block Plugin
The **block plugin** adds a positional hook next to every top-level node (paragraphs, headings, lists, etc.).
It is the foundation for features such as drag handles, quick-insert buttons or block toolbars.
In Milkdown this functionality lives in `@milkdown/plugin-block` and – consistent with tooltip & slash – consists of:
- a **BlockProvider** that deals with DOM positioning/lifecycle
- a **blockFactory** – _implemented internally_ – exposed as two ctx slices: `blockSpec`, `blockPlugin`
This guide covers:
- Understanding the provider/service architecture.
- Writing a **vanilla TypeScript** drag handle that lets you reorder blocks.
- Mounting custom UIs in **React** and **Vue**.
- Studying the production-ready _Block Handle_ feature inside Crepe.
---
## 1. Anatomy of a Block Plugin
Unlike tooltip/slash, `@milkdown/plugin-block` ships its factory slices directly:
```ts
import { blockSpec, blockPlugin } from "@milkdown/plugin-block";
```
You normally interact with **BlockProvider** which talks to an internal _BlockService_: the service listens to mouse / drag events, figures out which node is **active** and sends `show` / `hide` messages to the provider.
Your job is to decide how to render a UI for that active node.
Key APIs:
- `new BlockProvider({ ctx, content, ... })` – similar to Tooltip/Slash.
- `provider.active` – info about the currently focused block (`node`, `pos`, `el`).
- Optional callbacks: `getOffset`, `getPlacement`, `getPosition` for fine-grained positioning.
---
## 2. Minimal Vanilla Drag Handle
Below we build a small **drag handle** that appears on hover and lets you drag-n-drop any block.
```ts
import { block, blockPlugin } from "@milkdown/plugin-block";
import { BlockProvider } from "@milkdown/plugin-block/block-provider"; // path depending on bundler
import { Editor } from "@milkdown/kit/core";
import { commonmark } from "@milkdown/kit/preset/commonmark";
// 1️⃣ Create DOM element for the handle
const handle = document.createElement("div");
handle.className = "drag-handle";
handle.innerHTML = "≡";
handle.style.cssText = `
width:20px;height:20px;display:flex;align-items:center;justify-content:center;
cursor:grab;border-radius:4px;background:#f2f3f5;color:#555;user-select:none;
`;
// 2️⃣ Build provider – show only when mouse is over a block
const provider = (ctx: Ctx) => {
const provider = new BlockProvider({
ctx,
content: handle,
getOffset: () => 8,
});
return {
update: provider.update,
destroy: provider.destroy,
};
};
// 3️⃣ Wire provider to Milkdown
const blockConfig = (ctx: Ctx) => {
ctx.set(blockSpec.key, {
view: provider(ctx),
});
};
Editor.make().config(blockConfig).use(commonmark).use(block).create();
```
Drag & Drop:
The HTML element has `cursor:grab`. The internal `BlockService` automatically sets `draggable` and wires ProseMirror's drag-events so you can reorder blocks without extra code 👉 nice!
---
## 3. Framework Examples
### React
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/react-block"}
The React demo renders a `<BlockHandle/>` component, keeps drag state in hooks and feeds the root element to `BlockProvider`.
### Vue
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/vue-block"}
Vue's `<BlockHandle>` uses `Teleport` and reactive refs exactly like the tooltip/slash examples.
---
## 4. Real-world Feature – Crepe Block Handle
Crepe brings all the pieces together to create a **block edit** experience that combines a drag handle **and** a plus-button to open the slash menu:
```text
packages/crepe/src/feature/block-edit/handle/
```
Things worth exploring:
1. **Dynamic placement** via `getPlacement` (centred vs top-aligned depending on node height).
2. Filtering nodes with `blockConfig.filterNodes` so handles do not appear inside tables / math / blockquotes.
3. Programmatically showing the _slash menu_ after pressing the "+" button.
---
## 5. Summary & Next Steps
`@milkdown/plugin-block` is the Swiss-army knife for any block-level UI: drag handles, add-buttons, side toolbars…
Combine it with tooltip/slash to build sophisticated editors.
Hack on the examples, tweak positioning callbacks, and ship your own block goodies 🚀.
milkdown-docs/plugin/example-iframe-plugin.md
+159
View File
@@ -0,0 +1,159 @@
# Example: Iframe Plugin
This guide demonstrates how to create a custom iframe syntax plugin for Milkdown. This plugin allows you to embed iframes directly in your markdown content using a simple directive syntax.
## Overview
---
The iframe plugin enables you to embed external web content using the following syntax:
```markdown
::iframe{src="https://example.com"}
```
This will render as an embedded iframe in your document.
## Implementation Steps
---
To create a custom syntax plugin in Milkdown, we need to implement five key components:
1. **Remark Plugin**: Parse the custom syntax
2. **Schema Definition**: Define the node structure
3. **Parser**: Convert markdown to ProseMirror nodes
4. **Serializer**: Convert ProseMirror nodes back to markdown
5. **Input Rules**: Handle user input
Let's implement each component:
## 1. Remark Plugin
---
First, we use the `remark-directive` plugin to support our custom syntax. This plugin allows us to define custom directives in markdown.
```typescript
import directive from "remark-directive";
import { $remark } from "@milkdown/kit/utils";
const remarkDirective = $remark("remarkDirective", () => directive);
```
## 2. Schema Definition
---
Next, we define the schema for our iframe node. The schema specifies how the node behaves and appears in the editor.
```typescript
import { $node } from "@milkdown/kit/utils";
import { Node } from "@milkdown/kit/prose/model";
const iframeNode = $node("iframe", () => ({
group: "block", // Block-level node
atom: true, // Cannot be split
isolating: true, // Cannot be merged with adjacent nodes
marks: "", // No marks allowed
attrs: {
src: { default: null }, // URL attribute
},
parseDOM: [
{
tag: "iframe",
getAttrs: (dom) => ({
src: (dom as HTMLElement).getAttribute("src"),
}),
},
],
toDOM: (node: Node) => [
"iframe",
{ ...node.attrs, contenteditable: false }, // Prevent editing iframe content
0,
],
}));
```
## 3. Parser
---
The parser converts our markdown syntax into ProseMirror nodes. It looks for the `leafDirective` type with the name "iframe".
```typescript
parseMarkdown: {
match: (node) => node.type === 'leafDirective' && node.name === 'iframe',
runner: (state, node, type) => {
state.addNode(type, { src: (node.attributes as { src: string }).src });
},
},
```
## 4. Serializer
---
The serializer converts ProseMirror nodes back to markdown format.
```typescript
toMarkdown: {
match: (node) => node.type.name === 'iframe',
runner: (state, node) => {
state.addNode('leafDirective', undefined, undefined, {
name: 'iframe',
attributes: { src: node.attrs.src },
});
},
},
```
## 5. Input Rules
---
Input rules handle user typing and convert the syntax into an iframe node.
```typescript
import { InputRule } from "@milkdown/kit/prose";
import { $inputRule } from "@milkdown/kit/utils";
const iframeInputRule = $inputRule(
() =>
new InputRule(
/::iframe\{src\="(?<src>[^"]+)?"?\}/,
(state, match, start, end) => {
const [okay, src = ""] = match;
const { tr } = state;
if (okay) {
tr.replaceWith(start - 1, end, iframeNode.type().create({ src }));
}
return tr;
},
),
);
```
## Usage
---
To use the iframe plugin, add it to your Milkdown editor configuration:
```typescript
import { Editor } from "@milkdown/kit/core";
import { commonmark } from "@milkdown/kit/preset/commonmark";
Editor.make()
.use([remarkDirective, iframeNode, iframeInputRule])
.use(commonmark)
.create();
```
## Example
---
Here's a complete example of the iframe plugin in action:
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/vanilla-iframe-syntax"}
milkdown-docs/plugin/example-marker-plugin.md
+189
View File
@@ -0,0 +1,189 @@
# Example: Marker Plugin
This guide demonstrates how to create a custom marker syntax plugin for Milkdown. This plugin allows you to mark text with custom colors using a simple markdown syntax.
## Overview
---
The marker plugin enables you to mark text using the following syntax:
```markdown
==marked text==
=={#EE4B2B}marked text with color==
```
This will render as marked text in your document, with the option to specify custom colors.
## Implementation Steps
---
To create a custom marker syntax plugin in Milkdown, we need to implement several components:
1. **Remark Plugin**: Parse the custom syntax
2. **Schema Definition**: Define the mark structure
3. **Parser**: Convert markdown to ProseMirror marks
4. **Serializer**: Convert ProseMirror marks back to markdown
5. **Input Rules**: Handle user input
6. **Color Picker**: Add UI for color selection
Let's implement each component:
## 1. Remark Plugin
---
First, we create a remark plugin to handle our custom marker syntax:
> ⚠️ The real implementation is more complex, but we simplify it for the sake of the example.
> Under the hood, you'll need to write a [micromark extension](https://github.com/micromark/micromark) to make it works correctly.
```typescript
import { $remark } from "@milkdown/kit/utils";
const remarkMarkColor = () => {
return (tree: any) => {
visit(tree, "text", (node: any, index: number, parent: any) => {
const match = node.value.match(/==(?:{#([^}]+)})?([^=]+)==/);
if (match) {
const [_, color, text] = match;
const mark = {
type: "mark",
data: { color },
children: [{ type: "text", value: text }],
};
parent.children.splice(index, 1, mark);
}
});
};
};
const milkdownMarkColorPlugin = $remark("markColor", () => remarkMarkColor);
```
## 2. Schema Definition
---
Next, we define the schema for our marker:
```typescript
import { $markSchema } from "@milkdown/kit/utils";
import { Mark } from "mdast";
export const DEFAULT_COLOR = "#ffff00";
export const markSchema = $markSchema("mark", () => ({
attrs: {
color: {
default: DEFAULT_COLOR,
validate: "string",
},
},
parseDOM: [
{
tag: "mark",
getAttrs: (node: HTMLElement) => ({
color: node.style.backgroundColor,
}),
},
],
toDOM: (mark) => ["mark", { style: `background-color: ${mark.attrs.color}` }],
parseMarkdown: {
match: (node) => node.type === "mark",
runner: (state, node, markType) => {
const color = (node as Mark).data?.color;
state.openMark(markType, { color });
state.next(node.children);
state.closeMark(markType);
},
},
toMarkdown: {
match: (node) => node.type.name === "mark",
runner: (state, mark) => {
let color = mark.attrs.color;
if (color?.toLowerCase() === DEFAULT_COLOR.toLowerCase()) {
color = undefined;
}
state.withMark(mark, "mark", undefined, {
data: { color },
});
},
},
}));
```
## 3. Input Rules
---
We add input rules to handle user typing:
```typescript
import { $inputRule } from "@milkdown/kit/utils";
import { InputRule } from "@milkdown/kit/prose";
const markInputRule = $inputRule(
() =>
new InputRule(/==(?:{#([^}]+)})?([^=]+)==/, (state, match, start, end) => {
const [okay, color, text] = match;
const { tr } = state;
if (okay) {
tr.addMark(
start,
end,
markSchema.type().create({ color: color || DEFAULT_COLOR }),
);
}
return tr;
}),
);
```
## 4. Color Picker Tooltip
---
To enhance the user experience, we add a color picker tooltip:
```typescript
export const colorPickerTooltip = tooltipFactory("color-picker");
class TooltipPluginView {
// ... implementation
}
export const colorPickerTooltipConfig = (ctx: Ctx) => {
ctx.set(colorPickerTooltip.key, {
view: () => new TooltipPluginView(ctx),
});
};
```
## Usage
---
To use the marker plugin, add it to your Milkdown editor configuration:
```typescript
import { Editor } from "@milkdown/kit/core";
import { commonmark } from "@milkdown/kit/preset/commonmark";
Editor.make()
.use(milkdownMarkColorPlugin)
.use(markSchema)
.use(markInputRule)
.use(colorPickerTooltip)
.use(commonmark)
.create();
```
## Example
---
Here's a complete example of the marker plugin in action:
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/vanilla-highlight-syntax"}
milkdown-docs/plugin/example-slash-plugin.md
+137
View File
@@ -0,0 +1,137 @@
# Example: Slash Plugin
After reading the tooltip guide you already know how Milkdown separates **positioning logic** (provider) from **editor wiring** (ctx slices produced by a factory).
The `@milkdown/plugin-slash` package applies exactly the same idea but focuses on _command palettes_ triggered by a character – familiar to `/` menus in modern editors.
This document shows you how to:
- Understand what the slash plugin gives you out-of-the-box.
- Build a **vanilla TypeScript** implementation of a basic `/` menu.
- Use the slash provider with **React** and **Vue**.
- Explore a full-blown menu feature that ships inside Milkdown's Crepe UI.
---
## 1. Anatomy of a Slash Plugin
`@milkdown/plugin-slash` exports two utilities:
1. **`SlashProvider`** – Measures the caret position and manages show / hide of your menu.
2. **`slashFactory(id)`** – Generates a ctx slice & ProseMirror plugin pair that plugs the provider into the editor.
```ts
import { slashFactory } from "@milkdown/plugin-slash";
export const [mySlashSpec, mySlashPlugin] = slashFactory("my");
```
Just like the tooltip factory:
- `mySlashSpec` is where you put a `PluginSpec` (what ProseMirror needs).
- `mySlashPlugin` turns that spec into a runtime plugin.
---
## 2. A Minimal Vanilla `/` Menu
Below we create a small menu that suggests two commands whenever the user types `/`.
```ts
import { SlashProvider, slashFactory } from "@milkdown/plugin-slash";
import { Editor } from "@milkdown/kit/core";
import { commonmark } from "@milkdown/kit/preset/commonmark";
// DOM content of the menu – plain HTML for the demo
const menu = document.createElement("div");
menu.className = "slash-menu";
menu.style.cssText = `
position:absolute;padding:4px 0;background:white;border:1px solid #eee;
box-shadow:0 2px 8px rgba(0,0,0,.15);border-radius:6px;font-size:14px;
`;
menu.innerHTML = `<ul style="margin:0;padding:0;list-style:none">
<li data-cmd="h1" style="padding:4px 12px;cursor:pointer">Heading 1</li>
<li data-cmd="bullet" style="padding:4px 12px;cursor:pointer">Bullet List</li>
</ul>`;
// Click handler – replace with real commands
menu.addEventListener("click", (e) => {
const target = e.target as HTMLElement;
const cmd = target.dataset.cmd;
alert(`Run command: ${cmd}`);
});
// Provider positions & shows above DOM element
const provider = new SlashProvider({
content: menu,
// show the menu when the last character before caret is '/'
shouldShow(view) {
return provider.getContent(view)?.endsWith("/") ?? false;
},
offset: 8,
});
const slash = slashFactory("demo");
const slashConfig = (ctx: Ctx) => {
ctx.set(slash.key, {
view: () => ({
update: provider.update,
destroy: provider.destroy,
}),
});
};
Editor.make().config(slashConfig).use(commonmark).use(slash).create();
```
Key takeaways:
- `SlashProvider` has a helper `getContent(view)` to fetch text before the caret – handy for filtering.
- You decide **when to show** the menu via the `shouldShow` callback (default: when last char is `/`).
- The provider only manipulates **position + visibility**; rendering & commands are completely yours.
---
## 3. Framework Examples
### React
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/react-slash"}
Highlights:
1. A `<SlashMenu/>` React component renders the list.
2. The component root is passed to `SlashProvider` (just like the tooltip demo).
3. React hooks manage internal focus & keyboard navigation.
### Vue
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/vue-slash"}
The Vue version uses `Teleport` to append the menu to `document.body` and `ref` / `watch` for reactivity.
---
## 4. Real-world Feature – Crepe Block Menu
Milkdown's **Crepe** UI implements an extensible block-level menu on top of the slash plugin. You'll find the source code at:
```text
packages/crepe/src/feature/block-edit/menu/
```
Notable patterns to look for:
- **Context slices** (`menu` / `menuAPI`) to expose imperative `show` & `hide` methods.
- Filtering commands based on the current text after `/`.
- Preventing the menu inside `code` blocks or lists.
Studying this folder is a great next step once you master the basics.
---
## 5. Summary & Next Steps
- `@milkdown/plugin-slash` gives you caret detection + positioning – nothing else.
- UI, behaviour, and commands are fully customisable.
Fork one of the examples above, add your own commands, and you'll have a modern `/` command palette in minutes ✨.
milkdown-docs/plugin/example-tooltip-plugin.md
+140
View File
@@ -0,0 +1,140 @@
# Example: Tooltip Plugin
This guide walks you through creating and using **tooltip-based plugins** in Milkdown.
You will learn how the low-level `@milkdown/plugin-tooltip` works and how to build richer experiences on top of it in **vanilla TypeScript**, **React**, and **Vue**.
> **TL;DR** – A tooltip in Milkdown is nothing more than a ProseMirror plugin created by `tooltipFactory(id)`.
> It receives position information from the editor and renders any DOM of your choice.
> Everything else (buttons, inputs, styling, framework bindings) can be composed on top of that.
## 1. Anatomy of a Tooltip
---
At its core the tooltip plugin exported from `@milkdown/plugin-tooltip` contains two helpers:
1. **`TooltipProvider`** – An utility class powered by [floating-ui](https://floating-ui.com/) to calculate the tooltip position.
2. **`tooltipFactory(id)`** – A factory that returns a pair of Milkdown plugin slices which wire the provider into the editor.
The factory is extremely small (≈40 lines):
```ts
import { tooltipFactory } from "@milkdown/plugin-tooltip";
// Create a tooltip identified by the string "my".
export const [myTooltipSpec, myTooltipPlugin] = tooltipFactory("my");
```
The first element (`myTooltipSpec`) is a **ctx slice** that stores a `PluginSpec`, while the second one (`myTooltipPlugin`) is the real ProseMirror plugin which consumes that spec.
## 2. A Minimal Vanilla Tooltip
---
Below is the complete code for a tooltip that shows the **length of the current selection**.
```ts
import { Editor } from "@milkdown/kit/core";
import { commonmark } from "@milkdown/kit/preset/commonmark";
import { TooltipProvider, tooltipFactory } from "@milkdown/plugin-tooltip";
// 1) Prepare DOM that we will mount into the page.
const el = document.createElement("div");
el.className = "selection-length";
el.style.cssText = `
pointer-events:none;
background:#333;color:#fff;padding:2px 6px;border-radius:4px;font-size:12px;
`;
// 2) Build a provider which updates the content.
const provider = new TooltipProvider({
content: el,
shouldShow: (view) => !!view.state.selection.content().size,
});
// 3) Bridge provider & editor.
const tooltip = tooltipFactory("sel-length");
const tooltipConfig = (ctx: Ctx) => {
ctx.set(selectionTooltipSpec.key, {
view: () => ({
update: provider.update,
destroy: provider.destroy,
}),
});
};
Editor.make().config(tooltipConfig).use(commonmark).use(tooltip).create();
```
Key points:
- We **create** any DOM element we like (`el`).
- `TooltipProvider` tracks the editor position and moves the element.
- `tooltipFactory` wraps the provider into a pluggable slice.
## 3. Framework Examples
---
Sometimes building UI is easier in your favourite framework.
Because the tooltip provider only deals with **DOM elements**, you can freely render React, Vue or Svelte components and pass their root node to the provider.
### React
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/react-tooltip"}
The React example shows how to:
1. Create a React component (`<SelectionTooltip/>`).
2. Render it into a portal and give the root HTML element to `TooltipProvider`.
3. Re-use React state/hooks while Milkdown takes care of positioning.
### Vue
::iframe{src="https://stackblitz.com/github/Milkdown/examples/tree/main/vue-tooltip"}
The Vue example follows the same pattern with `defineComponent` and `teleport`.
## 4. Real-world Examples
---
### 4-1. Link Tooltip (_@milkdown/component/link-tooltip_)
The [link tooltip](https://github.com/Milkdown/milkdown/tree/main/packages/components/src/link-tooltip) demonstrates how to:
- Maintain UI **state** (`preview` vs `edit`) in ctx slices.
- Communicate with the editor through an **API slice** (add / edit / remove links).
- Render framework-agnostic UI inside a tooltip provider.
Have a look at the files below to see those techniques in action:
```text
packages/components/src/link-tooltip/
├── slices.ts # state & API slices
├── tooltips.ts # preview & edit providers
└── component.tsx # (framework examples)
```
### 4-2. Toolbar Feature (_@milkdown/crepe/feature/toolbar_)
The toolbar in the [crepe](https://github.com/Milkdown/milkdown/tree/main/packages/crepe) package pushes the idea further by:
- Using multiple tooltip instances (one per button group).
- Rendering the UI with Vue _inside_ the provider.
- Sharing configuration via ctx slices so that every button is extensible by third-party plugins.
You can browse the implementation starting from
```text
packages/crepe/src/feature/toolbar/component.tsx
```
## 5. Summary & Next Steps
---
- `@milkdown/plugin-tooltip` offers **just enough** abstraction: positioning & lifecycle.
- Everything else – **state, styling, framework integration** – is totally up to you.
Try to customise one of the examples above, then ship your own tooltip-powered features 🤟.
milkdown-docs/plugin/plugins-101.md
+174
View File
@@ -0,0 +1,174 @@
# Plugins 101
In this section we will show you the basic information of the plugin.
In most cases, you will not need to write plugins without helpers.
But it can help you understand the plugin system and what happens under the hood.
## Structure Overview
Generally speaking, a plugin will have following structure:
```typescript
import { MilkdownPlugin } from "@milkdown/kit/ctx";
const myPlugin: MilkdownPlugin = (ctx) => {
// #1 prepare plugin
return async () => {
// #2 run plugin
return async () => {
// #3 clean up plugin
};
};
};
```
Each plugin is composed by three parts:
1. _Prepare_: this part will be executed when plugin is registered in milkdown by `.use` method.
2. _Run_: this part will be executed when plugin is actually loaded.
3. _Post_: this part will be executed when plugin is removed by `.remove` method or editor is destroyed.
## Timer
Timer can be used to decide when to load the current plugin and how current plugin can influence other plugin's loading status.
You can use `ctx.wait` to wait a timer to finish.
```typescript
import { MilkdownPlugin, Complete } from "@milkdown/kit/core";
const myPlugin: MilkdownPlugin = (ctx) => {
return async () => {
const start = Date.now();
await ctx.wait(Complete);
const end = Date.now();
console.log("Milkdown load duration: ", end - start);
};
};
```
You can also create your own timer and influence other plugins load time.
For example, let's create a plugin that will fetch markdown content from remote server as editor's default value.
```typescript
import {
MilkdownPlugin,
editorStateTimerCtx,
defaultValueCtx,
createTimer,
} from "@milkdown/kit/core";
const RemoteTimer = createTimer("RemoteTimer");
const remotePlugin: MilkdownPlugin = (ctx) => {
// register timer
ctx.record(RemoteTimer);
return async () => {
// the editorState plugin will wait for this timer to finish before initialize editor state.
ctx.update(editorStateTimerCtx, (timers) => timers.concat(RemoteTimer));
const defaultMarkdown = await fetchMarkdownAPI();
ctx.set(defaultValueCtx, defaultMarkdown);
// mark timer as complete
ctx.done(RemoteTimer);
return async () => {
await SomeAPI();
// remove timer when plugin is removed
ctx.clearTimer(RemoteTimer);
};
};
};
```
It has following steps:
1. We use `createTimer` to create a timer, and use `pre.record` to register it into milkdown.
2. We update `editorStateTimerCtx` to tell the internal `editorState` plugin that before initialize editor state, it should wait our remote fetch process finished.
3. After we get value from `fetchMarkdownAPI`, we set it as `defaultValue` and use `ctx.done` to mark a timer as complete.
## Ctx
We have used `ctx` several times in the above example, now we can try to understand what it is.
Ctx is a data container which is shared in the entire editor instance. It's composed by a lot of slices. Every `slice` has a unique key and a value. You can change the value of a slice by `ctx.set` and `ctx.update`. And you can get the value of a slice by `ctx.get` with the slice key or name. Last but not least, you can remove a slice by `post.remove`.
```typescript
import { MilkdownPlugin, createSlice } from "@milkdown/kit/ctx";
const counterCtx = createSlice(0, "counter");
const counterPlugin: MilkdownPlugin = (ctx) => {
ctx.inject(counterCtx);
return () => {
// count is 0
const count0 = ctx.get(counterCtx);
// set count to 1
ctx.set(counterCtx, 1);
// now count is 1
const count1 = ctx.get(counterCtx);
// set count to n + 2
ctx.update(counterCtx, (prev) => prev + 2);
// now count is 3
const count2 = ctx.get(counterCtx);
// we can also get value by the slice name
const count3 = ctx.get("counter");
return () => {
// remove the slice
ctx.remove(counterCtx);
};
};
};
```
We can use `createSlice` to create a ctx, and use `pre.inject` to inject the ctx into the editor.
And when plugin processing, `ctx.get` can get the value of a ctx, `ctx.set` can set the value of a ctx, and `ctx.update` can update a ctx using callback function.
So, we can use `ctx` combine with `timer` to decide when should a plugin be processed.
```typescript
import {
MilkdownPlugin,
SchemaReady,
Timer,
createSlice,
} from "@milkdown/kit/core";
const examplePluginTimersCtx = createSlice<Timer[]>([], "example-timer");
const examplePlugin: MilkdownPlugin = (ctx) => {
ctx.inject(examplePluginTimersCtx, [SchemaReady]);
return async () => {
await Promise.all(
ctx.get(examplePluginTimersCtx).map((timer) => ctx.wait(timer)),
);
// or we can use a simplified syntax sugar
await ctx.waitTimers(examplePluginTimersCtx);
// do something
};
};
```
With this pattern, if other plugins want to delay the process of `examplePlugin`, all they need to do is just add a timer into `examplePluginTimersCtx` with `ctx.update`.
## Summary
Now let's go back to the plugin structure. Since we have the knowledge of `timer` and `ctx`, we can understand what we should do in each part of a plugin.
1. In `prepare` stage of the plugin, we can use `ctx.record` to register a timer, and use `ctx.inject` to inject a slice.
2. In `run` stage of the plugin, we can use `ctx.wait` to wait a timer to finish, and use `ctx.get` to get the value of a slice. We can also change values of slices by `ctx.set` and `ctx.update`. And we can use `ctx.done` to mark a timer as complete.
3. In `post` stage of the plugin, we can use `ctx.clearTimer` to clear a timer, and use `ctx.remove` to remove a slice.

Some files were not shown because too many files have changed in this diff Show More