2154 lines
72 KiB
Markdown
2154 lines
72 KiB
Markdown
# 汽车环车视频旧伤检测 Skill Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** 新增一个生产可用的 `vehicle-damage-inspection` compute-entry Skill，从环车视频中抽帧、检测漆面旧伤、定位标注损伤区域并筛选最佳证据帧。
|
||
|
||
**Architecture:** 后端先补强 `SkillExecutorService`，为 compute-entry 子进程注入 Windows 安装包生产所需的 `dataDir/workspace/uploads` 非密钥运行时环境变量。业务能力作为一个大 Skill 放在 `packages/backend/skills/vehicle-damage-inspection`，Agent 通过 `execute_skill` 调一次，Skill 内部用 Node CommonJS 脚本按固定流水线执行抽帧、视觉模型检测、bbox 标注和最佳帧筛选。
|
||
|
||
**Tech Stack:** TypeScript, Midway service, Jest, Node CommonJS, ffmpeg/ffprobe, fluent-ffmpeg, axios, sharp, 火山方舟 OpenAI-compatible Chat Completions, 豆包 `doubao-seed-2-0-pro-260215`
|
||
|
||
**Spec:** `docs/superpowers/specs/2026-05-04-vehicle-damage-inspection-skill-design.md`
|
||
|
||
---
|
||
|
||
## File Map
|
||
|
||
| 文件 | 职责 | 操作 |
|
||
|------|------|------|
|
||
| `packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts` | 根据 `pDataPath()` / `pUploadPath()` 构造 compute-entry skill 运行目录 env | 新增 |
|
||
| `packages/backend/src/modules/netaclaw/service/skill_env_schema.ts` | 从 `skill.config.yaml env` 同步管理页和 secrets 使用的 env schema | 新增 |
|
||
| `packages/backend/src/modules/netaclaw/service/skill_executor.ts` | 注入 runtime env，确保 Windows 安装包产物落在 dataDir | 修改 |
|
||
| `packages/backend/test/skill_runtime_env.test.ts` | 测试 runtime env 默认值和 skill secrets 覆盖顺序 | 新增 |
|
||
| `packages/backend/test/skill_env_schema.test.ts` | 测试 `skill.config.yaml env` 到 DB `envSchema` 的同步规则 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/SKILL.md` | Agent 可读 Skill 说明 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/skill.config.yaml` | compute-entry 配置、模型 env、依赖、接口 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/README.md` | 人工调试和 Windows 安装包注意事项 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/prompts/damage_detect.md` | 环车旧伤候选检测 prompt | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/prompts/grounding.md` | bbox/grounding prompt | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/prompts/dedup.md` | 损伤去重 prompt | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/prompts/best_frame.md` | 最佳帧筛选 prompt | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/index.cjs` | stdin/stdout JSON 入口和流水线编排 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/make.sh` | `check/run/demo/fix` 本地调试入口 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1` | Windows installer setup 入口；当前 installer 用 `powershell -File` 执行 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/lib/json_utils.cjs` | JSON、bbox、数字 clamp、sleep、并发限制等通用函数 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/lib/workspace.cjs` | task workspace、路径解析、JSON artifact I/O | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/lib/frame_extractor.cjs` | ffmpeg/ffprobe 抽帧 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/lib/vision_client.cjs` | OpenAI-compatible 视觉 API 调用、重试、图片压缩编码 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_detector.cjs` | 分批候选旧伤检测 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/lib/image_marker.cjs` | sharp 画 bbox 标注框 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_grounding.cjs` | 损伤定位标注 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/scripts/lib/best_frame_selector.cjs` | 损伤去重和最佳帧筛选 | 新增 |
|
||
| `packages/backend/skills/vehicle-damage-inspection/tests/*.test.cjs` | Skill 纯函数和 mock 流水线测试 | 新增 |
|
||
|
||
---
|
||
|
||
### Task 1: Skill env schema 同步
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/src/modules/netaclaw/service/skill_env_schema.ts`
|
||
- Modify: `packages/backend/src/modules/netaclaw/service/skill_loader.ts`
|
||
- Test: `packages/backend/test/skill_env_schema.test.ts`
|
||
|
||
- [ ] **Step 1: 写失败测试，覆盖从 skill.config.yaml env 同步默认模型**
|
||
|
||
Create `packages/backend/test/skill_env_schema.test.ts`:
|
||
|
||
```typescript
|
||
import { buildSkillEnvSchema } from '../src/modules/netaclaw/service/skill_env_schema.js';
|
||
|
||
describe('buildSkillEnvSchema', () => {
|
||
it('uses skill.config.yaml env as the envSchema source of truth', () => {
|
||
const schema = buildSkillEnvSchema({
|
||
env: [
|
||
{
|
||
name: 'ARK_API_KEY',
|
||
required: true,
|
||
description: '火山方舟/豆包 OpenAI-compatible API Key',
|
||
},
|
||
{
|
||
name: 'DAMAGE_DETECT_MODEL',
|
||
required: true,
|
||
default: 'doubao-seed-2-0-pro-260215',
|
||
description: '用于环车旧伤候选检测的多模态模型 ID',
|
||
},
|
||
],
|
||
});
|
||
|
||
expect(schema).toEqual([
|
||
{
|
||
name: 'ARK_API_KEY',
|
||
required: true,
|
||
description: '火山方舟/豆包 OpenAI-compatible API Key',
|
||
default: undefined,
|
||
},
|
||
{
|
||
name: 'DAMAGE_DETECT_MODEL',
|
||
required: true,
|
||
description: '用于环车旧伤候选检测的多模态模型 ID',
|
||
default: 'doubao-seed-2-0-pro-260215',
|
||
},
|
||
]);
|
||
});
|
||
|
||
it('returns null when the skill has no env declaration', () => {
|
||
expect(buildSkillEnvSchema({})).toBeNull();
|
||
});
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `pnpm --filter @neta/backend test -- --runInBand test/skill_env_schema.test.ts`
|
||
|
||
Expected: FAIL with module not found for `skill_env_schema.js`.
|
||
|
||
- [ ] **Step 3: 新增 skill_env_schema helper**
|
||
|
||
Create `packages/backend/src/modules/netaclaw/service/skill_env_schema.ts`:
|
||
|
||
```typescript
|
||
import type { SkillConfig } from './skill_config.js';
|
||
|
||
export interface SkillEnvSchemaItem {
|
||
name: string;
|
||
required: boolean;
|
||
description?: string;
|
||
default?: string;
|
||
}
|
||
|
||
export function buildSkillEnvSchema(config: Pick<SkillConfig, 'env'>): SkillEnvSchemaItem[] | null {
|
||
if (!Array.isArray(config.env) || config.env.length === 0) return null;
|
||
return config.env.map(item => ({
|
||
name: item.name,
|
||
required: !!item.required,
|
||
description: item.description || undefined,
|
||
default: item.default || undefined,
|
||
}));
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: 接入 SkillLoaderService.getSkillMetas**
|
||
|
||
Modify `packages/backend/src/modules/netaclaw/service/skill_loader.ts`.
|
||
|
||
Add import near existing service imports:
|
||
|
||
```typescript
|
||
import { buildSkillEnvSchema } from './skill_env_schema.js';
|
||
```
|
||
|
||
Inside `getSkillMetas()`, before `await this.skillRepo.save(...)`, add:
|
||
|
||
```typescript
|
||
const config = this.skillConfig.getConfig(fs.name);
|
||
const envSchema = buildSkillEnvSchema(config ?? {});
|
||
```
|
||
|
||
Replace the `envSchema` assignment in the save object:
|
||
|
||
```typescript
|
||
envSchema: Array.isArray((fs.metadata as any)?.env) ? (fs.metadata as any).env.map((e: any) => ({
|
||
name: e.name,
|
||
required: !!e.required,
|
||
description: e.description || undefined,
|
||
default: e.default || undefined,
|
||
})) : null,
|
||
```
|
||
|
||
with:
|
||
|
||
```typescript
|
||
envSchema,
|
||
```
|
||
|
||
This makes `skill.config.yaml env` the single source of truth for model defaults and secret fields. Do not duplicate env declarations in `SKILL.md metadata.env`.
|
||
|
||
- [ ] **Step 5: 运行测试确认通过**
|
||
|
||
Run: `pnpm --filter @neta/backend test -- --runInBand test/skill_env_schema.test.ts`
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 6: 验证后端编译**
|
||
|
||
Run: `pnpm --filter @neta/backend build`
|
||
|
||
Expected: build completes without TypeScript errors.
|
||
|
||
- [ ] **Step 7: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/src/modules/netaclaw/service/skill_env_schema.ts packages/backend/src/modules/netaclaw/service/skill_loader.ts packages/backend/test/skill_env_schema.test.ts
|
||
git commit -m "feat(netaclaw): sync skill config env schema"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: 宿主 runtime env 注入
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts`
|
||
- Modify: `packages/backend/src/modules/netaclaw/service/skill_executor.ts`
|
||
- Test: `packages/backend/test/skill_runtime_env.test.ts`
|
||
|
||
- [ ] **Step 1: 写失败测试，覆盖 Windows dataDir env 默认值**
|
||
|
||
Create `packages/backend/test/skill_runtime_env.test.ts`:
|
||
|
||
```typescript
|
||
import * as path from 'path';
|
||
import { buildSkillRuntimeEnv } from '../src/modules/netaclaw/service/skill_runtime_env.js';
|
||
|
||
describe('buildSkillRuntimeEnv', () => {
|
||
it('injects dataDir, workspace root and upload root for vehicle damage inspection', () => {
|
||
const dataDir = path.join('C:', 'RZYX_AI_DATA');
|
||
const uploadRoot = path.join(dataDir, 'uploads');
|
||
|
||
const env = buildSkillRuntimeEnv({
|
||
skillName: 'vehicle-damage-inspection',
|
||
dataDir,
|
||
uploadRoot,
|
||
});
|
||
|
||
expect(env.RZYX_AI_DATA_DIR).toBe(path.resolve(dataDir));
|
||
expect(env.RZYX_AI_UPLOAD_ROOT).toBe(path.resolve(uploadRoot));
|
||
expect(env.RZYX_AI_WORKSPACE_ROOT).toBe(
|
||
path.join(path.resolve(dataDir), 'workspace', 'vehicle-damage-inspection'),
|
||
);
|
||
});
|
||
|
||
it('uses generic workspace directory for other compute-entry skills', () => {
|
||
const dataDir = path.join('C:', 'RZYX_AI_DATA');
|
||
const env = buildSkillRuntimeEnv({
|
||
skillName: 'ocr-reader',
|
||
dataDir,
|
||
uploadRoot: path.join(dataDir, 'uploads'),
|
||
});
|
||
|
||
expect(env.RZYX_AI_WORKSPACE_ROOT).toBe(
|
||
path.join(path.resolve(dataDir), 'workspace', 'ocr-reader'),
|
||
);
|
||
});
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `pnpm --filter @neta/backend test -- --runInBand test/skill_runtime_env.test.ts`
|
||
|
||
Expected: FAIL with module not found for `skill_runtime_env.js`.
|
||
|
||
- [ ] **Step 3: 新增 runtime env helper**
|
||
|
||
Create `packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts`:
|
||
|
||
```typescript
|
||
import * as path from 'path';
|
||
|
||
export interface BuildSkillRuntimeEnvParams {
|
||
skillName: string;
|
||
dataDir: string;
|
||
uploadRoot: string;
|
||
}
|
||
|
||
export function buildSkillRuntimeEnv(params: BuildSkillRuntimeEnvParams): Record<string, string> {
|
||
const dataDir = path.resolve(params.dataDir);
|
||
const uploadRoot = path.resolve(params.uploadRoot);
|
||
const workspaceRoot = path.join(dataDir, 'workspace', params.skillName);
|
||
|
||
return {
|
||
RZYX_AI_DATA_DIR: dataDir,
|
||
RZYX_AI_UPLOAD_ROOT: uploadRoot,
|
||
RZYX_AI_WORKSPACE_ROOT: workspaceRoot,
|
||
};
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: 接入 SkillExecutorService**
|
||
|
||
Modify `packages/backend/src/modules/netaclaw/service/skill_executor.ts`.
|
||
|
||
Add imports near the existing imports:
|
||
|
||
```typescript
|
||
import { pDataPath, pUploadPath } from '../../../comm/path.js';
|
||
import { buildSkillRuntimeEnv } from './skill_runtime_env.js';
|
||
```
|
||
|
||
Replace:
|
||
|
||
```typescript
|
||
const skillEnv = await this.skillSecret.resolveEnv(skillName);
|
||
const env = { ...baseEnv, ...skillEnv };
|
||
```
|
||
|
||
with:
|
||
|
||
```typescript
|
||
const runtimeEnv = buildSkillRuntimeEnv({
|
||
skillName,
|
||
dataDir: pDataPath(),
|
||
uploadRoot: pUploadPath(),
|
||
});
|
||
const skillEnv = await this.skillSecret.resolveEnv(skillName);
|
||
const env = { ...baseEnv, ...runtimeEnv, ...skillEnv };
|
||
```
|
||
|
||
This order lets skill secrets override runtime defaults when an administrator explicitly configures a path.
|
||
|
||
- [ ] **Step 5: 运行测试确认通过**
|
||
|
||
Run: `pnpm --filter @neta/backend test -- --runInBand test/skill_runtime_env.test.ts`
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 6: 验证后端编译**
|
||
|
||
Run: `pnpm --filter @neta/backend build`
|
||
|
||
Expected: build completes without TypeScript errors.
|
||
|
||
- [ ] **Step 7: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts packages/backend/src/modules/netaclaw/service/skill_executor.ts packages/backend/test/skill_runtime_env.test.ts
|
||
git commit -m "feat(netaclaw): inject runtime data paths into compute skills"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Skill scaffold and metadata
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/SKILL.md`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/skill.config.yaml`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/README.md`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/prompts/damage_detect.md`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/prompts/grounding.md`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/prompts/dedup.md`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/prompts/best_frame.md`
|
||
|
||
- [ ] **Step 1: 创建 SKILL.md**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/SKILL.md`:
|
||
|
||
```markdown
|
||
---
|
||
name: vehicle-damage-inspection
|
||
description: >
|
||
汽车环车视频旧伤检测。输入一个环车视频文件或 /upload/... 路径，自动抽帧、
|
||
检测车身漆面旧伤、定位 bbox 并输出红框标注图和最佳证据帧。
|
||
license: MIT
|
||
metadata:
|
||
version: "1.0.0"
|
||
category: vehicle-inspection
|
||
skillType: multimodal
|
||
tags: ["vehicle", "video", "damage", "bbox"]
|
||
---
|
||
|
||
# 汽车环车视频旧伤检测
|
||
|
||
当用户需要从汽车环车视频中识别漆面旧伤、划痕、凹陷、掉漆、锈蚀等外观损伤时，使用此 skill。
|
||
|
||
这是 compute-entry skill。不要手动拆步骤调用；直接使用 `execute_skill`：
|
||
|
||
```json
|
||
{
|
||
"name": "vehicle-damage-inspection",
|
||
"input": {
|
||
"videoUrl": "/upload/20260504/car.mp4",
|
||
"fps": 3,
|
||
"topN": 1
|
||
}
|
||
}
|
||
```
|
||
|
||
## 输入
|
||
|
||
- `videoUrl`: 必填。视频本地路径或 `/upload/...` 路径；第一版不支持 http(s) URL。
|
||
- `taskId`: 可选。未传时自动生成。
|
||
- `fps`: 可选，默认 3。
|
||
- `quality`: 可选，默认 90。
|
||
- `batchSize`: 可选，默认 12。
|
||
- `concurrency`: 可选，默认 2。
|
||
- `groundingWindow`: 可选，默认 1.5 秒。
|
||
- `topN`: 可选，默认每处损伤 1 张最佳帧。
|
||
- `mode`: 可选，`full`、`frames-only` 或 `detect-only`。
|
||
|
||
## 输出
|
||
|
||
返回 JSON，包含 `success`、`taskId`、`workspacePath`、`summary`、`damages` 和 `artifacts`。
|
||
|
||
生产环境中，所有检测产物写入后端 dataDir 下的 `workspace/vehicle-damage-inspection/{taskId}`。
|
||
```
|
||
|
||
- [ ] **Step 2: 创建 skill.config.yaml**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/skill.config.yaml`:
|
||
|
||
```yaml
|
||
runtime: node
|
||
entrypoint: scripts/index.cjs
|
||
timeout: 900000
|
||
|
||
dependencies:
|
||
system:
|
||
- name: ffmpeg
|
||
check: "ffmpeg -version"
|
||
- name: ffprobe
|
||
check: "ffprobe -version"
|
||
- name: node
|
||
check: "node --version"
|
||
node:
|
||
packages:
|
||
- axios
|
||
- sharp
|
||
- fluent-ffmpeg
|
||
- "@ffmpeg-installer/ffmpeg"
|
||
- "@ffprobe-installer/ffprobe"
|
||
|
||
setup:
|
||
posix: scripts/make.sh fix
|
||
win32: scripts/setup.ps1
|
||
|
||
env:
|
||
- name: ARK_API_KEY
|
||
required: true
|
||
description: 火山方舟/豆包 OpenAI-compatible API Key
|
||
- name: ARK_API_URL
|
||
required: false
|
||
default: https://ark.cn-beijing.volces.com/api/v3/chat/completions
|
||
description: OpenAI-compatible chat completions endpoint
|
||
- name: DAMAGE_DETECT_MODEL
|
||
required: true
|
||
default: doubao-seed-2-0-pro-260215
|
||
description: 用于环车旧伤候选检测的多模态模型 ID
|
||
- name: DAMAGE_GROUNDING_MODEL
|
||
required: true
|
||
default: doubao-seed-2-0-pro-260215
|
||
description: 支持 bbox/grounding 输出的视觉模型 ID
|
||
- name: BEST_FRAME_MODEL
|
||
required: false
|
||
default: doubao-seed-2-0-pro-260215
|
||
description: 用于损伤去重和最佳帧筛选的模型 ID
|
||
- name: RZYX_AI_WORKSPACE_ROOT
|
||
required: false
|
||
description: 生产环境由宿主注入，默认 dataDir/workspace/vehicle-damage-inspection
|
||
- name: RZYX_AI_UPLOAD_ROOT
|
||
required: false
|
||
description: 生产环境由宿主注入，默认 dataDir/uploads
|
||
- name: RZYX_AI_DATA_DIR
|
||
required: false
|
||
description: 后端运行时 dataDir，由宿主注入
|
||
|
||
interface:
|
||
input:
|
||
videoUrl:
|
||
type: string
|
||
required: true
|
||
description: 视频本地路径或 /upload/... 路径；第一版不支持 http(s) URL
|
||
taskId:
|
||
type: string
|
||
required: false
|
||
description: 可选任务 ID
|
||
fps:
|
||
type: number
|
||
required: false
|
||
default: "3"
|
||
description: 抽帧帧率
|
||
quality:
|
||
type: number
|
||
required: false
|
||
default: "90"
|
||
description: 输出 JPG 质量，范围 1-100
|
||
batchSize:
|
||
type: number
|
||
required: false
|
||
default: "12"
|
||
description: 每批发送给检测模型的帧数
|
||
concurrency:
|
||
type: number
|
||
required: false
|
||
default: "2"
|
||
description: 多模态 API 并发数
|
||
groundingWindow:
|
||
type: number
|
||
required: false
|
||
default: "1.5"
|
||
description: 每个损伤时间点前后取帧窗口，单位秒
|
||
topN:
|
||
type: number
|
||
required: false
|
||
default: "1"
|
||
description: 每处损伤返回的最佳证据帧数量
|
||
mode:
|
||
type: string
|
||
required: false
|
||
default: full
|
||
description: full | frames-only | detect-only
|
||
output:
|
||
success:
|
||
type: boolean
|
||
taskId:
|
||
type: string
|
||
workspacePath:
|
||
type: string
|
||
summary:
|
||
type: object
|
||
damages:
|
||
type: array
|
||
artifacts:
|
||
type: object
|
||
```
|
||
|
||
- [ ] **Step 3: 创建 prompt 文件**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/prompts/damage_detect.md`:
|
||
|
||
```markdown
|
||
你是专业汽车环车视频旧伤检测专家。请检查给定视频帧中的车身外观可见区域，识别漆面旧伤，包括划痕、凹陷、掉漆、裂纹、锈蚀。
|
||
|
||
边界：
|
||
- 只把可见且明确的车身外观损伤列为 damages。
|
||
- 反光、污渍、阴影、雨水、压缩噪声、拍摄模糊不要直接判定为损伤。
|
||
- 玻璃、轮胎、车灯、车牌等非漆面问题如明显可见，可标记为 non_paint_damage，但不要混入 paint damage。
|
||
- 不确定时设置 uncertain=true，confidence="low"。
|
||
|
||
只输出 JSON：
|
||
{
|
||
"damages": [
|
||
{
|
||
"time_second": 12.4,
|
||
"part": "左前门",
|
||
"type": "划痕",
|
||
"severity": "轻微",
|
||
"description": "左前门中部可见细长白色划痕",
|
||
"confidence": "medium",
|
||
"uncertain": false
|
||
}
|
||
],
|
||
"summary": "本批检测简述"
|
||
}
|
||
```
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/prompts/grounding.md`:
|
||
|
||
```markdown
|
||
请在图中定位指定汽车外观损伤区域，输出 bbox。bbox 坐标使用 0-1000 归一化坐标，格式：
|
||
|
||
[12.40 second] <bbox>x1 y1 x2 y2</bbox>
|
||
|
||
如果某帧看不到该损伤，不要输出该帧 bbox。
|
||
```
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/prompts/dedup.md`:
|
||
|
||
```markdown
|
||
你是汽车旧伤记录去重专家。以下损伤记录可能来自连续视频帧中的同一处物理损伤。
|
||
|
||
合并规则：
|
||
- 位置相同或相近、类型相同，且时间相近，视为同一处物理损伤。
|
||
- 位置描述略有差异但明显指同一部位，也应合并。
|
||
- 不同部位或不同损伤类型不要合并。
|
||
|
||
只输出 JSON：
|
||
{
|
||
"groups": [
|
||
{
|
||
"merged_part": "左前门",
|
||
"merged_type": "划痕",
|
||
"merged_severity": "轻微",
|
||
"merged_description": "左前门中部细长划痕",
|
||
"member_indices": [0, 2]
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/prompts/best_frame.md`:
|
||
|
||
```markdown
|
||
以下是同一处汽车外观损伤的多张标注图。请选出最适合作为证据展示的帧。
|
||
|
||
优先级：
|
||
1. 红框准确框住损伤，不偏移、不过大、不遗漏。
|
||
2. 损伤区域清晰，不模糊、不遮挡。
|
||
3. 光照和角度能看清损伤全貌。
|
||
4. 避免反光、过曝、过暗。
|
||
|
||
只输出 JSON：
|
||
{
|
||
"best_timestamps": [12.4],
|
||
"reasons": ["红框准确，划痕清晰可见"]
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: 创建 README.md**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/README.md`:
|
||
|
||
```markdown
|
||
# vehicle-damage-inspection
|
||
|
||
生产形态：Windows 安装包中由后端复制到 `{dataDir}/skills/vehicle-damage-inspection` 后执行。
|
||
|
||
运行产物必须写入 `{dataDir}/workspace/vehicle-damage-inspection/{taskId}`，不要写入 skill 代码目录。
|
||
|
||
## 本地检查
|
||
|
||
```bash
|
||
bash scripts/make.sh check
|
||
```
|
||
|
||
## 本地运行
|
||
|
||
```bash
|
||
ARK_API_KEY=xxx bash scripts/make.sh run --video /path/to/car.mp4 --fps 3
|
||
```
|
||
|
||
## 必需模型
|
||
|
||
默认模型为 `doubao-seed-2-0-pro-260215`，用于检测、bbox 定位和最佳帧筛选。
|
||
```
|
||
|
||
- [ ] **Step 5: 验证 Skill Loader 能识别配置**
|
||
|
||
Run: `pnpm --filter @neta/backend test -- --runInBand test/entity_exports.test.ts`
|
||
|
||
Expected: PASS. This is a smoke check that backend tests still load.
|
||
|
||
- [ ] **Step 6: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection
|
||
git commit -m "feat(skills): scaffold vehicle damage inspection skill"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: JSON utilities and workspace module
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/lib/json_utils.cjs`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/lib/workspace.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs`
|
||
|
||
- [ ] **Step 1: 写 json_utils 测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const { parseJsonLoose, parseBboxes, clampNumber, runWithConcurrency } = require('../scripts/lib/json_utils.cjs');
|
||
|
||
(async () => {
|
||
assert.deepStrictEqual(parseJsonLoose('```json\n{"a":1}\n```'), { a: 1 });
|
||
assert.deepStrictEqual(parseBboxes('[1.0 second] <bbox>10 20 30 40</bbox>'), [
|
||
{ x1: 10, y1: 20, x2: 30, y2: 40 },
|
||
]);
|
||
assert.strictEqual(clampNumber('9', 1, 5, 3), 5);
|
||
assert.strictEqual(clampNumber(undefined, 1, 5, 3), 3);
|
||
|
||
let active = 0;
|
||
let maxActive = 0;
|
||
const tasks = Array.from({ length: 5 }, (_, i) => async () => {
|
||
active += 1;
|
||
maxActive = Math.max(maxActive, active);
|
||
await new Promise(resolve => setTimeout(resolve, 5));
|
||
active -= 1;
|
||
return i;
|
||
});
|
||
const result = await runWithConcurrency(tasks, 2);
|
||
assert.deepStrictEqual(result, [0, 1, 2, 3, 4]);
|
||
assert.ok(maxActive <= 2);
|
||
})();
|
||
```
|
||
|
||
- [ ] **Step 2: 写 workspace 测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const fs = require('node:fs');
|
||
const os = require('node:os');
|
||
const path = require('node:path');
|
||
const { createWorkspace, writeJson, readJson, resolveVideoPath } = require('../scripts/lib/workspace.cjs');
|
||
|
||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'vehicle-skill-workspace-'));
|
||
const uploads = path.join(root, 'uploads');
|
||
fs.mkdirSync(path.join(uploads, '20260504'), { recursive: true });
|
||
const video = path.join(uploads, '20260504', 'car.mp4');
|
||
fs.writeFileSync(video, 'fake');
|
||
|
||
try {
|
||
const ws = createWorkspace({ taskId: 'task-a', workspaceRoot: path.join(root, 'workspace') });
|
||
assert.strictEqual(ws.taskId, 'task-a');
|
||
assert.ok(fs.existsSync(path.join(ws.workspacePath, 'frames')));
|
||
writeJson(ws, 'video_info.json', { ok: true });
|
||
assert.deepStrictEqual(readJson(ws, 'video_info.json'), { ok: true });
|
||
assert.strictEqual(resolveVideoPath('/upload/20260504/car.mp4', { uploadRoot: uploads }), video);
|
||
assert.throws(() => resolveVideoPath('/upload/../secret.mp4', { uploadRoot: uploads }), /Invalid upload path/);
|
||
assert.throws(() => resolveVideoPath('https://example.test/car.mp4', { uploadRoot: uploads }), /Remote video URLs are not supported/);
|
||
} finally {
|
||
fs.rmSync(root, { recursive: true, force: true });
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 3: 运行测试确认失败**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
node packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs
|
||
node packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs
|
||
```
|
||
|
||
Expected: both FAIL with missing module errors.
|
||
|
||
- [ ] **Step 4: 实现 json_utils.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/lib/json_utils.cjs`:
|
||
|
||
```javascript
|
||
function parseJsonLoose(text) {
|
||
if (typeof text !== 'string') return null;
|
||
let value = text.trim();
|
||
if (value.startsWith('```')) {
|
||
value = value.split('\n').filter(line => !line.trim().startsWith('```')).join('\n').trim();
|
||
}
|
||
try {
|
||
return JSON.parse(value);
|
||
} catch {
|
||
const start = value.indexOf('{');
|
||
const end = value.lastIndexOf('}');
|
||
if (start >= 0 && end > start) {
|
||
try {
|
||
return JSON.parse(value.slice(start, end + 1));
|
||
} catch {
|
||
return null;
|
||
}
|
||
}
|
||
return null;
|
||
}
|
||
}
|
||
|
||
function parseBboxes(text) {
|
||
const result = [];
|
||
const pattern = /<bbox>\s*(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s*<\/bbox>/g;
|
||
let match;
|
||
while ((match = pattern.exec(String(text || ''))) !== null) {
|
||
result.push({
|
||
x1: Number(match[1]),
|
||
y1: Number(match[2]),
|
||
x2: Number(match[3]),
|
||
y2: Number(match[4]),
|
||
});
|
||
}
|
||
return result;
|
||
}
|
||
|
||
function clampNumber(value, min, max, fallback) {
|
||
const n = Number(value);
|
||
if (!Number.isFinite(n)) return fallback;
|
||
return Math.max(min, Math.min(max, n));
|
||
}
|
||
|
||
function sleep(ms) {
|
||
return new Promise(resolve => setTimeout(resolve, ms));
|
||
}
|
||
|
||
async function runWithConcurrency(tasks, limit) {
|
||
const results = new Array(tasks.length);
|
||
let nextIndex = 0;
|
||
async function worker() {
|
||
while (nextIndex < tasks.length) {
|
||
const index = nextIndex++;
|
||
results[index] = await tasks[index]();
|
||
}
|
||
}
|
||
await Promise.all(Array.from({ length: Math.min(Math.max(1, limit), tasks.length) }, () => worker()));
|
||
return results;
|
||
}
|
||
|
||
module.exports = {
|
||
parseJsonLoose,
|
||
parseBboxes,
|
||
clampNumber,
|
||
sleep,
|
||
runWithConcurrency,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 5: 实现 workspace.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/lib/workspace.cjs`:
|
||
|
||
```javascript
|
||
const fs = require('node:fs');
|
||
const path = require('node:path');
|
||
const crypto = require('node:crypto');
|
||
|
||
function sanitizeTaskId(taskId) {
|
||
const id = taskId || `${new Date().toISOString().replace(/[-:.TZ]/g, '').slice(0, 14)}-${crypto.randomUUID().slice(0, 8)}`;
|
||
if (!/^[a-zA-Z0-9_-]+$/.test(id)) throw new Error(`Invalid taskId: ${id}`);
|
||
return id;
|
||
}
|
||
|
||
function createWorkspace({ taskId, workspaceRoot }) {
|
||
const root = path.resolve(workspaceRoot);
|
||
const id = sanitizeTaskId(taskId);
|
||
const workspacePath = path.join(root, id);
|
||
fs.mkdirSync(path.join(workspacePath, 'source'), { recursive: true });
|
||
fs.mkdirSync(path.join(workspacePath, 'frames'), { recursive: true });
|
||
fs.mkdirSync(path.join(workspacePath, 'marked_frames'), { recursive: true });
|
||
return { taskId: id, root, workspacePath };
|
||
}
|
||
|
||
function ensureInside(root, candidate) {
|
||
const rootPath = path.resolve(root);
|
||
const fullPath = path.resolve(candidate);
|
||
if (fullPath !== rootPath && !fullPath.startsWith(rootPath + path.sep)) {
|
||
throw new Error(`Path escapes root: ${candidate}`);
|
||
}
|
||
return fullPath;
|
||
}
|
||
|
||
function writeJson(workspace, relativePath, value) {
|
||
const filePath = ensureInside(workspace.workspacePath, path.join(workspace.workspacePath, relativePath));
|
||
fs.mkdirSync(path.dirname(filePath), { recursive: true });
|
||
fs.writeFileSync(filePath, JSON.stringify(value, null, 2), 'utf8');
|
||
return filePath;
|
||
}
|
||
|
||
function readJson(workspace, relativePath) {
|
||
const filePath = ensureInside(workspace.workspacePath, path.join(workspace.workspacePath, relativePath));
|
||
if (!fs.existsSync(filePath)) return null;
|
||
return JSON.parse(fs.readFileSync(filePath, 'utf8'));
|
||
}
|
||
|
||
function resolveVideoPath(videoUrl, options) {
|
||
if (!videoUrl || typeof videoUrl !== 'string') throw new Error('videoUrl is required');
|
||
if (fs.existsSync(videoUrl)) return path.resolve(videoUrl);
|
||
|
||
const uploadMatch = videoUrl.match(/^\/upload\/(.+)$/);
|
||
if (uploadMatch) {
|
||
if (!options.uploadRoot) throw new Error('RZYX_AI_UPLOAD_ROOT is required for /upload paths');
|
||
const relative = uploadMatch[1].replace(/[\\/]+/g, path.sep);
|
||
if (relative.split(path.sep).includes('..')) throw new Error('Invalid upload path');
|
||
const filePath = ensureInside(options.uploadRoot, path.join(options.uploadRoot, relative));
|
||
if (!fs.existsSync(filePath)) throw new Error(`Video file not found: ${filePath}`);
|
||
return filePath;
|
||
}
|
||
|
||
if (/^https?:\/\//i.test(videoUrl)) {
|
||
throw new Error('Remote video URLs are not supported in the first version; use a local path or /upload/... path');
|
||
}
|
||
throw new Error(`Video file not found: ${videoUrl}`);
|
||
}
|
||
|
||
module.exports = {
|
||
createWorkspace,
|
||
writeJson,
|
||
readJson,
|
||
resolveVideoPath,
|
||
ensureInside,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 6: 运行测试确认通过**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
node packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs
|
||
node packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs
|
||
```
|
||
|
||
Expected: both commands exit 0.
|
||
|
||
- [ ] **Step 7: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/json_utils.cjs packages/backend/skills/vehicle-damage-inspection/scripts/lib/workspace.cjs packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs
|
||
git commit -m "feat(skills): add vehicle inspection workspace utilities"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Frame extraction module
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/lib/frame_extractor.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs`
|
||
|
||
- [ ] **Step 1: 写 ffmpeg command builder 测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const { buildFrameName, qualityToFfmpegQv, estimateFrameTimestamp } = require('../scripts/lib/frame_extractor.cjs');
|
||
|
||
assert.strictEqual(buildFrameName(1, 0), 'frame_000001_0.00s.jpg');
|
||
assert.strictEqual(buildFrameName(38, 12.4), 'frame_000038_12.40s.jpg');
|
||
assert.strictEqual(qualityToFfmpegQv(100), 1);
|
||
assert.strictEqual(qualityToFfmpegQv(1), 31);
|
||
assert.strictEqual(estimateFrameTimestamp(6, 3), 2);
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs`
|
||
|
||
Expected: FAIL with missing module.
|
||
|
||
- [ ] **Step 3: 实现 frame_extractor.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/lib/frame_extractor.cjs`:
|
||
|
||
```javascript
|
||
const fs = require('node:fs');
|
||
const path = require('node:path');
|
||
const ffmpeg = require('fluent-ffmpeg');
|
||
const ffmpegInstaller = require('@ffmpeg-installer/ffmpeg');
|
||
const ffprobeInstaller = require('@ffprobe-installer/ffprobe');
|
||
const { writeJson } = require('./workspace.cjs');
|
||
|
||
ffmpeg.setFfmpegPath(process.env.FFMPEG_PATH || ffmpegInstaller.path);
|
||
ffmpeg.setFfprobePath(process.env.FFPROBE_PATH || ffprobeInstaller.path);
|
||
|
||
function buildFrameName(index, timestamp) {
|
||
return `frame_${String(index).padStart(6, '0')}_${Number(timestamp).toFixed(2)}s.jpg`;
|
||
}
|
||
|
||
function qualityToFfmpegQv(quality) {
|
||
const q = Math.max(1, Math.min(100, Number(quality) || 90));
|
||
return Math.max(1, Math.min(31, Math.round(31 - (q / 100) * 30)));
|
||
}
|
||
|
||
function estimateFrameTimestamp(indexZeroBased, fps) {
|
||
return Number((indexZeroBased / fps).toFixed(2));
|
||
}
|
||
|
||
function probeVideo(videoPath) {
|
||
return new Promise((resolve, reject) => {
|
||
ffmpeg.ffprobe(videoPath, (err, data) => err ? reject(err) : resolve(data));
|
||
});
|
||
}
|
||
|
||
async function extractFrames({ workspace, videoPath, fps, quality }) {
|
||
const framesDir = path.join(workspace.workspacePath, 'frames');
|
||
fs.rmSync(framesDir, { recursive: true, force: true });
|
||
fs.mkdirSync(framesDir, { recursive: true });
|
||
|
||
const probe = await probeVideo(videoPath);
|
||
const stream = (probe.streams || []).find(item => item.codec_type === 'video');
|
||
if (!stream) throw new Error('No video stream found');
|
||
|
||
const duration = Number(probe.format && probe.format.duration ? probe.format.duration : 0);
|
||
const width = stream.width || 0;
|
||
const height = stream.height || 0;
|
||
const fpsValue = Number(fps) || 3;
|
||
const qv = qualityToFfmpegQv(quality);
|
||
const tempPattern = path.join(framesDir, 'raw_%06d.jpg');
|
||
|
||
await new Promise((resolve, reject) => {
|
||
ffmpeg(videoPath)
|
||
.outputOptions(['-vf', `fps=${fpsValue}`, '-q:v', String(qv)])
|
||
.output(tempPattern)
|
||
.on('end', resolve)
|
||
.on('error', reject)
|
||
.run();
|
||
});
|
||
|
||
const rawFiles = fs.readdirSync(framesDir).filter(name => /^raw_\d+\.jpg$/.test(name)).sort();
|
||
if (rawFiles.length === 0) throw new Error('No frames extracted');
|
||
|
||
const frames = rawFiles.map((name, index) => {
|
||
const timestamp = estimateFrameTimestamp(index, fpsValue);
|
||
const finalName = buildFrameName(index + 1, timestamp);
|
||
fs.renameSync(path.join(framesDir, name), path.join(framesDir, finalName));
|
||
return {
|
||
index,
|
||
timestamp,
|
||
relativePath: `frames/${finalName}`,
|
||
path: path.join(framesDir, finalName),
|
||
};
|
||
});
|
||
|
||
const videoInfo = {
|
||
duration: Number(duration.toFixed(2)),
|
||
videoFps: stream.r_frame_rate || '',
|
||
resolution: `${width}x${height}`,
|
||
extractFps: fpsValue,
|
||
extractedFrames: frames.length,
|
||
};
|
||
|
||
writeJson(workspace, 'video_info.json', {
|
||
videoPath,
|
||
videoInfo,
|
||
frames: frames.map(({ index, timestamp, relativePath }) => ({ index, timestamp, relativePath })),
|
||
});
|
||
|
||
return { videoInfo, frames };
|
||
}
|
||
|
||
module.exports = {
|
||
buildFrameName,
|
||
qualityToFfmpegQv,
|
||
estimateFrameTimestamp,
|
||
extractFrames,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs`
|
||
|
||
Expected: exits 0.
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/frame_extractor.cjs packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs
|
||
git commit -m "feat(skills): add vehicle video frame extractor"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Vision client module
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/lib/vision_client.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs`
|
||
|
||
- [ ] **Step 1: 写 vision client 纯函数测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const fs = require('node:fs');
|
||
const os = require('node:os');
|
||
const path = require('node:path');
|
||
const { resolveModelConfig, encodeImageDataUri, createTimestampedImageContent } = require('../scripts/lib/vision_client.cjs');
|
||
|
||
const env = {
|
||
ARK_API_KEY: 'key',
|
||
ARK_API_URL: 'https://example.test/v1/chat/completions',
|
||
DAMAGE_DETECT_MODEL: 'detect-model',
|
||
DAMAGE_GROUNDING_MODEL: 'ground-model',
|
||
};
|
||
|
||
assert.deepStrictEqual(resolveModelConfig(env), {
|
||
apiKey: 'key',
|
||
apiUrl: 'https://example.test/v1/chat/completions',
|
||
detectModel: 'detect-model',
|
||
groundingModel: 'ground-model',
|
||
bestFrameModel: 'detect-model',
|
||
});
|
||
|
||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vision-client-'));
|
||
try {
|
||
const img = path.join(dir, 'a.jpg');
|
||
fs.writeFileSync(img, Buffer.from([1, 2, 3]));
|
||
assert.strictEqual(encodeImageDataUri(img), 'data:image/jpeg;base64,AQID');
|
||
const content = createTimestampedImageContent([{ timestamp: 1.25, path: img }]);
|
||
assert.strictEqual(content[0].text, '[1.25 second]');
|
||
assert.ok(content[1].image_url.url.startsWith('data:image/jpeg;base64,'));
|
||
} finally {
|
||
fs.rmSync(dir, { recursive: true, force: true });
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs`
|
||
|
||
Expected: FAIL with missing module.
|
||
|
||
- [ ] **Step 3: 实现 vision_client.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/lib/vision_client.cjs`:
|
||
|
||
```javascript
|
||
const fs = require('node:fs');
|
||
const axios = require('axios');
|
||
const { sleep } = require('./json_utils.cjs');
|
||
|
||
function resolveModelConfig(env) {
|
||
const apiKey = env.ARK_API_KEY;
|
||
if (!apiKey) throw new Error('Missing required env: ARK_API_KEY');
|
||
const apiUrl = env.ARK_API_URL || 'https://ark.cn-beijing.volces.com/api/v3/chat/completions';
|
||
const detectModel = env.DAMAGE_DETECT_MODEL || 'doubao-seed-2-0-pro-260215';
|
||
const groundingModel = env.DAMAGE_GROUNDING_MODEL || detectModel;
|
||
const bestFrameModel = env.BEST_FRAME_MODEL || detectModel;
|
||
return { apiKey, apiUrl, detectModel, groundingModel, bestFrameModel };
|
||
}
|
||
|
||
function encodeImageDataUri(filePath) {
|
||
const buffer = fs.readFileSync(filePath);
|
||
return `data:image/jpeg;base64,${buffer.toString('base64')}`;
|
||
}
|
||
|
||
function createTimestampedImageContent(frames) {
|
||
const content = [];
|
||
for (const frame of frames) {
|
||
content.push({ type: 'text', text: `[${Number(frame.timestamp).toFixed(2)} second]` });
|
||
content.push({ type: 'image_url', image_url: { url: encodeImageDataUri(frame.path) } });
|
||
}
|
||
return content;
|
||
}
|
||
|
||
async function callVisionApi({ config, model, messages, maxTokens = 4096, temperature = 0.1, timeoutMs = 600000, maxRetries = 3 }) {
|
||
let lastError = '';
|
||
for (let attempt = 0; attempt <= maxRetries; attempt += 1) {
|
||
try {
|
||
const response = await axios.post(
|
||
config.apiUrl,
|
||
{ model, messages, max_tokens: maxTokens, temperature },
|
||
{
|
||
headers: {
|
||
Authorization: `Bearer ${config.apiKey}`,
|
||
'Content-Type': 'application/json',
|
||
},
|
||
timeout: timeoutMs,
|
||
maxBodyLength: Infinity,
|
||
maxContentLength: Infinity,
|
||
},
|
||
);
|
||
return {
|
||
ok: true,
|
||
content: response.data && response.data.choices && response.data.choices[0] && response.data.choices[0].message
|
||
? response.data.choices[0].message.content || ''
|
||
: '',
|
||
usage: response.data ? response.data.usage : undefined,
|
||
};
|
||
} catch (error) {
|
||
const status = error.response && error.response.status;
|
||
const body = error.response && error.response.data ? JSON.stringify(error.response.data).slice(0, 500) : error.message;
|
||
lastError = `API调用失败(${status || 'network'}): ${body}`;
|
||
const retryable = status === 429 || status === 500 || status === 502 || status === 503 || status === 504;
|
||
if (retryable && attempt < maxRetries) {
|
||
await sleep(1000 * Math.pow(2, attempt));
|
||
continue;
|
||
}
|
||
break;
|
||
}
|
||
}
|
||
return { ok: false, error: lastError };
|
||
}
|
||
|
||
module.exports = {
|
||
resolveModelConfig,
|
||
encodeImageDataUri,
|
||
createTimestampedImageContent,
|
||
callVisionApi,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs`
|
||
|
||
Expected: exits 0.
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/vision_client.cjs packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs
|
||
git commit -m "feat(skills): add doubao vision client for vehicle inspection"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: Damage candidate detector
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_detector.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs`
|
||
|
||
- [ ] **Step 1: 写 mock 检测测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const fs = require('node:fs');
|
||
const os = require('node:os');
|
||
const path = require('node:path');
|
||
const { createWorkspace, readJson } = require('../scripts/lib/workspace.cjs');
|
||
const { detectDamageCandidates } = require('../scripts/lib/damage_detector.cjs');
|
||
|
||
(async () => {
|
||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'damage-detector-'));
|
||
try {
|
||
const workspace = createWorkspace({ taskId: 'task-detect', workspaceRoot: root });
|
||
const framePath = path.join(workspace.workspacePath, 'frames', 'frame_000001_0.00s.jpg');
|
||
fs.writeFileSync(framePath, Buffer.from([1, 2, 3]));
|
||
const frames = [{ index: 0, timestamp: 0, path: framePath, relativePath: 'frames/frame_000001_0.00s.jpg' }];
|
||
const mockVision = async () => ({
|
||
ok: true,
|
||
content: JSON.stringify({
|
||
damages: [{ time_second: 0, part: '左前门', type: '划痕', severity: '轻微', description: '细长划痕', confidence: 'medium' }],
|
||
summary: '发现一处损伤',
|
||
}),
|
||
});
|
||
|
||
const result = await detectDamageCandidates({
|
||
workspace,
|
||
frames,
|
||
modelConfig: { detectModel: 'm', apiKey: 'k', apiUrl: 'u' },
|
||
prompt: 'prompt',
|
||
batchSize: 1,
|
||
concurrency: 1,
|
||
callVisionApi: mockVision,
|
||
});
|
||
|
||
assert.strictEqual(result.candidates.length, 1);
|
||
assert.strictEqual(result.candidates[0].id, 'cand_001');
|
||
assert.strictEqual(readJson(workspace, 'damage_candidates.json').candidates.length, 1);
|
||
} finally {
|
||
fs.rmSync(root, { recursive: true, force: true });
|
||
}
|
||
})();
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs`
|
||
|
||
Expected: FAIL with missing module.
|
||
|
||
- [ ] **Step 3: 实现 damage_detector.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_detector.cjs`:
|
||
|
||
```javascript
|
||
const { parseJsonLoose, runWithConcurrency } = require('./json_utils.cjs');
|
||
const { createTimestampedImageContent, callVisionApi: defaultCallVisionApi } = require('./vision_client.cjs');
|
||
const { writeJson } = require('./workspace.cjs');
|
||
|
||
function normalizeCandidate(raw, index, batchNumber) {
|
||
return {
|
||
id: `cand_${String(index + 1).padStart(3, '0')}`,
|
||
timestamp: Number(raw.time_second ?? raw.timestamp ?? 0),
|
||
part: raw.part || raw.location || '',
|
||
type: raw.type || '',
|
||
severity: raw.severity || '',
|
||
description: raw.description || '',
|
||
confidence: raw.confidence || 'medium',
|
||
uncertain: Boolean(raw.uncertain),
|
||
batch: batchNumber,
|
||
};
|
||
}
|
||
|
||
async function detectDamageCandidates(params) {
|
||
const {
|
||
workspace,
|
||
frames,
|
||
modelConfig,
|
||
prompt,
|
||
batchSize,
|
||
concurrency,
|
||
callVisionApi = defaultCallVisionApi,
|
||
} = params;
|
||
|
||
const batches = [];
|
||
for (let start = 0; start < frames.length; start += batchSize) {
|
||
batches.push(frames.slice(start, start + batchSize));
|
||
}
|
||
|
||
const batchRecords = [];
|
||
const collected = [];
|
||
|
||
await runWithConcurrency(batches.map((batch, batchIndex) => async () => {
|
||
const content = [
|
||
{ type: 'text', text: prompt },
|
||
...createTimestampedImageContent(batch),
|
||
{ type: 'text', text: '请只输出 JSON。' },
|
||
];
|
||
const result = await callVisionApi({
|
||
config: modelConfig,
|
||
model: modelConfig.detectModel,
|
||
messages: [{ role: 'user', content }],
|
||
});
|
||
const record = {
|
||
batch: batchIndex + 1,
|
||
frameStart: batch[0].index,
|
||
frameEnd: batch[batch.length - 1].index,
|
||
status: result.ok ? 'success' : 'error',
|
||
error: result.ok ? undefined : result.error,
|
||
};
|
||
batchRecords[batchIndex] = record;
|
||
if (!result.ok) return;
|
||
const parsed = parseJsonLoose(result.content || '');
|
||
const damages = Array.isArray(parsed && parsed.damages) ? parsed.damages : [];
|
||
for (const damage of damages) {
|
||
collected.push({ damage, batchNumber: batchIndex + 1 });
|
||
}
|
||
}), concurrency);
|
||
|
||
const candidates = collected.map((item, index) => normalizeCandidate(item.damage, index, item.batchNumber));
|
||
const output = {
|
||
totalFrames: frames.length,
|
||
batches: batchRecords,
|
||
candidates,
|
||
};
|
||
writeJson(workspace, 'damage_candidates.json', output);
|
||
return output;
|
||
}
|
||
|
||
module.exports = {
|
||
detectDamageCandidates,
|
||
normalizeCandidate,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs`
|
||
|
||
Expected: exits 0.
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_detector.cjs packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs
|
||
git commit -m "feat(skills): detect vehicle damage candidates from frames"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: Image marker and damage grounding
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/lib/image_marker.cjs`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_grounding.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs`
|
||
|
||
- [ ] **Step 1: 写 image marker 测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const { normalizedToPixel } = require('../scripts/lib/image_marker.cjs');
|
||
|
||
assert.deepStrictEqual(
|
||
normalizedToPixel({ x1: 100, y1: 200, x2: 300, y2: 400 }, { width: 1000, height: 500 }),
|
||
{ x1: 100, y1: 100, x2: 300, y2: 200 },
|
||
);
|
||
```
|
||
|
||
- [ ] **Step 2: 写 grounding mock 测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const fs = require('node:fs');
|
||
const os = require('node:os');
|
||
const path = require('node:path');
|
||
const { createWorkspace, readJson } = require('../scripts/lib/workspace.cjs');
|
||
const { groundDamages } = require('../scripts/lib/damage_grounding.cjs');
|
||
|
||
(async () => {
|
||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'damage-grounding-'));
|
||
try {
|
||
const workspace = createWorkspace({ taskId: 'task-ground', workspaceRoot: root });
|
||
const framePath = path.join(workspace.workspacePath, 'frames', 'frame_000001_0.00s.jpg');
|
||
fs.writeFileSync(framePath, Buffer.from([1, 2, 3]));
|
||
const frames = [{ index: 0, timestamp: 0, path: framePath, relativePath: 'frames/frame_000001_0.00s.jpg' }];
|
||
const candidates = [{ id: 'cand_001', timestamp: 0, part: '左前门', type: '划痕', severity: '轻微', description: '细长划痕' }];
|
||
const mockVision = async () => ({ ok: true, content: '[0.00 second] <bbox>10 20 30 40</bbox>' });
|
||
const mockMarker = async ({ dstPath }) => {
|
||
fs.writeFileSync(dstPath, 'marked');
|
||
return true;
|
||
};
|
||
|
||
const result = await groundDamages({
|
||
workspace,
|
||
frames,
|
||
candidates,
|
||
modelConfig: { groundingModel: 'm', apiKey: 'k', apiUrl: 'u' },
|
||
prompt: 'prompt',
|
||
groundingWindow: 1.5,
|
||
callVisionApi: mockVision,
|
||
drawBbox: mockMarker,
|
||
});
|
||
|
||
assert.strictEqual(result.annotations.length, 1);
|
||
assert.strictEqual(result.annotations[0].markedFrames.length, 1);
|
||
assert.strictEqual(readJson(workspace, 'damage_annotations.json').annotations.length, 1);
|
||
} finally {
|
||
fs.rmSync(root, { recursive: true, force: true });
|
||
}
|
||
})();
|
||
```
|
||
|
||
- [ ] **Step 3: 运行测试确认失败**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
node packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs
|
||
node packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs
|
||
```
|
||
|
||
Expected: both FAIL with missing module.
|
||
|
||
- [ ] **Step 4: 实现 image_marker.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/lib/image_marker.cjs`:
|
||
|
||
```javascript
|
||
const fs = require('node:fs');
|
||
const path = require('node:path');
|
||
const sharp = require('sharp');
|
||
|
||
function normalizedToPixel(box, metadata) {
|
||
const width = metadata.width || 1;
|
||
const height = metadata.height || 1;
|
||
return {
|
||
x1: Math.round((box.x1 * width) / 1000),
|
||
y1: Math.round((box.y1 * height) / 1000),
|
||
x2: Math.round((box.x2 * width) / 1000),
|
||
y2: Math.round((box.y2 * height) / 1000),
|
||
};
|
||
}
|
||
|
||
async function drawBbox({ srcPath, dstPath, bboxes, label }) {
|
||
fs.mkdirSync(path.dirname(dstPath), { recursive: true });
|
||
const metadata = await sharp(srcPath).metadata();
|
||
const width = metadata.width || 1;
|
||
const height = metadata.height || 1;
|
||
const parts = [];
|
||
for (const box of bboxes) {
|
||
const pixel = normalizedToPixel(box, { width, height });
|
||
const rectWidth = Math.max(1, pixel.x2 - pixel.x1);
|
||
const rectHeight = Math.max(1, pixel.y2 - pixel.y1);
|
||
parts.push(`<rect x="${pixel.x1}" y="${pixel.y1}" width="${rectWidth}" height="${rectHeight}" fill="none" stroke="red" stroke-width="4"/>`);
|
||
if (label) {
|
||
const safeLabel = String(label).replace(/[<>&"]/g, '');
|
||
parts.push(`<text x="${pixel.x1}" y="${Math.max(18, pixel.y1 - 6)}" font-size="18" fill="red">${safeLabel}</text>`);
|
||
}
|
||
}
|
||
const overlay = Buffer.from(`<svg width="${width}" height="${height}" xmlns="http://www.w3.org/2000/svg">${parts.join('')}</svg>`);
|
||
await sharp(srcPath).composite([{ input: overlay, top: 0, left: 0 }]).jpeg({ quality: 90 }).toFile(dstPath);
|
||
return true;
|
||
}
|
||
|
||
module.exports = {
|
||
normalizedToPixel,
|
||
drawBbox,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 5: 实现 damage_grounding.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_grounding.cjs`:
|
||
|
||
```javascript
|
||
const fs = require('node:fs');
|
||
const path = require('node:path');
|
||
const { parseBboxes } = require('./json_utils.cjs');
|
||
const { createTimestampedImageContent, callVisionApi: defaultCallVisionApi } = require('./vision_client.cjs');
|
||
const { drawBbox: defaultDrawBbox, normalizedToPixel } = require('./image_marker.cjs');
|
||
const { writeJson } = require('./workspace.cjs');
|
||
|
||
function selectNearbyFrames(frames, timestamp, windowSeconds) {
|
||
return frames.filter(frame => Math.abs(Number(frame.timestamp) - Number(timestamp)) <= windowSeconds);
|
||
}
|
||
|
||
async function groundDamages(params) {
|
||
const {
|
||
workspace,
|
||
frames,
|
||
candidates,
|
||
modelConfig,
|
||
prompt,
|
||
groundingWindow,
|
||
callVisionApi = defaultCallVisionApi,
|
||
drawBbox = defaultDrawBbox,
|
||
} = params;
|
||
|
||
const annotations = [];
|
||
for (let i = 0; i < candidates.length; i += 1) {
|
||
const candidate = candidates[i];
|
||
const nearbyFrames = selectNearbyFrames(frames, candidate.timestamp, groundingWindow);
|
||
const content = [
|
||
...createTimestampedImageContent(nearbyFrames),
|
||
{ type: 'text', text: `${prompt}\n损伤描述：${candidate.part} ${candidate.type} ${candidate.description}` },
|
||
];
|
||
const result = await callVisionApi({
|
||
config: modelConfig,
|
||
model: modelConfig.groundingModel,
|
||
messages: [{ role: 'user', content }],
|
||
maxTokens: 2048,
|
||
temperature: 0.1,
|
||
});
|
||
const bboxes = result.ok ? parseBboxes(result.content || '') : [];
|
||
const markedFrames = [];
|
||
for (const frame of nearbyFrames) {
|
||
const markedName = `dmg_${String(i + 1).padStart(3, '0')}_${Number(frame.timestamp).toFixed(2)}s.jpg`;
|
||
const dstPath = path.join(workspace.workspacePath, 'marked_frames', markedName);
|
||
if (bboxes.length > 0) {
|
||
await drawBbox({ srcPath: frame.path, dstPath, bboxes, label: `${candidate.part}-${candidate.type}` });
|
||
} else {
|
||
fs.copyFileSync(frame.path, dstPath);
|
||
}
|
||
const pixel = bboxes[0] ? normalizedToPixel(bboxes[0], { width: 1000, height: 1000 }) : null;
|
||
markedFrames.push({
|
||
timestamp: frame.timestamp,
|
||
sourceRelativePath: frame.relativePath,
|
||
markedRelativePath: `marked_frames/${markedName}`,
|
||
path: dstPath,
|
||
bbox: bboxes[0] ? { normalized: bboxes[0], pixel } : null,
|
||
groundingStatus: bboxes.length > 0 ? 'bbox' : 'no_bbox',
|
||
groundingRaw: result.ok ? result.content : result.error,
|
||
});
|
||
}
|
||
annotations.push({
|
||
candidateId: candidate.id,
|
||
damageId: `dmg_${String(i + 1).padStart(3, '0')}`,
|
||
part: candidate.part,
|
||
type: candidate.type,
|
||
severity: candidate.severity,
|
||
description: candidate.description,
|
||
markedFrames,
|
||
});
|
||
}
|
||
|
||
const output = { annotations };
|
||
writeJson(workspace, 'damage_annotations.json', output);
|
||
return output;
|
||
}
|
||
|
||
module.exports = {
|
||
selectNearbyFrames,
|
||
groundDamages,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 6: 运行测试确认通过**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
node packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs
|
||
node packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs
|
||
```
|
||
|
||
Expected: both commands exit 0.
|
||
|
||
- [ ] **Step 7: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/image_marker.cjs packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_grounding.cjs packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs
|
||
git commit -m "feat(skills): ground and mark vehicle damage frames"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 9: Best frame selector
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/lib/best_frame_selector.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs`
|
||
|
||
- [ ] **Step 1: 写最佳帧筛选测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const fs = require('node:fs');
|
||
const os = require('node:os');
|
||
const path = require('node:path');
|
||
const { createWorkspace, readJson } = require('../scripts/lib/workspace.cjs');
|
||
const { heuristicFrameScore, selectBestFrames, buildBestFramePrompt } = require('../scripts/lib/best_frame_selector.cjs');
|
||
|
||
(async () => {
|
||
assert.ok(heuristicFrameScore({ bbox: { normalized: { x1: 100, y1: 100, x2: 300, y2: 300 } } }) > heuristicFrameScore({ bbox: null }));
|
||
|
||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'best-frame-'));
|
||
try {
|
||
const workspace = createWorkspace({ taskId: 'task-best', workspaceRoot: root });
|
||
const annotations = [{
|
||
damageId: 'dmg_001',
|
||
part: '左前门',
|
||
type: '划痕',
|
||
severity: '轻微',
|
||
description: '细长划痕',
|
||
markedFrames: [
|
||
{ timestamp: 0, markedRelativePath: 'marked_frames/a.jpg', path: 'a.jpg', bbox: null },
|
||
{ timestamp: 1, markedRelativePath: 'marked_frames/b.jpg', path: 'b.jpg', bbox: { normalized: { x1: 100, y1: 100, x2: 300, y2: 300 } } },
|
||
],
|
||
}];
|
||
const result = await selectBestFrames({
|
||
workspace,
|
||
annotations,
|
||
topN: 1,
|
||
modelConfig: { bestFrameModel: 'doubao-seed-2-0-pro-260215' },
|
||
prompt: '选择最佳证据帧',
|
||
callVisionApi: async () => ({ ok: true, content: '{"best_timestamps":[1],"reasons":["bbox准确"]}' }),
|
||
});
|
||
assert.ok(buildBestFramePrompt(annotations[0], '选择最佳证据帧').includes('dmg_001'));
|
||
assert.strictEqual(result.damages[0].bestFrames[0].timestamp, 1);
|
||
assert.strictEqual(result.damages[0].bestFrames[0].selectionReason, 'bbox准确');
|
||
assert.strictEqual(readJson(workspace, 'best_frames.json').bestFrameCount, 1);
|
||
} finally {
|
||
fs.rmSync(root, { recursive: true, force: true });
|
||
}
|
||
})();
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs`
|
||
|
||
Expected: FAIL with missing module.
|
||
|
||
- [ ] **Step 3: 实现 best_frame_selector.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/lib/best_frame_selector.cjs`:
|
||
|
||
```javascript
|
||
const { writeJson } = require('./workspace.cjs');
|
||
const { parseJsonLoose } = require('./json_utils.cjs');
|
||
const { callVisionApi: defaultCallVisionApi } = require('./vision_client.cjs');
|
||
|
||
function heuristicFrameScore(frame) {
|
||
let score = 0;
|
||
if (frame.bbox && frame.bbox.normalized) {
|
||
score += 100;
|
||
const box = frame.bbox.normalized;
|
||
const area = Math.max(0, box.x2 - box.x1) * Math.max(0, box.y2 - box.y1);
|
||
if (area >= 10000 && area <= 250000) score += 30;
|
||
if (box.x1 > 10 && box.y1 > 10 && box.x2 < 990 && box.y2 < 990) score += 20;
|
||
}
|
||
return score;
|
||
}
|
||
|
||
function dedupeByTimestamp(frames) {
|
||
const map = new Map();
|
||
for (const frame of frames) {
|
||
const key = Number(frame.timestamp).toFixed(2);
|
||
const existing = map.get(key);
|
||
if (!existing || heuristicFrameScore(frame) > heuristicFrameScore(existing)) {
|
||
map.set(key, frame);
|
||
}
|
||
}
|
||
return Array.from(map.values());
|
||
}
|
||
|
||
function buildBestFramePrompt(annotation, prompt) {
|
||
const frames = (annotation.markedFrames || []).map(frame => ({
|
||
timestamp: frame.timestamp,
|
||
hasBbox: !!frame.bbox,
|
||
bbox: frame.bbox ? frame.bbox.normalized : null,
|
||
}));
|
||
return `${prompt}\n损伤ID：${annotation.damageId}\n部位：${annotation.part}\n类型：${annotation.type}\n描述：${annotation.description}\n候选帧：${JSON.stringify(frames)}`;
|
||
}
|
||
|
||
function parseBestFrameSelection(content) {
|
||
const parsed = parseJsonLoose(content || '');
|
||
if (!parsed || !Array.isArray(parsed.best_timestamps)) return { timestamps: [], reasons: [] };
|
||
return {
|
||
timestamps: parsed.best_timestamps.map(Number).filter(Number.isFinite),
|
||
reasons: Array.isArray(parsed.reasons) ? parsed.reasons.map(String) : [],
|
||
};
|
||
}
|
||
|
||
async function chooseFramesWithModel({ annotation, candidates, topN, modelConfig, prompt, callVisionApi }) {
|
||
if (!modelConfig || !modelConfig.bestFrameModel || candidates.length <= 1) return null;
|
||
const result = await callVisionApi({
|
||
config: modelConfig,
|
||
model: modelConfig.bestFrameModel,
|
||
messages: [{ role: 'user', content: [{ type: 'text', text: buildBestFramePrompt(annotation, prompt) }] }],
|
||
maxTokens: 1024,
|
||
temperature: 0.1,
|
||
});
|
||
if (!result.ok) return null;
|
||
const selection = parseBestFrameSelection(result.content);
|
||
const picked = [];
|
||
for (const timestamp of selection.timestamps) {
|
||
const frame = candidates.find(item => Math.abs(Number(item.timestamp) - timestamp) < 0.01);
|
||
if (frame && !picked.includes(frame)) picked.push(frame);
|
||
if (picked.length >= topN) break;
|
||
}
|
||
if (picked.length === 0) return null;
|
||
return picked.map((frame, index) => ({ frame, reason: selection.reasons[index] || 'BEST_FRAME_MODEL selected this frame' }));
|
||
}
|
||
|
||
async function selectBestFrames({ workspace, annotations, topN, modelConfig, prompt = '', callVisionApi = defaultCallVisionApi }) {
|
||
const damages = [];
|
||
for (let index = 0; index < annotations.length; index += 1) {
|
||
const annotation = annotations[index];
|
||
const candidates = dedupeByTimestamp(annotation.markedFrames || []);
|
||
const modelSelected = await chooseFramesWithModel({ annotation, candidates, topN, modelConfig, prompt, callVisionApi });
|
||
const selected = modelSelected || candidates
|
||
.slice()
|
||
.sort((a, b) => heuristicFrameScore(b) - heuristicFrameScore(a))
|
||
.slice(0, topN)
|
||
.map(frame => ({ frame, reason: 'heuristic fallback' }));
|
||
const bestFrames = selected.map(({ frame, reason }) => ({
|
||
timestamp: frame.timestamp,
|
||
path: frame.path,
|
||
relativePath: frame.markedRelativePath,
|
||
bbox: frame.bbox ? frame.bbox.normalized : null,
|
||
selectionReason: reason,
|
||
}));
|
||
damages.push({
|
||
id: annotation.damageId || `dmg_${String(index + 1).padStart(3, '0')}`,
|
||
part: annotation.part,
|
||
type: annotation.type,
|
||
severity: annotation.severity,
|
||
description: annotation.description,
|
||
timestamps: candidates.map(frame => frame.timestamp),
|
||
bestFrames,
|
||
});
|
||
}
|
||
|
||
const bestFrameCount = damages.reduce((sum, item) => sum + item.bestFrames.length, 0);
|
||
const output = {
|
||
damageCount: damages.length,
|
||
bestFrameCount,
|
||
damages,
|
||
};
|
||
writeJson(workspace, 'best_frames.json', output);
|
||
return output;
|
||
}
|
||
|
||
module.exports = {
|
||
heuristicFrameScore,
|
||
dedupeByTimestamp,
|
||
buildBestFramePrompt,
|
||
parseBestFrameSelection,
|
||
selectBestFrames,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs`
|
||
|
||
Expected: exits 0.
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/best_frame_selector.cjs packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs
|
||
git commit -m "feat(skills): select best vehicle damage evidence frames"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 10: Compute-entry pipeline entrypoint
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/index.cjs`
|
||
- Test: `packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs`
|
||
|
||
- [ ] **Step 1: 写 frames-only 入口测试**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs`:
|
||
|
||
```javascript
|
||
const assert = require('node:assert');
|
||
const { normalizeInput, buildFinalOutput } = require('../scripts/index.cjs');
|
||
|
||
const input = normalizeInput({ videoUrl: '/upload/a.mp4', fps: '20', quality: '120', mode: 'frames-only' });
|
||
assert.strictEqual(input.videoUrl, '/upload/a.mp4');
|
||
assert.strictEqual(input.fps, 10);
|
||
assert.strictEqual(input.quality, 100);
|
||
assert.strictEqual(input.mode, 'frames-only');
|
||
|
||
const output = buildFinalOutput({
|
||
workspace: { taskId: 'task', workspacePath: 'C:/data/workspace/task' },
|
||
videoInfo: { duration: 1, resolution: '1x1', extractedFrames: 3 },
|
||
candidates: [],
|
||
damages: [],
|
||
artifacts: { videoInfo: 'video_info.json' },
|
||
});
|
||
assert.strictEqual(output.success, true);
|
||
assert.strictEqual(output.summary.frameCount, 3);
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs`
|
||
|
||
Expected: FAIL with missing module.
|
||
|
||
- [ ] **Step 3: 实现 scripts/index.cjs**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/index.cjs`:
|
||
|
||
```javascript
|
||
const fs = require('node:fs');
|
||
const path = require('node:path');
|
||
const { clampNumber } = require('./lib/json_utils.cjs');
|
||
const { createWorkspace, resolveVideoPath, writeJson } = require('./lib/workspace.cjs');
|
||
const { extractFrames } = require('./lib/frame_extractor.cjs');
|
||
const { resolveModelConfig } = require('./lib/vision_client.cjs');
|
||
const { detectDamageCandidates } = require('./lib/damage_detector.cjs');
|
||
const { groundDamages } = require('./lib/damage_grounding.cjs');
|
||
const { selectBestFrames } = require('./lib/best_frame_selector.cjs');
|
||
|
||
function normalizeInput(raw) {
|
||
if (!raw || typeof raw !== 'object') throw new Error('Input JSON object is required');
|
||
if (!raw.videoUrl) throw new Error('videoUrl is required');
|
||
return {
|
||
videoUrl: String(raw.videoUrl),
|
||
taskId: raw.taskId ? String(raw.taskId) : undefined,
|
||
fps: clampNumber(raw.fps, 0.2, 10, 3),
|
||
quality: clampNumber(raw.quality, 1, 100, 90),
|
||
batchSize: clampNumber(raw.batchSize, 1, 20, 12),
|
||
concurrency: clampNumber(raw.concurrency, 1, 4, 2),
|
||
groundingWindow: clampNumber(raw.groundingWindow, 0.5, 5, 1.5),
|
||
topN: clampNumber(raw.topN, 1, 5, 1),
|
||
mode: ['full', 'frames-only', 'detect-only'].includes(raw.mode) ? raw.mode : 'full',
|
||
};
|
||
}
|
||
|
||
function readPrompt(name) {
|
||
return fs.readFileSync(path.join(__dirname, '..', 'prompts', name), 'utf8');
|
||
}
|
||
|
||
function resolveWorkspaceRoot(env) {
|
||
if (env.RZYX_AI_WORKSPACE_ROOT) return env.RZYX_AI_WORKSPACE_ROOT;
|
||
if (env.RZYX_AI_DATA_DIR) return path.join(env.RZYX_AI_DATA_DIR, 'workspace', 'vehicle-damage-inspection');
|
||
if (env.NODE_ENV === 'production') throw new Error('RZYX_AI_DATA_DIR or RZYX_AI_WORKSPACE_ROOT is required in production');
|
||
return path.join(__dirname, '..', '.workspace');
|
||
}
|
||
|
||
function buildFinalOutput({ workspace, videoInfo, candidates, damages, artifacts }) {
|
||
const bestFrameCount = damages.reduce((sum, item) => sum + (item.bestFrames || []).length, 0);
|
||
return {
|
||
success: true,
|
||
taskId: workspace.taskId,
|
||
workspacePath: workspace.workspacePath,
|
||
summary: {
|
||
duration: videoInfo.duration,
|
||
resolution: videoInfo.resolution,
|
||
frameCount: videoInfo.extractedFrames,
|
||
candidateDamageCount: candidates.length,
|
||
mergedDamageCount: damages.length,
|
||
bestFrameCount,
|
||
},
|
||
damages,
|
||
artifacts,
|
||
};
|
||
}
|
||
|
||
async function run(rawInput, env = process.env) {
|
||
const input = normalizeInput(rawInput);
|
||
const workspaceRoot = resolveWorkspaceRoot(env);
|
||
const workspace = createWorkspace({ taskId: input.taskId, workspaceRoot });
|
||
const uploadRoot = env.RZYX_AI_UPLOAD_ROOT || (env.RZYX_AI_DATA_DIR ? path.join(env.RZYX_AI_DATA_DIR, 'uploads') : undefined);
|
||
const videoPath = resolveVideoPath(input.videoUrl, { uploadRoot });
|
||
|
||
const { videoInfo, frames } = await extractFrames({
|
||
workspace,
|
||
videoPath,
|
||
fps: input.fps,
|
||
quality: input.quality,
|
||
});
|
||
|
||
if (input.mode === 'frames-only') {
|
||
const output = buildFinalOutput({
|
||
workspace,
|
||
videoInfo,
|
||
candidates: [],
|
||
damages: [],
|
||
artifacts: { videoInfo: 'video_info.json' },
|
||
});
|
||
writeJson(workspace, 'run_summary.json', output);
|
||
return output;
|
||
}
|
||
|
||
const modelConfig = resolveModelConfig(env);
|
||
const detection = await detectDamageCandidates({
|
||
workspace,
|
||
frames,
|
||
modelConfig,
|
||
prompt: readPrompt('damage_detect.md'),
|
||
batchSize: input.batchSize,
|
||
concurrency: input.concurrency,
|
||
});
|
||
|
||
if (input.mode === 'detect-only' || detection.candidates.length === 0) {
|
||
const output = buildFinalOutput({
|
||
workspace,
|
||
videoInfo,
|
||
candidates: detection.candidates,
|
||
damages: [],
|
||
artifacts: { videoInfo: 'video_info.json', damageCandidates: 'damage_candidates.json' },
|
||
});
|
||
writeJson(workspace, 'run_summary.json', output);
|
||
return output;
|
||
}
|
||
|
||
const grounding = await groundDamages({
|
||
workspace,
|
||
frames,
|
||
candidates: detection.candidates,
|
||
modelConfig,
|
||
prompt: readPrompt('grounding.md'),
|
||
groundingWindow: input.groundingWindow,
|
||
});
|
||
|
||
const best = await selectBestFrames({
|
||
workspace,
|
||
annotations: grounding.annotations,
|
||
topN: input.topN,
|
||
modelConfig,
|
||
prompt: readPrompt('best_frame.md'),
|
||
});
|
||
|
||
const output = buildFinalOutput({
|
||
workspace,
|
||
videoInfo,
|
||
candidates: detection.candidates,
|
||
damages: best.damages,
|
||
artifacts: {
|
||
videoInfo: 'video_info.json',
|
||
damageCandidates: 'damage_candidates.json',
|
||
damageAnnotations: 'damage_annotations.json',
|
||
bestFrames: 'best_frames.json',
|
||
runSummary: 'run_summary.json',
|
||
},
|
||
});
|
||
writeJson(workspace, 'run_summary.json', output);
|
||
return output;
|
||
}
|
||
|
||
async function main() {
|
||
try {
|
||
const stdin = fs.readFileSync(0, 'utf8');
|
||
const input = JSON.parse(stdin || '{}');
|
||
const output = await run(input);
|
||
process.stdout.write(JSON.stringify(output));
|
||
} catch (error) {
|
||
process.stdout.write(JSON.stringify({ success: false, error: error.message }));
|
||
process.exitCode = 0;
|
||
}
|
||
}
|
||
|
||
if (require.main === module) {
|
||
main();
|
||
}
|
||
|
||
module.exports = {
|
||
normalizeInput,
|
||
buildFinalOutput,
|
||
run,
|
||
};
|
||
```
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `node packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs`
|
||
|
||
Expected: exits 0.
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection/scripts/index.cjs packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs
|
||
git commit -m "feat(skills): wire vehicle inspection compute entrypoint"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 11: make.sh and Windows setup diagnostics
|
||
|
||
**Files:**
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/make.sh`
|
||
- Create: `packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1`
|
||
|
||
- [ ] **Step 1: 创建 make.sh**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/make.sh`:
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
set -euo pipefail
|
||
|
||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||
|
||
cmd_check() {
|
||
local ok=true
|
||
command -v node >/dev/null 2>&1 || { echo "node missing"; ok=false; }
|
||
command -v ffmpeg >/dev/null 2>&1 || echo "ffmpeg not found in PATH; npm installer fallback may be used"
|
||
command -v ffprobe >/dev/null 2>&1 || echo "ffprobe not found in PATH; npm installer fallback may be used"
|
||
node -e "require('axios'); require('sharp'); require('fluent-ffmpeg')" || ok=false
|
||
$ok
|
||
}
|
||
|
||
cmd_fix() {
|
||
cd "$ROOT"
|
||
npm install axios sharp fluent-ffmpeg @ffmpeg-installer/ffmpeg @ffprobe-installer/ffprobe
|
||
}
|
||
|
||
cmd_run() {
|
||
local video=""
|
||
local fps="3"
|
||
local mode="full"
|
||
while [[ $# -gt 0 ]]; do
|
||
case "$1" in
|
||
--video) video="$2"; shift 2 ;;
|
||
--fps) fps="$2"; shift 2 ;;
|
||
--mode) mode="$2"; shift 2 ;;
|
||
*) echo "Unknown option: $1"; exit 1 ;;
|
||
esac
|
||
done
|
||
if [[ -z "$video" ]]; then
|
||
echo "Usage: bash scripts/make.sh run --video FILE [--fps 3] [--mode full]"
|
||
exit 1
|
||
fi
|
||
printf '{"videoUrl":%s,"fps":%s,"mode":%s}\n' \
|
||
"$(node -e "process.stdout.write(JSON.stringify(process.argv[1]))" "$video")" \
|
||
"$fps" \
|
||
"$(node -e "process.stdout.write(JSON.stringify(process.argv[1]))" "$mode")" \
|
||
| node "$ROOT/scripts/index.cjs"
|
||
}
|
||
|
||
cmd_tests() {
|
||
for file in "$ROOT"/tests/*.test.cjs; do
|
||
node "$file"
|
||
done
|
||
}
|
||
|
||
case "${1:-}" in
|
||
check) cmd_check ;;
|
||
fix) cmd_fix ;;
|
||
run) shift; cmd_run "$@" ;;
|
||
test) cmd_tests ;;
|
||
*) echo "Usage: bash scripts/make.sh check|fix|run|test"; exit 1 ;;
|
||
esac
|
||
```
|
||
|
||
- [ ] **Step 2: 运行本地测试集合**
|
||
|
||
Run: `bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh test`
|
||
|
||
Expected: every `*.test.cjs` exits 0.
|
||
|
||
- [ ] **Step 3: 运行 check**
|
||
|
||
Run: `bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh check`
|
||
|
||
Expected: exits 0 if node packages are installed. If it fails because packages are not installed in the skill directory, run `bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh fix`, then rerun check.
|
||
|
||
- [ ] **Step 4: 提交**
|
||
|
||
- [ ] **Step 4: 创建 Windows setup.ps1**
|
||
|
||
Create `packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1`:
|
||
|
||
```powershell
|
||
$ErrorActionPreference = "Stop"
|
||
|
||
$SkillRoot = Resolve-Path (Join-Path $PSScriptRoot "..")
|
||
Set-Location $SkillRoot
|
||
|
||
npm install axios sharp fluent-ffmpeg @ffmpeg-installer/ffmpeg @ffprobe-installer/ffprobe
|
||
```
|
||
|
||
- [ ] **Step 5: 验证 Windows setup 脚本语法**
|
||
|
||
Run: `powershell -NoProfile -ExecutionPolicy Bypass -File packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1`
|
||
|
||
Expected: npm dependencies install successfully in the skill directory. For production packaging, run this during build/install, not during every skill execution.
|
||
|
||
- [ ] **Step 6: 提交**
|
||
|
||
```bash
|
||
git add packages/backend/skills/vehicle-damage-inspection/scripts/make.sh packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1
|
||
git commit -m "feat(skills): add vehicle inspection skill cli"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 12: End-to-end backend integration validation
|
||
|
||
**Files:**
|
||
- Test only: no source files expected unless validation exposes a bug.
|
||
|
||
- [ ] **Step 1: 运行 Skill 纯测试**
|
||
|
||
Run: `bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh test`
|
||
|
||
Expected: exits 0.
|
||
|
||
- [ ] **Step 2: 运行后端 runtime env 测试**
|
||
|
||
Run: `pnpm --filter @neta/backend test -- --runInBand test/skill_runtime_env.test.ts`
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 3: 运行后端 build**
|
||
|
||
Run: `pnpm --filter @neta/backend build`
|
||
|
||
Expected: build completes without TypeScript errors.
|
||
|
||
- [ ] **Step 4: 模拟 Windows dataDir 路径运行 frames-only**
|
||
|
||
Prepare a small MP4 file. If no sample is available, use ffmpeg to generate one:
|
||
|
||
```bash
|
||
mkdir -p C:/tmp/rzyx-ai-skill-test/uploads/20260504
|
||
ffmpeg -f lavfi -i testsrc=duration=2:size=320x240:rate=10 C:/tmp/rzyx-ai-skill-test/uploads/20260504/sample.mp4 -y
|
||
```
|
||
|
||
Run:
|
||
|
||
```bash
|
||
RZYX_AI_DATA_DIR=C:/tmp/rzyx-ai-skill-test \
|
||
RZYX_AI_UPLOAD_ROOT=C:/tmp/rzyx-ai-skill-test/uploads \
|
||
RZYX_AI_WORKSPACE_ROOT=C:/tmp/rzyx-ai-skill-test/workspace/vehicle-damage-inspection \
|
||
bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh run \
|
||
--video /upload/20260504/sample.mp4 \
|
||
--fps 1 \
|
||
--mode frames-only
|
||
```
|
||
|
||
Expected:
|
||
|
||
- stdout is valid JSON with `"success":true`.
|
||
- `C:/tmp/rzyx-ai-skill-test/workspace/vehicle-damage-inspection/.../video_info.json` exists.
|
||
- No `workspace` directory is created under `packages/backend/skills/vehicle-damage-inspection` during this production-path simulation.
|
||
|
||
- [ ] **Step 5: 提交 validation fixes only if needed**
|
||
|
||
If any validation step required code fixes, first inspect the exact modified files:
|
||
|
||
```bash
|
||
git status --short
|
||
```
|
||
|
||
Then add only files related to this skill implementation. The expected fix scope is one or more of these paths:
|
||
|
||
```bash
|
||
git add packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts packages/backend/src/modules/netaclaw/service/skill_executor.ts packages/backend/test/skill_runtime_env.test.ts packages/backend/skills/vehicle-damage-inspection
|
||
git commit -m "fix(skills): stabilize vehicle inspection integration"
|
||
```
|
||
|
||
If no files changed, skip this commit.
|
||
|
||
---
|
||
|
||
## Self-Review Checklist
|
||
|
||
- [x] Spec coverage: Tasks cover host env injection, Skill metadata, prompts, workspace protocol, frame extraction, vision client, detection, grounding, image marking, best-frame selection, compute entrypoint, CLI, tests, and Windows dataDir validation.
|
||
- [x] Placeholder scan: The plan contains no unresolved markers, no undefined file paths, and no deferred-implementation instructions.
|
||
- [x] Type consistency: The public Skill name is consistently `vehicle-damage-inspection`; runtime env names are consistently `RZYX_AI_DATA_DIR`, `RZYX_AI_WORKSPACE_ROOT`, and `RZYX_AI_UPLOAD_ROOT`; default model is consistently `doubao-seed-2-0-pro-260215`.
|