2026-05-20 21:39:12 +08:00

72 KiB

Raw Blame History

汽车环车视频旧伤检测 Skill Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: 新增一个生产可用的 vehicle-damage-inspection compute-entry Skill，从环车视频中抽帧、检测漆面旧伤、定位标注损伤区域并筛选最佳证据帧。

Architecture: 后端先补强 SkillExecutorService，为 compute-entry 子进程注入 Windows 安装包生产所需的 dataDir/workspace/uploads 非密钥运行时环境变量。业务能力作为一个大 Skill 放在 packages/backend/skills/vehicle-damage-inspection，Agent 通过 execute_skill 调一次，Skill 内部用 Node CommonJS 脚本按固定流水线执行抽帧、视觉模型检测、bbox 标注和最佳帧筛选。

Tech Stack: TypeScript, Midway service, Jest, Node CommonJS, ffmpeg/ffprobe, fluent-ffmpeg, axios, sharp, 火山方舟 OpenAI-compatible Chat Completions, 豆包 doubao-seed-2-0-pro-260215

Spec: docs/superpowers/specs/2026-05-04-vehicle-damage-inspection-skill-design.md

File Map

文件	职责	操作
`packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts`	根据 `pDataPath()` / `pUploadPath()` 构造 compute-entry skill 运行目录 env	新增
`packages/backend/src/modules/netaclaw/service/skill_env_schema.ts`	从 `skill.config.yaml env` 同步管理页和 secrets 使用的 env schema	新增
`packages/backend/src/modules/netaclaw/service/skill_executor.ts`	注入 runtime env，确保 Windows 安装包产物落在 dataDir	修改
`packages/backend/test/skill_runtime_env.test.ts`	测试 runtime env 默认值和 skill secrets 覆盖顺序	新增
`packages/backend/test/skill_env_schema.test.ts`	测试 `skill.config.yaml env` 到 DB `envSchema` 的同步规则	新增
`packages/backend/skills/vehicle-damage-inspection/SKILL.md`	Agent 可读 Skill 说明	新增
`packages/backend/skills/vehicle-damage-inspection/skill.config.yaml`	compute-entry 配置、模型 env、依赖、接口	新增
`packages/backend/skills/vehicle-damage-inspection/README.md`	人工调试和 Windows 安装包注意事项	新增
`packages/backend/skills/vehicle-damage-inspection/prompts/damage_detect.md`	环车旧伤候选检测 prompt	新增
`packages/backend/skills/vehicle-damage-inspection/prompts/grounding.md`	bbox/grounding prompt	新增
`packages/backend/skills/vehicle-damage-inspection/prompts/dedup.md`	损伤去重 prompt	新增
`packages/backend/skills/vehicle-damage-inspection/prompts/best_frame.md`	最佳帧筛选 prompt	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/index.cjs`	stdin/stdout JSON 入口和流水线编排	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/make.sh`	`check/run/demo/fix` 本地调试入口	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1`	Windows installer setup 入口；当前 installer 用 `powershell -File` 执行	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/lib/json_utils.cjs`	JSON、bbox、数字 clamp、sleep、并发限制等通用函数	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/lib/workspace.cjs`	task workspace、路径解析、JSON artifact I/O	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/lib/frame_extractor.cjs`	ffmpeg/ffprobe 抽帧	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/lib/vision_client.cjs`	OpenAI-compatible 视觉 API 调用、重试、图片压缩编码	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_detector.cjs`	分批候选旧伤检测	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/lib/image_marker.cjs`	sharp 画 bbox 标注框	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_grounding.cjs`	损伤定位标注	新增
`packages/backend/skills/vehicle-damage-inspection/scripts/lib/best_frame_selector.cjs`	损伤去重和最佳帧筛选	新增
`packages/backend/skills/vehicle-damage-inspection/tests/*.test.cjs`	Skill 纯函数和 mock 流水线测试	新增

Task 1: Skill env schema 同步

Files:

Create: packages/backend/src/modules/netaclaw/service/skill_env_schema.ts
Modify: packages/backend/src/modules/netaclaw/service/skill_loader.ts
Test: packages/backend/test/skill_env_schema.test.ts
Step 1: 写失败测试，覆盖从 skill.config.yaml env 同步默认模型

Create packages/backend/test/skill_env_schema.test.ts:

import { buildSkillEnvSchema } from '../src/modules/netaclaw/service/skill_env_schema.js';

describe('buildSkillEnvSchema', () => {
  it('uses skill.config.yaml env as the envSchema source of truth', () => {
    const schema = buildSkillEnvSchema({
      env: [
        {
          name: 'ARK_API_KEY',
          required: true,
          description: '火山方舟/豆包 OpenAI-compatible API Key',
        },
        {
          name: 'DAMAGE_DETECT_MODEL',
          required: true,
          default: 'doubao-seed-2-0-pro-260215',
          description: '用于环车旧伤候选检测的多模态模型 ID',
        },
      ],
    });

    expect(schema).toEqual([
      {
        name: 'ARK_API_KEY',
        required: true,
        description: '火山方舟/豆包 OpenAI-compatible API Key',
        default: undefined,
      },
      {
        name: 'DAMAGE_DETECT_MODEL',
        required: true,
        description: '用于环车旧伤候选检测的多模态模型 ID',
        default: 'doubao-seed-2-0-pro-260215',
      },
    ]);
  });

  it('returns null when the skill has no env declaration', () => {
    expect(buildSkillEnvSchema({})).toBeNull();
  });
});

Step 2: 运行测试确认失败

Run: pnpm --filter @neta/backend test -- --runInBand test/skill_env_schema.test.ts

Expected: FAIL with module not found for skill_env_schema.js.

Step 3: 新增 skill_env_schema helper

Create packages/backend/src/modules/netaclaw/service/skill_env_schema.ts:

import type { SkillConfig } from './skill_config.js';

export interface SkillEnvSchemaItem {
  name: string;
  required: boolean;
  description?: string;
  default?: string;
}

export function buildSkillEnvSchema(config: Pick<SkillConfig, 'env'>): SkillEnvSchemaItem[] | null {
  if (!Array.isArray(config.env) || config.env.length === 0) return null;
  return config.env.map(item => ({
    name: item.name,
    required: !!item.required,
    description: item.description || undefined,
    default: item.default || undefined,
  }));
}

Step 4: 接入 SkillLoaderService.getSkillMetas

Modify packages/backend/src/modules/netaclaw/service/skill_loader.ts.

Add import near existing service imports:

import { buildSkillEnvSchema } from './skill_env_schema.js';

Inside getSkillMetas(), before await this.skillRepo.save(...), add:

        const config = this.skillConfig.getConfig(fs.name);
        const envSchema = buildSkillEnvSchema(config ?? {});

Replace the envSchema assignment in the save object:

          envSchema: Array.isArray((fs.metadata as any)?.env) ? (fs.metadata as any).env.map((e: any) => ({
            name: e.name,
            required: !!e.required,
            description: e.description || undefined,
            default: e.default || undefined,
          })) : null,

with:

          envSchema,

This makes skill.config.yaml env the single source of truth for model defaults and secret fields. Do not duplicate env declarations in SKILL.md metadata.env.

Step 5: 运行测试确认通过

Run: pnpm --filter @neta/backend test -- --runInBand test/skill_env_schema.test.ts

Expected: PASS.

Step 6: 验证后端编译

Run: pnpm --filter @neta/backend build

Expected: build completes without TypeScript errors.

Step 7: 提交

git add packages/backend/src/modules/netaclaw/service/skill_env_schema.ts packages/backend/src/modules/netaclaw/service/skill_loader.ts packages/backend/test/skill_env_schema.test.ts
git commit -m "feat(netaclaw): sync skill config env schema"

Task 2: 宿主 runtime env 注入

Files:

Create: packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts
Modify: packages/backend/src/modules/netaclaw/service/skill_executor.ts
Test: packages/backend/test/skill_runtime_env.test.ts
Step 1: 写失败测试，覆盖 Windows dataDir env 默认值

Create packages/backend/test/skill_runtime_env.test.ts:

import * as path from 'path';
import { buildSkillRuntimeEnv } from '../src/modules/netaclaw/service/skill_runtime_env.js';

describe('buildSkillRuntimeEnv', () => {
  it('injects dataDir, workspace root and upload root for vehicle damage inspection', () => {
    const dataDir = path.join('C:', 'RZYX_AI_DATA');
    const uploadRoot = path.join(dataDir, 'uploads');

    const env = buildSkillRuntimeEnv({
      skillName: 'vehicle-damage-inspection',
      dataDir,
      uploadRoot,
    });

    expect(env.RZYX_AI_DATA_DIR).toBe(path.resolve(dataDir));
    expect(env.RZYX_AI_UPLOAD_ROOT).toBe(path.resolve(uploadRoot));
    expect(env.RZYX_AI_WORKSPACE_ROOT).toBe(
      path.join(path.resolve(dataDir), 'workspace', 'vehicle-damage-inspection'),
    );
  });

  it('uses generic workspace directory for other compute-entry skills', () => {
    const dataDir = path.join('C:', 'RZYX_AI_DATA');
    const env = buildSkillRuntimeEnv({
      skillName: 'ocr-reader',
      dataDir,
      uploadRoot: path.join(dataDir, 'uploads'),
    });

    expect(env.RZYX_AI_WORKSPACE_ROOT).toBe(
      path.join(path.resolve(dataDir), 'workspace', 'ocr-reader'),
    );
  });
});

Step 2: 运行测试确认失败

Run: pnpm --filter @neta/backend test -- --runInBand test/skill_runtime_env.test.ts

Expected: FAIL with module not found for skill_runtime_env.js.

Step 3: 新增 runtime env helper

Create packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts:

import * as path from 'path';

export interface BuildSkillRuntimeEnvParams {
  skillName: string;
  dataDir: string;
  uploadRoot: string;
}

export function buildSkillRuntimeEnv(params: BuildSkillRuntimeEnvParams): Record<string, string> {
  const dataDir = path.resolve(params.dataDir);
  const uploadRoot = path.resolve(params.uploadRoot);
  const workspaceRoot = path.join(dataDir, 'workspace', params.skillName);

  return {
    RZYX_AI_DATA_DIR: dataDir,
    RZYX_AI_UPLOAD_ROOT: uploadRoot,
    RZYX_AI_WORKSPACE_ROOT: workspaceRoot,
  };
}

Step 4: 接入 SkillExecutorService

Modify packages/backend/src/modules/netaclaw/service/skill_executor.ts.

Add imports near the existing imports:

import { pDataPath, pUploadPath } from '../../../comm/path.js';
import { buildSkillRuntimeEnv } from './skill_runtime_env.js';

Replace:

    const skillEnv = await this.skillSecret.resolveEnv(skillName);
    const env = { ...baseEnv, ...skillEnv };

with:

    const runtimeEnv = buildSkillRuntimeEnv({
      skillName,
      dataDir: pDataPath(),
      uploadRoot: pUploadPath(),
    });
    const skillEnv = await this.skillSecret.resolveEnv(skillName);
    const env = { ...baseEnv, ...runtimeEnv, ...skillEnv };

This order lets skill secrets override runtime defaults when an administrator explicitly configures a path.

Step 5: 运行测试确认通过

Run: pnpm --filter @neta/backend test -- --runInBand test/skill_runtime_env.test.ts

Expected: PASS.

Step 6: 验证后端编译

Run: pnpm --filter @neta/backend build

Expected: build completes without TypeScript errors.

Step 7: 提交

git add packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts packages/backend/src/modules/netaclaw/service/skill_executor.ts packages/backend/test/skill_runtime_env.test.ts
git commit -m "feat(netaclaw): inject runtime data paths into compute skills"

Task 3: Skill scaffold and metadata

Files:

Create: packages/backend/skills/vehicle-damage-inspection/SKILL.md
Create: packages/backend/skills/vehicle-damage-inspection/skill.config.yaml
Create: packages/backend/skills/vehicle-damage-inspection/README.md
Create: packages/backend/skills/vehicle-damage-inspection/prompts/damage_detect.md
Create: packages/backend/skills/vehicle-damage-inspection/prompts/grounding.md
Create: packages/backend/skills/vehicle-damage-inspection/prompts/dedup.md
Create: packages/backend/skills/vehicle-damage-inspection/prompts/best_frame.md
Step 1: 创建 SKILL.md

Create packages/backend/skills/vehicle-damage-inspection/SKILL.md:

---
name: vehicle-damage-inspection
description: >
  汽车环车视频旧伤检测。输入一个环车视频文件或 /upload/... 路径，自动抽帧、
  检测车身漆面旧伤、定位 bbox 并输出红框标注图和最佳证据帧。
license: MIT
metadata:
  version: "1.0.0"
  category: vehicle-inspection
  skillType: multimodal
  tags: ["vehicle", "video", "damage", "bbox"]
---

# 汽车环车视频旧伤检测

当用户需要从汽车环车视频中识别漆面旧伤、划痕、凹陷、掉漆、锈蚀等外观损伤时，使用此 skill。

这是 compute-entry skill。不要手动拆步骤调用；直接使用 `execute_skill`：

```json
{
  "name": "vehicle-damage-inspection",
  "input": {
    "videoUrl": "/upload/20260504/car.mp4",
    "fps": 3,
    "topN": 1
  }
}

输入

videoUrl: 必填。视频本地路径或 /upload/... 路径；第一版不支持 http(s) URL。
taskId: 可选。未传时自动生成。
fps: 可选，默认 3。
quality: 可选，默认 90。
batchSize: 可选，默认 12。
concurrency: 可选，默认 2。
groundingWindow: 可选，默认 1.5 秒。
topN: 可选，默认每处损伤 1 张最佳帧。
mode: 可选，full、frames-only 或 detect-only。

输出

返回 JSON，包含 success、taskId、workspacePath、summary、damages 和 artifacts。

生产环境中，所有检测产物写入后端 dataDir 下的 workspace/vehicle-damage-inspection/{taskId}。


- [ ] **Step 2: 创建 skill.config.yaml**

Create `packages/backend/skills/vehicle-damage-inspection/skill.config.yaml`:

```yaml
runtime: node
entrypoint: scripts/index.cjs
timeout: 900000

dependencies:
  system:
    - name: ffmpeg
      check: "ffmpeg -version"
    - name: ffprobe
      check: "ffprobe -version"
    - name: node
      check: "node --version"
  node:
    packages:
      - axios
      - sharp
      - fluent-ffmpeg
      - "@ffmpeg-installer/ffmpeg"
      - "@ffprobe-installer/ffprobe"

setup:
  posix: scripts/make.sh fix
  win32: scripts/setup.ps1

env:
  - name: ARK_API_KEY
    required: true
    description: 火山方舟/豆包 OpenAI-compatible API Key
  - name: ARK_API_URL
    required: false
    default: https://ark.cn-beijing.volces.com/api/v3/chat/completions
    description: OpenAI-compatible chat completions endpoint
  - name: DAMAGE_DETECT_MODEL
    required: true
    default: doubao-seed-2-0-pro-260215
    description: 用于环车旧伤候选检测的多模态模型 ID
  - name: DAMAGE_GROUNDING_MODEL
    required: true
    default: doubao-seed-2-0-pro-260215
    description: 支持 bbox/grounding 输出的视觉模型 ID
  - name: BEST_FRAME_MODEL
    required: false
    default: doubao-seed-2-0-pro-260215
    description: 用于损伤去重和最佳帧筛选的模型 ID
  - name: RZYX_AI_WORKSPACE_ROOT
    required: false
    description: 生产环境由宿主注入，默认 dataDir/workspace/vehicle-damage-inspection
  - name: RZYX_AI_UPLOAD_ROOT
    required: false
    description: 生产环境由宿主注入，默认 dataDir/uploads
  - name: RZYX_AI_DATA_DIR
    required: false
    description: 后端运行时 dataDir，由宿主注入

interface:
  input:
    videoUrl:
      type: string
      required: true
      description: 视频本地路径或 /upload/... 路径；第一版不支持 http(s) URL
    taskId:
      type: string
      required: false
      description: 可选任务 ID
    fps:
      type: number
      required: false
      default: "3"
      description: 抽帧帧率
    quality:
      type: number
      required: false
      default: "90"
      description: 输出 JPG 质量，范围 1-100
    batchSize:
      type: number
      required: false
      default: "12"
      description: 每批发送给检测模型的帧数
    concurrency:
      type: number
      required: false
      default: "2"
      description: 多模态 API 并发数
    groundingWindow:
      type: number
      required: false
      default: "1.5"
      description: 每个损伤时间点前后取帧窗口，单位秒
    topN:
      type: number
      required: false
      default: "1"
      description: 每处损伤返回的最佳证据帧数量
    mode:
      type: string
      required: false
      default: full
      description: full | frames-only | detect-only
  output:
    success:
      type: boolean
    taskId:
      type: string
    workspacePath:
      type: string
    summary:
      type: object
    damages:
      type: array
    artifacts:
      type: object

Step 3: 创建 prompt 文件

Create packages/backend/skills/vehicle-damage-inspection/prompts/damage_detect.md:

你是专业汽车环车视频旧伤检测专家。请检查给定视频帧中的车身外观可见区域，识别漆面旧伤，包括划痕、凹陷、掉漆、裂纹、锈蚀。

边界：
- 只把可见且明确的车身外观损伤列为 damages。
- 反光、污渍、阴影、雨水、压缩噪声、拍摄模糊不要直接判定为损伤。
- 玻璃、轮胎、车灯、车牌等非漆面问题如明显可见，可标记为 non_paint_damage，但不要混入 paint damage。
- 不确定时设置 uncertain=true，confidence="low"。

只输出 JSON：
{
  "damages": [
    {
      "time_second": 12.4,
      "part": "左前门",
      "type": "划痕",
      "severity": "轻微",
      "description": "左前门中部可见细长白色划痕",
      "confidence": "medium",
      "uncertain": false
    }
  ],
  "summary": "本批检测简述"
}

Create packages/backend/skills/vehicle-damage-inspection/prompts/grounding.md:

请在图中定位指定汽车外观损伤区域，输出 bbox。bbox 坐标使用 0-1000 归一化坐标，格式：

[12.40 second] <bbox>x1 y1 x2 y2</bbox>

如果某帧看不到该损伤，不要输出该帧 bbox。

Create packages/backend/skills/vehicle-damage-inspection/prompts/dedup.md:

你是汽车旧伤记录去重专家。以下损伤记录可能来自连续视频帧中的同一处物理损伤。

合并规则：
- 位置相同或相近、类型相同，且时间相近，视为同一处物理损伤。
- 位置描述略有差异但明显指同一部位，也应合并。
- 不同部位或不同损伤类型不要合并。

只输出 JSON：
{
  "groups": [
    {
      "merged_part": "左前门",
      "merged_type": "划痕",
      "merged_severity": "轻微",
      "merged_description": "左前门中部细长划痕",
      "member_indices": [0, 2]
    }
  ]
}

Create packages/backend/skills/vehicle-damage-inspection/prompts/best_frame.md:

以下是同一处汽车外观损伤的多张标注图。请选出最适合作为证据展示的帧。

优先级：
1. 红框准确框住损伤，不偏移、不过大、不遗漏。
2. 损伤区域清晰，不模糊、不遮挡。
3. 光照和角度能看清损伤全貌。
4. 避免反光、过曝、过暗。

只输出 JSON：
{
  "best_timestamps": [12.4],
  "reasons": ["红框准确，划痕清晰可见"]
}

Step 4: 创建 README.md

Create packages/backend/skills/vehicle-damage-inspection/README.md:

# vehicle-damage-inspection

生产形态：Windows 安装包中由后端复制到 `{dataDir}/skills/vehicle-damage-inspection` 后执行。

运行产物必须写入 `{dataDir}/workspace/vehicle-damage-inspection/{taskId}`，不要写入 skill 代码目录。

## 本地检查

```bash
bash scripts/make.sh check

本地运行

ARK_API_KEY=xxx bash scripts/make.sh run --video /path/to/car.mp4 --fps 3

必需模型

默认模型为 doubao-seed-2-0-pro-260215，用于检测、bbox 定位和最佳帧筛选。


- [ ] **Step 5: 验证 Skill Loader 能识别配置**

Run: `pnpm --filter @neta/backend test -- --runInBand test/entity_exports.test.ts`

Expected: PASS. This is a smoke check that backend tests still load.

- [ ] **Step 6: 提交**

```bash
git add packages/backend/skills/vehicle-damage-inspection
git commit -m "feat(skills): scaffold vehicle damage inspection skill"

Task 4: JSON utilities and workspace module

Files:

Create: packages/backend/skills/vehicle-damage-inspection/scripts/lib/json_utils.cjs
Create: packages/backend/skills/vehicle-damage-inspection/scripts/lib/workspace.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs
Step 1: 写 json_utils 测试

Create packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs:

const assert = require('node:assert');
const { parseJsonLoose, parseBboxes, clampNumber, runWithConcurrency } = require('../scripts/lib/json_utils.cjs');

(async () => {
  assert.deepStrictEqual(parseJsonLoose('```json\n{"a":1}\n```'), { a: 1 });
  assert.deepStrictEqual(parseBboxes('[1.0 second] <bbox>10 20 30 40</bbox>'), [
    { x1: 10, y1: 20, x2: 30, y2: 40 },
  ]);
  assert.strictEqual(clampNumber('9', 1, 5, 3), 5);
  assert.strictEqual(clampNumber(undefined, 1, 5, 3), 3);

  let active = 0;
  let maxActive = 0;
  const tasks = Array.from({ length: 5 }, (_, i) => async () => {
    active += 1;
    maxActive = Math.max(maxActive, active);
    await new Promise(resolve => setTimeout(resolve, 5));
    active -= 1;
    return i;
  });
  const result = await runWithConcurrency(tasks, 2);
  assert.deepStrictEqual(result, [0, 1, 2, 3, 4]);
  assert.ok(maxActive <= 2);
})();

Step 2: 写 workspace 测试

Create packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs:

const assert = require('node:assert');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { createWorkspace, writeJson, readJson, resolveVideoPath } = require('../scripts/lib/workspace.cjs');

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'vehicle-skill-workspace-'));
const uploads = path.join(root, 'uploads');
fs.mkdirSync(path.join(uploads, '20260504'), { recursive: true });
const video = path.join(uploads, '20260504', 'car.mp4');
fs.writeFileSync(video, 'fake');

try {
  const ws = createWorkspace({ taskId: 'task-a', workspaceRoot: path.join(root, 'workspace') });
  assert.strictEqual(ws.taskId, 'task-a');
  assert.ok(fs.existsSync(path.join(ws.workspacePath, 'frames')));
  writeJson(ws, 'video_info.json', { ok: true });
  assert.deepStrictEqual(readJson(ws, 'video_info.json'), { ok: true });
  assert.strictEqual(resolveVideoPath('/upload/20260504/car.mp4', { uploadRoot: uploads }), video);
  assert.throws(() => resolveVideoPath('/upload/../secret.mp4', { uploadRoot: uploads }), /Invalid upload path/);
  assert.throws(() => resolveVideoPath('https://example.test/car.mp4', { uploadRoot: uploads }), /Remote video URLs are not supported/);
} finally {
  fs.rmSync(root, { recursive: true, force: true });
}

Step 3: 运行测试确认失败

Run:

node packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs
node packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs

Expected: both FAIL with missing module errors.

Step 4: 实现 json_utils.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/lib/json_utils.cjs:

function parseJsonLoose(text) {
  if (typeof text !== 'string') return null;
  let value = text.trim();
  if (value.startsWith('```')) {
    value = value.split('\n').filter(line => !line.trim().startsWith('```')).join('\n').trim();
  }
  try {
    return JSON.parse(value);
  } catch {
    const start = value.indexOf('{');
    const end = value.lastIndexOf('}');
    if (start >= 0 && end > start) {
      try {
        return JSON.parse(value.slice(start, end + 1));
      } catch {
        return null;
      }
    }
    return null;
  }
}

function parseBboxes(text) {
  const result = [];
  const pattern = /<bbox>\s*(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s*<\/bbox>/g;
  let match;
  while ((match = pattern.exec(String(text || ''))) !== null) {
    result.push({
      x1: Number(match[1]),
      y1: Number(match[2]),
      x2: Number(match[3]),
      y2: Number(match[4]),
    });
  }
  return result;
}

function clampNumber(value, min, max, fallback) {
  const n = Number(value);
  if (!Number.isFinite(n)) return fallback;
  return Math.max(min, Math.min(max, n));
}

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function runWithConcurrency(tasks, limit) {
  const results = new Array(tasks.length);
  let nextIndex = 0;
  async function worker() {
    while (nextIndex < tasks.length) {
      const index = nextIndex++;
      results[index] = await tasks[index]();
    }
  }
  await Promise.all(Array.from({ length: Math.min(Math.max(1, limit), tasks.length) }, () => worker()));
  return results;
}

module.exports = {
  parseJsonLoose,
  parseBboxes,
  clampNumber,
  sleep,
  runWithConcurrency,
};

Step 5: 实现 workspace.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/lib/workspace.cjs:

const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');

function sanitizeTaskId(taskId) {
  const id = taskId || `${new Date().toISOString().replace(/[-:.TZ]/g, '').slice(0, 14)}-${crypto.randomUUID().slice(0, 8)}`;
  if (!/^[a-zA-Z0-9_-]+$/.test(id)) throw new Error(`Invalid taskId: ${id}`);
  return id;
}

function createWorkspace({ taskId, workspaceRoot }) {
  const root = path.resolve(workspaceRoot);
  const id = sanitizeTaskId(taskId);
  const workspacePath = path.join(root, id);
  fs.mkdirSync(path.join(workspacePath, 'source'), { recursive: true });
  fs.mkdirSync(path.join(workspacePath, 'frames'), { recursive: true });
  fs.mkdirSync(path.join(workspacePath, 'marked_frames'), { recursive: true });
  return { taskId: id, root, workspacePath };
}

function ensureInside(root, candidate) {
  const rootPath = path.resolve(root);
  const fullPath = path.resolve(candidate);
  if (fullPath !== rootPath && !fullPath.startsWith(rootPath + path.sep)) {
    throw new Error(`Path escapes root: ${candidate}`);
  }
  return fullPath;
}

function writeJson(workspace, relativePath, value) {
  const filePath = ensureInside(workspace.workspacePath, path.join(workspace.workspacePath, relativePath));
  fs.mkdirSync(path.dirname(filePath), { recursive: true });
  fs.writeFileSync(filePath, JSON.stringify(value, null, 2), 'utf8');
  return filePath;
}

function readJson(workspace, relativePath) {
  const filePath = ensureInside(workspace.workspacePath, path.join(workspace.workspacePath, relativePath));
  if (!fs.existsSync(filePath)) return null;
  return JSON.parse(fs.readFileSync(filePath, 'utf8'));
}

function resolveVideoPath(videoUrl, options) {
  if (!videoUrl || typeof videoUrl !== 'string') throw new Error('videoUrl is required');
  if (fs.existsSync(videoUrl)) return path.resolve(videoUrl);

  const uploadMatch = videoUrl.match(/^\/upload\/(.+)$/);
  if (uploadMatch) {
    if (!options.uploadRoot) throw new Error('RZYX_AI_UPLOAD_ROOT is required for /upload paths');
    const relative = uploadMatch[1].replace(/[\\/]+/g, path.sep);
    if (relative.split(path.sep).includes('..')) throw new Error('Invalid upload path');
    const filePath = ensureInside(options.uploadRoot, path.join(options.uploadRoot, relative));
    if (!fs.existsSync(filePath)) throw new Error(`Video file not found: ${filePath}`);
    return filePath;
  }

  if (/^https?:\/\//i.test(videoUrl)) {
    throw new Error('Remote video URLs are not supported in the first version; use a local path or /upload/... path');
  }
  throw new Error(`Video file not found: ${videoUrl}`);
}

module.exports = {
  createWorkspace,
  writeJson,
  readJson,
  resolveVideoPath,
  ensureInside,
};

Step 6: 运行测试确认通过

Run:

node packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs
node packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs

Expected: both commands exit 0.

Step 7: 提交

git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/json_utils.cjs packages/backend/skills/vehicle-damage-inspection/scripts/lib/workspace.cjs packages/backend/skills/vehicle-damage-inspection/tests/json_utils.test.cjs packages/backend/skills/vehicle-damage-inspection/tests/workspace.test.cjs
git commit -m "feat(skills): add vehicle inspection workspace utilities"

Task 5: Frame extraction module

Files:

Create: packages/backend/skills/vehicle-damage-inspection/scripts/lib/frame_extractor.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs
Step 1: 写 ffmpeg command builder 测试

Create packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs:

const assert = require('node:assert');
const { buildFrameName, qualityToFfmpegQv, estimateFrameTimestamp } = require('../scripts/lib/frame_extractor.cjs');

assert.strictEqual(buildFrameName(1, 0), 'frame_000001_0.00s.jpg');
assert.strictEqual(buildFrameName(38, 12.4), 'frame_000038_12.40s.jpg');
assert.strictEqual(qualityToFfmpegQv(100), 1);
assert.strictEqual(qualityToFfmpegQv(1), 31);
assert.strictEqual(estimateFrameTimestamp(6, 3), 2);

Step 2: 运行测试确认失败

Run: node packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs

Expected: FAIL with missing module.

Step 3: 实现 frame_extractor.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/lib/frame_extractor.cjs:

const fs = require('node:fs');
const path = require('node:path');
const ffmpeg = require('fluent-ffmpeg');
const ffmpegInstaller = require('@ffmpeg-installer/ffmpeg');
const ffprobeInstaller = require('@ffprobe-installer/ffprobe');
const { writeJson } = require('./workspace.cjs');

ffmpeg.setFfmpegPath(process.env.FFMPEG_PATH || ffmpegInstaller.path);
ffmpeg.setFfprobePath(process.env.FFPROBE_PATH || ffprobeInstaller.path);

function buildFrameName(index, timestamp) {
  return `frame_${String(index).padStart(6, '0')}_${Number(timestamp).toFixed(2)}s.jpg`;
}

function qualityToFfmpegQv(quality) {
  const q = Math.max(1, Math.min(100, Number(quality) || 90));
  return Math.max(1, Math.min(31, Math.round(31 - (q / 100) * 30)));
}

function estimateFrameTimestamp(indexZeroBased, fps) {
  return Number((indexZeroBased / fps).toFixed(2));
}

function probeVideo(videoPath) {
  return new Promise((resolve, reject) => {
    ffmpeg.ffprobe(videoPath, (err, data) => err ? reject(err) : resolve(data));
  });
}

async function extractFrames({ workspace, videoPath, fps, quality }) {
  const framesDir = path.join(workspace.workspacePath, 'frames');
  fs.rmSync(framesDir, { recursive: true, force: true });
  fs.mkdirSync(framesDir, { recursive: true });

  const probe = await probeVideo(videoPath);
  const stream = (probe.streams || []).find(item => item.codec_type === 'video');
  if (!stream) throw new Error('No video stream found');

  const duration = Number(probe.format && probe.format.duration ? probe.format.duration : 0);
  const width = stream.width || 0;
  const height = stream.height || 0;
  const fpsValue = Number(fps) || 3;
  const qv = qualityToFfmpegQv(quality);
  const tempPattern = path.join(framesDir, 'raw_%06d.jpg');

  await new Promise((resolve, reject) => {
    ffmpeg(videoPath)
      .outputOptions(['-vf', `fps=${fpsValue}`, '-q:v', String(qv)])
      .output(tempPattern)
      .on('end', resolve)
      .on('error', reject)
      .run();
  });

  const rawFiles = fs.readdirSync(framesDir).filter(name => /^raw_\d+\.jpg$/.test(name)).sort();
  if (rawFiles.length === 0) throw new Error('No frames extracted');

  const frames = rawFiles.map((name, index) => {
    const timestamp = estimateFrameTimestamp(index, fpsValue);
    const finalName = buildFrameName(index + 1, timestamp);
    fs.renameSync(path.join(framesDir, name), path.join(framesDir, finalName));
    return {
      index,
      timestamp,
      relativePath: `frames/${finalName}`,
      path: path.join(framesDir, finalName),
    };
  });

  const videoInfo = {
    duration: Number(duration.toFixed(2)),
    videoFps: stream.r_frame_rate || '',
    resolution: `${width}x${height}`,
    extractFps: fpsValue,
    extractedFrames: frames.length,
  };

  writeJson(workspace, 'video_info.json', {
    videoPath,
    videoInfo,
    frames: frames.map(({ index, timestamp, relativePath }) => ({ index, timestamp, relativePath })),
  });

  return { videoInfo, frames };
}

module.exports = {
  buildFrameName,
  qualityToFfmpegQv,
  estimateFrameTimestamp,
  extractFrames,
};

Step 4: 运行测试确认通过

Run: node packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs

Expected: exits 0.

Step 5: 提交

git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/frame_extractor.cjs packages/backend/skills/vehicle-damage-inspection/tests/frame_extractor.test.cjs
git commit -m "feat(skills): add vehicle video frame extractor"

Task 6: Vision client module

Files:

Create: packages/backend/skills/vehicle-damage-inspection/scripts/lib/vision_client.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs
Step 1: 写 vision client 纯函数测试

Create packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs:

const assert = require('node:assert');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { resolveModelConfig, encodeImageDataUri, createTimestampedImageContent } = require('../scripts/lib/vision_client.cjs');

const env = {
  ARK_API_KEY: 'key',
  ARK_API_URL: 'https://example.test/v1/chat/completions',
  DAMAGE_DETECT_MODEL: 'detect-model',
  DAMAGE_GROUNDING_MODEL: 'ground-model',
};

assert.deepStrictEqual(resolveModelConfig(env), {
  apiKey: 'key',
  apiUrl: 'https://example.test/v1/chat/completions',
  detectModel: 'detect-model',
  groundingModel: 'ground-model',
  bestFrameModel: 'detect-model',
});

const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vision-client-'));
try {
  const img = path.join(dir, 'a.jpg');
  fs.writeFileSync(img, Buffer.from([1, 2, 3]));
  assert.strictEqual(encodeImageDataUri(img), 'data:image/jpeg;base64,AQID');
  const content = createTimestampedImageContent([{ timestamp: 1.25, path: img }]);
  assert.strictEqual(content[0].text, '[1.25 second]');
  assert.ok(content[1].image_url.url.startsWith('data:image/jpeg;base64,'));
} finally {
  fs.rmSync(dir, { recursive: true, force: true });
}

Step 2: 运行测试确认失败

Run: node packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs

Expected: FAIL with missing module.

Step 3: 实现 vision_client.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/lib/vision_client.cjs:

const fs = require('node:fs');
const axios = require('axios');
const { sleep } = require('./json_utils.cjs');

function resolveModelConfig(env) {
  const apiKey = env.ARK_API_KEY;
  if (!apiKey) throw new Error('Missing required env: ARK_API_KEY');
  const apiUrl = env.ARK_API_URL || 'https://ark.cn-beijing.volces.com/api/v3/chat/completions';
  const detectModel = env.DAMAGE_DETECT_MODEL || 'doubao-seed-2-0-pro-260215';
  const groundingModel = env.DAMAGE_GROUNDING_MODEL || detectModel;
  const bestFrameModel = env.BEST_FRAME_MODEL || detectModel;
  return { apiKey, apiUrl, detectModel, groundingModel, bestFrameModel };
}

function encodeImageDataUri(filePath) {
  const buffer = fs.readFileSync(filePath);
  return `data:image/jpeg;base64,${buffer.toString('base64')}`;
}

function createTimestampedImageContent(frames) {
  const content = [];
  for (const frame of frames) {
    content.push({ type: 'text', text: `[${Number(frame.timestamp).toFixed(2)} second]` });
    content.push({ type: 'image_url', image_url: { url: encodeImageDataUri(frame.path) } });
  }
  return content;
}

async function callVisionApi({ config, model, messages, maxTokens = 4096, temperature = 0.1, timeoutMs = 600000, maxRetries = 3 }) {
  let lastError = '';
  for (let attempt = 0; attempt <= maxRetries; attempt += 1) {
    try {
      const response = await axios.post(
        config.apiUrl,
        { model, messages, max_tokens: maxTokens, temperature },
        {
          headers: {
            Authorization: `Bearer ${config.apiKey}`,
            'Content-Type': 'application/json',
          },
          timeout: timeoutMs,
          maxBodyLength: Infinity,
          maxContentLength: Infinity,
        },
      );
      return {
        ok: true,
        content: response.data && response.data.choices && response.data.choices[0] && response.data.choices[0].message
          ? response.data.choices[0].message.content || ''
          : '',
        usage: response.data ? response.data.usage : undefined,
      };
    } catch (error) {
      const status = error.response && error.response.status;
      const body = error.response && error.response.data ? JSON.stringify(error.response.data).slice(0, 500) : error.message;
      lastError = `API调用失败(${status || 'network'}): ${body}`;
      const retryable = status === 429 || status === 500 || status === 502 || status === 503 || status === 504;
      if (retryable && attempt < maxRetries) {
        await sleep(1000 * Math.pow(2, attempt));
        continue;
      }
      break;
    }
  }
  return { ok: false, error: lastError };
}

module.exports = {
  resolveModelConfig,
  encodeImageDataUri,
  createTimestampedImageContent,
  callVisionApi,
};

Step 4: 运行测试确认通过

Run: node packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs

Expected: exits 0.

Step 5: 提交

git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/vision_client.cjs packages/backend/skills/vehicle-damage-inspection/tests/vision_client.test.cjs
git commit -m "feat(skills): add doubao vision client for vehicle inspection"

Task 7: Damage candidate detector

Files:

Create: packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_detector.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs
Step 1: 写 mock 检测测试

Create packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs:

const assert = require('node:assert');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { createWorkspace, readJson } = require('../scripts/lib/workspace.cjs');
const { detectDamageCandidates } = require('../scripts/lib/damage_detector.cjs');

(async () => {
  const root = fs.mkdtempSync(path.join(os.tmpdir(), 'damage-detector-'));
  try {
    const workspace = createWorkspace({ taskId: 'task-detect', workspaceRoot: root });
    const framePath = path.join(workspace.workspacePath, 'frames', 'frame_000001_0.00s.jpg');
    fs.writeFileSync(framePath, Buffer.from([1, 2, 3]));
    const frames = [{ index: 0, timestamp: 0, path: framePath, relativePath: 'frames/frame_000001_0.00s.jpg' }];
    const mockVision = async () => ({
      ok: true,
      content: JSON.stringify({
        damages: [{ time_second: 0, part: '左前门', type: '划痕', severity: '轻微', description: '细长划痕', confidence: 'medium' }],
        summary: '发现一处损伤',
      }),
    });

    const result = await detectDamageCandidates({
      workspace,
      frames,
      modelConfig: { detectModel: 'm', apiKey: 'k', apiUrl: 'u' },
      prompt: 'prompt',
      batchSize: 1,
      concurrency: 1,
      callVisionApi: mockVision,
    });

    assert.strictEqual(result.candidates.length, 1);
    assert.strictEqual(result.candidates[0].id, 'cand_001');
    assert.strictEqual(readJson(workspace, 'damage_candidates.json').candidates.length, 1);
  } finally {
    fs.rmSync(root, { recursive: true, force: true });
  }
})();

Step 2: 运行测试确认失败

Run: node packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs

Expected: FAIL with missing module.

Step 3: 实现 damage_detector.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_detector.cjs:

const { parseJsonLoose, runWithConcurrency } = require('./json_utils.cjs');
const { createTimestampedImageContent, callVisionApi: defaultCallVisionApi } = require('./vision_client.cjs');
const { writeJson } = require('./workspace.cjs');

function normalizeCandidate(raw, index, batchNumber) {
  return {
    id: `cand_${String(index + 1).padStart(3, '0')}`,
    timestamp: Number(raw.time_second ?? raw.timestamp ?? 0),
    part: raw.part || raw.location || '',
    type: raw.type || '',
    severity: raw.severity || '',
    description: raw.description || '',
    confidence: raw.confidence || 'medium',
    uncertain: Boolean(raw.uncertain),
    batch: batchNumber,
  };
}

async function detectDamageCandidates(params) {
  const {
    workspace,
    frames,
    modelConfig,
    prompt,
    batchSize,
    concurrency,
    callVisionApi = defaultCallVisionApi,
  } = params;

  const batches = [];
  for (let start = 0; start < frames.length; start += batchSize) {
    batches.push(frames.slice(start, start + batchSize));
  }

  const batchRecords = [];
  const collected = [];

  await runWithConcurrency(batches.map((batch, batchIndex) => async () => {
    const content = [
      { type: 'text', text: prompt },
      ...createTimestampedImageContent(batch),
      { type: 'text', text: '请只输出 JSON。' },
    ];
    const result = await callVisionApi({
      config: modelConfig,
      model: modelConfig.detectModel,
      messages: [{ role: 'user', content }],
    });
    const record = {
      batch: batchIndex + 1,
      frameStart: batch[0].index,
      frameEnd: batch[batch.length - 1].index,
      status: result.ok ? 'success' : 'error',
      error: result.ok ? undefined : result.error,
    };
    batchRecords[batchIndex] = record;
    if (!result.ok) return;
    const parsed = parseJsonLoose(result.content || '');
    const damages = Array.isArray(parsed && parsed.damages) ? parsed.damages : [];
    for (const damage of damages) {
      collected.push({ damage, batchNumber: batchIndex + 1 });
    }
  }), concurrency);

  const candidates = collected.map((item, index) => normalizeCandidate(item.damage, index, item.batchNumber));
  const output = {
    totalFrames: frames.length,
    batches: batchRecords,
    candidates,
  };
  writeJson(workspace, 'damage_candidates.json', output);
  return output;
}

module.exports = {
  detectDamageCandidates,
  normalizeCandidate,
};

Step 4: 运行测试确认通过

Run: node packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs

Expected: exits 0.

Step 5: 提交

git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_detector.cjs packages/backend/skills/vehicle-damage-inspection/tests/damage_detector.test.cjs
git commit -m "feat(skills): detect vehicle damage candidates from frames"

Task 8: Image marker and damage grounding

Files:

Create: packages/backend/skills/vehicle-damage-inspection/scripts/lib/image_marker.cjs
Create: packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_grounding.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs
Step 1: 写 image marker 测试

Create packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs:

const assert = require('node:assert');
const { normalizedToPixel } = require('../scripts/lib/image_marker.cjs');

assert.deepStrictEqual(
  normalizedToPixel({ x1: 100, y1: 200, x2: 300, y2: 400 }, { width: 1000, height: 500 }),
  { x1: 100, y1: 100, x2: 300, y2: 200 },
);

Step 2: 写 grounding mock 测试

Create packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs:

const assert = require('node:assert');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { createWorkspace, readJson } = require('../scripts/lib/workspace.cjs');
const { groundDamages } = require('../scripts/lib/damage_grounding.cjs');

(async () => {
  const root = fs.mkdtempSync(path.join(os.tmpdir(), 'damage-grounding-'));
  try {
    const workspace = createWorkspace({ taskId: 'task-ground', workspaceRoot: root });
    const framePath = path.join(workspace.workspacePath, 'frames', 'frame_000001_0.00s.jpg');
    fs.writeFileSync(framePath, Buffer.from([1, 2, 3]));
    const frames = [{ index: 0, timestamp: 0, path: framePath, relativePath: 'frames/frame_000001_0.00s.jpg' }];
    const candidates = [{ id: 'cand_001', timestamp: 0, part: '左前门', type: '划痕', severity: '轻微', description: '细长划痕' }];
    const mockVision = async () => ({ ok: true, content: '[0.00 second] <bbox>10 20 30 40</bbox>' });
    const mockMarker = async ({ dstPath }) => {
      fs.writeFileSync(dstPath, 'marked');
      return true;
    };

    const result = await groundDamages({
      workspace,
      frames,
      candidates,
      modelConfig: { groundingModel: 'm', apiKey: 'k', apiUrl: 'u' },
      prompt: 'prompt',
      groundingWindow: 1.5,
      callVisionApi: mockVision,
      drawBbox: mockMarker,
    });

    assert.strictEqual(result.annotations.length, 1);
    assert.strictEqual(result.annotations[0].markedFrames.length, 1);
    assert.strictEqual(readJson(workspace, 'damage_annotations.json').annotations.length, 1);
  } finally {
    fs.rmSync(root, { recursive: true, force: true });
  }
})();

Step 3: 运行测试确认失败

Run:

node packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs
node packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs

Expected: both FAIL with missing module.

Step 4: 实现 image_marker.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/lib/image_marker.cjs:

const fs = require('node:fs');
const path = require('node:path');
const sharp = require('sharp');

function normalizedToPixel(box, metadata) {
  const width = metadata.width || 1;
  const height = metadata.height || 1;
  return {
    x1: Math.round((box.x1 * width) / 1000),
    y1: Math.round((box.y1 * height) / 1000),
    x2: Math.round((box.x2 * width) / 1000),
    y2: Math.round((box.y2 * height) / 1000),
  };
}

async function drawBbox({ srcPath, dstPath, bboxes, label }) {
  fs.mkdirSync(path.dirname(dstPath), { recursive: true });
  const metadata = await sharp(srcPath).metadata();
  const width = metadata.width || 1;
  const height = metadata.height || 1;
  const parts = [];
  for (const box of bboxes) {
    const pixel = normalizedToPixel(box, { width, height });
    const rectWidth = Math.max(1, pixel.x2 - pixel.x1);
    const rectHeight = Math.max(1, pixel.y2 - pixel.y1);
    parts.push(`<rect x="${pixel.x1}" y="${pixel.y1}" width="${rectWidth}" height="${rectHeight}" fill="none" stroke="red" stroke-width="4"/>`);
    if (label) {
      const safeLabel = String(label).replace(/[<>&"]/g, '');
      parts.push(`<text x="${pixel.x1}" y="${Math.max(18, pixel.y1 - 6)}" font-size="18" fill="red">${safeLabel}</text>`);
    }
  }
  const overlay = Buffer.from(`<svg width="${width}" height="${height}" xmlns="http://www.w3.org/2000/svg">${parts.join('')}</svg>`);
  await sharp(srcPath).composite([{ input: overlay, top: 0, left: 0 }]).jpeg({ quality: 90 }).toFile(dstPath);
  return true;
}

module.exports = {
  normalizedToPixel,
  drawBbox,
};

Step 5: 实现 damage_grounding.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_grounding.cjs:

const fs = require('node:fs');
const path = require('node:path');
const { parseBboxes } = require('./json_utils.cjs');
const { createTimestampedImageContent, callVisionApi: defaultCallVisionApi } = require('./vision_client.cjs');
const { drawBbox: defaultDrawBbox, normalizedToPixel } = require('./image_marker.cjs');
const { writeJson } = require('./workspace.cjs');

function selectNearbyFrames(frames, timestamp, windowSeconds) {
  return frames.filter(frame => Math.abs(Number(frame.timestamp) - Number(timestamp)) <= windowSeconds);
}

async function groundDamages(params) {
  const {
    workspace,
    frames,
    candidates,
    modelConfig,
    prompt,
    groundingWindow,
    callVisionApi = defaultCallVisionApi,
    drawBbox = defaultDrawBbox,
  } = params;

  const annotations = [];
  for (let i = 0; i < candidates.length; i += 1) {
    const candidate = candidates[i];
    const nearbyFrames = selectNearbyFrames(frames, candidate.timestamp, groundingWindow);
    const content = [
      ...createTimestampedImageContent(nearbyFrames),
      { type: 'text', text: `${prompt}\n损伤描述：${candidate.part} ${candidate.type} ${candidate.description}` },
    ];
    const result = await callVisionApi({
      config: modelConfig,
      model: modelConfig.groundingModel,
      messages: [{ role: 'user', content }],
      maxTokens: 2048,
      temperature: 0.1,
    });
    const bboxes = result.ok ? parseBboxes(result.content || '') : [];
    const markedFrames = [];
    for (const frame of nearbyFrames) {
      const markedName = `dmg_${String(i + 1).padStart(3, '0')}_${Number(frame.timestamp).toFixed(2)}s.jpg`;
      const dstPath = path.join(workspace.workspacePath, 'marked_frames', markedName);
      if (bboxes.length > 0) {
        await drawBbox({ srcPath: frame.path, dstPath, bboxes, label: `${candidate.part}-${candidate.type}` });
      } else {
        fs.copyFileSync(frame.path, dstPath);
      }
      const pixel = bboxes[0] ? normalizedToPixel(bboxes[0], { width: 1000, height: 1000 }) : null;
      markedFrames.push({
        timestamp: frame.timestamp,
        sourceRelativePath: frame.relativePath,
        markedRelativePath: `marked_frames/${markedName}`,
        path: dstPath,
        bbox: bboxes[0] ? { normalized: bboxes[0], pixel } : null,
        groundingStatus: bboxes.length > 0 ? 'bbox' : 'no_bbox',
        groundingRaw: result.ok ? result.content : result.error,
      });
    }
    annotations.push({
      candidateId: candidate.id,
      damageId: `dmg_${String(i + 1).padStart(3, '0')}`,
      part: candidate.part,
      type: candidate.type,
      severity: candidate.severity,
      description: candidate.description,
      markedFrames,
    });
  }

  const output = { annotations };
  writeJson(workspace, 'damage_annotations.json', output);
  return output;
}

module.exports = {
  selectNearbyFrames,
  groundDamages,
};

Step 6: 运行测试确认通过

Run:

node packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs
node packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs

Expected: both commands exit 0.

Step 7: 提交

git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/image_marker.cjs packages/backend/skills/vehicle-damage-inspection/scripts/lib/damage_grounding.cjs packages/backend/skills/vehicle-damage-inspection/tests/image_marker.test.cjs packages/backend/skills/vehicle-damage-inspection/tests/damage_grounding.test.cjs
git commit -m "feat(skills): ground and mark vehicle damage frames"

Task 9: Best frame selector

Files:

Create: packages/backend/skills/vehicle-damage-inspection/scripts/lib/best_frame_selector.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs
Step 1: 写最佳帧筛选测试

Create packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs:

const assert = require('node:assert');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { createWorkspace, readJson } = require('../scripts/lib/workspace.cjs');
const { heuristicFrameScore, selectBestFrames, buildBestFramePrompt } = require('../scripts/lib/best_frame_selector.cjs');

(async () => {
  assert.ok(heuristicFrameScore({ bbox: { normalized: { x1: 100, y1: 100, x2: 300, y2: 300 } } }) > heuristicFrameScore({ bbox: null }));

  const root = fs.mkdtempSync(path.join(os.tmpdir(), 'best-frame-'));
  try {
    const workspace = createWorkspace({ taskId: 'task-best', workspaceRoot: root });
    const annotations = [{
      damageId: 'dmg_001',
      part: '左前门',
      type: '划痕',
      severity: '轻微',
      description: '细长划痕',
      markedFrames: [
        { timestamp: 0, markedRelativePath: 'marked_frames/a.jpg', path: 'a.jpg', bbox: null },
        { timestamp: 1, markedRelativePath: 'marked_frames/b.jpg', path: 'b.jpg', bbox: { normalized: { x1: 100, y1: 100, x2: 300, y2: 300 } } },
      ],
    }];
    const result = await selectBestFrames({
      workspace,
      annotations,
      topN: 1,
      modelConfig: { bestFrameModel: 'doubao-seed-2-0-pro-260215' },
      prompt: '选择最佳证据帧',
      callVisionApi: async () => ({ ok: true, content: '{"best_timestamps":[1],"reasons":["bbox准确"]}' }),
    });
    assert.ok(buildBestFramePrompt(annotations[0], '选择最佳证据帧').includes('dmg_001'));
    assert.strictEqual(result.damages[0].bestFrames[0].timestamp, 1);
    assert.strictEqual(result.damages[0].bestFrames[0].selectionReason, 'bbox准确');
    assert.strictEqual(readJson(workspace, 'best_frames.json').bestFrameCount, 1);
  } finally {
    fs.rmSync(root, { recursive: true, force: true });
  }
})();

Step 2: 运行测试确认失败

Run: node packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs

Expected: FAIL with missing module.

Step 3: 实现 best_frame_selector.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/lib/best_frame_selector.cjs:

const { writeJson } = require('./workspace.cjs');
const { parseJsonLoose } = require('./json_utils.cjs');
const { callVisionApi: defaultCallVisionApi } = require('./vision_client.cjs');

function heuristicFrameScore(frame) {
  let score = 0;
  if (frame.bbox && frame.bbox.normalized) {
    score += 100;
    const box = frame.bbox.normalized;
    const area = Math.max(0, box.x2 - box.x1) * Math.max(0, box.y2 - box.y1);
    if (area >= 10000 && area <= 250000) score += 30;
    if (box.x1 > 10 && box.y1 > 10 && box.x2 < 990 && box.y2 < 990) score += 20;
  }
  return score;
}

function dedupeByTimestamp(frames) {
  const map = new Map();
  for (const frame of frames) {
    const key = Number(frame.timestamp).toFixed(2);
    const existing = map.get(key);
    if (!existing || heuristicFrameScore(frame) > heuristicFrameScore(existing)) {
      map.set(key, frame);
    }
  }
  return Array.from(map.values());
}

function buildBestFramePrompt(annotation, prompt) {
  const frames = (annotation.markedFrames || []).map(frame => ({
    timestamp: frame.timestamp,
    hasBbox: !!frame.bbox,
    bbox: frame.bbox ? frame.bbox.normalized : null,
  }));
  return `${prompt}\n损伤ID：${annotation.damageId}\n部位：${annotation.part}\n类型：${annotation.type}\n描述：${annotation.description}\n候选帧：${JSON.stringify(frames)}`;
}

function parseBestFrameSelection(content) {
  const parsed = parseJsonLoose(content || '');
  if (!parsed || !Array.isArray(parsed.best_timestamps)) return { timestamps: [], reasons: [] };
  return {
    timestamps: parsed.best_timestamps.map(Number).filter(Number.isFinite),
    reasons: Array.isArray(parsed.reasons) ? parsed.reasons.map(String) : [],
  };
}

async function chooseFramesWithModel({ annotation, candidates, topN, modelConfig, prompt, callVisionApi }) {
  if (!modelConfig || !modelConfig.bestFrameModel || candidates.length <= 1) return null;
  const result = await callVisionApi({
    config: modelConfig,
    model: modelConfig.bestFrameModel,
    messages: [{ role: 'user', content: [{ type: 'text', text: buildBestFramePrompt(annotation, prompt) }] }],
    maxTokens: 1024,
    temperature: 0.1,
  });
  if (!result.ok) return null;
  const selection = parseBestFrameSelection(result.content);
  const picked = [];
  for (const timestamp of selection.timestamps) {
    const frame = candidates.find(item => Math.abs(Number(item.timestamp) - timestamp) < 0.01);
    if (frame && !picked.includes(frame)) picked.push(frame);
    if (picked.length >= topN) break;
  }
  if (picked.length === 0) return null;
  return picked.map((frame, index) => ({ frame, reason: selection.reasons[index] || 'BEST_FRAME_MODEL selected this frame' }));
}

async function selectBestFrames({ workspace, annotations, topN, modelConfig, prompt = '', callVisionApi = defaultCallVisionApi }) {
  const damages = [];
  for (let index = 0; index < annotations.length; index += 1) {
    const annotation = annotations[index];
    const candidates = dedupeByTimestamp(annotation.markedFrames || []);
    const modelSelected = await chooseFramesWithModel({ annotation, candidates, topN, modelConfig, prompt, callVisionApi });
    const selected = modelSelected || candidates
      .slice()
      .sort((a, b) => heuristicFrameScore(b) - heuristicFrameScore(a))
      .slice(0, topN)
      .map(frame => ({ frame, reason: 'heuristic fallback' }));
    const bestFrames = selected.map(({ frame, reason }) => ({
        timestamp: frame.timestamp,
        path: frame.path,
        relativePath: frame.markedRelativePath,
        bbox: frame.bbox ? frame.bbox.normalized : null,
        selectionReason: reason,
      }));
    damages.push({
      id: annotation.damageId || `dmg_${String(index + 1).padStart(3, '0')}`,
      part: annotation.part,
      type: annotation.type,
      severity: annotation.severity,
      description: annotation.description,
      timestamps: candidates.map(frame => frame.timestamp),
      bestFrames,
    });
  }

  const bestFrameCount = damages.reduce((sum, item) => sum + item.bestFrames.length, 0);
  const output = {
    damageCount: damages.length,
    bestFrameCount,
    damages,
  };
  writeJson(workspace, 'best_frames.json', output);
  return output;
}

module.exports = {
  heuristicFrameScore,
  dedupeByTimestamp,
  buildBestFramePrompt,
  parseBestFrameSelection,
  selectBestFrames,
};

Step 4: 运行测试确认通过

Run: node packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs

Expected: exits 0.

Step 5: 提交

git add packages/backend/skills/vehicle-damage-inspection/scripts/lib/best_frame_selector.cjs packages/backend/skills/vehicle-damage-inspection/tests/best_frame_selector.test.cjs
git commit -m "feat(skills): select best vehicle damage evidence frames"

Task 10: Compute-entry pipeline entrypoint

Files:

Create: packages/backend/skills/vehicle-damage-inspection/scripts/index.cjs
Test: packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs
Step 1: 写 frames-only 入口测试

Create packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs:

const assert = require('node:assert');
const { normalizeInput, buildFinalOutput } = require('../scripts/index.cjs');

const input = normalizeInput({ videoUrl: '/upload/a.mp4', fps: '20', quality: '120', mode: 'frames-only' });
assert.strictEqual(input.videoUrl, '/upload/a.mp4');
assert.strictEqual(input.fps, 10);
assert.strictEqual(input.quality, 100);
assert.strictEqual(input.mode, 'frames-only');

const output = buildFinalOutput({
  workspace: { taskId: 'task', workspacePath: 'C:/data/workspace/task' },
  videoInfo: { duration: 1, resolution: '1x1', extractedFrames: 3 },
  candidates: [],
  damages: [],
  artifacts: { videoInfo: 'video_info.json' },
});
assert.strictEqual(output.success, true);
assert.strictEqual(output.summary.frameCount, 3);

Step 2: 运行测试确认失败

Run: node packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs

Expected: FAIL with missing module.

Step 3: 实现 scripts/index.cjs

Create packages/backend/skills/vehicle-damage-inspection/scripts/index.cjs:

const fs = require('node:fs');
const path = require('node:path');
const { clampNumber } = require('./lib/json_utils.cjs');
const { createWorkspace, resolveVideoPath, writeJson } = require('./lib/workspace.cjs');
const { extractFrames } = require('./lib/frame_extractor.cjs');
const { resolveModelConfig } = require('./lib/vision_client.cjs');
const { detectDamageCandidates } = require('./lib/damage_detector.cjs');
const { groundDamages } = require('./lib/damage_grounding.cjs');
const { selectBestFrames } = require('./lib/best_frame_selector.cjs');

function normalizeInput(raw) {
  if (!raw || typeof raw !== 'object') throw new Error('Input JSON object is required');
  if (!raw.videoUrl) throw new Error('videoUrl is required');
  return {
    videoUrl: String(raw.videoUrl),
    taskId: raw.taskId ? String(raw.taskId) : undefined,
    fps: clampNumber(raw.fps, 0.2, 10, 3),
    quality: clampNumber(raw.quality, 1, 100, 90),
    batchSize: clampNumber(raw.batchSize, 1, 20, 12),
    concurrency: clampNumber(raw.concurrency, 1, 4, 2),
    groundingWindow: clampNumber(raw.groundingWindow, 0.5, 5, 1.5),
    topN: clampNumber(raw.topN, 1, 5, 1),
    mode: ['full', 'frames-only', 'detect-only'].includes(raw.mode) ? raw.mode : 'full',
  };
}

function readPrompt(name) {
  return fs.readFileSync(path.join(__dirname, '..', 'prompts', name), 'utf8');
}

function resolveWorkspaceRoot(env) {
  if (env.RZYX_AI_WORKSPACE_ROOT) return env.RZYX_AI_WORKSPACE_ROOT;
  if (env.RZYX_AI_DATA_DIR) return path.join(env.RZYX_AI_DATA_DIR, 'workspace', 'vehicle-damage-inspection');
  if (env.NODE_ENV === 'production') throw new Error('RZYX_AI_DATA_DIR or RZYX_AI_WORKSPACE_ROOT is required in production');
  return path.join(__dirname, '..', '.workspace');
}

function buildFinalOutput({ workspace, videoInfo, candidates, damages, artifacts }) {
  const bestFrameCount = damages.reduce((sum, item) => sum + (item.bestFrames || []).length, 0);
  return {
    success: true,
    taskId: workspace.taskId,
    workspacePath: workspace.workspacePath,
    summary: {
      duration: videoInfo.duration,
      resolution: videoInfo.resolution,
      frameCount: videoInfo.extractedFrames,
      candidateDamageCount: candidates.length,
      mergedDamageCount: damages.length,
      bestFrameCount,
    },
    damages,
    artifacts,
  };
}

async function run(rawInput, env = process.env) {
  const input = normalizeInput(rawInput);
  const workspaceRoot = resolveWorkspaceRoot(env);
  const workspace = createWorkspace({ taskId: input.taskId, workspaceRoot });
  const uploadRoot = env.RZYX_AI_UPLOAD_ROOT || (env.RZYX_AI_DATA_DIR ? path.join(env.RZYX_AI_DATA_DIR, 'uploads') : undefined);
  const videoPath = resolveVideoPath(input.videoUrl, { uploadRoot });

  const { videoInfo, frames } = await extractFrames({
    workspace,
    videoPath,
    fps: input.fps,
    quality: input.quality,
  });

  if (input.mode === 'frames-only') {
    const output = buildFinalOutput({
      workspace,
      videoInfo,
      candidates: [],
      damages: [],
      artifacts: { videoInfo: 'video_info.json' },
    });
    writeJson(workspace, 'run_summary.json', output);
    return output;
  }

  const modelConfig = resolveModelConfig(env);
  const detection = await detectDamageCandidates({
    workspace,
    frames,
    modelConfig,
    prompt: readPrompt('damage_detect.md'),
    batchSize: input.batchSize,
    concurrency: input.concurrency,
  });

  if (input.mode === 'detect-only' || detection.candidates.length === 0) {
    const output = buildFinalOutput({
      workspace,
      videoInfo,
      candidates: detection.candidates,
      damages: [],
      artifacts: { videoInfo: 'video_info.json', damageCandidates: 'damage_candidates.json' },
    });
    writeJson(workspace, 'run_summary.json', output);
    return output;
  }

  const grounding = await groundDamages({
    workspace,
    frames,
    candidates: detection.candidates,
    modelConfig,
    prompt: readPrompt('grounding.md'),
    groundingWindow: input.groundingWindow,
  });

  const best = await selectBestFrames({
    workspace,
    annotations: grounding.annotations,
    topN: input.topN,
    modelConfig,
    prompt: readPrompt('best_frame.md'),
  });

  const output = buildFinalOutput({
    workspace,
    videoInfo,
    candidates: detection.candidates,
    damages: best.damages,
    artifacts: {
      videoInfo: 'video_info.json',
      damageCandidates: 'damage_candidates.json',
      damageAnnotations: 'damage_annotations.json',
      bestFrames: 'best_frames.json',
      runSummary: 'run_summary.json',
    },
  });
  writeJson(workspace, 'run_summary.json', output);
  return output;
}

async function main() {
  try {
    const stdin = fs.readFileSync(0, 'utf8');
    const input = JSON.parse(stdin || '{}');
    const output = await run(input);
    process.stdout.write(JSON.stringify(output));
  } catch (error) {
    process.stdout.write(JSON.stringify({ success: false, error: error.message }));
    process.exitCode = 0;
  }
}

if (require.main === module) {
  main();
}

module.exports = {
  normalizeInput,
  buildFinalOutput,
  run,
};

Step 4: 运行测试确认通过

Run: node packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs

Expected: exits 0.

Step 5: 提交

git add packages/backend/skills/vehicle-damage-inspection/scripts/index.cjs packages/backend/skills/vehicle-damage-inspection/tests/index_frames_only.test.cjs
git commit -m "feat(skills): wire vehicle inspection compute entrypoint"

Task 11: make.sh and Windows setup diagnostics

Files:

Create: packages/backend/skills/vehicle-damage-inspection/scripts/make.sh
Create: packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1
Step 1: 创建 make.sh

Create packages/backend/skills/vehicle-damage-inspection/scripts/make.sh:

#!/usr/bin/env bash
set -euo pipefail

ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"

cmd_check() {
  local ok=true
  command -v node >/dev/null 2>&1 || { echo "node missing"; ok=false; }
  command -v ffmpeg >/dev/null 2>&1 || echo "ffmpeg not found in PATH; npm installer fallback may be used"
  command -v ffprobe >/dev/null 2>&1 || echo "ffprobe not found in PATH; npm installer fallback may be used"
  node -e "require('axios'); require('sharp'); require('fluent-ffmpeg')" || ok=false
  $ok
}

cmd_fix() {
  cd "$ROOT"
  npm install axios sharp fluent-ffmpeg @ffmpeg-installer/ffmpeg @ffprobe-installer/ffprobe
}

cmd_run() {
  local video=""
  local fps="3"
  local mode="full"
  while [[ $# -gt 0 ]]; do
    case "$1" in
      --video) video="$2"; shift 2 ;;
      --fps) fps="$2"; shift 2 ;;
      --mode) mode="$2"; shift 2 ;;
      *) echo "Unknown option: $1"; exit 1 ;;
    esac
  done
  if [[ -z "$video" ]]; then
    echo "Usage: bash scripts/make.sh run --video FILE [--fps 3] [--mode full]"
    exit 1
  fi
  printf '{"videoUrl":%s,"fps":%s,"mode":%s}\n' \
    "$(node -e "process.stdout.write(JSON.stringify(process.argv[1]))" "$video")" \
    "$fps" \
    "$(node -e "process.stdout.write(JSON.stringify(process.argv[1]))" "$mode")" \
    | node "$ROOT/scripts/index.cjs"
}

cmd_tests() {
  for file in "$ROOT"/tests/*.test.cjs; do
    node "$file"
  done
}

case "${1:-}" in
  check) cmd_check ;;
  fix) cmd_fix ;;
  run) shift; cmd_run "$@" ;;
  test) cmd_tests ;;
  *) echo "Usage: bash scripts/make.sh check|fix|run|test"; exit 1 ;;
esac

Step 2: 运行本地测试集合

Run: bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh test

Expected: every *.test.cjs exits 0.

Step 3: 运行 check

Run: bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh check

Expected: exits 0 if node packages are installed. If it fails because packages are not installed in the skill directory, run bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh fix, then rerun check.

Step 4: 提交
Step 4: 创建 Windows setup.ps1

Create packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1:

$ErrorActionPreference = "Stop"

$SkillRoot = Resolve-Path (Join-Path $PSScriptRoot "..")
Set-Location $SkillRoot

npm install axios sharp fluent-ffmpeg @ffmpeg-installer/ffmpeg @ffprobe-installer/ffprobe

Step 5: 验证 Windows setup 脚本语法

Run: powershell -NoProfile -ExecutionPolicy Bypass -File packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1

Expected: npm dependencies install successfully in the skill directory. For production packaging, run this during build/install, not during every skill execution.

Step 6: 提交

git add packages/backend/skills/vehicle-damage-inspection/scripts/make.sh packages/backend/skills/vehicle-damage-inspection/scripts/setup.ps1
git commit -m "feat(skills): add vehicle inspection skill cli"

Task 12: End-to-end backend integration validation

Files:

Test only: no source files expected unless validation exposes a bug.
Step 1: 运行 Skill 纯测试

Run: bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh test

Expected: exits 0.

Step 2: 运行后端 runtime env 测试

Run: pnpm --filter @neta/backend test -- --runInBand test/skill_runtime_env.test.ts

Expected: PASS.

Step 3: 运行后端 build

Run: pnpm --filter @neta/backend build

Expected: build completes without TypeScript errors.

Step 4: 模拟 Windows dataDir 路径运行 frames-only

Prepare a small MP4 file. If no sample is available, use ffmpeg to generate one:

mkdir -p C:/tmp/rzyx-ai-skill-test/uploads/20260504
ffmpeg -f lavfi -i testsrc=duration=2:size=320x240:rate=10 C:/tmp/rzyx-ai-skill-test/uploads/20260504/sample.mp4 -y

Run:

RZYX_AI_DATA_DIR=C:/tmp/rzyx-ai-skill-test \
RZYX_AI_UPLOAD_ROOT=C:/tmp/rzyx-ai-skill-test/uploads \
RZYX_AI_WORKSPACE_ROOT=C:/tmp/rzyx-ai-skill-test/workspace/vehicle-damage-inspection \
bash packages/backend/skills/vehicle-damage-inspection/scripts/make.sh run \
  --video /upload/20260504/sample.mp4 \
  --fps 1 \
  --mode frames-only

Expected:

stdout is valid JSON with "success":true.
C:/tmp/rzyx-ai-skill-test/workspace/vehicle-damage-inspection/.../video_info.json exists.
No workspace directory is created under packages/backend/skills/vehicle-damage-inspection during this production-path simulation.
Step 5: 提交 validation fixes only if needed

If any validation step required code fixes, first inspect the exact modified files:

git status --short

Then add only files related to this skill implementation. The expected fix scope is one or more of these paths:

git add packages/backend/src/modules/netaclaw/service/skill_runtime_env.ts packages/backend/src/modules/netaclaw/service/skill_executor.ts packages/backend/test/skill_runtime_env.test.ts packages/backend/skills/vehicle-damage-inspection
git commit -m "fix(skills): stabilize vehicle inspection integration"

If no files changed, skip this commit.

Self-Review Checklist

Spec coverage: Tasks cover host env injection, Skill metadata, prompts, workspace protocol, frame extraction, vision client, detection, grounding, image marking, best-frame selection, compute entrypoint, CLI, tests, and Windows dataDir validation.
Placeholder scan: The plan contains no unresolved markers, no undefined file paths, and no deferred-implementation instructions.
Type consistency: The public Skill name is consistently vehicle-damage-inspection; runtime env names are consistently RZYX_AI_DATA_DIR, RZYX_AI_WORKSPACE_ROOT, and RZYX_AI_UPLOAD_ROOT; default model is consistently doubao-seed-2-0-pro-260215.

72 KiB Raw Blame History Unescape Escape

汽车环车视频旧伤检测 Skill Implementation Plan

File Map

Task 1: Skill env schema 同步

Task 2: 宿主 runtime env 注入

Task 3: Skill scaffold and metadata

输入

输出

本地运行

必需模型

Task 4: JSON utilities and workspace module

Task 5: Frame extraction module

Task 6: Vision client module

Task 7: Damage candidate detector

Task 8: Image marker and damage grounding

Task 9: Best frame selector

Task 10: Compute-entry pipeline entrypoint

Task 11: make.sh and Windows setup diagnostics

Task 12: End-to-end backend integration validation

Self-Review Checklist

72 KiB

Raw Blame History