docs(plans): add goods data cleaning feature plan and pipeline folder
This commit is contained in:
yeuimu
@@ -0,0 +1,280 @@
|
||||
# Goods Data Cleaning Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 用「备份 → 导出 → 脚本转换 → 导入 → 验证」的流水线，把右侧 SDS 原产品库（`origin_goods`）按规则批量配置为左侧官网商品（`goods`），替代后台手动逐个点击。直接作用于生产库。
|
||||
|
||||
**Background:**
|
||||
- 后台 GoodsView 已实现手动配置能力：右侧树选原产品 → 选国家/品类/标签 → 生成 `goods` 记录。
|
||||
- 手动逐个点击效率低，需要脚本化批量处理。
|
||||
- 同步规则（品类映射 / 国家分配 / 标签分配 / 优先级策略）**尚未确定**，将在后续数据清洗计划中明确后填入 `data-cleaning/sync-rules.config.mjs`。
|
||||
|
||||
**Architecture:** 三步式流水线，复用现有 export/import 脚本，只新写转换脚本与独立规则文件。规则文件占位先行——改规则不改代码。转换脚本为纯函数式（JSON in → JSON out），不直接连接数据库；导入前输出变更摘要报告作为人工检查点；导入前必须完成全量备份保证可回滚。
|
||||
|
||||
**Tech Stack:** Node.js ESM 脚本、@prisma/client（仅 export/import 使用）、PostgreSQL。
|
||||
|
||||
**目录约定：**
|
||||
- 清洗相关脚本/规则/产物统一放根目录 `data-cleaning/`（见 [data-cleaning/README.md](../../data-cleaning/README.md)）
|
||||
- 复用现有 [apps/api/scripts/export-data.mjs](../../apps/api/scripts/export-data.mjs) 与 [apps/api/scripts/import-data.mjs](../../apps/api/scripts/import-data.mjs)
|
||||
|
||||
---
|
||||
|
||||
## 流水线总览
|
||||
|
||||
```
|
||||
[生产库]
|
||||
│ ① pg_dump 全量备份
|
||||
▼
|
||||
② node scripts/export-data.mjs data-cleaning/runs/<date>/export.json
|
||||
▼
|
||||
③ node data-cleaning/sync-transform.mjs data-cleaning/runs/<date>/export.json
|
||||
→ 输出 transformed.json + change-report.md（人工检查点）
|
||||
▼
|
||||
④ node scripts/import-data.mjs data-cleaning/runs/<date>/transformed.json
|
||||
▼
|
||||
⑤ 验证：行数对比 / 抽查商品配置 / 官网公开 API 抽查
|
||||
```
|
||||
|
||||
### Task 1: 创建规则文件占位 `sync-rules.config.mjs`
|
||||
|
||||
**Files:**
|
||||
- Create: `data-cleaning/sync-rules.config.mjs`
|
||||
- Test: 无（纯数据文件，由 Task 2 的测试覆盖加载逻辑）
|
||||
|
||||
- [x] **Step 1: 创建规则文件骨架**
|
||||
|
||||
```js
|
||||
/**
|
||||
* 商品数据清洗 — 同步规则配置
|
||||
*
|
||||
* 规则尚未确定。确定后只修改本文件，不改 sync-transform.mjs。
|
||||
* 字段语义在规则确定时补充说明。
|
||||
*/
|
||||
export const rules = {
|
||||
/** 品类映射：SDS 品类 → 本地品类（待定） */
|
||||
categoryMapping: {
|
||||
// '<sds_category_id 或名称>': '<本地 category_id 或名称>',
|
||||
},
|
||||
|
||||
/** 国家分配：每条原产品生成哪些国家的 good（待定） */
|
||||
countryAssignment: {
|
||||
mode: 'none', // none | all | fixed | perCategory
|
||||
fixedCountryIds: [],
|
||||
perCategory: {},
|
||||
},
|
||||
|
||||
/** 标签分配：新 good 挂哪些 tag（待定） */
|
||||
tagAssignment: {
|
||||
mode: 'none', // none | fixed | perCategory
|
||||
fixedTagIds: [],
|
||||
perCategory: {},
|
||||
},
|
||||
|
||||
/** 优先级策略（待定） */
|
||||
priority: {
|
||||
defaultPriority: 0,
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
- [x] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add data-cleaning/sync-rules.config.mjs
|
||||
git commit -m "feat(data-cleaning): add sync rules config placeholder"
|
||||
```
|
||||
|
||||
### Task 2: 创建转换脚本 `sync-transform.mjs`（TDD）
|
||||
|
||||
**Files:**
|
||||
- Create: `data-cleaning/sync-transform.mjs`
|
||||
- Test: `data-cleaning/sync-transform.test.mjs`
|
||||
|
||||
- [x] **Step 1: 写失败测试**
|
||||
|
||||
```js
|
||||
// data-cleaning/sync-transform.test.mjs
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { transform } from './sync-transform.mjs';
|
||||
import { rules } from './sync-rules.config.mjs';
|
||||
|
||||
const baseDump = () => ({
|
||||
exportedAt: new Date().toISOString(),
|
||||
tables: {
|
||||
users: [], countries: [], categories: [], tag_groups: [], tags: [],
|
||||
positions: [],
|
||||
origin_goods: [
|
||||
{ id: '1', sds_good_id: 'SDS-A', good_name: 'A', delisted: 'false', is_custom: 'false' },
|
||||
{ id: '2', sds_good_id: 'SDS-B', good_name: 'B', delisted: 'true', is_custom: 'false' },
|
||||
],
|
||||
origin_good_variants: [], origin_good_details: [],
|
||||
goods: [], good_tags: [], sync_logs: [],
|
||||
},
|
||||
});
|
||||
|
||||
describe('transform', () => {
|
||||
it('rules 为空时原样返回且报告零变更', () => {
|
||||
const dump = baseDump();
|
||||
const { result, report } = transform(dump, rules);
|
||||
expect(result.tables.goods).toHaveLength(0);
|
||||
expect(report.created).toBe(0);
|
||||
expect(result.tables.origin_goods).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('跳过已下架原产品', () => {
|
||||
const dump = baseDump();
|
||||
const testRules = { ...rules, countryAssignment: { mode: 'all' } };
|
||||
const { report } = transform(dump, testRules);
|
||||
// 只有未下架的 id=1 会生成 good
|
||||
expect(report.created).toBe(1);
|
||||
expect(report.skippedDelisted).toBe(1);
|
||||
});
|
||||
|
||||
it('幂等：已存在的 (originGoodId,countryId) 不重复创建', () => {
|
||||
const dump = baseDump();
|
||||
dump.tables.goods = [
|
||||
{ id: '100', origin_good_id: '1', country_id: '9', good_name: 'A', good_priority: '0' },
|
||||
];
|
||||
dump.tables.countries = [{ id: '9', country_name: 'US' }];
|
||||
const testRules = {
|
||||
...rules,
|
||||
countryAssignment: { mode: 'fixed', fixedCountryIds: ['9'] },
|
||||
};
|
||||
const { report } = transform(dump, testRules);
|
||||
expect(report.created).toBe(0);
|
||||
expect(report.duplicates).toBe(1);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
> 注：若仓库未配置 vitest 运行 `.mjs`，可用 `node --test`（node:test）替代，断言等价改写。
|
||||
|
||||
- [x] **Step 2: 运行测试确认 RED**
|
||||
|
||||
Run: `cd apps/api && npx vitest run ../../data-cleaning/sync-transform.test.mjs`（或 `node --test data-cleaning/`）
|
||||
Expected: FAIL（模块不存在）
|
||||
|
||||
- [x] **Step 3: 最小实现**
|
||||
|
||||
```js
|
||||
// data-cleaning/sync-transform.mjs
|
||||
/**
|
||||
* 数据转换脚本（纯函数式）：读取导出 JSON + 规则文件，
|
||||
* 输出待导入 JSON（transformed.json）与变更摘要（change-report.md）。
|
||||
*
|
||||
* Usage:
|
||||
* node data-cleaning/sync-transform.mjs <export.json>
|
||||
*/
|
||||
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
|
||||
import { dirname } from 'node:path';
|
||||
import { rules } from './sync-rules.config.mjs';
|
||||
|
||||
/** 纯函数：dump + rules → { result, report } */
|
||||
export function transform(dump, rulesConfig) {
|
||||
const tables = JSON.parse(JSON.stringify(dump.tables)); // deep clone
|
||||
const countries = tables.countries;
|
||||
const existingKeys = new Set(
|
||||
tables.goods.map((g) => `${g.origin_good_id}:${g.country_id}`),
|
||||
);
|
||||
|
||||
const report = { created: 0, duplicates: 0, skippedDelisted: 0, details: [] };
|
||||
const activeCountries = countries.filter((c) => c.id);
|
||||
|
||||
for (const og of tables.origin_goods) {
|
||||
if (og.delisted === 'true' || og.delisted === true) {
|
||||
if (og.is_custom !== 'true' && og.is_custom !== true) report.skippedDelisted++;
|
||||
continue;
|
||||
}
|
||||
let targets = [];
|
||||
if (rulesConfig.countryAssignment.mode === 'all') {
|
||||
targets = activeCountries.map((c) => c.id);
|
||||
} else if (rulesConfig.countryAssignment.mode === 'fixed') {
|
||||
targets = rulesConfig.countryAssignment.fixedCountryIds;
|
||||
}
|
||||
|
||||
for (const countryId of targets) {
|
||||
const key = `${og.id}:${countryId}`;
|
||||
if (existingKeys.has(key)) { report.duplicates++; continue; }
|
||||
const newId = String(
|
||||
tables.goods.reduce((m, g) => Math.max(m, Number(g.id || 0)), 0) + 1,
|
||||
);
|
||||
tables.goods.push({
|
||||
id: newId,
|
||||
origin_good_id: og.id,
|
||||
country_id: countryId,
|
||||
category_id: rulesConfig.categoryMapping.default ?? '',
|
||||
good_name: og.good_name,
|
||||
good_priority: String(rulesConfig.priority.defaultPriority ?? 0),
|
||||
});
|
||||
existingKeys.add(key);
|
||||
report.created++;
|
||||
report.details.push(`+ good[${newId}] origin=${og.sds_good_id} country=${countryId}`);
|
||||
}
|
||||
}
|
||||
return { result: { exportedAt: dump.exportedAt, tables }, report };
|
||||
}
|
||||
|
||||
function main() {
|
||||
const input = process.argv[2];
|
||||
if (!input) {
|
||||
console.error('Usage: node data-cleaning/sync-transform.mjs <export.json>');
|
||||
process.exit(1);
|
||||
}
|
||||
const dump = JSON.parse(readFileSync(input, 'utf8'));
|
||||
const { result, report } = transform(dump, rules);
|
||||
|
||||
mkdirSync(`${dirname(input)}/out`, { recursive: true });
|
||||
writeFileSync(`${dirname(input)}/out/transformed.json`, JSON.stringify(result));
|
||||
writeFileSync(
|
||||
`${dirname(input)}/out/change-report.md`,
|
||||
['# 变更摘要', `- 新增 goods: ${report.created}`, `- 重复跳过: ${report.duplicates}`,
|
||||
`- 下架跳过: ${report.skippedDelisted}`, '', ...report.details.map((d) => `- ${d}`)].join('\n'),
|
||||
);
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
}
|
||||
|
||||
// 测试环境下不自动执行 main
|
||||
if (process.env.NODE_ENV !== 'test' && import.meta.url === `file://${process.argv[1]}`) {
|
||||
main();
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 4: 运行测试确认 GREEN**
|
||||
|
||||
Run: `cd apps/api && npx vitest run ../../data-cleaning/sync-transform.test.mjs`
|
||||
Expected: PASS（3 个用例全过）
|
||||
|
||||
- [x] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add data-cleaning/sync-transform.mjs data-cleaning/sync-transform.test.mjs
|
||||
git commit -m "feat(data-cleaning): add pure transform script with tests"
|
||||
```
|
||||
|
||||
### Task 3: 生产执行手册写入 README 并联调演练
|
||||
|
||||
**Files:**
|
||||
- Modify: `data-cleaning/README.md`
|
||||
|
||||
- [x] **Step 1: 补充执行命令段**（备份 / 导出 / 转换 / 人工检查 / 导入 / 验证 六个步骤的确切命令与预期输出）
|
||||
|
||||
- [x] **Step 2: 本地或预发演练一次全流程**（规则为空应零变更），确认 change-report 为空、行数一致
|
||||
|
||||
- [x] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add data-cleaning/README.md
|
||||
git commit -m "docs(data-cleaning): add production runbook"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 待定事项（规则确定后回填）
|
||||
|
||||
| 项 | 状态 | 回填位置 |
|
||||
|----|------|----------|
|
||||
| 品类映射规则 | 未定 | `data-cleaning/sync-rules.config.mjs#categoryMapping` |
|
||||
| 国家分配策略 | 未定 | `...#countryAssignment` |
|
||||
| 标签分配策略 | 未定 | `...#tagAssignment` |
|
||||
| 优先级策略 | 未定 | `...#priority` |
|
||||
| 规则细节文档 | 未定 | 后续数据清洗计划文件夹（本目录）内新建规则说明 md |
|
||||
Reference in New Issue
Block a user