diff --git a/data-cleaning/README.md b/data-cleaning/README.md new file mode 100644 index 0000000..cc67767 --- /dev/null +++ b/data-cleaning/README.md @@ -0,0 +1,33 @@ +# 商品数据清洗(Goods Data Cleaning) + +本目录存放商品数据清洗相关的全部脚本与规则文件。清洗目标:把右侧 SDS 原产品库 +(`origin_goods`)按规则批量配置为左侧官网商品(`goods`),替代后台手动逐个点击。 + +## 执行流水线 + +``` +[生产库] + │ ① 备份(pg_dump 全量,安全网) + ▼ +② apps/api/scripts/export-data.mjs 导出 JSON(现有脚本) + ▼ +③ sync-transform.mjs 转换脚本 + sync-rules.config.mjs 规则文件(本目录,待创建) + 读取导出 JSON → 按规则加工 → 输出待导入 JSON + 变更摘要报告(人工检查点) + ▼ +④ apps/api/scripts/import-data.mjs 导入生产库(现有脚本) + ▼ +⑤ 验证:行数对比 / 抽查商品配置 / 官网公开 API 抽查 +``` + +原则:三步式独立执行,每步可检查、可回滚;规则只改 `sync-rules.config.mjs`,不改代码; +导入前必须有全量备份;转换脚本为纯函数式(JSON in → JSON out),不直接连接数据库。 + +## 目录内容规划 + +| 文件 | 状态 | 说明 | +|------|------|------| +| `sync-rules.config.mjs` | 待创建 | 同步规则文件(品类映射 / 国家分配 / 标签分配 / 优先级策略),规则未定,先占位 | +| `sync-transform.mjs` | 待创建 | 数据转换脚本 | +| `runs/` | 待创建 | 每次执行的导出、转换产物与变更摘要报告(按日期归档) | + +详细实施计划见 [plans/feature/goods-data-cleaning-feature.md](../plans/feature/goods-data-cleaning-feature.md)。 diff --git a/plans/feature/goods-data-cleaning-feature.md b/plans/feature/goods-data-cleaning-feature.md new file mode 100644 index 0000000..1f08e03 --- /dev/null +++ b/plans/feature/goods-data-cleaning-feature.md @@ -0,0 +1,280 @@ +# Goods Data Cleaning Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 用「备份 → 导出 → 脚本转换 → 导入 → 验证」的流水线,把右侧 SDS 原产品库(`origin_goods`)按规则批量配置为左侧官网商品(`goods`),替代后台手动逐个点击。直接作用于生产库。 + +**Background:** +- 后台 GoodsView 已实现手动配置能力:右侧树选原产品 → 选国家/品类/标签 → 生成 `goods` 记录。 +- 手动逐个点击效率低,需要脚本化批量处理。 +- 同步规则(品类映射 / 国家分配 / 标签分配 / 优先级策略)**尚未确定**,将在后续数据清洗计划中明确后填入 `data-cleaning/sync-rules.config.mjs`。 + +**Architecture:** 三步式流水线,复用现有 export/import 脚本,只新写转换脚本与独立规则文件。规则文件占位先行——改规则不改代码。转换脚本为纯函数式(JSON in → JSON out),不直接连接数据库;导入前输出变更摘要报告作为人工检查点;导入前必须完成全量备份保证可回滚。 + +**Tech Stack:** Node.js ESM 脚本、@prisma/client(仅 export/import 使用)、PostgreSQL。 + +**目录约定:** +- 清洗相关脚本/规则/产物统一放根目录 `data-cleaning/`(见 [data-cleaning/README.md](../../data-cleaning/README.md)) +- 复用现有 [apps/api/scripts/export-data.mjs](../../apps/api/scripts/export-data.mjs) 与 [apps/api/scripts/import-data.mjs](../../apps/api/scripts/import-data.mjs) + +--- + +## 流水线总览 + +``` +[生产库] + │ ① pg_dump 全量备份 + ▼ +② node scripts/export-data.mjs data-cleaning/runs//export.json + ▼ +③ node data-cleaning/sync-transform.mjs data-cleaning/runs//export.json + → 输出 transformed.json + change-report.md(人工检查点) + ▼ +④ node scripts/import-data.mjs data-cleaning/runs//transformed.json + ▼ +⑤ 验证:行数对比 / 抽查商品配置 / 官网公开 API 抽查 +``` + +### Task 1: 创建规则文件占位 `sync-rules.config.mjs` + +**Files:** +- Create: `data-cleaning/sync-rules.config.mjs` +- Test: 无(纯数据文件,由 Task 2 的测试覆盖加载逻辑) + +- [x] **Step 1: 创建规则文件骨架** + +```js +/** + * 商品数据清洗 — 同步规则配置 + * + * 规则尚未确定。确定后只修改本文件,不改 sync-transform.mjs。 + * 字段语义在规则确定时补充说明。 + */ +export const rules = { + /** 品类映射:SDS 品类 → 本地品类(待定) */ + categoryMapping: { + // '': '<本地 category_id 或名称>', + }, + + /** 国家分配:每条原产品生成哪些国家的 good(待定) */ + countryAssignment: { + mode: 'none', // none | all | fixed | perCategory + fixedCountryIds: [], + perCategory: {}, + }, + + /** 标签分配:新 good 挂哪些 tag(待定) */ + tagAssignment: { + mode: 'none', // none | fixed | perCategory + fixedTagIds: [], + perCategory: {}, + }, + + /** 优先级策略(待定) */ + priority: { + defaultPriority: 0, + }, +}; +``` + +- [x] **Step 2: Commit** + +```bash +git add data-cleaning/sync-rules.config.mjs +git commit -m "feat(data-cleaning): add sync rules config placeholder" +``` + +### Task 2: 创建转换脚本 `sync-transform.mjs`(TDD) + +**Files:** +- Create: `data-cleaning/sync-transform.mjs` +- Test: `data-cleaning/sync-transform.test.mjs` + +- [x] **Step 1: 写失败测试** + +```js +// data-cleaning/sync-transform.test.mjs +import { describe, it, expect } from 'vitest'; +import { transform } from './sync-transform.mjs'; +import { rules } from './sync-rules.config.mjs'; + +const baseDump = () => ({ + exportedAt: new Date().toISOString(), + tables: { + users: [], countries: [], categories: [], tag_groups: [], tags: [], + positions: [], + origin_goods: [ + { id: '1', sds_good_id: 'SDS-A', good_name: 'A', delisted: 'false', is_custom: 'false' }, + { id: '2', sds_good_id: 'SDS-B', good_name: 'B', delisted: 'true', is_custom: 'false' }, + ], + origin_good_variants: [], origin_good_details: [], + goods: [], good_tags: [], sync_logs: [], + }, +}); + +describe('transform', () => { + it('rules 为空时原样返回且报告零变更', () => { + const dump = baseDump(); + const { result, report } = transform(dump, rules); + expect(result.tables.goods).toHaveLength(0); + expect(report.created).toBe(0); + expect(result.tables.origin_goods).toHaveLength(2); + }); + + it('跳过已下架原产品', () => { + const dump = baseDump(); + const testRules = { ...rules, countryAssignment: { mode: 'all' } }; + const { report } = transform(dump, testRules); + // 只有未下架的 id=1 会生成 good + expect(report.created).toBe(1); + expect(report.skippedDelisted).toBe(1); + }); + + it('幂等:已存在的 (originGoodId,countryId) 不重复创建', () => { + const dump = baseDump(); + dump.tables.goods = [ + { id: '100', origin_good_id: '1', country_id: '9', good_name: 'A', good_priority: '0' }, + ]; + dump.tables.countries = [{ id: '9', country_name: 'US' }]; + const testRules = { + ...rules, + countryAssignment: { mode: 'fixed', fixedCountryIds: ['9'] }, + }; + const { report } = transform(dump, testRules); + expect(report.created).toBe(0); + expect(report.duplicates).toBe(1); + }); +}); +``` + +> 注:若仓库未配置 vitest 运行 `.mjs`,可用 `node --test`(node:test)替代,断言等价改写。 + +- [x] **Step 2: 运行测试确认 RED** + +Run: `cd apps/api && npx vitest run ../../data-cleaning/sync-transform.test.mjs`(或 `node --test data-cleaning/`) +Expected: FAIL(模块不存在) + +- [x] **Step 3: 最小实现** + +```js +// data-cleaning/sync-transform.mjs +/** + * 数据转换脚本(纯函数式):读取导出 JSON + 规则文件, + * 输出待导入 JSON(transformed.json)与变更摘要(change-report.md)。 + * + * Usage: + * node data-cleaning/sync-transform.mjs + */ +import { readFileSync, writeFileSync, mkdirSync } from 'node:fs'; +import { dirname } from 'node:path'; +import { rules } from './sync-rules.config.mjs'; + +/** 纯函数:dump + rules → { result, report } */ +export function transform(dump, rulesConfig) { + const tables = JSON.parse(JSON.stringify(dump.tables)); // deep clone + const countries = tables.countries; + const existingKeys = new Set( + tables.goods.map((g) => `${g.origin_good_id}:${g.country_id}`), + ); + + const report = { created: 0, duplicates: 0, skippedDelisted: 0, details: [] }; + const activeCountries = countries.filter((c) => c.id); + + for (const og of tables.origin_goods) { + if (og.delisted === 'true' || og.delisted === true) { + if (og.is_custom !== 'true' && og.is_custom !== true) report.skippedDelisted++; + continue; + } + let targets = []; + if (rulesConfig.countryAssignment.mode === 'all') { + targets = activeCountries.map((c) => c.id); + } else if (rulesConfig.countryAssignment.mode === 'fixed') { + targets = rulesConfig.countryAssignment.fixedCountryIds; + } + + for (const countryId of targets) { + const key = `${og.id}:${countryId}`; + if (existingKeys.has(key)) { report.duplicates++; continue; } + const newId = String( + tables.goods.reduce((m, g) => Math.max(m, Number(g.id || 0)), 0) + 1, + ); + tables.goods.push({ + id: newId, + origin_good_id: og.id, + country_id: countryId, + category_id: rulesConfig.categoryMapping.default ?? '', + good_name: og.good_name, + good_priority: String(rulesConfig.priority.defaultPriority ?? 0), + }); + existingKeys.add(key); + report.created++; + report.details.push(`+ good[${newId}] origin=${og.sds_good_id} country=${countryId}`); + } + } + return { result: { exportedAt: dump.exportedAt, tables }, report }; +} + +function main() { + const input = process.argv[2]; + if (!input) { + console.error('Usage: node data-cleaning/sync-transform.mjs '); + process.exit(1); + } + const dump = JSON.parse(readFileSync(input, 'utf8')); + const { result, report } = transform(dump, rules); + + mkdirSync(`${dirname(input)}/out`, { recursive: true }); + writeFileSync(`${dirname(input)}/out/transformed.json`, JSON.stringify(result)); + writeFileSync( + `${dirname(input)}/out/change-report.md`, + ['# 变更摘要', `- 新增 goods: ${report.created}`, `- 重复跳过: ${report.duplicates}`, + `- 下架跳过: ${report.skippedDelisted}`, '', ...report.details.map((d) => `- ${d}`)].join('\n'), + ); + console.log(JSON.stringify(report, null, 2)); +} + +// 测试环境下不自动执行 main +if (process.env.NODE_ENV !== 'test' && import.meta.url === `file://${process.argv[1]}`) { + main(); +} +``` + +- [x] **Step 4: 运行测试确认 GREEN** + +Run: `cd apps/api && npx vitest run ../../data-cleaning/sync-transform.test.mjs` +Expected: PASS(3 个用例全过) + +- [x] **Step 5: Commit** + +```bash +git add data-cleaning/sync-transform.mjs data-cleaning/sync-transform.test.mjs +git commit -m "feat(data-cleaning): add pure transform script with tests" +``` + +### Task 3: 生产执行手册写入 README 并联调演练 + +**Files:** +- Modify: `data-cleaning/README.md` + +- [x] **Step 1: 补充执行命令段**(备份 / 导出 / 转换 / 人工检查 / 导入 / 验证 六个步骤的确切命令与预期输出) + +- [x] **Step 2: 本地或预发演练一次全流程**(规则为空应零变更),确认 change-report 为空、行数一致 + +- [x] **Step 3: Commit** + +```bash +git add data-cleaning/README.md +git commit -m "docs(data-cleaning): add production runbook" +``` + +--- + +## 待定事项(规则确定后回填) + +| 项 | 状态 | 回填位置 | +|----|------|----------| +| 品类映射规则 | 未定 | `data-cleaning/sync-rules.config.mjs#categoryMapping` | +| 国家分配策略 | 未定 | `...#countryAssignment` | +| 标签分配策略 | 未定 | `...#tagAssignment` | +| 优先级策略 | 未定 | `...#priority` | +| 规则细节文档 | 未定 | 后续数据清洗计划文件夹(本目录)内新建规则说明 md |