init: Prefect ETL sync project (RIIN/SDS → PG) with 12 sync flows
This commit is contained in:
@@ -0,0 +1,9 @@
|
|||||||
|
# PostgreSQL (ETL 目标库) - 复制为 .env 并填入真实值
|
||||||
|
DB_HOST=192.168.124.92
|
||||||
|
DB_PORT=5432
|
||||||
|
DB_USER=postgres
|
||||||
|
DB_PASSWORD=your-password-here
|
||||||
|
DB_NAME=inkreach-dmp
|
||||||
|
|
||||||
|
# 完整连接串(供 SQLAlchemy / prefect 使用)
|
||||||
|
DB_DSN=postgresql://postgres:your-password-here@192.168.124.92:5432/inkreach-dmp
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
.env
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
.venv/
|
||||||
|
*.csv
|
||||||
|
external_projects/
|
||||||
|
*.log
|
||||||
|
.prefect/
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
# AGENTS.md
|
||||||
|
|
||||||
|
ETL 工作流项目,使用 [Prefect](https://docs.prefect.org.cn/v3/get-started) 编排。
|
||||||
|
任务:REST API → PostgreSQL(Extract / Transform / Load)。
|
||||||
|
|
||||||
|
## 设计理念(简述)
|
||||||
|
|
||||||
|
- **高内聚、低耦合、关注点分离**。
|
||||||
|
- `api/` 是纯客户端层(httpx + pydantic,**禁止 import prefect**),封装 RIIN/SDS 两个外部系统的 REST API,可独立复用/测试。
|
||||||
|
- `flows/` 是编排层(依赖 `api/`,使用 prefect),负责重试/缓存/并发/调度/密钥。
|
||||||
|
- 共享行为(token 注入、401 重登、分页、批量、缓存)放 `api/_common/`,用组合式 helper,不过度抽象。
|
||||||
|
- 新增系统是加法式:加一个 `api/<系统>/` + 一个 flow,不碰旧代码。
|
||||||
|
- 详细架构说明见 [`docs/dev/systematic-design.md`](docs/dev/systematic-design.md)。
|
||||||
|
|
||||||
|
## 依赖管理:使用 uv(不要用 requirements.txt)
|
||||||
|
|
||||||
|
本项目用 [uv](https://docs.astral.sh/uv/) 管理依赖,锁定文件为 `uv.lock`。
|
||||||
|
**没有 requirements.txt,也不要新建。**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv sync # 按 uv.lock 安装依赖到 .venv(首次/拉取代码后执行)
|
||||||
|
uv add <package> # 添加依赖(自动写入 pyproject.toml + 更新 uv.lock)
|
||||||
|
uv add --dev <package> # 添加开发依赖
|
||||||
|
uv remove <package> # 移除依赖
|
||||||
|
uv lock --upgrade # 升级所有依赖的锁定版本
|
||||||
|
uv run python <file> # 在项目环境中运行(无需手动激活 venv)
|
||||||
|
uv run prefect <cmd> # 运行 prefect CLI
|
||||||
|
```
|
||||||
|
|
||||||
|
- Python 版本固定为 `>=3.12`(见 `pyproject.toml`)。
|
||||||
|
- 所有命令都用 `uv run ...` 前缀,**不要**直接用系统 `python` / `pip`。
|
||||||
|
- 提交代码时一并提交 `pyproject.toml` 和 `uv.lock`;**不要提交 `.venv/`**。
|
||||||
|
|
||||||
|
## 运行环境
|
||||||
|
|
||||||
|
- **Prefect server**:自托管,通过 Docker Compose 运行,数据持久化。
|
||||||
|
```bash
|
||||||
|
docker compose up -d # 启动 server(UI: http://192.168.124.92:4200)
|
||||||
|
docker compose ps # 查看状态
|
||||||
|
docker compose logs -f # 查看日志
|
||||||
|
docker compose down # 停止
|
||||||
|
```
|
||||||
|
- **API 地址**:`http://localhost:4200/api`(本机)/ `http://192.168.124.92:4200`(局域网)。
|
||||||
|
- 客户端已配置:`uv run prefect config set PREFECT_API_URL="http://localhost:4200/api"`。
|
||||||
|
|
||||||
|
## 密钥管理(标准做法)
|
||||||
|
|
||||||
|
- **不要把密钥写进代码或 pyproject.toml**。
|
||||||
|
- 数据库凭据用 Prefect Block 加密存储,Block 名:`pg-inkreach-dmp`(`SqlAlchemyConnector` 类型)。
|
||||||
|
```python
|
||||||
|
from prefect_sqlalchemy import SqlAlchemyConnector
|
||||||
|
connector = SqlAlchemyConnector.load("pg-inkreach-dmp")
|
||||||
|
engine = connector.get_engine()
|
||||||
|
```
|
||||||
|
- RIIN / SDS 登录凭据用 `Secret` Block 加密存储,值为 JSON:`{"username":"...","password":"..."}`。
|
||||||
|
Block 名:`riin-credentials`、`sds-credentials`。flow 在 extract task 内部读取,不外泄。
|
||||||
|
- 本地开发备用:`.env`(已 gitignore)含 `DB_DSN`,加载用 `python-dotenv` 或 `os.environ`。
|
||||||
|
|
||||||
|
## 编写/运行工作流
|
||||||
|
|
||||||
|
- 同步任务(对齐 inkreach-bi-auth-mng 的 sync tasks)放在 `flows/riin/` 和 `flows/sds/`,每系统各 6 个:
|
||||||
|
- `order_sync`(按时间窗口)、`incremental_sync`(按 DB 最近下单时间增量,无参)、
|
||||||
|
`recent_sync`(近 N 天)、`refresh_pending`(刷新未完结)、`by_orders`(RIIN 指定单号)/ `refresh_100_prefix`(SDS 100 前缀)、`verify`(逐天核验修复)。
|
||||||
|
- 共享层在 `flows/shared/`:`mappers.py`(RIIN/SDS→统一 Order/Goods)、`db.py`(PG upsert `SyncDB`)、`clients.py`(客户端工厂+顺序分页+重试+渠道字典)。
|
||||||
|
- 运行单个 flow:`uv run python flows/riin/<file>.py`(或 `flows/sds/<file>.py`)
|
||||||
|
- 部署调度:flow 末尾 `flow.serve(name=..., cron=...)`,保持 serve 进程运行。
|
||||||
|
- 写入目标:PG 的 `Orders` / `Goods` / `Merchant` 表(主键 `Orders=("orderNo","createdAt")`、`Goods=("orderNo","productionNo")`),`createdAt` 写入前以库中已有值覆盖以保证幂等。
|
||||||
|
|
||||||
|
## 参考文档
|
||||||
|
|
||||||
|
- Prefect 入门:https://docs.prefect.org.cn/v3/get-started
|
||||||
|
- API 驱动 ETL 示例:https://docs.prefect.org.cn/v3/examples/run-api-sourced-etl
|
||||||
|
- 密钥存储:https://docs.prefect.org.cn/v3/how-to-guides/configuration/store-secrets
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
"""Inkreach 外部系统 REST API 客户端集合。
|
||||||
|
|
||||||
|
子模块:
|
||||||
|
- api.riin : RIIN 分销平台(RiinDisClient)
|
||||||
|
- api.sds : SDS POD 平台(SdsMngClient)
|
||||||
|
- api._common : 共享 HTTP / 缓存 / 并发 / 批量 工具
|
||||||
|
|
||||||
|
这些客户端是纯 httpx + pydantic 实现,不依赖 Prefect。
|
||||||
|
Prefect 编排在项目根的 flows/ 目录。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from api.riin import RiinDisClient
|
||||||
|
from api.sds import SdsMngClient
|
||||||
|
|
||||||
|
__all__ = ["RiinDisClient", "SdsMngClient"]
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
"""共享基础设施工具:HTTP 基类、TTL 缓存、并发、批量切分。"""
|
||||||
|
|
||||||
|
from api._common.batch import chunk, parse_id_text, split_lines
|
||||||
|
from api._common.cache import TTLCache
|
||||||
|
from api._common.concurrency import run_concurrent
|
||||||
|
from api._common.http import TokenHttpClient
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"TokenHttpClient",
|
||||||
|
"TTLCache",
|
||||||
|
"run_concurrent",
|
||||||
|
"parse_id_text",
|
||||||
|
"split_lines",
|
||||||
|
"chunk",
|
||||||
|
]
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
"""批量切分辅助:对应 TS SDK 里订单 OID/快递单号、SDS 订单关键词的批量拆分。"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
_MULTI_SEP = re.compile(r"[,,\s\n\r]+")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_id_text(text: str) -> list[str]:
|
||||||
|
"""把逗号/空白/换行分隔的字符串解析为去重的 ID 列表(RIIN 用)。"""
|
||||||
|
items = _MULTI_SEP.split(text or "")
|
||||||
|
seen: dict[str, None] = {}
|
||||||
|
for item in items:
|
||||||
|
item = item.strip()
|
||||||
|
if item and item not in seen:
|
||||||
|
seen[item] = None
|
||||||
|
return list(seen.keys())
|
||||||
|
|
||||||
|
|
||||||
|
def split_lines(text: str) -> list[str]:
|
||||||
|
"""把换行分隔的字符串解析为去重的列表(SDS 订单号 keywords 用)。"""
|
||||||
|
items = re.split(r"[\n\r]+", text or "")
|
||||||
|
seen: dict[str, None] = {}
|
||||||
|
for item in items:
|
||||||
|
item = item.strip()
|
||||||
|
if item and item not in seen:
|
||||||
|
seen[item] = None
|
||||||
|
return list(seen.keys())
|
||||||
|
|
||||||
|
|
||||||
|
def chunk(items: list[str], size: int) -> list[list[str]]:
|
||||||
|
"""把列表按 size 切成多批。"""
|
||||||
|
if not items:
|
||||||
|
return []
|
||||||
|
return [items[i : i + size] for i in range(0, len(items), size)]
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
"""简单的 TTL 内存缓存(纯 Python,无第三方依赖)。
|
||||||
|
|
||||||
|
对应 TS SDK 的 MemoryCache,用于主数据等就近缓存,生命周期跟随 client 实例。
|
||||||
|
注意:这是客户端层缓存,与 Prefect 的 task 缓存(cache_key_fn)是两回事。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
from typing import Any, Generic, TypeVar
|
||||||
|
|
||||||
|
T = TypeVar("T")
|
||||||
|
|
||||||
|
_DEFAULT_TTL_SECONDS = 300
|
||||||
|
|
||||||
|
|
||||||
|
class TTLCache(Generic[T]):
|
||||||
|
def __init__(self, ttl_seconds: int | float = _DEFAULT_TTL_SECONDS) -> None:
|
||||||
|
self._ttl = ttl_seconds
|
||||||
|
self._store: dict[str, tuple[T, float]] = {}
|
||||||
|
|
||||||
|
def get(self, key: str) -> T | None:
|
||||||
|
item = self._store.get(key)
|
||||||
|
if item is None:
|
||||||
|
return None
|
||||||
|
value, expires_at = item
|
||||||
|
if time.monotonic() > expires_at:
|
||||||
|
self._store.pop(key, None)
|
||||||
|
return None
|
||||||
|
return value
|
||||||
|
|
||||||
|
def set(self, key: str, value: T) -> None:
|
||||||
|
self._store[key] = (value, time.monotonic() + self._ttl)
|
||||||
|
|
||||||
|
def clear(self) -> None:
|
||||||
|
self._store.clear()
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
"""并发执行辅助:线程池 + 信号量,对应 TS SDK 的 ConcurrentLimiter(默认 10)。"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
from typing import Any, TypeVar
|
||||||
|
|
||||||
|
T = TypeVar("T")
|
||||||
|
|
||||||
|
_DEFAULT_CONCURRENCY = 10
|
||||||
|
|
||||||
|
|
||||||
|
def run_concurrent(
|
||||||
|
funcs: list[Callable[[], T]],
|
||||||
|
limit: int = _DEFAULT_CONCURRENCY,
|
||||||
|
) -> list[T]:
|
||||||
|
"""并发执行一批无参 callable(每个返回一个结果),保持输入顺序返回结果。"""
|
||||||
|
if not funcs:
|
||||||
|
return []
|
||||||
|
with ThreadPoolExecutor(max_workers=max(1, min(limit, len(funcs)))) as pool:
|
||||||
|
return list(pool.map(lambda fn: fn(), funcs))
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
"""基础 HTTP 客户端:token 注入 + 401 自动重登。
|
||||||
|
|
||||||
|
两个外部系统(RIIN / SDS)共享这套行为,仅以下差异由子类配置:
|
||||||
|
- base_url
|
||||||
|
- token 写入的请求头名(RIIN: Authorization,SDS: access-token)
|
||||||
|
- 固定请求头(RIIN 带 X-Time-Zone / X-Requested-With,SDS 无)
|
||||||
|
- 登录实现(子类实现 _login)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
|
||||||
|
class TokenHttpClient(ABC):
|
||||||
|
"""带 token 鉴权的 HTTP 客户端基类(组合式,非强制耦合)。"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
base_url: str,
|
||||||
|
*,
|
||||||
|
token_header: str,
|
||||||
|
fixed_headers: dict[str, str] | None = None,
|
||||||
|
login_markers: tuple[str, ...] = (),
|
||||||
|
timeout: float = 30.0,
|
||||||
|
) -> None:
|
||||||
|
self._token: str | None = None
|
||||||
|
self._token_header = token_header
|
||||||
|
self._login_markers = login_markers
|
||||||
|
self._http = httpx.Client(
|
||||||
|
base_url=base_url,
|
||||||
|
headers=fixed_headers or {},
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def _login(self) -> str:
|
||||||
|
"""执行登录并返回新 token。子类实现具体认证逻辑。"""
|
||||||
|
|
||||||
|
# ----- token 生命周期 -----
|
||||||
|
|
||||||
|
def _ensure_token(self) -> str:
|
||||||
|
if not self._token:
|
||||||
|
self._token = self._login()
|
||||||
|
return self._token
|
||||||
|
|
||||||
|
def _clear_token(self) -> None:
|
||||||
|
self._token = None
|
||||||
|
|
||||||
|
def _is_login_call(self, url: str) -> bool:
|
||||||
|
return any(marker in url for marker in self._login_markers)
|
||||||
|
|
||||||
|
# ----- 请求 -----
|
||||||
|
|
||||||
|
def _send(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
*,
|
||||||
|
params: dict[str, Any] | None = None,
|
||||||
|
body: dict[str, Any] | None = None,
|
||||||
|
_retry: bool = True,
|
||||||
|
) -> Any:
|
||||||
|
headers: dict[str, str] = {}
|
||||||
|
if not self._is_login_call(url):
|
||||||
|
headers[self._token_header] = self._ensure_token()
|
||||||
|
|
||||||
|
response = self._http.request(method, url, params=params, json=body, headers=headers)
|
||||||
|
|
||||||
|
if response.status_code == 401 and _retry and not self._is_login_call(url):
|
||||||
|
self._clear_token()
|
||||||
|
return self._send(method, url, params=params, body=body, _retry=False)
|
||||||
|
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
def get(self, url: str, params: dict[str, Any] | None = None) -> Any:
|
||||||
|
return self._send("GET", url, params=params)
|
||||||
|
|
||||||
|
def post(
|
||||||
|
self,
|
||||||
|
url: str,
|
||||||
|
body: dict[str, Any] | None = None,
|
||||||
|
params: dict[str, Any] | None = None,
|
||||||
|
) -> Any:
|
||||||
|
return self._send("POST", url, body=body, params=params)
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
self._http.close()
|
||||||
|
|
||||||
|
def __enter__(self) -> "TokenHttpClient":
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *exc: Any) -> None:
|
||||||
|
self.close()
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
"""RIIN 分销平台 REST API 客户端。"""
|
||||||
|
|
||||||
|
from api.riin.client import RiinDisClient
|
||||||
|
from api.riin.models import OrderPageRequest
|
||||||
|
|
||||||
|
__all__ = ["RiinDisClient", "OrderPageRequest"]
|
||||||
@@ -0,0 +1,329 @@
|
|||||||
|
"""RIIN 分销平台 REST API 客户端(移植自 @inkreach/riin-dis)。
|
||||||
|
|
||||||
|
Base URL: https://tshirt.riin.com
|
||||||
|
认证: username + MD5(password) -> Bearer token(Authorization 头),401 自动重登。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from pydantic import BaseModel, ConfigDict
|
||||||
|
from pydantic.alias_generators import to_camel
|
||||||
|
|
||||||
|
from api._common import TTLCache, chunk, parse_id_text, run_concurrent
|
||||||
|
from api._common.http import TokenHttpClient
|
||||||
|
from api.riin.models import (
|
||||||
|
Article,
|
||||||
|
Color,
|
||||||
|
CustomerListResponse,
|
||||||
|
CustomerShopListResponse,
|
||||||
|
ExpressInfo,
|
||||||
|
FactoryInfoListResponse,
|
||||||
|
LoginResult,
|
||||||
|
OrderGoodsResponse,
|
||||||
|
OrderListCountResponse,
|
||||||
|
OrderLogResponse,
|
||||||
|
OrderPageRequest,
|
||||||
|
OrderPageResponse,
|
||||||
|
Size,
|
||||||
|
ThirdInterfaceCustomer,
|
||||||
|
)
|
||||||
|
|
||||||
|
BASE_URL = "https://tshirt.riin.com"
|
||||||
|
BATCH_SIZE = 500
|
||||||
|
|
||||||
|
|
||||||
|
class _ListResponse(BaseModel):
|
||||||
|
"""通用列表响应壳(辅助主数据用)。"""
|
||||||
|
|
||||||
|
model_config = ConfigDict(extra="allow", populate_by_name=True, alias_generator=to_camel)
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: list[Any] = []
|
||||||
|
|
||||||
|
|
||||||
|
class _PageResponse(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="allow", populate_by_name=True, alias_generator=to_camel)
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: dict[str, Any] = {}
|
||||||
|
|
||||||
|
|
||||||
|
class RiinDisClient(TokenHttpClient):
|
||||||
|
"""RIIN 分销系统客户端:认证 / 订单 / 店铺 / 辅助主数据。"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
username: str,
|
||||||
|
password: str,
|
||||||
|
*,
|
||||||
|
cache_ttl_seconds: int | float = 300,
|
||||||
|
concurrency: int = 10,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(
|
||||||
|
BASE_URL,
|
||||||
|
token_header="Authorization",
|
||||||
|
fixed_headers={"X-Time-Zone": "UTC+8", "X-Requested-With": "XMLHttpRequest"},
|
||||||
|
login_markers=("auth/api/auth/login",),
|
||||||
|
)
|
||||||
|
self._username = username
|
||||||
|
self._password = password
|
||||||
|
self._cache: TTLCache[Any] = TTLCache(cache_ttl_seconds)
|
||||||
|
self._concurrency = concurrency
|
||||||
|
|
||||||
|
# ----- 认证 -----
|
||||||
|
|
||||||
|
def _login(self) -> str:
|
||||||
|
md5_password = hashlib.md5(self._password.encode()).hexdigest()
|
||||||
|
data = self.post("auth/api/auth/login", body={"username": self._username, "password": md5_password})
|
||||||
|
result = LoginResult.model_validate(data["data"])
|
||||||
|
self._token = result.token
|
||||||
|
return self._token
|
||||||
|
|
||||||
|
# ----- 订单 -----
|
||||||
|
|
||||||
|
def _build_order_page_body(self, params: OrderPageRequest) -> dict[str, Any]:
|
||||||
|
has_oid_or_courier = bool(params.platform_oid_list or params.courier_number_list)
|
||||||
|
|
||||||
|
body: dict[str, Any] = {
|
||||||
|
"pageIndex": params.page_index if params.page_index is not None else 1,
|
||||||
|
"pageSize": params.page_size if params.page_size is not None else 20,
|
||||||
|
"shopCodeList": params.shop_code_list or [],
|
||||||
|
"orderStatus": params.order_status if params.order_status is not None else None,
|
||||||
|
"dayRangeGroup": params.day_range_group if params.day_range_group is not None else None,
|
||||||
|
"hasRefund": params.has_refund if params.has_refund is not None else False,
|
||||||
|
}
|
||||||
|
|
||||||
|
if not has_oid_or_courier:
|
||||||
|
body["timeType"] = params.time_type if params.time_type is not None else 3
|
||||||
|
if params.start_time:
|
||||||
|
body["startTime"] = params.start_time
|
||||||
|
if params.end_time:
|
||||||
|
body["endTime"] = params.end_time
|
||||||
|
elif params.time_type is not None:
|
||||||
|
body["timeType"] = params.time_type
|
||||||
|
|
||||||
|
if params.platform_oid_list:
|
||||||
|
body["platformOidList"] = parse_id_text(params.platform_oid_list)
|
||||||
|
if params.courier_number_list:
|
||||||
|
body["courierNumberList"] = parse_id_text(params.courier_number_list)
|
||||||
|
if params.sub_id_list:
|
||||||
|
body["subIdList"] = params.sub_id_list
|
||||||
|
if params.goods_title:
|
||||||
|
body["goodsTitle"] = params.goods_title
|
||||||
|
if params.mapping_sku_code:
|
||||||
|
body["mappingSkuCode"] = params.mapping_sku_code
|
||||||
|
if params.platform_type is not None:
|
||||||
|
body["platformType"] = params.platform_type
|
||||||
|
if params.order_source is not None:
|
||||||
|
body["orderSource"] = params.order_source
|
||||||
|
if params.factory_ids:
|
||||||
|
body["factoryIds"] = params.factory_ids
|
||||||
|
if params.express_source is not None:
|
||||||
|
body["expressSource"] = params.express_source
|
||||||
|
if params.delivery_courier:
|
||||||
|
body["deliveryCourier"] = params.delivery_courier
|
||||||
|
if params.receiver_country:
|
||||||
|
body["receiverCountry"] = params.receiver_country
|
||||||
|
if params.region:
|
||||||
|
body["region"] = params.region
|
||||||
|
if params.receiver_province:
|
||||||
|
body["receiverProvince"] = params.receiver_province
|
||||||
|
if params.craft_type is not None:
|
||||||
|
body["craftType"] = params.craft_type
|
||||||
|
if params.style_code_list:
|
||||||
|
body["styleCodeList"] = params.style_code_list
|
||||||
|
if params.color_code_list:
|
||||||
|
body["colorCodeList"] = params.color_code_list
|
||||||
|
if params.size_code_list:
|
||||||
|
body["sizeCodeList"] = params.size_code_list
|
||||||
|
if params.seller_remark:
|
||||||
|
body["sellerRemark"] = params.seller_remark
|
||||||
|
if params.urgent_order is not None:
|
||||||
|
body["urgentOrder"] = params.urgent_order
|
||||||
|
if params.third_interface_customer_id:
|
||||||
|
body["thirdInterfaceCustomerId"] = params.third_interface_customer_id
|
||||||
|
if params.platform_status_list:
|
||||||
|
body["platformStatusList"] = params.platform_status_list
|
||||||
|
if params.is_third_flag is not None:
|
||||||
|
body["isThirdFlag"] = params.is_third_flag
|
||||||
|
if params.push_failed is not None:
|
||||||
|
body["pushFailed"] = params.push_failed
|
||||||
|
if params.assigned_fail_reason:
|
||||||
|
body["assignedFailReason"] = params.assigned_fail_reason
|
||||||
|
if params.push_fail_reason:
|
||||||
|
body["pushFailReason"] = params.push_fail_reason
|
||||||
|
if params.factory_remark_param:
|
||||||
|
body["factoryRemarkParam"] = params.factory_remark_param
|
||||||
|
if params.distributor_remark_param:
|
||||||
|
body["distributorRemarkParam"] = params.distributor_remark_param
|
||||||
|
return body
|
||||||
|
|
||||||
|
def _get_single_page(self, params: OrderPageRequest) -> OrderPageResponse:
|
||||||
|
body = self._build_order_page_body(
|
||||||
|
params.model_copy(update={"page_index": params.page_index or 1, "page_size": params.page_size or 20})
|
||||||
|
)
|
||||||
|
data = self.post("distribution/order/page", body=body)
|
||||||
|
return OrderPageResponse.model_validate(data)
|
||||||
|
|
||||||
|
def _get_all_pages(self, params: OrderPageRequest) -> OrderPageResponse:
|
||||||
|
page_size = params.page_size or 200
|
||||||
|
first = self._get_single_page(params.model_copy(update={"page_index": 1, "page_size": page_size}))
|
||||||
|
total_count = first.data.total
|
||||||
|
total_pages = -(-total_count // page_size) # ceil
|
||||||
|
if total_pages <= 1:
|
||||||
|
return first
|
||||||
|
|
||||||
|
remaining_indices = list(range(2, total_pages + 1))
|
||||||
|
|
||||||
|
def fetch(i: int) -> OrderPageResponse:
|
||||||
|
return self._get_single_page(params.model_copy(update={"page_index": i, "page_size": page_size}))
|
||||||
|
|
||||||
|
remaining = run_concurrent([lambda i=i: fetch(i) for i in remaining_indices], limit=self._concurrency)
|
||||||
|
records = list(first.data.records)
|
||||||
|
for r in remaining:
|
||||||
|
records.extend(r.data.records)
|
||||||
|
return OrderPageResponse(
|
||||||
|
successful=True,
|
||||||
|
data={"records": records, "total": total_count},
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_orders(self, params: OrderPageRequest | None = None) -> OrderPageResponse:
|
||||||
|
"""获取订单。支持三种模式:limit(取首页 N 条)/ all(全量分页)/ 单页。"""
|
||||||
|
params = params or OrderPageRequest()
|
||||||
|
|
||||||
|
needs_batching = (
|
||||||
|
(params.platform_oid_list and len(parse_id_text(params.platform_oid_list)) > BATCH_SIZE)
|
||||||
|
or (params.courier_number_list and len(parse_id_text(params.courier_number_list)) > BATCH_SIZE)
|
||||||
|
)
|
||||||
|
if needs_batching:
|
||||||
|
return self._get_orders_batched(params)
|
||||||
|
|
||||||
|
if params.limit:
|
||||||
|
return self._get_single_page(params.model_copy(update={"page_index": 1, "page_size": params.limit}))
|
||||||
|
if params.all:
|
||||||
|
return self._get_all_pages(params)
|
||||||
|
return self._get_single_page(params)
|
||||||
|
|
||||||
|
def _get_orders_batched(self, params: OrderPageRequest) -> OrderPageResponse:
|
||||||
|
oid_batches = chunk(parse_id_text(params.platform_oid_list), BATCH_SIZE) if params.platform_oid_list else [None]
|
||||||
|
courier_batches = (
|
||||||
|
chunk(parse_id_text(params.courier_number_list), BATCH_SIZE) if params.courier_number_list else [None]
|
||||||
|
)
|
||||||
|
|
||||||
|
def fetch(oid_batch: list[str] | None, courier_batch: list[str] | None) -> OrderPageResponse:
|
||||||
|
update: dict[str, Any] = {}
|
||||||
|
if oid_batch is not None:
|
||||||
|
update["platform_oid_list"] = ",".join(oid_batch)
|
||||||
|
if courier_batch is not None:
|
||||||
|
update["courier_number_list"] = ",".join(courier_batch)
|
||||||
|
batch_params = params.model_copy(update=update)
|
||||||
|
if params.limit:
|
||||||
|
return self._get_single_page(
|
||||||
|
batch_params.model_copy(update={"page_index": 1, "page_size": params.limit})
|
||||||
|
)
|
||||||
|
return self._get_all_pages(batch_params)
|
||||||
|
|
||||||
|
calls = [
|
||||||
|
(lambda o=o, c=c: fetch(o, c)) for o in oid_batches for c in courier_batches
|
||||||
|
]
|
||||||
|
results = run_concurrent(calls, limit=self._concurrency)
|
||||||
|
|
||||||
|
records: list[Any] = []
|
||||||
|
total = 0
|
||||||
|
for r in results:
|
||||||
|
records.extend(r.data.records)
|
||||||
|
total += r.data.total
|
||||||
|
if params.limit:
|
||||||
|
records = records[: params.limit]
|
||||||
|
return OrderPageResponse(successful=True, data={"records": records, "total": total})
|
||||||
|
|
||||||
|
def get_order_goods(self, order_id: str) -> OrderGoodsResponse:
|
||||||
|
data = self.post("distribution/order/goods", body={"orderId": order_id})
|
||||||
|
return OrderGoodsResponse.model_validate(data)
|
||||||
|
|
||||||
|
def get_order_count(
|
||||||
|
self, params: OrderPageRequest | None = None
|
||||||
|
) -> OrderListCountResponse:
|
||||||
|
params = params or OrderPageRequest()
|
||||||
|
cache_key = f"orderCount:{params.model_dump_json(exclude_none=True)}"
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
body = self._build_order_page_body(params)
|
||||||
|
body.pop("pageIndex", None)
|
||||||
|
body.pop("pageSize", None)
|
||||||
|
data = self.post("distribution/order/list/count", body=body)
|
||||||
|
result = OrderListCountResponse.model_validate(data)
|
||||||
|
self._cache.set(cache_key, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def get_order_logs(self, order_id: str, platform_oid: str) -> OrderLogResponse:
|
||||||
|
data = self.post("distribution/order/logs", body={"orderId": order_id, "platformOid": platform_oid})
|
||||||
|
return OrderLogResponse.model_validate(data)
|
||||||
|
|
||||||
|
# ----- 店铺 / 工厂 / 客户 -----
|
||||||
|
|
||||||
|
def get_customer_shop_list(self) -> CustomerShopListResponse:
|
||||||
|
cache_key = "shops:customerShopList"
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
data = self.post("distribution/order/customer/shop/list", body={})
|
||||||
|
result = CustomerShopListResponse.model_validate(data)
|
||||||
|
self._cache.set(cache_key, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def get_factory_list(self) -> FactoryInfoListResponse:
|
||||||
|
cache_key = "shops:factoryList"
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
data = self.get("distribution/tenant/getFactoryInfoList")
|
||||||
|
result = FactoryInfoListResponse.model_validate(data)
|
||||||
|
self._cache.set(cache_key, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def get_customer_list(self) -> CustomerListResponse:
|
||||||
|
cache_key = "shops:customerList"
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
data = self.post("distribution/customer/listTenantNoDisable", body={})
|
||||||
|
result = CustomerListResponse.model_validate(data)
|
||||||
|
self._cache.set(cache_key, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
# ----- 辅助主数据 -----
|
||||||
|
|
||||||
|
def get_articles(self, params: dict[str, Any] | None = None) -> list[Article]:
|
||||||
|
data = self.post("distribution/articleManagement/list", body=params or {})
|
||||||
|
resp = _ListResponse.model_validate(data)
|
||||||
|
return [Article.model_validate(item) for item in resp.data]
|
||||||
|
|
||||||
|
def get_sizes(self, params: dict[str, Any] | None = None) -> list[Size]:
|
||||||
|
data = self.post("distribution/sizeManagement/size/list", body=params or {})
|
||||||
|
resp = _ListResponse.model_validate(data)
|
||||||
|
return [Size.model_validate(item) for item in resp.data]
|
||||||
|
|
||||||
|
def get_colors(self, params: dict[str, Any] | None = None) -> list[Color]:
|
||||||
|
data = self.post("distribution/colorManagement/list", body=params or {})
|
||||||
|
resp = _ListResponse.model_validate(data)
|
||||||
|
return [Color.model_validate(item) for item in resp.data]
|
||||||
|
|
||||||
|
def get_third_party_customers(
|
||||||
|
self, params: dict[str, Any] | None = None
|
||||||
|
) -> list[ThirdInterfaceCustomer]:
|
||||||
|
data = self.post("distribution/thirdInterfaceCustomer/page", body=params or {})
|
||||||
|
resp = _PageResponse.model_validate(data)
|
||||||
|
records = resp.data.get("records", []) if isinstance(resp.data, dict) else []
|
||||||
|
return [ThirdInterfaceCustomer.model_validate(item) for item in records]
|
||||||
|
|
||||||
|
def get_express_info(self) -> list[ExpressInfo]:
|
||||||
|
data = self.post("distribution/order/tenant/delivery/express", body={})
|
||||||
|
resp = _ListResponse.model_validate(data)
|
||||||
|
return [ExpressInfo.model_validate(item) for item in resp.data]
|
||||||
@@ -0,0 +1,409 @@
|
|||||||
|
"""RIIN 分销平台 —— 数据模型(pydantic v2)。
|
||||||
|
|
||||||
|
忠实还原 @inkreach/riin-dis 的 TypeScript 类型,extra='allow' 兼容未知字段。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from enum import IntEnum
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from pydantic import BaseModel, ConfigDict
|
||||||
|
from pydantic.alias_generators import to_camel
|
||||||
|
|
||||||
|
|
||||||
|
class PlatformType(IntEnum):
|
||||||
|
TIKTOK = 11
|
||||||
|
TEMU = 13
|
||||||
|
SHEIN = 17
|
||||||
|
WALMART = 19
|
||||||
|
|
||||||
|
|
||||||
|
class ShopType(IntEnum):
|
||||||
|
TIKTOK = 11
|
||||||
|
TEMU = 13
|
||||||
|
SHEIN = 17
|
||||||
|
WALMART = 19
|
||||||
|
|
||||||
|
|
||||||
|
class OrderStatusValue(IntEnum):
|
||||||
|
NOT_PUSHED = -4
|
||||||
|
PENDING_REVIEW = -5
|
||||||
|
ASSIGN_FAILED = -6
|
||||||
|
UNDER_REVIEW = -3
|
||||||
|
PENDING_PUSH = 1
|
||||||
|
UNDER_REVIEW_DISTRIBUTOR = 3
|
||||||
|
PUSHED = 4
|
||||||
|
IN_PRODUCTION = 5
|
||||||
|
SHIPPED = 12
|
||||||
|
COMPLETED = 13
|
||||||
|
CANCELLED = 14
|
||||||
|
CLOSED = 15
|
||||||
|
DAY_RANGE_3 = 101
|
||||||
|
DAY_RANGE_3_7 = 102
|
||||||
|
DAY_RANGE_7_15 = 103
|
||||||
|
DAY_RANGE_15_PLUS = 104
|
||||||
|
|
||||||
|
|
||||||
|
class ExpressSource(IntEnum):
|
||||||
|
FACTORY = 1
|
||||||
|
PLATFORM_LABEL = 2
|
||||||
|
SELF_LABEL = 3
|
||||||
|
|
||||||
|
|
||||||
|
class TimeType(IntEnum):
|
||||||
|
ORDER_CREATE = 1
|
||||||
|
PUSH_FACTORY = 5
|
||||||
|
MERCHANT_PUSH = 3
|
||||||
|
SHIP = 4
|
||||||
|
|
||||||
|
|
||||||
|
class _RiinModel(BaseModel):
|
||||||
|
model_config = ConfigDict(
|
||||||
|
extra="allow",
|
||||||
|
populate_by_name=True,
|
||||||
|
alias_generator=to_camel,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 认证 -----
|
||||||
|
|
||||||
|
|
||||||
|
class LoginResult(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
tenant_id: str | None = None
|
||||||
|
sub_id: str | None = None
|
||||||
|
tenant_name: str | None = None
|
||||||
|
username: str | None = None
|
||||||
|
nickname: str | None = None
|
||||||
|
user_type: str | None = None
|
||||||
|
token: str
|
||||||
|
token_flag: str | None = None
|
||||||
|
invite_code: str | None = None
|
||||||
|
main_account: bool | None = None
|
||||||
|
factory: bool | None = None
|
||||||
|
distribution: bool | None = None
|
||||||
|
sub_distribution: bool | None = None
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 订单 -----
|
||||||
|
|
||||||
|
|
||||||
|
class ExpressItem(_RiinModel):
|
||||||
|
express_name: str | None = None
|
||||||
|
ship_logistics_type: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Order(_RiinModel):
|
||||||
|
id: str
|
||||||
|
tenant_id: str | None = None
|
||||||
|
sub_id: str | None = None
|
||||||
|
sub_name: str | None = None
|
||||||
|
source_platform_oid: str | None = None
|
||||||
|
platform_oid: str | None = None
|
||||||
|
platform_order_name: str | None = None
|
||||||
|
order_status: int | None = None
|
||||||
|
order_type: int | None = None
|
||||||
|
goods_total_qty: int | None = None
|
||||||
|
platform_order_status: str | None = None
|
||||||
|
platform_refund_status: str | None = None
|
||||||
|
platform_shop_code: str | None = None
|
||||||
|
platform_shop_id: str | None = None
|
||||||
|
platform_shop_name: str | None = None
|
||||||
|
platform_type: int | None = None
|
||||||
|
sub_platform_type: int | None = None
|
||||||
|
seller_flag: str | None = None
|
||||||
|
factory_remark: str | None = None
|
||||||
|
distributor_remark: str | None = None
|
||||||
|
seller_remark: str | None = None
|
||||||
|
buyer_remark: str | None = None
|
||||||
|
delivery_courier: str | None = None
|
||||||
|
express_code: str | None = None
|
||||||
|
express_fee: float | None = None
|
||||||
|
express_source: int | None = None
|
||||||
|
courier_number: str | None = None
|
||||||
|
consignee_name: str | None = None
|
||||||
|
phone: str | None = None
|
||||||
|
email: str | None = None
|
||||||
|
address: str | None = None
|
||||||
|
receiver_country: str | None = None
|
||||||
|
receiver_province: str | None = None
|
||||||
|
receiver_city: str | None = None
|
||||||
|
receiver_district: str | None = None
|
||||||
|
receiver_town: str | None = None
|
||||||
|
post_code: str | None = None
|
||||||
|
currency: str | None = None
|
||||||
|
currency_type: str | None = None
|
||||||
|
total_fee: float | None = None
|
||||||
|
payment: float | None = None
|
||||||
|
received_payment: float | None = None
|
||||||
|
total_price: float | None = None
|
||||||
|
total_sell_price: float | None = None
|
||||||
|
post_fee: float | None = None
|
||||||
|
order_create_time: str | None = None
|
||||||
|
pre_shipping_time: str | None = None
|
||||||
|
shipping_time: str | None = None
|
||||||
|
order_pay_time: str | None = None
|
||||||
|
order_time: str | None = None
|
||||||
|
finish_time: str | None = None
|
||||||
|
cancel_time: str | None = None
|
||||||
|
audit_time: str | None = None
|
||||||
|
push_time: str | None = None
|
||||||
|
distributor_push_time: str | None = None
|
||||||
|
distributor_push_target_name: str | None = None
|
||||||
|
except_last_shipping_time: str | None = None
|
||||||
|
push_msg: str | None = None
|
||||||
|
pushed_failed: str | None = None
|
||||||
|
assigned_fail_reason: str | None = None
|
||||||
|
urgent_order: int | None = None
|
||||||
|
is_third_flag: bool | None = None
|
||||||
|
is_custom_product: bool | None = None
|
||||||
|
express_list: list[ExpressItem] = []
|
||||||
|
|
||||||
|
|
||||||
|
class OrderPageData(_RiinModel):
|
||||||
|
records: list[Order] = []
|
||||||
|
total: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
class OrderPageResponse(_RiinModel):
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: OrderPageData = OrderPageData()
|
||||||
|
|
||||||
|
|
||||||
|
class PrintingImage(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
type: int | None = None
|
||||||
|
url: str | None = None
|
||||||
|
name: str | None = None
|
||||||
|
code: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class OrderGoodsImageObjs(_RiinModel):
|
||||||
|
sku_image: str | None = None
|
||||||
|
printing_image: str | None = None
|
||||||
|
printing_images: list[PrintingImage] = []
|
||||||
|
sku_images: list[str] | None = None
|
||||||
|
print_urls: list[str] = []
|
||||||
|
sku_urls: list[str] = []
|
||||||
|
|
||||||
|
|
||||||
|
class OrderGoods(_RiinModel):
|
||||||
|
id: str
|
||||||
|
order_id: str | None = None
|
||||||
|
platform_oid: str | None = None
|
||||||
|
platform_spu_id: str | None = None
|
||||||
|
platform_sku_id: str | None = None
|
||||||
|
title: str | None = None
|
||||||
|
specification: str | None = None
|
||||||
|
goods_status: str | None = None
|
||||||
|
refund_status: str | None = None
|
||||||
|
price: float | None = None
|
||||||
|
sell_price: float | None = None
|
||||||
|
num: int | None = None
|
||||||
|
total_price: float | None = None
|
||||||
|
total_sell_price: float | None = None
|
||||||
|
sku_image: str | None = None
|
||||||
|
mapping_sku_code: str | None = None
|
||||||
|
factory_mapping_sku_code: str | None = None
|
||||||
|
style_code: str | None = None
|
||||||
|
image_code: str | None = None
|
||||||
|
size_code: str | None = None
|
||||||
|
color_code: str | None = None
|
||||||
|
style_name: str | None = None
|
||||||
|
image_name: str | None = None
|
||||||
|
size_name: str | None = None
|
||||||
|
color_name: str | None = None
|
||||||
|
product_code: str | None = None
|
||||||
|
craft_type: int | None = None
|
||||||
|
print_position: str | None = None
|
||||||
|
image_objs: Any | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class OrderGoodsResponse(_RiinModel):
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: list[OrderGoods] = []
|
||||||
|
|
||||||
|
|
||||||
|
class OrderListCount(_RiinModel):
|
||||||
|
assigned_suceessed: int = 0
|
||||||
|
factory_auditing: int = 0
|
||||||
|
in_producing: int = 0
|
||||||
|
shipped: int = 0
|
||||||
|
closed: int = 0
|
||||||
|
refunding: int = 0
|
||||||
|
refunded: int = 0
|
||||||
|
rejecting: int = 0
|
||||||
|
rejected_to_shop: int = 0
|
||||||
|
day_range_group1: int = 0
|
||||||
|
day_range_group2: int = 0
|
||||||
|
day_range_group3: int = 0
|
||||||
|
day_range_group4: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
class OrderListCountResponse(_RiinModel):
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: OrderListCount = OrderListCount()
|
||||||
|
|
||||||
|
|
||||||
|
class OrderLog(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
order_id: str | None = None
|
||||||
|
platform_oid: str | None = None
|
||||||
|
operate_type: str | None = None
|
||||||
|
operate_msg: str | None = None
|
||||||
|
operator_name: str | None = None
|
||||||
|
operate_time: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class OrderLogResponse(_RiinModel):
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: list[OrderLog] = []
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 店铺 / 工厂 / 客户 -----
|
||||||
|
|
||||||
|
|
||||||
|
class CustomerShop(_RiinModel):
|
||||||
|
tenant_id: str | None = None
|
||||||
|
tenant_name: str | None = None
|
||||||
|
shop_id: str | None = None
|
||||||
|
shop_code: str | None = None
|
||||||
|
shop_name: str | None = None
|
||||||
|
shop_type: int | None = None
|
||||||
|
sub_id: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class CustomerShopListResponse(_RiinModel):
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: list[CustomerShop] = []
|
||||||
|
|
||||||
|
|
||||||
|
class FactoryInfo(_RiinModel):
|
||||||
|
factory_id: str | None = None
|
||||||
|
phone: str | None = None
|
||||||
|
contact: str | None = None
|
||||||
|
address: str | None = None
|
||||||
|
name: str | None = None
|
||||||
|
logo_url: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FactoryInfoListResponse(_RiinModel):
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: list[FactoryInfo] = []
|
||||||
|
|
||||||
|
|
||||||
|
class Customer(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
tenant_id: str | None = None
|
||||||
|
customer_tenant_id: str | None = None
|
||||||
|
customer_name: str | None = None
|
||||||
|
username: str | None = None
|
||||||
|
contacts: str | None = None
|
||||||
|
mobile: str | None = None
|
||||||
|
settlement_currency: int | None = None
|
||||||
|
disable: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class CustomerListResponse(_RiinModel):
|
||||||
|
successful: bool | None = None
|
||||||
|
message: str | None = None
|
||||||
|
error_code: str | None = None
|
||||||
|
data: list[Customer] = []
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 辅助主数据 -----
|
||||||
|
|
||||||
|
|
||||||
|
class Article(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
code: str | None = None
|
||||||
|
name: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Size(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
code: str | None = None
|
||||||
|
name: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Color(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
code: str | None = None
|
||||||
|
name: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ThirdInterfaceCustomer(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
name: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ExpressInfo(_RiinModel):
|
||||||
|
id: str | None = None
|
||||||
|
name: str | None = None
|
||||||
|
code: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class PageData(_RiinModel):
|
||||||
|
records: list[Any] = []
|
||||||
|
total: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 查询参数(请求 schema)-----
|
||||||
|
|
||||||
|
|
||||||
|
class OrderPageRequest(BaseModel):
|
||||||
|
"""RIIN 订单分页查询参数。timeType/startTime/endTime 等可选。"""
|
||||||
|
|
||||||
|
model_config = ConfigDict(extra="allow")
|
||||||
|
|
||||||
|
time_type: int | None = None
|
||||||
|
start_time: str | None = None
|
||||||
|
end_time: str | None = None
|
||||||
|
platform_oid_list: str | None = None
|
||||||
|
shop_code_list: list[str] | None = None
|
||||||
|
courier_number_list: str | None = None
|
||||||
|
sub_id_list: list[str] | None = None
|
||||||
|
goods_title: str | None = None
|
||||||
|
mapping_sku_code: str | None = None
|
||||||
|
platform_type: int | None = None
|
||||||
|
order_source: int | None = None
|
||||||
|
factory_ids: list[str] | None = None
|
||||||
|
express_source: int | None = None
|
||||||
|
delivery_courier: str | None = None
|
||||||
|
receiver_country: str | None = None
|
||||||
|
region: str | None = None
|
||||||
|
receiver_province: str | None = None
|
||||||
|
craft_type: int | None = None
|
||||||
|
style_code_list: list[str] | None = None
|
||||||
|
color_code_list: list[str] | None = None
|
||||||
|
size_code_list: list[str] | None = None
|
||||||
|
seller_remark: str | None = None
|
||||||
|
urgent_order: int | None = None
|
||||||
|
third_interface_customer_id: str | None = None
|
||||||
|
platform_status_list: list[str] | None = None
|
||||||
|
is_third_flag: bool | None = None
|
||||||
|
page_index: int | None = None
|
||||||
|
page_size: int | None = None
|
||||||
|
order_status: int | None = None
|
||||||
|
day_range_group: int | None = None
|
||||||
|
has_refund: bool | None = None
|
||||||
|
push_failed: bool | None = None
|
||||||
|
assigned_fail_reason: str | None = None
|
||||||
|
push_fail_reason: str | None = None
|
||||||
|
factory_remark_param: str | None = None
|
||||||
|
distributor_remark_param: str | None = None
|
||||||
|
limit: int | None = None
|
||||||
|
all: bool | None = None
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
"""SDS POD 平台 REST API 客户端。"""
|
||||||
|
|
||||||
|
from api.sds.client import SdsMngClient
|
||||||
|
from api.sds.models import OrderPageRequest
|
||||||
|
|
||||||
|
__all__ = ["SdsMngClient", "OrderPageRequest"]
|
||||||
@@ -0,0 +1,354 @@
|
|||||||
|
"""SDS POD 平台 REST API 客户端(移植自 @inkreach/sds-mng)。
|
||||||
|
|
||||||
|
Base URL: https://pod-api.sdspod.com
|
||||||
|
认证: username + password + 公司名(印美达) -> access-token 头,401 自动重登。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from api._common import TTLCache, chunk, run_concurrent, split_lines
|
||||||
|
from api._common.http import TokenHttpClient
|
||||||
|
from api.sds.models import (
|
||||||
|
FiltratePanelResponse,
|
||||||
|
IssuingBayAreaResponse,
|
||||||
|
LogisticsItem,
|
||||||
|
LogisticsPageRequest,
|
||||||
|
LogisticsPageResponse,
|
||||||
|
LoginResult,
|
||||||
|
OrderPageRequest,
|
||||||
|
OrderPageResponse,
|
||||||
|
OrderQueryParams,
|
||||||
|
OrderStatistics,
|
||||||
|
OrderStatus,
|
||||||
|
ProductListRequest,
|
||||||
|
ProductListResponse,
|
||||||
|
)
|
||||||
|
|
||||||
|
BASE_URL = "https://pod-api.sdspod.com"
|
||||||
|
COMPANY_NO = "印美达"
|
||||||
|
KEYWORDS_BATCH_SIZE = 200
|
||||||
|
FILTERS_FIELDS = (
|
||||||
|
"factoryList,merchantList,logisticsList,earlyWarningList,"
|
||||||
|
"issuingBayAreaList,logisticsSourceList,extendStatusList"
|
||||||
|
)
|
||||||
|
EXTRA_INFO = (
|
||||||
|
"eyJpcCI6IjM4LjcxLjEyNy4xODkiLCJsb2NhdGlvbiI6eyJsYXQiOjM4Ljg4MzMsImxuZyI6LTc3fSwiYWRfaW5mbyI6"
|
||||||
|
"eyJuYXRpb24iOiLnrKw0OSIsInByb3ZpbmNlIjoiIiwiY2l0eSI6IiIsImRpc3RyaWN0IjoiIiwiYWRjb2RlIjotMSwi"
|
||||||
|
"bmF0aW9uX2NvZGUiOjg0MH0sImZpbmdlcnByaW50SWQiOiI3YzdjOWUwNTFhZTY0OTM5MTMwZjk5YjQ2OGJjYWFhNSIs"
|
||||||
|
"InVzZXJBZ2VudCI6Ik1vemlsbGEvNS4wIChXaW5kb3dzIE5UIDEwLjA7IFdpbjY0OyB4NjQpIEFwcGxlV2ViS2l0LzUz"
|
||||||
|
"Ny4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS8xMzguMC4wLjAgU2FmYXJpLzUzNy4zNiIsImhyZWYiOiJodHRw"
|
||||||
|
"czovL21hbmFnZS5zZHNwb2QuY29tL3VzZXIvbG9naW4ifQ=="
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SdsMngClient(TokenHttpClient):
|
||||||
|
"""SDS POD 系统客户端:认证 / 订单 / 产品 / 物流。"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
username: str,
|
||||||
|
password: str,
|
||||||
|
*,
|
||||||
|
cache_ttl_seconds: int | float = 300,
|
||||||
|
concurrency: int = 10,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(
|
||||||
|
BASE_URL,
|
||||||
|
token_header="access-token",
|
||||||
|
fixed_headers=None,
|
||||||
|
login_markers=("/pod/auth/login",),
|
||||||
|
)
|
||||||
|
self._username = username
|
||||||
|
self._password = password
|
||||||
|
self._cache: TTLCache[Any] = TTLCache(cache_ttl_seconds)
|
||||||
|
self._concurrency = concurrency
|
||||||
|
|
||||||
|
# ----- 认证 -----
|
||||||
|
|
||||||
|
def _login(self) -> str:
|
||||||
|
data = self.post(
|
||||||
|
f"/pod/auth/login?t={int(time.time() * 1000)}",
|
||||||
|
body={
|
||||||
|
"no": COMPANY_NO,
|
||||||
|
"password": self._password,
|
||||||
|
"username": self._username,
|
||||||
|
"extraInfo": EXTRA_INFO,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
result = LoginResult.model_validate(data["data"])
|
||||||
|
self._token = result.access_token
|
||||||
|
return self._token
|
||||||
|
|
||||||
|
# ----- 订单 -----
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _build_order_params(params: OrderPageRequest) -> dict[str, Any]:
|
||||||
|
query: dict[str, Any] = {"productionType": 2, "t": int(time.time() * 1000)}
|
||||||
|
if params.size is not None:
|
||||||
|
query["size"] = params.size
|
||||||
|
if params.page is not None:
|
||||||
|
query["page"] = params.page
|
||||||
|
if params.keywords is not None:
|
||||||
|
query["keywords"] = params.keywords
|
||||||
|
if params.created_date_begin is not None:
|
||||||
|
query["createdDateBegin"] = params.created_date_begin
|
||||||
|
if params.created_date_end is not None:
|
||||||
|
query["createdDateEnd"] = params.created_date_end
|
||||||
|
if params.pay_date_begin is not None:
|
||||||
|
query["payDateBegin"] = params.pay_date_begin
|
||||||
|
if params.pay_date_end is not None:
|
||||||
|
query["payDateEnd"] = params.pay_date_end
|
||||||
|
if params.confirm_date_begin is not None:
|
||||||
|
query["confirmDateBegin"] = params.confirm_date_begin
|
||||||
|
if params.confirm_date_end is not None:
|
||||||
|
query["confirmDateEnd"] = params.confirm_date_end
|
||||||
|
if params.finish_date_begin is not None:
|
||||||
|
query["finishDateBegin"] = params.finish_date_begin
|
||||||
|
if params.finish_date_end is not None:
|
||||||
|
query["finishDateEnd"] = params.finish_date_end
|
||||||
|
if params.order_status_code is not None:
|
||||||
|
query["orderStatusCode"] = params.order_status_code
|
||||||
|
if params.product_keywords is not None:
|
||||||
|
query["productKeywords"] = params.product_keywords
|
||||||
|
if params.order_time_range_type is not None:
|
||||||
|
query["orderTimeRangeType"] = params.order_time_range_type
|
||||||
|
if params.factory_ids is not None:
|
||||||
|
query["factoryIds"] = params.factory_ids
|
||||||
|
if params.merchant_ids is not None:
|
||||||
|
query["merchantIds"] = params.merchant_ids
|
||||||
|
if params.logistics_ids is not None:
|
||||||
|
query["logisticsIds"] = params.logistics_ids
|
||||||
|
if params.issuing_bay_area_id is not None:
|
||||||
|
query["issuingBayAreaId"] = params.issuing_bay_area_id
|
||||||
|
if params.extend_status is not None:
|
||||||
|
query["extendStatus"] = params.extend_status
|
||||||
|
if params.early_warning_type is not None:
|
||||||
|
query["earlyWarningType"] = params.early_warning_type
|
||||||
|
if params.item_num_type is not None:
|
||||||
|
query["itemNumType"] = params.item_num_type
|
||||||
|
return query
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _build_query_params(params: OrderQueryParams | None) -> dict[str, Any]:
|
||||||
|
params = params or OrderQueryParams()
|
||||||
|
query: dict[str, Any] = {"t": int(time.time() * 1000)}
|
||||||
|
for py_name, api_name in [
|
||||||
|
("created_date_begin", "createdDateBegin"),
|
||||||
|
("created_date_end", "createdDateEnd"),
|
||||||
|
("pay_date_begin", "payDateBegin"),
|
||||||
|
("pay_date_end", "payDateEnd"),
|
||||||
|
("confirm_date_begin", "confirmDateBegin"),
|
||||||
|
("confirm_date_end", "confirmDateEnd"),
|
||||||
|
("finish_date_begin", "finishDateBegin"),
|
||||||
|
("finish_date_end", "finishDateEnd"),
|
||||||
|
("order_time_range_type", "orderTimeRangeType"),
|
||||||
|
("product_keywords", "productKeywords"),
|
||||||
|
("factory_ids", "factoryIds"),
|
||||||
|
("merchant_ids", "merchantIds"),
|
||||||
|
("logistics_ids", "logisticsIds"),
|
||||||
|
("issuing_bay_area_id", "issuingBayAreaId"),
|
||||||
|
("extend_status", "extendStatus"),
|
||||||
|
("early_warning_type", "earlyWarningType"),
|
||||||
|
("item_num_type", "itemNumType"),
|
||||||
|
]:
|
||||||
|
value = getattr(params, py_name, None)
|
||||||
|
if value:
|
||||||
|
query[api_name] = value
|
||||||
|
return query
|
||||||
|
|
||||||
|
def _get_single_page(self, params: OrderPageRequest) -> OrderPageResponse:
|
||||||
|
query = self._build_order_params(
|
||||||
|
params.model_copy(update={"size": params.size or 50, "page": params.page or 1})
|
||||||
|
)
|
||||||
|
data = self.get("/pod/orders/page", params=query)
|
||||||
|
return OrderPageResponse.model_validate(data)
|
||||||
|
|
||||||
|
def _get_all_pages(self, params: OrderPageRequest) -> OrderPageResponse:
|
||||||
|
page_size = params.size or 200
|
||||||
|
first = self._get_single_page(params.model_copy(update={"size": page_size, "page": 1}))
|
||||||
|
total_pages = -(-first.total_count // page_size)
|
||||||
|
if total_pages <= 1:
|
||||||
|
return first
|
||||||
|
|
||||||
|
def fetch(i: int) -> OrderPageResponse:
|
||||||
|
return self._get_single_page(params.model_copy(update={"size": page_size, "page": i}))
|
||||||
|
|
||||||
|
remaining = run_concurrent(
|
||||||
|
[lambda i=i: fetch(i) for i in range(2, total_pages + 1)], limit=self._concurrency
|
||||||
|
)
|
||||||
|
merged = list(first.records)
|
||||||
|
for r in remaining:
|
||||||
|
merged.extend(r.records)
|
||||||
|
return OrderPageResponse(total_count=first.total_count, records=merged)
|
||||||
|
|
||||||
|
def get_orders(self, params: OrderPageRequest | None = None) -> OrderPageResponse:
|
||||||
|
"""获取订单。keywords>200 自动分批;limit 取首页 N 条;all 全量分页。"""
|
||||||
|
params = params or OrderPageRequest()
|
||||||
|
|
||||||
|
if params.keywords:
|
||||||
|
return self._get_orders_with_keywords(params)
|
||||||
|
|
||||||
|
if params.limit:
|
||||||
|
return self._get_single_page(params.model_copy(update={"size": params.limit, "page": 1}))
|
||||||
|
if params.all:
|
||||||
|
return self._get_all_pages(params)
|
||||||
|
return self._get_single_page(params)
|
||||||
|
|
||||||
|
def _get_orders_with_keywords(self, params: OrderPageRequest) -> OrderPageResponse:
|
||||||
|
batches = chunk(split_lines(params.keywords or ""), KEYWORDS_BATCH_SIZE) or [[]]
|
||||||
|
|
||||||
|
def fetch(batch: list[str]) -> OrderPageResponse:
|
||||||
|
batch_params = params.model_copy(update={"keywords": "\n".join(batch)})
|
||||||
|
if params.limit:
|
||||||
|
return self._get_single_page(
|
||||||
|
batch_params.model_copy(update={"size": params.limit, "page": 1})
|
||||||
|
)
|
||||||
|
return self._get_all_pages(batch_params)
|
||||||
|
|
||||||
|
results = run_concurrent([lambda b=b: fetch(b) for b in batches], limit=self._concurrency)
|
||||||
|
merged: list[Any] = []
|
||||||
|
total = 0
|
||||||
|
for r in results:
|
||||||
|
merged.extend(r.records)
|
||||||
|
total += r.total_count
|
||||||
|
if params.limit:
|
||||||
|
merged = merged[: params.limit]
|
||||||
|
return OrderPageResponse(total_count=total, list=merged)
|
||||||
|
|
||||||
|
def get_filters(self, params: OrderQueryParams | None = None) -> FiltratePanelResponse:
|
||||||
|
cache_key = f"filters:{(params.model_dump_json(exclude_none=True)) if params else '{}'}"
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
query = {"fields": FILTERS_FIELDS, **self._build_query_params(params)}
|
||||||
|
data = self.get("/pod/orders/filtratePanel", params=query)
|
||||||
|
result = FiltratePanelResponse.model_validate(data)
|
||||||
|
self._cache.set(cache_key, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def get_status(self, params: OrderQueryParams | None = None) -> list[OrderStatus]:
|
||||||
|
cache_key = f"status:{(params.model_dump_json(exclude_none=True)) if params else '{}'}"
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
query = {"productionType": 2, **self._build_query_params(params)}
|
||||||
|
data = self.get("/pod/orders/statistics", params=query)
|
||||||
|
result = [OrderStatus.model_validate(item) for item in data]
|
||||||
|
self._cache.set(cache_key, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def get_statistics(
|
||||||
|
self, params: OrderQueryParams | None = None, production_type: int = 2
|
||||||
|
) -> list[OrderStatistics]:
|
||||||
|
params = params or OrderQueryParams()
|
||||||
|
cache_key = f"statistics:{production_type}:{params.model_dump_json(exclude_none=True)}"
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
query = {
|
||||||
|
"fields": FILTERS_FIELDS,
|
||||||
|
"productionType": production_type,
|
||||||
|
**self._build_query_params(params),
|
||||||
|
}
|
||||||
|
data = self.get("/pod/orders/statistics", params=query)
|
||||||
|
result = [OrderStatistics.model_validate(item) for item in data]
|
||||||
|
self._cache.set(cache_key, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
# ----- 产品 -----
|
||||||
|
|
||||||
|
def get_products(self, params: ProductListRequest | None = None) -> ProductListResponse:
|
||||||
|
params = params or ProductListRequest()
|
||||||
|
if params.limit:
|
||||||
|
return self._get_products_page(params.model_copy(update={"size": params.limit, "page": 1}))
|
||||||
|
if params.all:
|
||||||
|
return self._get_all_products(params)
|
||||||
|
return self._get_products_page(params)
|
||||||
|
|
||||||
|
def _get_products_page(self, params: ProductListRequest) -> ProductListResponse:
|
||||||
|
query = {
|
||||||
|
"keyword": params.keyword or "",
|
||||||
|
"page": params.page or 1,
|
||||||
|
"size": params.size or 50,
|
||||||
|
"publicStatus": params.public_status if params.public_status is not None else 1,
|
||||||
|
"t": int(time.time() * 1000),
|
||||||
|
}
|
||||||
|
data = self.get("/pod/product/self", params=query)
|
||||||
|
return ProductListResponse.model_validate(data)
|
||||||
|
|
||||||
|
def _get_all_products(self, params: ProductListRequest) -> ProductListResponse:
|
||||||
|
page_size = params.size or 200
|
||||||
|
first = self._get_products_page(params.model_copy(update={"size": page_size, "page": 1}))
|
||||||
|
total_pages = -(-first.total_count // page_size)
|
||||||
|
if total_pages <= 1:
|
||||||
|
return first
|
||||||
|
remaining = run_concurrent(
|
||||||
|
[
|
||||||
|
lambda i=i: self._get_products_page(params.model_copy(update={"size": page_size, "page": i}))
|
||||||
|
for i in range(2, total_pages + 1)
|
||||||
|
],
|
||||||
|
limit=self._concurrency,
|
||||||
|
)
|
||||||
|
merged = list(first.records)
|
||||||
|
for r in remaining:
|
||||||
|
merged.extend(r.records)
|
||||||
|
return ProductListResponse(total_count=first.total_count, records=merged)
|
||||||
|
|
||||||
|
# ----- 物流 -----
|
||||||
|
|
||||||
|
def get_logistics_areas(self) -> IssuingBayAreaResponse:
|
||||||
|
cache_key = "logistics:areas"
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
data = self.get("/pod/issuing_bay_areas", params={"t": int(time.time() * 1000)})
|
||||||
|
result = IssuingBayAreaResponse.model_validate(data)
|
||||||
|
self._cache.set(cache_key, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def get_logistics_page(self, params: LogisticsPageRequest) -> LogisticsPageResponse:
|
||||||
|
if params.limit:
|
||||||
|
return self._get_logistics_single(
|
||||||
|
params.model_copy(update={"size": params.limit, "page": 1})
|
||||||
|
)
|
||||||
|
if params.all:
|
||||||
|
return self._get_logistics_all(params)
|
||||||
|
return self._get_logistics_single(params)
|
||||||
|
|
||||||
|
def _get_logistics_single(self, params: LogisticsPageRequest) -> LogisticsPageResponse:
|
||||||
|
query = {
|
||||||
|
"keyword": params.keyword or "",
|
||||||
|
"deliveryMethod": params.delivery_method or "FBM",
|
||||||
|
"issuingBayAreaId": params.issuing_bay_area_id,
|
||||||
|
"page": params.page or 1,
|
||||||
|
"size": params.size or 50,
|
||||||
|
"t": int(time.time() * 1000),
|
||||||
|
}
|
||||||
|
data = self.get("/pod/logistics/page", params=query)
|
||||||
|
return LogisticsPageResponse.model_validate(data)
|
||||||
|
|
||||||
|
def _get_logistics_all(self, params: LogisticsPageRequest) -> LogisticsPageResponse:
|
||||||
|
page_size = params.size or 200
|
||||||
|
first = self._get_logistics_single(
|
||||||
|
params.model_copy(update={"size": page_size, "page": 1})
|
||||||
|
)
|
||||||
|
total_count = first.total_count or first.total_count_alt or 0
|
||||||
|
total_pages = -(-total_count // page_size)
|
||||||
|
if total_pages <= 1:
|
||||||
|
return first
|
||||||
|
remaining = run_concurrent(
|
||||||
|
[
|
||||||
|
lambda i=i: self._get_logistics_single(params.model_copy(update={"size": page_size, "page": i}))
|
||||||
|
for i in range(2, total_pages + 1)
|
||||||
|
],
|
||||||
|
limit=self._concurrency,
|
||||||
|
)
|
||||||
|
merged: list[LogisticsItem] = list(first.items)
|
||||||
|
for r in remaining:
|
||||||
|
merged.extend(r.items)
|
||||||
|
return LogisticsPageResponse(
|
||||||
|
items=merged, page=1, size=len(merged), total_count=total_count
|
||||||
|
)
|
||||||
@@ -0,0 +1,364 @@
|
|||||||
|
"""SDS POD 平台 —— 数据模型(pydantic v2)。
|
||||||
|
|
||||||
|
移植自 @inkreach/sds-mng。订单域为 camelCase;物流域混用 snake_case,
|
||||||
|
对 snake_case 字段显式标注 alias(alias 优先于 alias_generator)。
|
||||||
|
所有模型 extra='allow',不丢字段。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from pydantic import BaseModel, ConfigDict, Field
|
||||||
|
from pydantic.alias_generators import to_camel
|
||||||
|
|
||||||
|
|
||||||
|
class _SdsModel(BaseModel):
|
||||||
|
model_config = ConfigDict(
|
||||||
|
extra="allow",
|
||||||
|
populate_by_name=True,
|
||||||
|
alias_generator=to_camel,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 认证 -----
|
||||||
|
|
||||||
|
|
||||||
|
class LoginResult(_SdsModel):
|
||||||
|
access_token: str
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 订单域(camelCase)-----
|
||||||
|
|
||||||
|
|
||||||
|
class MerchantStore(_SdsModel):
|
||||||
|
id: int | None = None
|
||||||
|
name: str | None = None
|
||||||
|
platform_name: str | None = None
|
||||||
|
platform_code: str | None = None
|
||||||
|
status: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Address(_SdsModel):
|
||||||
|
id: int | None = None
|
||||||
|
country: str | None = None
|
||||||
|
consignee: str | None = None
|
||||||
|
city: str | None = None
|
||||||
|
detail: str | None = None
|
||||||
|
province: str | None = None
|
||||||
|
mobile_phone: str | None = None
|
||||||
|
postcode: str | None = None
|
||||||
|
email: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class IssuingBayArea(_SdsModel):
|
||||||
|
name: str | None = None
|
||||||
|
type: str | None = None
|
||||||
|
id: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class IssuingBayRef(_SdsModel):
|
||||||
|
id: int | None = None
|
||||||
|
name: str | None = None
|
||||||
|
city: str | None = None
|
||||||
|
province: str | None = None
|
||||||
|
detail: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ImportProduct(_SdsModel):
|
||||||
|
id: str | None = None
|
||||||
|
import_sku: str | None = None
|
||||||
|
import_product_name: str | None = None
|
||||||
|
import_color: str | None = None
|
||||||
|
import_size: str | None = None
|
||||||
|
import_img: str | None = None
|
||||||
|
import_style_number: str | None = None
|
||||||
|
import_sku_id: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class OrderItem(_SdsModel):
|
||||||
|
id: str | None = None
|
||||||
|
order_id: str | None = None
|
||||||
|
no: str | None = None
|
||||||
|
out_order_item_id: str | None = None
|
||||||
|
import_product: ImportProduct | None = None
|
||||||
|
product_id: int | None = None
|
||||||
|
factory_id: int | None = None
|
||||||
|
factory: Any | None = None
|
||||||
|
seller_sku: str | None = None
|
||||||
|
product_name: str | None = None
|
||||||
|
img_url: str | None = None
|
||||||
|
product_color_block: Any | None = None
|
||||||
|
product_size: str | None = None
|
||||||
|
num: int | None = None
|
||||||
|
status: int | None = None
|
||||||
|
origin_price: float | None = None
|
||||||
|
price: float | None = None
|
||||||
|
supply_price: float | None = None
|
||||||
|
amount: float | None = None
|
||||||
|
profit: float | None = None
|
||||||
|
refund_status: str | None = None
|
||||||
|
design_status: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Merchant(_SdsModel):
|
||||||
|
id: int | None = None
|
||||||
|
name: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class StatusHistory(_SdsModel):
|
||||||
|
status: int | None = None
|
||||||
|
gmt_update_time: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Compensation(_SdsModel):
|
||||||
|
status: int | None = None
|
||||||
|
amount: float | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Order(_SdsModel):
|
||||||
|
id: str
|
||||||
|
store_id: int | None = None
|
||||||
|
customer_id: int | None = None
|
||||||
|
no: str | None = None
|
||||||
|
out_order_no: str | None = None
|
||||||
|
username: str | None = None
|
||||||
|
origin: str | None = None
|
||||||
|
total_amount: float | None = None
|
||||||
|
product_amount: float | None = None
|
||||||
|
cost_amount: float | None = None
|
||||||
|
status: int | None = None
|
||||||
|
online_order_status_name: str | None = None
|
||||||
|
online_order_status_code: str | None = None
|
||||||
|
gmt_created: str | None = None
|
||||||
|
gmt_pay: str | None = None
|
||||||
|
pay_status: int | None = None
|
||||||
|
latest_ship_time: Any | None = None
|
||||||
|
refund_status: int | None = None
|
||||||
|
be_after_service_order: bool | None = None
|
||||||
|
fba_type: str | None = None
|
||||||
|
remark: str | None = None
|
||||||
|
carriage_name: str | None = None
|
||||||
|
parcel_list: list[Any] = []
|
||||||
|
merchant_store: MerchantStore | None = None
|
||||||
|
address: Address | None = None
|
||||||
|
issuing_bay_area: IssuingBayArea | None = None
|
||||||
|
issuing_bay: IssuingBayRef | None = None
|
||||||
|
items: list[OrderItem] = []
|
||||||
|
logistics_id: int | None = None
|
||||||
|
delivery_type: int | None = None
|
||||||
|
merchant: Merchant | None = None
|
||||||
|
finish_time: Any | None = None
|
||||||
|
status_history: list[StatusHistory] = []
|
||||||
|
compensation: Compensation | None = None
|
||||||
|
compensation_status: int | None = None
|
||||||
|
express_type: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class OrderPageResponse(_SdsModel):
|
||||||
|
total_count: int = Field(default=0, alias="totalCount")
|
||||||
|
current_page: int | None = None
|
||||||
|
page_size: int | None = None
|
||||||
|
records: list[Order] = Field(default=[], alias="list")
|
||||||
|
|
||||||
|
|
||||||
|
class OrderStatus(_SdsModel):
|
||||||
|
name: str | None = None
|
||||||
|
order_status_code: str | None = None
|
||||||
|
sort: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class OrderStatistics(_SdsModel):
|
||||||
|
name: str | None = None
|
||||||
|
order_num: int | None = None
|
||||||
|
order_status_code: str | None = None
|
||||||
|
sort: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 产品域(camelCase)-----
|
||||||
|
|
||||||
|
|
||||||
|
class ProductCategory(_SdsModel):
|
||||||
|
id: int | None = None
|
||||||
|
name: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ProductSupply(_SdsModel):
|
||||||
|
factory_id: int | None = None
|
||||||
|
name: str | None = None
|
||||||
|
supply_price: str | None = None
|
||||||
|
max_price: float | None = None
|
||||||
|
min_price: float | None = None
|
||||||
|
online_supply_num: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Product(_SdsModel):
|
||||||
|
id: int
|
||||||
|
img_url: str | None = None
|
||||||
|
thumb_img_url: str | None = None
|
||||||
|
category_id: int | None = None
|
||||||
|
category_list: list[ProductCategory] = []
|
||||||
|
on_sale_status: int | None = None
|
||||||
|
public_status: int | None = None
|
||||||
|
platform_price: str | None = None
|
||||||
|
on_sale_price: str | None = None
|
||||||
|
status: int | None = None
|
||||||
|
sku: str | None = None
|
||||||
|
name: str | None = None
|
||||||
|
english_name: str | None = None
|
||||||
|
product_supply: list[ProductSupply] = []
|
||||||
|
|
||||||
|
|
||||||
|
class ProductListResponse(_SdsModel):
|
||||||
|
total_count: int = Field(default=0, alias="totalCount")
|
||||||
|
records: list[Product] = Field(default=[], alias="list")
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 物流域(混用 snake_case,显式 alias)-----
|
||||||
|
|
||||||
|
|
||||||
|
class IssuingBay(_SdsModel):
|
||||||
|
id: int | None = None
|
||||||
|
name: str | None = None
|
||||||
|
type: str | None = None
|
||||||
|
country: str | None = None
|
||||||
|
country_name: str | None = Field(default=None, alias="country_name")
|
||||||
|
province: str | None = None
|
||||||
|
province_code: str | None = Field(default=None, alias="province_code")
|
||||||
|
city: str | None = None
|
||||||
|
city_code: str | None = Field(default=None, alias="city_code")
|
||||||
|
detail: str | None = None
|
||||||
|
priority: int | None = None
|
||||||
|
status: int | None = None
|
||||||
|
issuing_bay_area_id: int | None = Field(default=None, alias="issuing_bay_area_id")
|
||||||
|
created_time: int | None = Field(default=None, alias="created_time")
|
||||||
|
updated_time: int | None = Field(default=None, alias="updated_time")
|
||||||
|
|
||||||
|
|
||||||
|
class IssuingBayAreaDetail(_SdsModel):
|
||||||
|
id: int | None = None
|
||||||
|
name: str | None = None
|
||||||
|
type: str | None = None
|
||||||
|
country_code: str | None = None
|
||||||
|
address: str | None = None
|
||||||
|
icon_url: str | None = None
|
||||||
|
status: int | None = None
|
||||||
|
priority: int | None = None
|
||||||
|
created_time: int | None = Field(default=None, alias="created_time")
|
||||||
|
updated_time: int | None = Field(default=None, alias="updated_time")
|
||||||
|
issuing_bays: list[IssuingBay] = []
|
||||||
|
|
||||||
|
|
||||||
|
class IssuingBayAreaResponse(_SdsModel):
|
||||||
|
items: list[IssuingBayAreaDetail] = []
|
||||||
|
|
||||||
|
|
||||||
|
class LogisticsItem(IssuingBay):
|
||||||
|
factory_num: int | None = None
|
||||||
|
account_num: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class LogisticsPageResponse(_SdsModel):
|
||||||
|
items: list[LogisticsItem] = []
|
||||||
|
page: int | None = None
|
||||||
|
size: int | None = None
|
||||||
|
total_count: int = Field(default=0, alias="totalCount")
|
||||||
|
total_count_alt: int | None = Field(default=None, alias="total_count")
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 筛选面板(camelCase)-----
|
||||||
|
|
||||||
|
|
||||||
|
class FilterItem(_SdsModel):
|
||||||
|
name: str | None = None
|
||||||
|
id: Any | None = None
|
||||||
|
fp: Any | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class FiltratePanelResponse(_SdsModel):
|
||||||
|
factory_list: list[FilterItem] = []
|
||||||
|
merchant_list: list[FilterItem] = []
|
||||||
|
logistics_list: list[FilterItem] = []
|
||||||
|
early_warning_list: list[FilterItem] = []
|
||||||
|
issuing_bay_area_list: list[FilterItem] = []
|
||||||
|
extend_status_list: list[FilterItem] = []
|
||||||
|
|
||||||
|
|
||||||
|
# ----- 查询参数(请求 schema)-----
|
||||||
|
|
||||||
|
|
||||||
|
class OrderPageRequest(BaseModel):
|
||||||
|
"""SDS 订单分页查询参数。keywords(换行分隔订单号)/时间范围/状态等可选。"""
|
||||||
|
|
||||||
|
model_config = ConfigDict(extra="allow")
|
||||||
|
|
||||||
|
size: int | None = None
|
||||||
|
page: int | None = None
|
||||||
|
keywords: str | None = None
|
||||||
|
created_date_begin: str | None = None
|
||||||
|
created_date_end: str | None = None
|
||||||
|
pay_date_begin: str | None = None
|
||||||
|
pay_date_end: str | None = None
|
||||||
|
confirm_date_begin: str | None = None
|
||||||
|
confirm_date_end: str | None = None
|
||||||
|
finish_date_begin: str | None = None
|
||||||
|
finish_date_end: str | None = None
|
||||||
|
order_status_code: str | None = None
|
||||||
|
product_keywords: str | None = None
|
||||||
|
order_time_range_type: str | None = None
|
||||||
|
factory_ids: list[int] | None = None
|
||||||
|
merchant_ids: list[int] | None = None
|
||||||
|
logistics_ids: list[int] | None = None
|
||||||
|
issuing_bay_area_id: int | None = None
|
||||||
|
extend_status: str | None = None
|
||||||
|
early_warning_type: str | None = None
|
||||||
|
item_num_type: str | None = None
|
||||||
|
limit: int | None = None
|
||||||
|
all: bool | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class OrderQueryParams(BaseModel):
|
||||||
|
"""SDS 订单筛选/状态/统计查询参数(不含分页)。"""
|
||||||
|
|
||||||
|
model_config = ConfigDict(extra="allow")
|
||||||
|
|
||||||
|
created_date_begin: str | None = None
|
||||||
|
created_date_end: str | None = None
|
||||||
|
pay_date_begin: str | None = None
|
||||||
|
pay_date_end: str | None = None
|
||||||
|
confirm_date_begin: str | None = None
|
||||||
|
confirm_date_end: str | None = None
|
||||||
|
finish_date_begin: str | None = None
|
||||||
|
finish_date_end: str | None = None
|
||||||
|
order_time_range_type: str | None = None
|
||||||
|
product_keywords: str | None = None
|
||||||
|
factory_ids: list[int] | None = None
|
||||||
|
merchant_ids: list[int] | None = None
|
||||||
|
logistics_ids: list[int] | None = None
|
||||||
|
issuing_bay_area_id: int | None = None
|
||||||
|
extend_status: str | None = None
|
||||||
|
early_warning_type: str | None = None
|
||||||
|
item_num_type: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ProductListRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="allow")
|
||||||
|
|
||||||
|
keyword: str | None = None
|
||||||
|
page: int | None = None
|
||||||
|
size: int | None = None
|
||||||
|
public_status: int | None = None
|
||||||
|
limit: int | None = None
|
||||||
|
all: bool | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class LogisticsPageRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="allow")
|
||||||
|
|
||||||
|
keyword: str | None = None
|
||||||
|
delivery_method: str | None = None
|
||||||
|
issuing_bay_area_id: int
|
||||||
|
page: int | None = None
|
||||||
|
size: int | None = None
|
||||||
|
limit: int | None = None
|
||||||
|
all: bool | None = None
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
services:
|
||||||
|
prefect-server:
|
||||||
|
image: prefecthq/prefect:3-python3.12
|
||||||
|
container_name: prefect-server
|
||||||
|
command: prefect server start --host 0.0.0.0
|
||||||
|
environment:
|
||||||
|
PREFECT_UI_API_URL: http://192.168.124.92:4200/api
|
||||||
|
ports:
|
||||||
|
- "4200:4200"
|
||||||
|
volumes:
|
||||||
|
- prefect-data:/root/.prefect
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
prefect-data:
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
# 系统设计架构
|
||||||
|
|
||||||
|
本项目把两个外部业务系统(RIIN 分销、SDS POD)的 REST API 封装为可复用的 Python 客户端,
|
||||||
|
再用 Prefect 编排成 ETL 工作流(REST API → PostgreSQL)。
|
||||||
|
|
||||||
|
## 设计理念:高内聚、低耦合、关注点分离
|
||||||
|
|
||||||
|
核心原则贯穿整个项目:
|
||||||
|
|
||||||
|
1. **高内聚** —— 每个模块只承担一个职责。`api/riin/` 只管 RIIN,`api/sds/` 只管 SDS,
|
||||||
|
`flows/` 只管编排。一个模块内部的代码都围绕同一件事。
|
||||||
|
2. **低耦合** —— 客户端层(`api/`)零 Prefect 依赖,可独立使用、独立测试;
|
||||||
|
编排层(`flows/`)只通过客户端的公开方法调用它。两层之间是单向依赖。
|
||||||
|
3. **不过度抽象** —— 仅在行为稳定且重复时才抽取共享代码。当前 2 个系统,用组合式 helper
|
||||||
|
而非强制基类;等出现 3+ 个行为高度一致的源,再考虑抽象 Protocol 接口(YAGNI)。
|
||||||
|
|
||||||
|
## 分层结构
|
||||||
|
|
||||||
|
```
|
||||||
|
inkreach-prefect/
|
||||||
|
├── api/ # 客户端层:纯 httpx + pydantic,不 import prefect
|
||||||
|
│ ├── _common/ # 共享 helper(组合,非强制基类)
|
||||||
|
│ │ ├── http.py # TokenHttpClient:token 注入 + 401 自动重登
|
||||||
|
│ │ ├── cache.py # TTLCache:主数据就近缓存(随 client 生命周期)
|
||||||
|
│ │ ├── concurrency.py # run_concurrent:线程池 + 并发上限
|
||||||
|
│ │ └── batch.py # parse_id_text / split_lines / chunk
|
||||||
|
│ ├── riin/ # RIIN 分销平台
|
||||||
|
│ │ ├── client.py # RiinDisClient:认证/订单/店铺/辅助主数据
|
||||||
|
│ │ └── models.py # pydantic schema + 枚举
|
||||||
|
│ └── sds/ # SDS POD 平台
|
||||||
|
│ ├── client.py # SdsMngClient:认证/订单/产品/物流
|
||||||
|
│ └── models.py
|
||||||
|
├── flows/ # 编排层:依赖 api/,使用 prefect
|
||||||
|
│ ├── riin_orders_etl.py # @task 封装 → @flow: 抽取→转换→落 CSV(后接 PG)
|
||||||
|
│ ├── sds_orders_etl.py
|
||||||
|
│ └── shared/load.py # 共享 task:凭据加载 / CSV / PG 写入
|
||||||
|
└── pyproject.toml # build-system=hatchling,packages=["api"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 层次边界(关键)
|
||||||
|
|
||||||
|
| 层 | 依赖 | 职责 |
|
||||||
|
|----|------|------|
|
||||||
|
| `api/` 客户端层 | httpx、pydantic(**禁止** import prefect) | REST 封装、认证、分页、schema |
|
||||||
|
| `flows/` 编排层 | prefect、`api/`、pandas | 任务编排、重试、缓存、并发、调度、密钥 |
|
||||||
|
|
||||||
|
> 客户端层绝不反向依赖 Prefect。这样客户端能在 ETL 之外被复用(脚本、其他服务、单测)。
|
||||||
|
> 缓存/重试/并发这些**编排能力**留在 flows 层,客户端只保留自身需要的就近缓存(TTLCache)
|
||||||
|
> 和内部并发(run_concurrent,用于同一次调用内的多页并发)。
|
||||||
|
|
||||||
|
## 客户端层设计要点
|
||||||
|
|
||||||
|
### TokenHttpClient 基类(`_common/http.py`)
|
||||||
|
|
||||||
|
两个系统共享"token 注入 + 401 自动重登"这一稳定行为,故提取为基类(继承)。
|
||||||
|
差异通过构造参数由子类配置:
|
||||||
|
|
||||||
|
| 系统 | base_url | token 头 | 固定头 | 登录 |
|
||||||
|
|------|----------|---------|--------|------|
|
||||||
|
| RIIN | tshirt.riin.com | `Authorization` | X-Time-Zone / X-Requested-With | username + MD5(password) |
|
||||||
|
| SDS | pod-api.sdspod.com | `access-token` | 无 | username + password + 公司名 |
|
||||||
|
|
||||||
|
子类只需实现 `_login()`,其余 token 生命周期(缓存/失效/重试)由基类统一处理。
|
||||||
|
|
||||||
|
### 模式还原(忠实移植自 TS SDK)
|
||||||
|
|
||||||
|
- **分页三态**:单页 / `limit`(取首页 N 条)/ `all`(全量并发分页)
|
||||||
|
- **批量切分**:RIIN 订单 OID/快递单号 >500 自动分批;SDS 订单 keywords >200 自动分批
|
||||||
|
- **就近缓存**:主数据(店铺/工厂/客户/统计)用 TTLCache 缓存 ~5 分钟
|
||||||
|
- **并发**:多页/多批用 `run_concurrent`(线程池,默认 10)
|
||||||
|
|
||||||
|
### Schema
|
||||||
|
|
||||||
|
每个系统用 pydantic v2 建模参数与返回(`extra='allow'` 不丢字段)。
|
||||||
|
- RIIN 全 camelCase → `alias_generator=to_camel`。
|
||||||
|
- SDS 订单/产品 camelCase;物流域混用 snake_case,对 snake_case 字段显式 `Field(alias=...)`。
|
||||||
|
注意:响应字段名避免用 Python 内置名(如 `list` 会与 `list[Order]` 注解冲突,改用 `records` + `alias="list"`)。
|
||||||
|
|
||||||
|
## 编排层设计要点(对照 Prefect 文档)
|
||||||
|
|
||||||
|
| TS SDK 概念 | Prefect 对应 | 位置 |
|
||||||
|
|------------|-------------|------|
|
||||||
|
| 401 自动重试 | `@task(retries=3, retry_delay_seconds=[2,5,15])` | flows/*.py 的 extract task |
|
||||||
|
| 主数据缓存 | task 级 `cache_key_fn`(按需启用) | flows |
|
||||||
|
| 并发限制器 | Prefect 并发限制 / `.submit()`+`.map()` | flows |
|
||||||
|
| 凭据 | Prefect Secret Block(加密) | flows 内部读取 |
|
||||||
|
|
||||||
|
任务粒度遵循 Prefect"小而可观测"原则:extract / transform / load 各为独立 task。
|
||||||
|
凭据在 extract task **内部**读取 Secret,不作为参数传递,避免泄露到日志/UI。
|
||||||
|
|
||||||
|
## 密钥管理
|
||||||
|
|
||||||
|
- **PostgreSQL**:`SqlAlchemyConnector` Block,名 `pg-inkreach-dmp`。
|
||||||
|
- **RIIN / SDS**:JSON 存入 `Secret` Block,名 `riin-credentials` / `sds-credentials`,
|
||||||
|
值为 `{"username":"...","password":"..."}`。
|
||||||
|
- 本地开发备用:`.env`(已 gitignore)。
|
||||||
|
|
||||||
|
## 扩展性:如何接入第三个系统
|
||||||
|
|
||||||
|
加法式,不碰现有代码(开闭原则):
|
||||||
|
|
||||||
|
1. `api/<新系统>/` 新增 `client.py`(继承 TokenHttpClient)+ `models.py`。
|
||||||
|
2. `flows/<新系统>_orders_etl.py` 新增一个 flow,复用 `flows/shared/load.py`。
|
||||||
|
3. 注册该系统的凭据 Secret Block。
|
||||||
|
|
||||||
|
若新系统行为与现有高度一致,可白嫖 `api/_common/` 的 helper;差异大就自包含实现。
|
||||||
|
暂不强制统一 `Connector` 接口 —— 系统异质(REST 不同认证 vs DB vs 文件),强行抽象会变成漏抽象。
|
||||||
|
等行为趋同后再抽 Protocol。
|
||||||
|
|
||||||
|
## 同步任务层(flows/riin、flows/sds)
|
||||||
|
|
||||||
|
对齐 `inkreach-bi-auth-mng` 的 sync tasks,每系统 6 个,效果一致:
|
||||||
|
|
||||||
|
| 任务 | RIIN | SDS | 说明 |
|
||||||
|
|------|------|-----|------|
|
||||||
|
| 时间窗口同步 | `order_sync` | `order_sync` | 按时间范围(+筛选)拉取 |
|
||||||
|
| 增量同步 | `incremental_sync` | `incremental_sync` | 从 DB `MAX(orderedAt)` 增量,无参 |
|
||||||
|
| 近期同步 | `recent_sync` | `recent_sync` | 近 N 天 |
|
||||||
|
| 刷新未完结 | `refresh_pending` | `refresh_pending` | DB 未完结订单号批量刷新 |
|
||||||
|
| 指定单号/100前缀 | `by_orders` | `refresh_100_prefix` | RIIN 按单号;SDS 替换 100 前缀暂存 |
|
||||||
|
| 数据核验 | `verify` | `verify` | 逐天比对远程/本地计数并修复差异 |
|
||||||
|
|
||||||
|
### 共享编排层(flows/shared)
|
||||||
|
|
||||||
|
- `mappers.py`:RIIN/SDS 原始订单 → 统一 `OrderUpsertInput` / `GoodsUpsertInput`,含状态/平台/快递来源/国家归一映射。
|
||||||
|
- `db.py`:`SyncDB.sync_orders_and_goods()` —— Orders/Goods 批量 `ON CONFLICT` upsert,**写入前用库中已有 `createdAt` 覆盖**(保证幂等),Goods 写前删除该订单不再出现的行;另有 `get_latest_ordered_at` / `get_pending_orders` / `get_order_count_by_status` / `get_local_order_nos` / `get_local_goods_count` / `get_orders_with_100_prefix` 等查询。
|
||||||
|
- `clients.py`:客户端工厂(读 Secret Block)、`fetch_riin/sds_all_pages`(顺序分页,页间 500ms,仅对 5xx/429/超时重试)、RIIN 商品并发拉取(并发 20)、SDS 物流渠道字典(24h 缓存)。
|
||||||
|
|
||||||
|
### 关键差异点
|
||||||
|
|
||||||
|
- RIIN 订单与商品分离(需 `getGoods` 逐单拉取,并发 20);SDS 订单内联 `items`(直接映射,无需额外请求)。
|
||||||
|
- SDS 同步默认**跳过待付款**(refresh_pending 除外);RIIN 无此过滤。
|
||||||
|
- 两个系统时间维度不同:RIIN 用 `timeType=3`(商家推送)+ `startTime/endTime`;SDS 用 `payDate` 或 `createdDate`。
|
||||||
|
- `verify` 用各自的统计接口(RIIN `getCount`、SDS `getStatus`)按状态比对,排除 SDS 的"售后"。
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""RIIN 同步 flows:order_sync / incremental_sync / recent_sync / refresh_pending / by_orders / verify。"""
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
"""RIIN 同步公共管道:映射订单 → 并发拉商品 → upsert 到 PG。"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from flows.shared import fetch_goods_concurrent
|
||||||
|
from flows.shared.db import SyncDB
|
||||||
|
from flows.shared.mappers import map_riin_goods, map_riin_order
|
||||||
|
from api import RiinDisClient
|
||||||
|
|
||||||
|
|
||||||
|
def sync_riin_records(raw_orders: list[Any], client: RiinDisClient, db: SyncDB) -> dict[str, int]:
|
||||||
|
order_inputs: list[Any] = []
|
||||||
|
order_meta: dict[str, tuple[str, Any]] = {} # raw.id -> (orderNo, createdAt)
|
||||||
|
goods_failed = 0
|
||||||
|
|
||||||
|
for raw in raw_orders:
|
||||||
|
try:
|
||||||
|
o = map_riin_order(raw)
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
print(f"[RIIN] 映射订单失败 {getattr(raw, 'platform_oid', '?')}: {err}")
|
||||||
|
continue
|
||||||
|
order_inputs.append(o)
|
||||||
|
if raw.id:
|
||||||
|
order_meta[raw.id] = (o.orderNo, o.createdAt)
|
||||||
|
|
||||||
|
goods_inputs: list[Any] = []
|
||||||
|
if order_meta:
|
||||||
|
goods_map = fetch_goods_concurrent(client, list(order_meta.keys()))
|
||||||
|
for oid, (order_no, created_at) in order_meta.items():
|
||||||
|
items = goods_map.get(oid, [])
|
||||||
|
if not items:
|
||||||
|
goods_failed += 1
|
||||||
|
for g in items:
|
||||||
|
try:
|
||||||
|
goods_inputs.append(map_riin_goods(g, order_no, created_at))
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
print(f"[RIIN] 映射商品失败 order={order_no}: {err}")
|
||||||
|
|
||||||
|
stats = db.sync_orders_and_goods(order_inputs, goods_inputs)
|
||||||
|
stats["orderCount"] = len(order_inputs)
|
||||||
|
stats["goodsCount"] = len(goods_inputs)
|
||||||
|
stats["goodsFetchFailed"] = goods_failed
|
||||||
|
return stats
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
"""RIIN 指定单号同步:通过订单号列表(逗号/换行分隔)拉取指定订单。
|
||||||
|
|
||||||
|
对应 riin-by-orders。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from api.riin.models import OrderPageRequest
|
||||||
|
from flows.riin._pipeline import sync_riin_records
|
||||||
|
from flows.shared import SyncDB, create_riin_client
|
||||||
|
|
||||||
|
|
||||||
|
def _split_order_nos(text: str) -> list[str]:
|
||||||
|
seen: dict[str, None] = {}
|
||||||
|
for part in re.split(r"[\n\r,,\s]+", text or ""):
|
||||||
|
part = part.strip()
|
||||||
|
if part and part not in seen:
|
||||||
|
seen[part] = None
|
||||||
|
return list(seen.keys())
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def fetch_by_orders(order_nos: list[str]) -> list:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
resp = client.get_orders(OrderPageRequest(platform_oid_list=",".join(order_nos), all=True))
|
||||||
|
return list(resp.data.records)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list) -> dict:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
return sync_riin_records(raw_orders, client, SyncDB())
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="riin-by-orders-sync", log_prints=True)
|
||||||
|
def riin_by_orders_sync(order_nos: str) -> dict:
|
||||||
|
nos = _split_order_nos(order_nos)
|
||||||
|
print(f"RIIN 指定单号同步 输入 {len(nos)} 个")
|
||||||
|
if not nos:
|
||||||
|
return {"inputCount": 0, "matchedCount": 0}
|
||||||
|
|
||||||
|
raw = fetch_by_orders(nos)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 匹配 {len(raw)} | {stats}")
|
||||||
|
return {"inputCount": len(nos), "matchedCount": len(raw), **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
riin_by_orders_sync.serve(name="riin-by-orders-sync")
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""RIIN 增量同步:从数据库最近下单时间增量拉取(无参,全自动)。
|
||||||
|
|
||||||
|
对应 riin-incremental-sync。from = floorToHour(MAX(orderedAt)),to = now。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from flows.riin._pipeline import sync_riin_records
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_riin_client, fetch_riin_all_pages, floor_to_hour, format_date,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def extract_orders(params: dict) -> list:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
return fetch_riin_all_pages(client, params)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list) -> dict:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
return sync_riin_records(raw_orders, client, SyncDB())
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="riin-incremental-sync", log_prints=True)
|
||||||
|
def riin_incremental_sync() -> dict:
|
||||||
|
db = SyncDB()
|
||||||
|
latest = db.get_latest_ordered_at("RIIN")
|
||||||
|
now = datetime.now()
|
||||||
|
from_dt = floor_to_hour(latest) if latest else floor_to_hour(now)
|
||||||
|
to_dt = now
|
||||||
|
if from_dt >= to_dt:
|
||||||
|
from_dt = floor_to_hour(now)
|
||||||
|
|
||||||
|
params = {
|
||||||
|
"time_type": 3,
|
||||||
|
"start_time": format_date(from_dt),
|
||||||
|
"end_time": format_date(to_dt),
|
||||||
|
"page_size": 200,
|
||||||
|
}
|
||||||
|
print(f"RIIN 增量同步 {params['start_time']} ~ {params['end_time']}")
|
||||||
|
raw = extract_orders(params)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 订单 {stats['orderCount']} / 商品 {stats['goodsCount']}")
|
||||||
|
return {"syncRange": {"from": params["start_time"], "to": params["end_time"]}, **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
riin_incremental_sync.serve(name="riin-incremental-sync", cron="0 * * * *")
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
"""RIIN 订单同步:按时间窗口(+可选筛选)拉取订单与商品。
|
||||||
|
|
||||||
|
对应 riin-order-sync。timeType 固定 3(商家推送时间)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from api.riin.models import OrderPageRequest
|
||||||
|
from flows.riin._pipeline import sync_riin_records
|
||||||
|
from flows.shared import SyncDB, create_riin_client, fetch_riin_all_pages, format_date
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def extract_orders(params: dict[str, Any]) -> list[Any]:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
return fetch_riin_all_pages(client, params)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list[Any]) -> dict[str, int]:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
return sync_riin_records(raw_orders, client, SyncDB())
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="riin-order-sync", log_prints=True)
|
||||||
|
def riin_order_sync(
|
||||||
|
order_date_begin: str,
|
||||||
|
order_date_end: str,
|
||||||
|
platform_type: int | None = None,
|
||||||
|
order_status: int | None = None,
|
||||||
|
factory_ids: list[str] | None = None,
|
||||||
|
express_source: int | None = None,
|
||||||
|
urgent_order: int | None = None,
|
||||||
|
has_refund: bool | None = None,
|
||||||
|
page_size: int = 200,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
if not order_date_begin or not order_date_end:
|
||||||
|
raise ValueError("order_date_begin / order_date_end 为必填")
|
||||||
|
|
||||||
|
params: dict[str, Any] = {
|
||||||
|
"time_type": 3,
|
||||||
|
"start_time": order_date_begin,
|
||||||
|
"end_time": order_date_end,
|
||||||
|
"page_size": page_size,
|
||||||
|
}
|
||||||
|
if platform_type is not None:
|
||||||
|
params["platform_type"] = platform_type
|
||||||
|
if order_status is not None:
|
||||||
|
params["order_status"] = order_status
|
||||||
|
if factory_ids:
|
||||||
|
params["factory_ids"] = factory_ids
|
||||||
|
if express_source is not None:
|
||||||
|
params["express_source"] = express_source
|
||||||
|
if urgent_order is not None:
|
||||||
|
params["urgent_order"] = urgent_order
|
||||||
|
if has_refund is not None:
|
||||||
|
params["has_refund"] = has_refund
|
||||||
|
|
||||||
|
print(f"RIIN 订单同步 {order_date_begin} ~ {order_date_end}")
|
||||||
|
raw = extract_orders(params)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 订单 {stats['orderCount']} / 商品 {stats['goodsCount']} | {stats}")
|
||||||
|
return {"syncRange": {"from": order_date_begin, "to": order_date_end}, **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
riin_order_sync.serve(name="riin-order-sync")
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
"""RIIN 近期同步:拉取最近 N 天订单(默认 7 天,1~90)。
|
||||||
|
|
||||||
|
对应 riin-recent-sync。from = startOfDay(now - N days),to = now。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from flows.riin._pipeline import sync_riin_records
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_riin_client, fetch_riin_all_pages, format_date, start_of_day,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def extract_orders(params: dict) -> list:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
return fetch_riin_all_pages(client, params)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list) -> dict:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
return sync_riin_records(raw_orders, client, SyncDB())
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="riin-recent-sync", log_prints=True)
|
||||||
|
def riin_recent_sync(recent_days: int = 7) -> dict:
|
||||||
|
days = min(max(recent_days, 1), 90)
|
||||||
|
now = datetime.now()
|
||||||
|
from_dt = start_of_day(now - timedelta(days=days))
|
||||||
|
|
||||||
|
params = {
|
||||||
|
"time_type": 3,
|
||||||
|
"start_time": format_date(from_dt),
|
||||||
|
"end_time": format_date(now),
|
||||||
|
"page_size": 200,
|
||||||
|
}
|
||||||
|
print(f"RIIN 近期同步 近 {days} 天 {params['start_time']} ~ {params['end_time']}")
|
||||||
|
raw = extract_orders(params)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 订单 {stats['orderCount']} / 商品 {stats['goodsCount']}")
|
||||||
|
return {"syncRange": {"from": params["start_time"], "to": params["end_time"]}, "recentDays": days, **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
riin_recent_sync.serve(name="riin-recent-sync", cron="30 * * * *")
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
"""RIIN 刷新未完结:查 DB 中未完结订单号,按 platformOidList 批量(500)刷新。
|
||||||
|
|
||||||
|
对应 riin-refresh-pending。终态:已退款 / 已关闭 / 已发货。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from api.riin.models import OrderPageRequest
|
||||||
|
from flows.riin._pipeline import sync_riin_records
|
||||||
|
from flows.shared import SyncDB, create_riin_client
|
||||||
|
|
||||||
|
RIIN_FINISHED_STATUSES = ["已退款", "已关闭", "已发货"]
|
||||||
|
ORDER_BATCH = 500
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def fetch_pending(order_nos: list[str]) -> list:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
all_raw: list = []
|
||||||
|
for i in range(0, len(order_nos), ORDER_BATCH):
|
||||||
|
batch = order_nos[i : i + ORDER_BATCH]
|
||||||
|
resp = client.get_orders(
|
||||||
|
OrderPageRequest(platform_oid_list=",".join(batch), all=True)
|
||||||
|
)
|
||||||
|
all_raw.extend(resp.data.records)
|
||||||
|
return all_raw
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list) -> dict:
|
||||||
|
client = create_riin_client()
|
||||||
|
try:
|
||||||
|
return sync_riin_records(raw_orders, client, SyncDB())
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="riin-refresh-pending", log_prints=True)
|
||||||
|
def riin_refresh_pending() -> dict:
|
||||||
|
db = SyncDB()
|
||||||
|
order_nos = db.get_pending_orders("RIIN", RIIN_FINISHED_STATUSES)
|
||||||
|
print(f"RIIN 未完结订单 {len(order_nos)} 个")
|
||||||
|
if not order_nos:
|
||||||
|
return {"pendingCount": 0, "matchedCount": 0}
|
||||||
|
|
||||||
|
raw = fetch_pending(order_nos)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 匹配 {len(raw)} | {stats}")
|
||||||
|
return {"pendingCount": len(order_nos), "matchedCount": len(raw), **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
riin_refresh_pending.serve(name="riin-refresh-pending", cron="0 */3 * * *")
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
"""RIIN 数据核验:逐天比对远程/本地按状态计数,有差异则拉取修复。
|
||||||
|
|
||||||
|
对应 riin-verify。mode: recent(近 N 天,不含今天)/ range(指定区间)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from api.riin.models import OrderPageRequest
|
||||||
|
from flows.riin._pipeline import sync_riin_records
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_riin_client, fetch_riin_all_pages, format_date, start_of_day,
|
||||||
|
)
|
||||||
|
|
||||||
|
RIIN_COUNT_STATUS_MAP = {
|
||||||
|
"assignedSuceessed": "已分配",
|
||||||
|
"factoryAuditing": "工厂审核中",
|
||||||
|
"inProducing": "生产中",
|
||||||
|
"shipped": "已发货",
|
||||||
|
"closed": "已关闭",
|
||||||
|
"refunding": "退款中",
|
||||||
|
"refunded": "已退款",
|
||||||
|
"rejecting": "拒审中",
|
||||||
|
"rejectedToShop": "已退回店铺",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_days(mode: str, recent_days: int, date_range: str | None) -> list[datetime]:
|
||||||
|
today = start_of_day(datetime.now())
|
||||||
|
if mode == "range" and date_range:
|
||||||
|
parts = [p.strip() for p in date_range.split("~")]
|
||||||
|
from_dt = datetime.fromisoformat(parts[0])
|
||||||
|
to_dt = datetime.fromisoformat(parts[1]) if len(parts) > 1 and parts[1] else from_dt
|
||||||
|
days = []
|
||||||
|
cur = from_dt
|
||||||
|
while cur <= to_dt:
|
||||||
|
days.append(cur)
|
||||||
|
cur += timedelta(days=1)
|
||||||
|
return days
|
||||||
|
days_n = min(max(recent_days, 1), 90)
|
||||||
|
return [today - timedelta(days=i) for i in range(days_n, 0, -1)]
|
||||||
|
|
||||||
|
|
||||||
|
def _count_to_status_map(data: dict) -> dict[str, int]:
|
||||||
|
out: dict[str, int] = {}
|
||||||
|
for key, count in data.items():
|
||||||
|
if key.startswith("dayRange"):
|
||||||
|
continue
|
||||||
|
label = RIIN_COUNT_STATUS_MAP.get(key)
|
||||||
|
if label and count and count > 0:
|
||||||
|
out[label] = count
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _diff(remote: dict[str, int], local: dict[str, int]) -> tuple[dict[str, int], bool]:
|
||||||
|
diff: dict[str, int] = {}
|
||||||
|
has_diff = False
|
||||||
|
for status in set(remote) | set(local):
|
||||||
|
r = remote.get(status, 0)
|
||||||
|
l = local.get(status, 0)
|
||||||
|
if r != l:
|
||||||
|
diff[status] = l - r
|
||||||
|
has_diff = True
|
||||||
|
return diff, has_diff
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="riin-verify", log_prints=True)
|
||||||
|
def riin_verify(mode: str = "recent", recent_days: int = 1, date_range: str | None = None) -> dict:
|
||||||
|
days = _parse_days(mode, recent_days, date_range)
|
||||||
|
db = SyncDB()
|
||||||
|
client = create_riin_client()
|
||||||
|
|
||||||
|
totals = {k: 0 for k in ("ordersAdded", "ordersUpdated", "ordersRemoved",
|
||||||
|
"goodsAdded", "goodsUpdated", "goodsRemoved")}
|
||||||
|
days_with_diff = 0
|
||||||
|
day_reports = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
for day in days:
|
||||||
|
day_str = format_date(day).split(" ")[0]
|
||||||
|
day_start = start_of_day(day)
|
||||||
|
day_end = day_start + timedelta(days=1)
|
||||||
|
|
||||||
|
count_resp = client.get_order_count(
|
||||||
|
OrderPageRequest(time_type=3, start_time=format_date(day_start), end_time=format_date(day_end))
|
||||||
|
)
|
||||||
|
remote_map = _count_to_status_map(count_resp.data.model_dump(by_alias=False))
|
||||||
|
remote_total = sum(remote_map.values())
|
||||||
|
local_counts = db.get_order_count_by_status("RIIN", day_start, day_end)
|
||||||
|
local_goods = db.get_local_goods_count("RIIN", day_start, day_end)
|
||||||
|
diff, has_diff = _diff(remote_map, local_counts)
|
||||||
|
|
||||||
|
print(f"[核验 {day_str}] 远程 {remote_total} 本地 {sum(local_counts.values())}"
|
||||||
|
+ (f" 差异 {diff}" if has_diff else " 一致,跳过"))
|
||||||
|
if not has_diff:
|
||||||
|
day_reports.append({"date": day_str, "diff": diff, "ok": True})
|
||||||
|
continue
|
||||||
|
|
||||||
|
days_with_diff += 1
|
||||||
|
raw_orders = fetch_riin_all_pages(
|
||||||
|
client,
|
||||||
|
{"time_type": 3, "start_time": format_date(day_start), "end_time": format_date(day_end), "page_size": 200},
|
||||||
|
)
|
||||||
|
stats = sync_riin_records(raw_orders, client, db)
|
||||||
|
for k in ("ordersCreated", "ordersUpdated", "goodsCreated", "goodsUpdated"):
|
||||||
|
pass
|
||||||
|
totals["ordersAdded"] += stats["ordersCreated"]
|
||||||
|
totals["ordersUpdated"] += stats["ordersUpdated"]
|
||||||
|
totals["goodsAdded"] += stats["goodsCreated"]
|
||||||
|
totals["goodsUpdated"] += stats["goodsUpdated"]
|
||||||
|
day_reports.append({"date": day_str, "diff": diff, "ok": False,
|
||||||
|
"orders": stats["orderCount"], "goods": stats["goodsCount"]})
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
return {
|
||||||
|
"verifyReport": {"days": day_reports, "summary": {"totalDays": len(days), "daysWithDiff": days_with_diff, **totals}},
|
||||||
|
"syncRange": {"from": format_date(start_of_day(days[0])) if days else "", "to": format_date(start_of_day(days[-1])) if days else ""},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
riin_verify.serve(name="riin-verify", cron="0 6 * * *")
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""SDS 同步 flows:order_sync / incremental_sync / recent_sync / refresh_pending / refresh_100_prefix / verify。"""
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
"""SDS 同步公共管道:映射订单(含内联 items)→ upsert 到 PG。"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any, Callable
|
||||||
|
|
||||||
|
from flows.shared import resolve_courier_source
|
||||||
|
from flows.shared.db import SyncDB
|
||||||
|
from flows.shared.mappers import map_sds_goods, map_sds_order
|
||||||
|
|
||||||
|
|
||||||
|
def map_sds_records(
|
||||||
|
raw_orders: list[Any],
|
||||||
|
channel_map: dict[int, str | None],
|
||||||
|
channel_refresh: Callable[[], dict[int, str | None]] | None = None,
|
||||||
|
skip_unpaid: bool = True,
|
||||||
|
) -> tuple[list[Any], list[Any], int]:
|
||||||
|
"""映射 SDS 订单+商品。返回 (order_inputs, goods_inputs, skipped_unpaid)。"""
|
||||||
|
order_inputs: list[Any] = []
|
||||||
|
goods_inputs: list[Any] = []
|
||||||
|
refreshed: set[int] = set()
|
||||||
|
skipped_unpaid = 0
|
||||||
|
|
||||||
|
for raw in raw_orders:
|
||||||
|
courier = None
|
||||||
|
if channel_refresh is not None:
|
||||||
|
courier = resolve_courier_source(raw.logistics_id, channel_map, channel_refresh, refreshed)
|
||||||
|
try:
|
||||||
|
o = map_sds_order(raw, channel_map, courier)
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
print(f"[SDS] 映射订单失败 {getattr(raw, 'no', '?')}: {err}")
|
||||||
|
continue
|
||||||
|
if skip_unpaid and o.systemStatus == "待付款":
|
||||||
|
skipped_unpaid += 1
|
||||||
|
continue
|
||||||
|
order_inputs.append(o)
|
||||||
|
for item in raw.items or []:
|
||||||
|
try:
|
||||||
|
goods_inputs.append(map_sds_goods(item, o.orderNo, o.createdAt))
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
print(f"[SDS] 映射商品失败 order={o.orderNo}: {err}")
|
||||||
|
|
||||||
|
return order_inputs, goods_inputs, skipped_unpaid
|
||||||
|
|
||||||
|
|
||||||
|
def sync_sds_records(
|
||||||
|
raw_orders: list[Any], channel_map: dict[int, str | None], db: SyncDB,
|
||||||
|
channel_refresh: Callable[[], dict[int, str | None]] | None = None,
|
||||||
|
skip_unpaid: bool = True,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
order_inputs, goods_inputs, skipped = map_sds_records(
|
||||||
|
raw_orders, channel_map, channel_refresh, skip_unpaid
|
||||||
|
)
|
||||||
|
stats = db.sync_orders_and_goods(order_inputs, goods_inputs)
|
||||||
|
stats["orderCount"] = len(order_inputs)
|
||||||
|
stats["goodsCount"] = len(goods_inputs)
|
||||||
|
stats["skippedUnpaid"] = skipped
|
||||||
|
return stats
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
"""SDS 增量同步:从数据库最近下单时间增量拉取(无参,全自动)。
|
||||||
|
|
||||||
|
对应 sds-incremental-sync。from = floorToHour(MAX(orderedAt)),to = now,按 payDate。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from flows.sds._pipeline import sync_sds_records
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_sds_client, fetch_sds_all_pages, floor_to_hour, format_date,
|
||||||
|
get_channel_type_map, refresh_channel_type_map,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def extract_orders(params: dict) -> list:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
return fetch_sds_all_pages(client, params)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list) -> dict:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
channel_map = get_channel_type_map(client)
|
||||||
|
return sync_sds_records(raw_orders, channel_map, SyncDB(),
|
||||||
|
channel_refresh=lambda: refresh_channel_type_map(client))
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="sds-incremental-sync", log_prints=True)
|
||||||
|
def sds_incremental_sync() -> dict:
|
||||||
|
db = SyncDB()
|
||||||
|
latest = db.get_latest_ordered_at("SDS")
|
||||||
|
now = datetime.now()
|
||||||
|
from_dt = floor_to_hour(latest) if latest else floor_to_hour(now)
|
||||||
|
if from_dt >= now:
|
||||||
|
from_dt = floor_to_hour(now)
|
||||||
|
|
||||||
|
params = {"pay_date_begin": format_date(from_dt), "pay_date_end": format_date(now), "size": 500}
|
||||||
|
print(f"SDS 增量同步 {params['pay_date_begin']} ~ {params['pay_date_end']}")
|
||||||
|
raw = extract_orders(params)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 订单 {stats['orderCount']} / 商品 {stats['goodsCount']}")
|
||||||
|
return {"syncRange": {"from": params["pay_date_begin"], "to": params["pay_date_end"]}, **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sds_incremental_sync.serve(name="sds-incremental-sync", cron="15 * * * *")
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
"""SDS 订单同步:按下单时间窗口(+筛选)拉取订单与商品。
|
||||||
|
|
||||||
|
对应 sds-order-sync。主时间维度 payDate(下单时间)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from flows.sds._pipeline import sync_sds_records
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_sds_client, fetch_sds_all_pages, get_channel_type_map, refresh_channel_type_map,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def extract_orders(params: dict[str, Any]) -> list:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
return fetch_sds_all_pages(client, params)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list) -> dict:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
channel_map = get_channel_type_map(client)
|
||||||
|
return sync_sds_records(raw_orders, channel_map, SyncDB(),
|
||||||
|
channel_refresh=lambda: refresh_channel_type_map(client))
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="sds-order-sync", log_prints=True)
|
||||||
|
def sds_order_sync(
|
||||||
|
pay_date_begin: str,
|
||||||
|
pay_date_end: str,
|
||||||
|
order_status_code: str | None = None,
|
||||||
|
factory_ids: list[int] | None = None,
|
||||||
|
merchant_ids: list[int] | None = None,
|
||||||
|
logistics_ids: list[int] | None = None,
|
||||||
|
issuing_bay_area_id: int | None = None,
|
||||||
|
keywords: str | None = None,
|
||||||
|
page_size: int = 500,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
params: dict[str, Any] = {
|
||||||
|
"pay_date_begin": pay_date_begin,
|
||||||
|
"pay_date_end": pay_date_end,
|
||||||
|
"size": page_size,
|
||||||
|
}
|
||||||
|
if order_status_code is not None:
|
||||||
|
params["order_status_code"] = order_status_code
|
||||||
|
if factory_ids:
|
||||||
|
params["factory_ids"] = factory_ids
|
||||||
|
if merchant_ids:
|
||||||
|
params["merchant_ids"] = merchant_ids
|
||||||
|
if logistics_ids:
|
||||||
|
params["logistics_ids"] = logistics_ids
|
||||||
|
if issuing_bay_area_id is not None:
|
||||||
|
params["issuing_bay_area_id"] = issuing_bay_area_id
|
||||||
|
if keywords:
|
||||||
|
params["keywords"] = keywords
|
||||||
|
|
||||||
|
print(f"SDS 订单同步 {pay_date_begin} ~ {pay_date_end}")
|
||||||
|
raw = extract_orders(params)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 订单 {stats['orderCount']} / 商品 {stats['goodsCount']} / 跳过待付款 {stats.get('skippedUnpaid',0)}")
|
||||||
|
return {"syncRange": {"from": pay_date_begin, "to": pay_date_end}, **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sds_order_sync.serve(name="sds-order-sync")
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
"""SDS 近期同步:拉取最近 N 天订单(默认 7,1~90),按 createdDate。
|
||||||
|
|
||||||
|
对应 sds-recent-sync。from = startOfDay(now - N days),to = startOfDay(today)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from flows.sds._pipeline import sync_sds_records
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_sds_client, fetch_sds_all_pages, format_date, get_channel_type_map,
|
||||||
|
refresh_channel_type_map, start_of_day,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def extract_orders(params: dict) -> list:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
return fetch_sds_all_pages(client, params)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list) -> dict:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
channel_map = get_channel_type_map(client)
|
||||||
|
return sync_sds_records(raw_orders, channel_map, SyncDB(),
|
||||||
|
channel_refresh=lambda: refresh_channel_type_map(client))
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="sds-recent-sync", log_prints=True)
|
||||||
|
def sds_recent_sync(recent_days: int = 7) -> dict:
|
||||||
|
days = min(max(recent_days, 1), 90)
|
||||||
|
now = datetime.now()
|
||||||
|
from_dt = start_of_day(now - timedelta(days=days))
|
||||||
|
to_dt = start_of_day(now)
|
||||||
|
|
||||||
|
params = {"created_date_begin": format_date(from_dt), "created_date_end": format_date(to_dt), "size": 500}
|
||||||
|
print(f"SDS 近期同步 近 {days} 天 {params['created_date_begin']} ~ {params['created_date_end']}")
|
||||||
|
raw = extract_orders(params)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 订单 {stats['orderCount']} / 商品 {stats['goodsCount']}")
|
||||||
|
return {"syncRange": {"from": params["created_date_begin"], "to": format_date(now)},
|
||||||
|
"recentDays": days, **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sds_recent_sync.serve(name="sds-recent-sync", cron="45 * * * *")
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
"""SDS 100前缀刷新:备货中/待确认且 productionNo 以 100 开头的订单,工厂分配后用新数据替换。
|
||||||
|
|
||||||
|
对应 sds-refresh-100-prefix。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from api.sds.models import OrderPageRequest
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_sds_client, get_channel_type_map,
|
||||||
|
refresh_channel_type_map, resolve_courier_source,
|
||||||
|
)
|
||||||
|
from flows.shared.mappers import map_sds_goods, map_sds_order
|
||||||
|
|
||||||
|
TARGET_STATUSES = ["备货中", "待确认"]
|
||||||
|
KEYWORDS_BATCH = 200
|
||||||
|
|
||||||
|
|
||||||
|
def _has_100_prefix_item(raw) -> bool:
|
||||||
|
for item in raw.items or []:
|
||||||
|
prod_no = (item.no or item.id) if hasattr(item, "no") else None
|
||||||
|
if prod_no and str(prod_no).startswith("100"):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def fetch_remote(order_nos: list[str]) -> dict:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
remote: dict[str, object] = {}
|
||||||
|
for i in range(0, len(order_nos), KEYWORDS_BATCH):
|
||||||
|
batch = order_nos[i : i + KEYWORDS_BATCH]
|
||||||
|
resp = client.get_orders(OrderPageRequest(keywords="\n".join(batch), all=True))
|
||||||
|
for o in resp.records:
|
||||||
|
if o.no:
|
||||||
|
remote[o.no] = o
|
||||||
|
return remote
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(remote_map: dict, target_order_nos: list[str]) -> dict:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
channel_map = get_channel_type_map(client)
|
||||||
|
refreshed: set[int] = set()
|
||||||
|
|
||||||
|
order_inputs: list = []
|
||||||
|
goods_inputs: list = []
|
||||||
|
skipped_pending = skipped_unpaid = 0
|
||||||
|
|
||||||
|
for no in target_order_nos:
|
||||||
|
raw = remote_map.get(no)
|
||||||
|
if not raw:
|
||||||
|
continue
|
||||||
|
if _has_100_prefix_item(raw):
|
||||||
|
skipped_pending += 1
|
||||||
|
continue
|
||||||
|
courier = resolve_courier_source(raw.logistics_id, channel_map,
|
||||||
|
lambda: refresh_channel_type_map(client), refreshed)
|
||||||
|
try:
|
||||||
|
o = map_sds_order(raw, channel_map, courier)
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
print(f"[SDS 100] 映射失败 {no}: {err}")
|
||||||
|
continue
|
||||||
|
if o.systemStatus == "待付款":
|
||||||
|
skipped_unpaid += 1
|
||||||
|
continue
|
||||||
|
order_inputs.append(o)
|
||||||
|
for item in raw.items or []:
|
||||||
|
try:
|
||||||
|
goods_inputs.append(map_sds_goods(item, o.orderNo, o.createdAt))
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass
|
||||||
|
|
||||||
|
stats = SyncDB().sync_orders_and_goods(order_inputs, goods_inputs)
|
||||||
|
stats["skippedStillPending"] = skipped_pending
|
||||||
|
stats["skippedUnpaid"] = skipped_unpaid
|
||||||
|
return stats
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="sds-refresh-100-prefix", log_prints=True)
|
||||||
|
def sds_refresh_100_prefix() -> dict:
|
||||||
|
db = SyncDB()
|
||||||
|
targets = db.get_orders_with_100_prefix("SDS", TARGET_STATUSES)
|
||||||
|
order_nos = list({o for o, _ in targets})
|
||||||
|
print(f"SDS 100前缀目标订单 {len(order_nos)} 个")
|
||||||
|
if not order_nos:
|
||||||
|
return {"targetCount": 0, "refreshed": 0}
|
||||||
|
|
||||||
|
remote = fetch_remote(order_nos)
|
||||||
|
stats = upsert_records(remote, order_nos)
|
||||||
|
print(f"完成: 刷新 {stats['orderCount']} / 仍待分配 {stats.get('skippedStillPending',0)} / 待付款 {stats.get('skippedUnpaid',0)}")
|
||||||
|
return {"targetCount": len(order_nos), **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sds_refresh_100_prefix.serve(name="sds-refresh-100-prefix", cron="30 */2 * * *")
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
"""SDS 刷新未完结:查 DB 未完结订单号,按 keywords 批量刷新。
|
||||||
|
|
||||||
|
对应 sds-refresh-pending。终态:已完成 / 已取消。注意不跳过待付款。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from api.sds.models import OrderPageRequest
|
||||||
|
from flows.sds._pipeline import sync_sds_records
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_sds_client, get_channel_type_map, refresh_channel_type_map,
|
||||||
|
)
|
||||||
|
|
||||||
|
SDS_FINISHED_STATUSES = ["已完成", "已取消"]
|
||||||
|
KEYWORDS_BATCH = 500
|
||||||
|
|
||||||
|
|
||||||
|
@task(retries=3, retry_delay_seconds=[3, 6, 9])
|
||||||
|
def fetch_pending(order_nos: list[str]) -> list:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
all_raw: list = []
|
||||||
|
for i in range(0, len(order_nos), KEYWORDS_BATCH):
|
||||||
|
batch = order_nos[i : i + KEYWORDS_BATCH]
|
||||||
|
resp = client.get_orders(OrderPageRequest(keywords="\n".join(batch), all=True))
|
||||||
|
all_raw.extend(resp.records)
|
||||||
|
return all_raw
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@task
|
||||||
|
def upsert_records(raw_orders: list) -> dict:
|
||||||
|
client = create_sds_client()
|
||||||
|
try:
|
||||||
|
channel_map = get_channel_type_map(client)
|
||||||
|
return sync_sds_records(raw_orders, channel_map, SyncDB(),
|
||||||
|
channel_refresh=lambda: refresh_channel_type_map(client),
|
||||||
|
skip_unpaid=False)
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="sds-refresh-pending", log_prints=True)
|
||||||
|
def sds_refresh_pending() -> dict:
|
||||||
|
db = SyncDB()
|
||||||
|
order_nos = db.get_pending_orders("SDS", SDS_FINISHED_STATUSES)
|
||||||
|
print(f"SDS 未完结订单 {len(order_nos)} 个")
|
||||||
|
if not order_nos:
|
||||||
|
return {"pendingCount": 0, "matchedCount": 0}
|
||||||
|
|
||||||
|
raw = fetch_pending(order_nos)
|
||||||
|
stats = upsert_records(raw)
|
||||||
|
print(f"完成: 匹配 {len(raw)} | {stats}")
|
||||||
|
return {"pendingCount": len(order_nos), "matchedCount": len(raw), **stats}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sds_refresh_pending.serve(name="sds-refresh-pending", cron="0 */2 * * *")
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
"""SDS 数据核验:逐天比对远程/本地按状态计数(排除"售后"),有差异则拉取修复。
|
||||||
|
|
||||||
|
对应 sds-verify。mode: recent(近 N 天,不含今天)/ range(指定区间)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
from api.sds.models import OrderQueryParams
|
||||||
|
from flows.sds._pipeline import sync_sds_records
|
||||||
|
from flows.shared import (
|
||||||
|
SyncDB, create_sds_client, fetch_sds_all_pages, format_date, get_channel_type_map,
|
||||||
|
refresh_channel_type_map, start_of_day,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_days(mode: str, recent_days: int, date_range: str | None) -> list[datetime]:
|
||||||
|
today = start_of_day(datetime.now())
|
||||||
|
if mode == "range" and date_range:
|
||||||
|
parts = [p.strip() for p in date_range.split("~")]
|
||||||
|
from_dt = datetime.fromisoformat(parts[0])
|
||||||
|
to_dt = datetime.fromisoformat(parts[1]) if len(parts) > 1 and parts[1] else from_dt
|
||||||
|
days = []
|
||||||
|
cur = from_dt
|
||||||
|
while cur <= to_dt:
|
||||||
|
days.append(cur)
|
||||||
|
cur += timedelta(days=1)
|
||||||
|
return days
|
||||||
|
n = min(max(recent_days, 1), 90)
|
||||||
|
return [today - timedelta(days=i) for i in range(n, 0, -1)]
|
||||||
|
|
||||||
|
|
||||||
|
def _diff(remote_stats: list, local: dict[str, int]) -> tuple[dict[str, int], bool]:
|
||||||
|
diff: dict[str, int] = {}
|
||||||
|
has_diff = False
|
||||||
|
statuses = {getattr(s, "name", None) for s in remote_stats} | set(local)
|
||||||
|
statuses.discard("售后")
|
||||||
|
statuses.discard(None)
|
||||||
|
for status in statuses:
|
||||||
|
remote = next((getattr(s, "orderNum", 0) for s in remote_stats if getattr(s, "name", None) == status), 0)
|
||||||
|
local_n = local.get(status, 0)
|
||||||
|
if remote != local_n:
|
||||||
|
diff[status] = local_n - remote
|
||||||
|
has_diff = True
|
||||||
|
return diff, has_diff
|
||||||
|
|
||||||
|
|
||||||
|
@flow(name="sds-verify", log_prints=True)
|
||||||
|
def sds_verify(mode: str = "recent", recent_days: int = 1, date_range: str | None = None) -> dict:
|
||||||
|
days = _parse_days(mode, recent_days, date_range)
|
||||||
|
db = SyncDB()
|
||||||
|
client = create_sds_client()
|
||||||
|
channel_map = get_channel_type_map(client)
|
||||||
|
|
||||||
|
totals = {k: 0 for k in ("ordersAdded", "ordersUpdated", "goodsAdded", "goodsUpdated")}
|
||||||
|
days_with_diff = 0
|
||||||
|
day_reports = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
for day in days:
|
||||||
|
day_str = format_date(day).split(" ")[0]
|
||||||
|
day_start = start_of_day(day)
|
||||||
|
day_end = day_start + timedelta(days=1)
|
||||||
|
|
||||||
|
remote_stats = client.get_status(
|
||||||
|
OrderQueryParams(pay_date_begin=format_date(day_start), pay_date_end=format_date(day_end))
|
||||||
|
)
|
||||||
|
remote_total = sum(getattr(s, "orderNum", 0) or 0 for s in remote_stats)
|
||||||
|
local_counts = db.get_order_count_by_status("SDS", day_start, day_end)
|
||||||
|
diff, has_diff = _diff(remote_stats, local_counts)
|
||||||
|
|
||||||
|
print(f"[核验 {day_str}] 远程 {remote_total} 本地 {sum(local_counts.values())}"
|
||||||
|
+ (f" 差异 {diff}" if has_diff else " 一致,跳过"))
|
||||||
|
if not has_diff:
|
||||||
|
day_reports.append({"date": day_str, "diff": diff, "ok": True})
|
||||||
|
continue
|
||||||
|
|
||||||
|
days_with_diff += 1
|
||||||
|
raw = fetch_sds_all_pages(
|
||||||
|
client,
|
||||||
|
{"pay_date_begin": format_date(day_start), "pay_date_end": format_date(day_end), "size": 500},
|
||||||
|
)
|
||||||
|
stats = sync_sds_records(raw, channel_map, db,
|
||||||
|
channel_refresh=lambda: refresh_channel_type_map(client))
|
||||||
|
totals["ordersAdded"] += stats["ordersCreated"]
|
||||||
|
totals["ordersUpdated"] += stats["ordersUpdated"]
|
||||||
|
totals["goodsAdded"] += stats["goodsCreated"]
|
||||||
|
totals["goodsUpdated"] += stats["goodsUpdated"]
|
||||||
|
day_reports.append({"date": day_str, "diff": diff, "ok": False,
|
||||||
|
"orders": stats["orderCount"], "goods": stats["goodsCount"]})
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
return {
|
||||||
|
"verifyReport": {"days": day_reports, "summary": {"totalDays": len(days), "daysWithDiff": days_with_diff, **totals}},
|
||||||
|
"syncRange": {"from": format_date(start_of_day(days[0])) if days else "", "to": format_date(start_of_day(days[-1])) if days else ""},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sds_verify.serve(name="sds-verify", cron="0 7 * * *")
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
"""同步编排共享层:客户端/分页、映射、PG 写入。"""
|
||||||
|
|
||||||
|
from flows.shared import clients, db, mappers
|
||||||
|
from flows.shared.clients import (
|
||||||
|
create_riin_client,
|
||||||
|
create_sds_client,
|
||||||
|
fetch_goods_concurrent,
|
||||||
|
fetch_riin_all_pages,
|
||||||
|
fetch_sds_all_pages,
|
||||||
|
floor_to_hour,
|
||||||
|
format_date,
|
||||||
|
get_channel_type_map,
|
||||||
|
refresh_channel_type_map,
|
||||||
|
resolve_courier_source,
|
||||||
|
start_of_day,
|
||||||
|
)
|
||||||
|
from flows.shared.db import SyncDB
|
||||||
|
from flows.shared.mappers import GoodsUpsertInput, OrderUpsertInput
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"clients", "db", "mappers", "SyncDB",
|
||||||
|
"OrderUpsertInput", "GoodsUpsertInput",
|
||||||
|
"create_riin_client", "create_sds_client",
|
||||||
|
"fetch_riin_all_pages", "fetch_sds_all_pages", "fetch_goods_concurrent",
|
||||||
|
"get_channel_type_map", "refresh_channel_type_map", "resolve_courier_source",
|
||||||
|
"format_date", "start_of_day", "floor_to_hour",
|
||||||
|
]
|
||||||
@@ -0,0 +1,229 @@
|
|||||||
|
"""同步共享:客户端工厂、顺序分页(含重试/延迟)、RIIN 商品拉取、SDS 物流渠道字典、日期辅助。
|
||||||
|
|
||||||
|
行为对齐 inkreach-bi-auth-mng 的 sync/tasks/clients.ts:
|
||||||
|
- 顺序分页(页间 500ms),非并发,对外部 API 友好。
|
||||||
|
- fetch_with_retry:仅对 5xx / 429 / 超时重试,最多 3 次,退避 3s*次数。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from typing import Any, Callable, TypeVar
|
||||||
|
|
||||||
|
from prefect.blocks.system import Secret
|
||||||
|
|
||||||
|
from api import RiinDisClient, SdsMngClient
|
||||||
|
from api.riin.models import OrderPageRequest as RiinReq
|
||||||
|
from api.sds.models import LogisticsPageRequest, OrderPageRequest as SdsReq
|
||||||
|
from flows.shared.mappers import map_sds_channel_type
|
||||||
|
|
||||||
|
T = TypeVar("T")
|
||||||
|
|
||||||
|
GOODS_CONCURRENCY = 20
|
||||||
|
PAGE_DELAY_MS = 500
|
||||||
|
MAX_RETRIES = 3
|
||||||
|
RETRY_DELAY_MS = 3000
|
||||||
|
TZ_CN = timezone(timedelta(hours=8))
|
||||||
|
|
||||||
|
_RIIN_FINISHED = ["已退款", "已关闭", "已发货"]
|
||||||
|
_SDS_FINISHED = ["已完成", "已取消"]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 客户端工厂 ----------
|
||||||
|
|
||||||
|
def create_riin_client() -> RiinDisClient:
|
||||||
|
cred = _load_secret("riin-credentials")
|
||||||
|
return RiinDisClient(cred["username"], cred["password"])
|
||||||
|
|
||||||
|
|
||||||
|
def create_sds_client() -> SdsMngClient:
|
||||||
|
cred = _load_secret("sds-credentials")
|
||||||
|
return SdsMngClient(cred["username"], cred["password"])
|
||||||
|
|
||||||
|
|
||||||
|
def _load_secret(name: str) -> dict[str, str]:
|
||||||
|
value = Secret.load(name).get()
|
||||||
|
if isinstance(value, str):
|
||||||
|
value = json.loads(value)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 日期辅助 ----------
|
||||||
|
|
||||||
|
def format_date(d: datetime) -> str:
|
||||||
|
return d.strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
|
||||||
|
def start_of_day(d: datetime) -> datetime:
|
||||||
|
return d.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||||
|
|
||||||
|
|
||||||
|
def floor_to_hour(d: datetime) -> datetime:
|
||||||
|
return d.replace(minute=0, second=0, microsecond=0)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 重试 ----------
|
||||||
|
|
||||||
|
def fetch_with_retry(fn: Callable[[], T], label: str) -> T:
|
||||||
|
last_err: Exception | None = None
|
||||||
|
for attempt in range(1, MAX_RETRIES + 1):
|
||||||
|
try:
|
||||||
|
return fn()
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
last_err = err
|
||||||
|
status = _status_of(err)
|
||||||
|
msg = str(err)
|
||||||
|
is_timeout = "timeout" in msg.lower() or "timed out" in msg.lower()
|
||||||
|
retryable = status == 429 or (status is not None and status >= 500) or is_timeout
|
||||||
|
if attempt < MAX_RETRIES and retryable:
|
||||||
|
print(f"[sync] {label} 第 {attempt}/{MAX_RETRIES} 次重试 ({status}: {msg[:120]})")
|
||||||
|
time.sleep(RETRY_DELAY_MS * attempt / 1000)
|
||||||
|
continue
|
||||||
|
raise
|
||||||
|
assert last_err is not None
|
||||||
|
raise last_err
|
||||||
|
|
||||||
|
|
||||||
|
def _status_of(err: Exception) -> int | None:
|
||||||
|
for attr in ("status_code", "statusCode", "status"):
|
||||||
|
v = getattr(err, attr, None)
|
||||||
|
if isinstance(v, int):
|
||||||
|
return v
|
||||||
|
resp = getattr(err, "response", None)
|
||||||
|
return getattr(resp, "status_code", None) if resp is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 分页 ----------
|
||||||
|
|
||||||
|
def fetch_riin_all_pages(client: RiinDisClient, params: dict[str, Any], max_pages: int = 1000) -> list[Any]:
|
||||||
|
page_size = params.get("page_size", 200)
|
||||||
|
base = {**params, "page_size": page_size}
|
||||||
|
|
||||||
|
def fetch(idx: int):
|
||||||
|
return fetch_with_retry(lambda: client.get_orders(RiinReq(**{**base, "page_index": idx})), f"RIIN page {idx}")
|
||||||
|
|
||||||
|
first = fetch(1)
|
||||||
|
total = first.data.total
|
||||||
|
records = list(first.data.records)
|
||||||
|
total_pages = min(-(-total // page_size), max_pages)
|
||||||
|
for page in range(2, total_pages + 1):
|
||||||
|
resp = fetch(page)
|
||||||
|
records.extend(resp.data.records)
|
||||||
|
time.sleep(PAGE_DELAY_MS / 1000)
|
||||||
|
if total_pages == max_pages:
|
||||||
|
print(f"[RIIN sync] 达到最大页数限制 {max_pages},截断到 {len(records)} 条")
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_sds_all_pages(client: SdsMngClient, params: dict[str, Any]) -> list[Any]:
|
||||||
|
page_size = params.get("size", 500)
|
||||||
|
base = {**params, "size": page_size}
|
||||||
|
|
||||||
|
def fetch(idx: int):
|
||||||
|
return fetch_with_retry(lambda: client.get_orders(SdsReq(**{**base, "page": idx})), f"SDS page {idx}")
|
||||||
|
|
||||||
|
first = fetch(1)
|
||||||
|
records = list(first.records)
|
||||||
|
total_pages = -(-first.total_count // page_size)
|
||||||
|
for page in range(2, total_pages + 1):
|
||||||
|
resp = fetch(page)
|
||||||
|
records.extend(resp.records)
|
||||||
|
time.sleep(PAGE_DELAY_MS / 1000)
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- RIIN 商品 ----------
|
||||||
|
|
||||||
|
def fetch_goods_for_order(client: RiinDisClient, order_id: str) -> list[Any]:
|
||||||
|
try:
|
||||||
|
resp = client.get_order_goods(order_id)
|
||||||
|
return list(resp.data)
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
print(f"[RIIN sync] 订单 {order_id} 商品拉取失败: {err}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_goods_concurrent(client: RiinDisClient, order_ids: list[str]) -> dict[str, list[Any]]:
|
||||||
|
"""以 GOODS_CONCURRENCY 并发拉取多订单商品,返回 {order_id: [goods...]}。"""
|
||||||
|
from collections.abc import Callable
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
|
||||||
|
results: dict[str, list[Any]] = {}
|
||||||
|
if not order_ids:
|
||||||
|
return results
|
||||||
|
|
||||||
|
def make(oid: str) -> Callable[[], None]:
|
||||||
|
def run() -> None:
|
||||||
|
results[oid] = fetch_goods_for_order(client, oid)
|
||||||
|
return run
|
||||||
|
|
||||||
|
with ThreadPoolExecutor(max_workers=min(GOODS_CONCURRENCY, len(order_ids))) as pool:
|
||||||
|
list(pool.map(lambda fn: fn(), [make(oid) for oid in order_ids]))
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- SDS 物流渠道字典 ----------
|
||||||
|
|
||||||
|
_channel_cache: dict[str, Any] = {}
|
||||||
|
_CHANNEL_TTL = 24 * 3600
|
||||||
|
_CHANNEL_KEY = "all"
|
||||||
|
|
||||||
|
|
||||||
|
def get_channel_type_map(client: SdsMngClient) -> dict[int, str | None]:
|
||||||
|
cached = _channel_cache.get(_CHANNEL_KEY)
|
||||||
|
if cached and cached["expire_at"] > time.time():
|
||||||
|
return cached["data"]
|
||||||
|
mapping: dict[int, str | None] = {}
|
||||||
|
try:
|
||||||
|
areas = client.get_logistics_areas()
|
||||||
|
area_ids = [a.id for a in areas.items if a.id is not None]
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
print(f"[SDS sync] 物流区域拉取失败: {err}")
|
||||||
|
area_ids = []
|
||||||
|
for area_id in area_ids:
|
||||||
|
try:
|
||||||
|
resp = client.get_logistics_page(LogisticsPageRequest(issuing_bay_area_id=area_id, all=True, size=200))
|
||||||
|
for item in resp.items:
|
||||||
|
if item.id is None:
|
||||||
|
continue
|
||||||
|
raw = getattr(item, "channelType", None) or getattr(item, "channel_type", None)
|
||||||
|
label = map_sds_channel_type(raw)
|
||||||
|
if label is None:
|
||||||
|
continue
|
||||||
|
mapping[int(item.id)] = label
|
||||||
|
except Exception as err: # noqa: BLE001
|
||||||
|
print(f"[SDS sync] 区域 {area_id} 物流字典拉取失败: {err}")
|
||||||
|
_channel_cache[_CHANNEL_KEY] = {"data": mapping, "expire_at": time.time() + _CHANNEL_TTL}
|
||||||
|
return mapping
|
||||||
|
|
||||||
|
|
||||||
|
def refresh_channel_type_map(client: SdsMngClient) -> dict[int, str | None]:
|
||||||
|
_channel_cache.pop(_CHANNEL_KEY, None)
|
||||||
|
return get_channel_type_map(client)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_courier_source(
|
||||||
|
raw_logistics_id: Any,
|
||||||
|
mapping: dict[int, str | None],
|
||||||
|
refresh: Callable[[], dict[int, str | None]],
|
||||||
|
refreshed_ids: set[int],
|
||||||
|
) -> str | None:
|
||||||
|
if raw_logistics_id is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
lid = int(raw_logistics_id)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
if lid in mapping:
|
||||||
|
return mapping[lid]
|
||||||
|
if lid in refreshed_ids:
|
||||||
|
return None
|
||||||
|
refreshed_ids.add(lid)
|
||||||
|
try:
|
||||||
|
new_map = refresh()
|
||||||
|
mapping.update(new_map)
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass
|
||||||
|
return mapping.get(lid)
|
||||||
@@ -0,0 +1,365 @@
|
|||||||
|
"""PG 同步写入层:Orders / Goods / Merchant 的 upsert + 同步所需的查询。
|
||||||
|
|
||||||
|
忠实移植自 inkreach-bi-auth-mng 的 sync/lib/db.service.ts。
|
||||||
|
- Orders 主键:("orderNo", "createdAt")
|
||||||
|
- Goods 主键:("orderNo", "productionNo")
|
||||||
|
关键:写入前用库中已有 createdAt 覆盖,避免同一订单因 createdAt 不同产生重复主键行。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from psycopg2.extras import execute_values
|
||||||
|
|
||||||
|
from flows.shared.mappers import GoodsUpsertInput, OrderUpsertInput
|
||||||
|
|
||||||
|
ORDER_CHUNK = 1000
|
||||||
|
GOODS_CHUNK = 2000
|
||||||
|
LOOKUP_BATCH = 10000
|
||||||
|
|
||||||
|
|
||||||
|
class SyncDB:
|
||||||
|
def __init__(self, block_name: str = "pg-inkreach-dmp") -> None:
|
||||||
|
from prefect_sqlalchemy import SqlAlchemyConnector
|
||||||
|
|
||||||
|
self._engine = SqlAlchemyConnector.load(block_name).get_engine()
|
||||||
|
|
||||||
|
def _conn(self):
|
||||||
|
return self._engine.raw_connection()
|
||||||
|
|
||||||
|
# ---------- 查询辅助 ----------
|
||||||
|
|
||||||
|
def get_latest_ordered_at(self, system_name: str) -> datetime | None:
|
||||||
|
with self._conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT MAX(COALESCE("orderedAt","createdAt")) FROM "Orders" WHERE "systemName"=%s',
|
||||||
|
(system_name,),
|
||||||
|
)
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row and row[0] else None
|
||||||
|
|
||||||
|
def get_pending_orders(self, system_name: str, exclude_statuses: list[str]) -> list[str]:
|
||||||
|
with self._conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT DISTINCT "orderNo" FROM "Orders" WHERE "systemName"=%s '
|
||||||
|
'AND "systemStatus" IS DISTINCT FROM ALL %s',
|
||||||
|
(system_name, exclude_statuses),
|
||||||
|
)
|
||||||
|
rows = cur.fetchall()
|
||||||
|
return [r[0] for r in rows]
|
||||||
|
|
||||||
|
def get_order_count_by_status(self, system_name: str, start: datetime, end: datetime) -> dict[str, int]:
|
||||||
|
with self._conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT "systemStatus", COUNT(*) FROM "Orders" WHERE "systemName"=%s '
|
||||||
|
'AND "orderedAt">=%s AND "orderedAt"<%s GROUP BY "systemStatus"',
|
||||||
|
(system_name, start, end),
|
||||||
|
)
|
||||||
|
rows = cur.fetchall()
|
||||||
|
return {r[0]: int(r[1]) for r in rows}
|
||||||
|
|
||||||
|
def get_local_order_nos(self, system_name: str, start: datetime, end: datetime) -> list[str]:
|
||||||
|
with self._conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT DISTINCT "orderNo" FROM "Orders" WHERE "systemName"=%s '
|
||||||
|
'AND "orderedAt">=%s AND "orderedAt"<%s',
|
||||||
|
(system_name, start, end),
|
||||||
|
)
|
||||||
|
rows = cur.fetchall()
|
||||||
|
return [r[0] for r in rows]
|
||||||
|
|
||||||
|
def get_local_goods_count(self, system_name: str, start: datetime, end: datetime) -> int:
|
||||||
|
with self._conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT COUNT(*) FROM "Goods" g JOIN "Orders" o '
|
||||||
|
'ON g."orderNo"=o."orderNo" AND g."createdAt"=o."createdAt" '
|
||||||
|
'WHERE o."systemName"=%s AND o."orderedAt">=%s AND o."orderedAt"<%s',
|
||||||
|
(system_name, start, end),
|
||||||
|
)
|
||||||
|
row = cur.fetchone()
|
||||||
|
return int(row[0]) if row else 0
|
||||||
|
|
||||||
|
def get_orders_with_100_prefix(self, system_name: str, statuses: list[str]) -> list[tuple[str, datetime]]:
|
||||||
|
with self._conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT DISTINCT o."orderNo", o."createdAt" FROM "Orders" o '
|
||||||
|
'JOIN "Goods" g ON g."orderNo"=o."orderNo" AND g."createdAt"=o."createdAt" '
|
||||||
|
'WHERE o."systemName"=%s AND o."systemStatus"=ANY(%s) AND g."productionNo" LIKE \'100%\'',
|
||||||
|
(system_name, statuses),
|
||||||
|
)
|
||||||
|
rows = cur.fetchall()
|
||||||
|
return [(r[0], r[1]) for r in rows]
|
||||||
|
|
||||||
|
# ---------- 主 upsert ----------
|
||||||
|
|
||||||
|
def sync_orders_and_goods(
|
||||||
|
self,
|
||||||
|
orders: list[OrderUpsertInput],
|
||||||
|
goods: list[GoodsUpsertInput],
|
||||||
|
delete_order_nos: list[str] | None = None,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
result = {
|
||||||
|
"ordersCreated": 0, "ordersUpdated": 0, "goodsCreated": 0,
|
||||||
|
"goodsUpdated": 0, "goodsRemoved": 0, "ordersDeleted": 0,
|
||||||
|
}
|
||||||
|
if not orders and not goods and not delete_order_nos:
|
||||||
|
return result
|
||||||
|
|
||||||
|
conn = self._conn()
|
||||||
|
try:
|
||||||
|
system_name = orders[0].systemName if orders else None
|
||||||
|
all_order_nos = list({o.orderNo for o in orders} | {g.orderNo for g in goods})
|
||||||
|
|
||||||
|
# 1. 解析已有 createdAt,覆盖到 orders(保证幂等 upsert)
|
||||||
|
existing_created: dict[str, datetime] = {}
|
||||||
|
for i in range(0, len(all_order_nos), LOOKUP_BATCH):
|
||||||
|
chunk = all_order_nos[i : i + LOOKUP_BATCH]
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT "orderNo","createdAt" FROM "Orders" WHERE "orderNo"=ANY(%s)'
|
||||||
|
+ (f' AND "systemName"=%s' if system_name else ""),
|
||||||
|
(chunk, system_name) if system_name else (chunk,),
|
||||||
|
)
|
||||||
|
for r in cur.fetchall():
|
||||||
|
existing_created[r[0]] = r[1]
|
||||||
|
for o in orders:
|
||||||
|
if o.orderNo in existing_created:
|
||||||
|
o.createdAt = existing_created[o.orderNo]
|
||||||
|
|
||||||
|
# 2. 去重 orders / goods
|
||||||
|
seen_o: set[str] = set()
|
||||||
|
orders_dedup: list[OrderUpsertInput] = []
|
||||||
|
for o in orders:
|
||||||
|
key = f'{o.orderNo}|{o.createdAt.isoformat()}'
|
||||||
|
if key in seen_o:
|
||||||
|
continue
|
||||||
|
seen_o.add(key)
|
||||||
|
orders_dedup.append(o)
|
||||||
|
|
||||||
|
resolved_created: dict[str, datetime] = {}
|
||||||
|
for o in orders_dedup:
|
||||||
|
resolved_created.setdefault(o.orderNo, o.createdAt)
|
||||||
|
|
||||||
|
seen_g: set[str] = set()
|
||||||
|
goods_dedup: list[GoodsUpsertInput] = []
|
||||||
|
for g in goods:
|
||||||
|
if g.orderNo in resolved_created:
|
||||||
|
g.createdAt = resolved_created[g.orderNo]
|
||||||
|
key = f'{g.orderNo}|{g.productionNo}'
|
||||||
|
if key in seen_g:
|
||||||
|
continue
|
||||||
|
seen_g.add(key)
|
||||||
|
goods_dedup.append(g)
|
||||||
|
|
||||||
|
goods_by_order: dict[str, list[GoodsUpsertInput]] = {}
|
||||||
|
for g in goods_dedup:
|
||||||
|
goods_by_order.setdefault(g.orderNo, []).append(g)
|
||||||
|
|
||||||
|
# 3. 分批 upsert orders + 关联 goods
|
||||||
|
for i in range(0, len(orders_dedup), ORDER_CHUNK):
|
||||||
|
chunk = orders_dedup[i : i + ORDER_CHUNK]
|
||||||
|
related: list[GoodsUpsertInput] = []
|
||||||
|
for o in chunk:
|
||||||
|
related.extend(goods_by_order.get(o.orderNo, []))
|
||||||
|
stats = self._upsert_order_batch(conn, chunk, related, system_name)
|
||||||
|
for k, v in stats.items():
|
||||||
|
result[k] += v
|
||||||
|
|
||||||
|
# 4. 剩余 goods(其 order 不在本次 orders 列表)
|
||||||
|
orders_set = {o.orderNo for o in orders_dedup}
|
||||||
|
remaining = [g for g in goods_dedup if g.orderNo not in orders_set]
|
||||||
|
if remaining:
|
||||||
|
c, u = self._upsert_goods_only(conn, remaining, system_name)
|
||||||
|
result["goodsCreated"] += c
|
||||||
|
result["goodsUpdated"] += u
|
||||||
|
|
||||||
|
# 5. 删除订单
|
||||||
|
if delete_order_nos:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'DELETE FROM "Orders" WHERE "orderNo"=ANY(%s)'
|
||||||
|
+ (f' AND "systemName"=%s' if system_name else ""),
|
||||||
|
(delete_order_nos, system_name) if system_name else (delete_order_nos,),
|
||||||
|
)
|
||||||
|
result["ordersDeleted"] = cur.rowcount
|
||||||
|
|
||||||
|
# 6. 商家占位
|
||||||
|
self._ensure_merchants(conn, orders_dedup)
|
||||||
|
conn.commit()
|
||||||
|
except Exception:
|
||||||
|
conn.rollback()
|
||||||
|
raise
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
return result
|
||||||
|
|
||||||
|
def _upsert_order_batch(
|
||||||
|
self, conn, orders: list[OrderUpsertInput], goods: list[GoodsUpsertInput], system_name: str | None
|
||||||
|
) -> dict[str, int]:
|
||||||
|
stats = {"ordersCreated": 0, "ordersUpdated": 0, "goodsCreated": 0, "goodsUpdated": 0, "goodsRemoved": 0}
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
# 统计已存在 → created/updated
|
||||||
|
pairs = [(o.orderNo, o.createdAt) for o in orders]
|
||||||
|
cur.execute(
|
||||||
|
"SELECT t.order_no, t.created_at FROM unnest(%s::text[], %s::timestamp[]) AS t(order_no,created_at) "
|
||||||
|
"JOIN \"Orders\" o ON o.\"orderNo\"=t.order_no AND o.\"createdAt\"=t.created_at",
|
||||||
|
_unnest_pairs(pairs),
|
||||||
|
)
|
||||||
|
existing_set = {(r[0], r[1]) for r in cur.fetchall()}
|
||||||
|
updated = sum(1 for o in orders if (o.orderNo, o.createdAt) in existing_set)
|
||||||
|
stats["ordersUpdated"] = updated
|
||||||
|
stats["ordersCreated"] = len(orders) - updated
|
||||||
|
|
||||||
|
execute_values(
|
||||||
|
cur,
|
||||||
|
'''INSERT INTO "Orders" (
|
||||||
|
"orderNo","outOrderNo","courierSource","courierName","courierNo","merchantId",
|
||||||
|
"countryRegion","factoryName","systemName","systemStatus","platformName","platformStatus",
|
||||||
|
"createdAt","orderedAt","confirmedAt","completedAt","shipDeadline"
|
||||||
|
) VALUES %s
|
||||||
|
ON CONFLICT ("orderNo","createdAt") DO UPDATE SET
|
||||||
|
"outOrderNo"=EXCLUDED."outOrderNo","courierSource"=EXCLUDED."courierSource",
|
||||||
|
"courierName"=EXCLUDED."courierName","courierNo"=EXCLUDED."courierNo",
|
||||||
|
"merchantId"=EXCLUDED."merchantId","countryRegion"=EXCLUDED."countryRegion",
|
||||||
|
"factoryName"=EXCLUDED."factoryName","systemName"=EXCLUDED."systemName",
|
||||||
|
"systemStatus"=EXCLUDED."systemStatus","platformName"=EXCLUDED."platformName",
|
||||||
|
"platformStatus"=EXCLUDED."platformStatus","orderedAt"=EXCLUDED."orderedAt",
|
||||||
|
"confirmedAt"=EXCLUDED."confirmedAt","completedAt"=EXCLUDED."completedAt",
|
||||||
|
"shipDeadline"=EXCLUDED."shipDeadline"''',
|
||||||
|
[(
|
||||||
|
o.orderNo, o.outOrderNo, o.courierSource, o.courierName, o.courierNo, o.merchantId,
|
||||||
|
o.countryRegion, o.factoryName, o.systemName, o.systemStatus, o.platformName, o.platformStatus,
|
||||||
|
o.createdAt, o.orderedAt, o.confirmedAt, o.completedAt, o.shipDeadline,
|
||||||
|
) for o in orders],
|
||||||
|
page_size=ORDER_CHUNK,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 关联 goods:先删本次不再出现的,再 upsert
|
||||||
|
if goods:
|
||||||
|
order_nos = list({g.orderNo for g in goods})
|
||||||
|
prod_nos = [g.productionNo for g in goods]
|
||||||
|
cur.execute(
|
||||||
|
'''DELETE FROM "Goods" WHERE "orderNo"=ANY(%s) AND "productionNo" <> ALL(%s)
|
||||||
|
AND ("orderNo","createdAt") IN (
|
||||||
|
SELECT "orderNo","createdAt" FROM "Orders"
|
||||||
|
WHERE "orderNo"=ANY(%s)''' + (f' AND "systemName"=%s' if system_name else "") + ')',
|
||||||
|
(order_nos, prod_nos, order_nos, system_name) if system_name else (order_nos, prod_nos, order_nos),
|
||||||
|
)
|
||||||
|
stats["goodsRemoved"] += cur.rowcount
|
||||||
|
|
||||||
|
g_pairs = [(g.orderNo, g.productionNo) for g in goods]
|
||||||
|
cur.execute(
|
||||||
|
"SELECT t.order_no,t.production_no FROM unnest(%s::text[], %s::text[]) AS t(order_no,production_no) "
|
||||||
|
"JOIN \"Goods\" g ON g.\"orderNo\"=t.order_no AND g.\"productionNo\"=t.production_no",
|
||||||
|
_unnest_pairs(g_pairs),
|
||||||
|
)
|
||||||
|
g_existing = {(r[0], r[1]) for r in cur.fetchall()}
|
||||||
|
g_updated = sum(1 for g in goods if (g.orderNo, g.productionNo) in g_existing)
|
||||||
|
stats["goodsUpdated"] = g_updated
|
||||||
|
stats["goodsCreated"] = len(goods) - g_updated
|
||||||
|
|
||||||
|
execute_values(
|
||||||
|
cur,
|
||||||
|
'''INSERT INTO "Goods" (
|
||||||
|
"orderNo","productionNo","createdAt","outOrderNo","itemName","itemColor",
|
||||||
|
"itemSize","itemStyleNo","quantity","status"
|
||||||
|
) VALUES %s
|
||||||
|
ON CONFLICT ("orderNo","productionNo") DO UPDATE SET
|
||||||
|
"createdAt"=EXCLUDED."createdAt","outOrderNo"=EXCLUDED."outOrderNo",
|
||||||
|
"itemName"=EXCLUDED."itemName","itemColor"=EXCLUDED."itemColor",
|
||||||
|
"itemSize"=EXCLUDED."itemSize","itemStyleNo"=EXCLUDED."itemStyleNo",
|
||||||
|
"quantity"=EXCLUDED."quantity","status"=EXCLUDED."status"''',
|
||||||
|
[(
|
||||||
|
g.orderNo, g.productionNo, g.createdAt, g.outOrderNo, g.itemName, g.itemColor,
|
||||||
|
g.itemSize, g.itemStyleNo, g.quantity, g.status,
|
||||||
|
) for g in goods],
|
||||||
|
page_size=GOODS_CHUNK,
|
||||||
|
)
|
||||||
|
return stats
|
||||||
|
|
||||||
|
def _upsert_goods_only(
|
||||||
|
self, conn, goods: list[GoodsUpsertInput], system_name: str | None
|
||||||
|
) -> tuple[int, int]:
|
||||||
|
# 用库中实际 createdAt 覆盖
|
||||||
|
order_nos = list({g.orderNo for g in goods})
|
||||||
|
actual: dict[str, datetime] = {}
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT "orderNo","createdAt" FROM "Orders" WHERE "orderNo"=ANY(%s)'
|
||||||
|
+ (f' AND "systemName"=%s' if system_name else ""),
|
||||||
|
(order_nos, system_name) if system_name else (order_nos,),
|
||||||
|
)
|
||||||
|
for r in cur.fetchall():
|
||||||
|
actual[r[0]] = r[1]
|
||||||
|
for g in goods:
|
||||||
|
if g.orderNo in actual:
|
||||||
|
g.createdAt = actual[g.orderNo]
|
||||||
|
|
||||||
|
created = updated = 0
|
||||||
|
for i in range(0, len(goods), GOODS_CHUNK):
|
||||||
|
chunk = goods[i : i + GOODS_CHUNK]
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
c_order = list({g.orderNo for g in chunk})
|
||||||
|
c_prod = [g.productionNo for g in chunk]
|
||||||
|
cur.execute(
|
||||||
|
'''DELETE FROM "Goods" WHERE "orderNo"=ANY(%s) AND "productionNo" <> ALL(%s)'''
|
||||||
|
+ (f''' AND ("orderNo","createdAt") IN (SELECT "orderNo","createdAt" FROM "Orders"
|
||||||
|
WHERE "orderNo"=ANY(%s) AND "systemName"=%s)''' if system_name else ""),
|
||||||
|
(c_order, c_prod, c_order, system_name) if system_name else (c_order, c_prod),
|
||||||
|
)
|
||||||
|
g_pairs = [(g.orderNo, g.productionNo) for g in chunk]
|
||||||
|
cur.execute(
|
||||||
|
"SELECT t.order_no,t.production_no FROM unnest(%s::text[], %s::text[]) AS t(order_no,production_no) "
|
||||||
|
"JOIN \"Goods\" g ON g.\"orderNo\"=t.order_no AND g.\"productionNo\"=t.production_no",
|
||||||
|
_unnest_pairs(g_pairs),
|
||||||
|
)
|
||||||
|
g_existing = {(r[0], r[1]) for r in cur.fetchall()}
|
||||||
|
u = sum(1 for g in chunk if (g.orderNo, g.productionNo) in g_existing)
|
||||||
|
updated += u
|
||||||
|
created += len(chunk) - u
|
||||||
|
execute_values(
|
||||||
|
cur,
|
||||||
|
'''INSERT INTO "Goods" (
|
||||||
|
"orderNo","productionNo","createdAt","outOrderNo","itemName","itemColor",
|
||||||
|
"itemSize","itemStyleNo","quantity","status"
|
||||||
|
) VALUES %s
|
||||||
|
ON CONFLICT ("orderNo","productionNo") DO UPDATE SET
|
||||||
|
"createdAt"=EXCLUDED."createdAt","outOrderNo"=EXCLUDED."outOrderNo",
|
||||||
|
"itemName"=EXCLUDED."itemName","itemColor"=EXCLUDED."itemColor",
|
||||||
|
"itemSize"=EXCLUDED."itemSize","itemStyleNo"=EXCLUDED."itemStyleNo",
|
||||||
|
"quantity"=EXCLUDED."quantity","status"=EXCLUDED."status"''',
|
||||||
|
[(
|
||||||
|
g.orderNo, g.productionNo, g.createdAt, g.outOrderNo, g.itemName, g.itemColor,
|
||||||
|
g.itemSize, g.itemStyleNo, g.quantity, g.status,
|
||||||
|
) for g in chunk],
|
||||||
|
page_size=GOODS_CHUNK,
|
||||||
|
)
|
||||||
|
return created, updated
|
||||||
|
|
||||||
|
def _ensure_merchants(self, conn, orders: list[OrderUpsertInput]) -> None:
|
||||||
|
ids = list({o.merchantId for o in orders if o.merchantId and o.merchantId.strip()})
|
||||||
|
if not ids:
|
||||||
|
return
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
execute_values(
|
||||||
|
cur,
|
||||||
|
'''INSERT INTO "Merchant" ("merchantId","merchantGroup","responsibleName","responsibleId","tags")
|
||||||
|
VALUES %s ON CONFLICT ("merchantId") DO NOTHING''',
|
||||||
|
[(mid, "", None, None, "") for mid in ids],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _unnest_pairs(pairs: list[tuple]) -> tuple[list, list]:
|
||||||
|
"""构造 psycopg2 unnest 双数组参数:返回 (col_a_list, col_b_list)。
|
||||||
|
|
||||||
|
配合 SQL:unnest(%s::text[], %s::timestamp[]) 或 unnest(%s::text[], %s::text[])。
|
||||||
|
"""
|
||||||
|
return [p[0] for p in pairs], [p[1] for p in pairs]
|
||||||
@@ -0,0 +1,317 @@
|
|||||||
|
"""数据映射:RIIN / SDS 原始订单 → 统一 Order/Goods 写入格式。
|
||||||
|
|
||||||
|
忠实移植自 inkreach-bi-auth-mng 的 sync/mappers/{riin,sds,normalize-country}.ts。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
TZ_CN = timezone(timedelta(hours=8))
|
||||||
|
|
||||||
|
# ---------- RIIN 状态 / 平台 / 快递来源映射 ----------
|
||||||
|
|
||||||
|
RIIN_SYSTEM_STATUS_MAP: dict[int, str] = {
|
||||||
|
-4: "待推送", -3: "工厂反审", -5: "待审核", -6: "分配失败",
|
||||||
|
1: "待推送", 3: "已反审", 4: "生产中", 5: "生产中",
|
||||||
|
12: "已发货", 13: "已关闭", 14: "已取消", 15: "已关闭",
|
||||||
|
}
|
||||||
|
|
||||||
|
RIIN_PLATFORM_TYPE_MAP: dict[int, str] = {
|
||||||
|
2: "淘宝", 3: "拼多多", 6: "Shopee", 7: "LAZADA", 8: "1688",
|
||||||
|
11: "TIKTOK", 13: "TEMU", 15: "自建站(海外)", 16: "SHOPIFY",
|
||||||
|
17: "SHEIN", 18: "自建站(国内)", 19: "WALMART", 21: "AMAZON",
|
||||||
|
26: "OZON", 27: "Wildberries", 30: "ETSY", 31: "MERCADO",
|
||||||
|
}
|
||||||
|
|
||||||
|
RIIN_EXPRESS_SOURCE_MAP: dict[int, str] = {
|
||||||
|
1: "工厂发货", 2: "平台面单", 3: "自有面单", 4: "分销面单", 5: "电商面单",
|
||||||
|
}
|
||||||
|
|
||||||
|
RIIN_GOODS_STATUS_MAP: dict[str, str] = {
|
||||||
|
"NOT_SHIPPED": "待发货", "SHIPPED": "已发货", "CANCEL": "已取消",
|
||||||
|
"CANCELLED": "已取消", "REFUNDING": "退款中",
|
||||||
|
}
|
||||||
|
|
||||||
|
RIIN_PLATFORM_STATUS_MAP: dict[str, str] = {
|
||||||
|
"SHIPPED": "已发货", "NOT_SHIPPED": "待发货", "CANCEL": "已取消",
|
||||||
|
"PAUSE": "暂停", "COMPLETE": "已完成",
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- SDS 状态 / 物流渠道映射 ----------
|
||||||
|
|
||||||
|
SDS_SYSTEM_STATUS_MAP: dict[int, str] = {
|
||||||
|
1: "待付款", 2: "待确认", 3: "备货中", 4: "已完成", 5: "搁置中",
|
||||||
|
7: "待设计", 9: "部分发货", 98: "已取消",
|
||||||
|
}
|
||||||
|
SDS_GOODS_STATUS_MAP: dict[int, str] = {**SDS_SYSTEM_STATUS_MAP, 99: "已删除"}
|
||||||
|
|
||||||
|
SDS_CHANNEL_TYPE_LABEL_MAP: dict[str, str] = {
|
||||||
|
"online": "线上物流", "offline": "普通物流", "manual": "手动物流", "self_pickup": "自提",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 日期解析 ----------
|
||||||
|
|
||||||
|
def safe_date(value: Any, context: str = "") -> datetime:
|
||||||
|
"""解析日期字符串(无时区视为 +08:00),返回 tz-aware datetime。"""
|
||||||
|
if not value or not isinstance(value, str):
|
||||||
|
raise ValueError(f"Invalid date for {context}: {value!r}")
|
||||||
|
s = value.strip()
|
||||||
|
if "+" not in s and "T" not in s and not s.endswith("Z"):
|
||||||
|
s = f"{s}T00:00:00+08:00" if len(s) == 10 else f"{s}+08:00"
|
||||||
|
try:
|
||||||
|
dt = datetime.fromisoformat(s)
|
||||||
|
except ValueError:
|
||||||
|
raise ValueError(f"Invalid date for {context}: {value!r}")
|
||||||
|
return dt if dt.tzinfo else dt.replace(tzinfo=TZ_CN)
|
||||||
|
|
||||||
|
|
||||||
|
def _from_epoch_ms(value: Any) -> datetime | None:
|
||||||
|
try:
|
||||||
|
n = int(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
return datetime.fromtimestamp(n / 1000, tz=TZ_CN) if n > 0 else None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 国家/地区归一 ----------
|
||||||
|
|
||||||
|
_US_ALIASES = {"US", "UNITED STATES", "USA", "ESTADOS UNIDOS", "美国加州发货仓"}
|
||||||
|
_MX_ALIASES = {"MX", "MEXICO", "MÉXICO", "MEX", "ME"}
|
||||||
|
_CA_ALIASES = {"加拿大工厂", "CA"}
|
||||||
|
_PL_ALIASES = {"欧州波兰发货仓", "欧洲波兰法兰克福发货仓"}
|
||||||
|
_WAREHOUSE_MAP = {
|
||||||
|
"印美达澳洲发货仓": "澳洲", "欧洲德国发货仓": "德国", "欧洲匈牙利发货仓": "匈牙利",
|
||||||
|
"欧洲西班牙发货仓": "西班牙", "欧洲意大利发货仓": "意大利",
|
||||||
|
}
|
||||||
|
_DIRECT_MAP = {"Saudi Arabia": "沙特", "Japan": "日本"}
|
||||||
|
_EXCLUDED_COUNTRIES = {
|
||||||
|
"Croatia", "Czech Republic", "丹麦", "保加利亚", "克罗地亚", "卢森堡", "奥地利", "希腊",
|
||||||
|
"拉脱维亚", "捷克", "斯洛伐克", "斯洛文尼亚", "比利时", "法国", "爱尔兰", "爱沙尼亚",
|
||||||
|
"瑞典", "瑞士", "立陶宛", "罗马尼亚", "芬兰", "荷兰", "Denmark", "Austria", "Greece",
|
||||||
|
"Latvia", "Slovakia", "Slovenia", "Belgium", "France", "Ireland", "Estonia", "Sweden",
|
||||||
|
"Switzerland", "Lithuania", "Romania", "Finland", "Netherlands", "Luxembourg", "Bulgaria",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_country_region(raw: str | None, factory_name: str | None = None) -> str | None:
|
||||||
|
if factory_name == "广州印美达":
|
||||||
|
return "中国"
|
||||||
|
if not raw:
|
||||||
|
return None
|
||||||
|
trimmed = raw.strip()
|
||||||
|
if trimmed in _EXCLUDED_COUNTRIES:
|
||||||
|
return None
|
||||||
|
upper = trimmed.upper()
|
||||||
|
if upper in _US_ALIASES:
|
||||||
|
return "美国"
|
||||||
|
if upper in _MX_ALIASES:
|
||||||
|
return "墨西哥"
|
||||||
|
if upper in _CA_ALIASES:
|
||||||
|
return "加拿大"
|
||||||
|
if trimmed in _PL_ALIASES:
|
||||||
|
return "波兰"
|
||||||
|
if trimmed in _WAREHOUSE_MAP:
|
||||||
|
return _WAREHOUSE_MAP[trimmed]
|
||||||
|
if raw in _DIRECT_MAP:
|
||||||
|
return _DIRECT_MAP[raw]
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 统一写入模型 ----------
|
||||||
|
|
||||||
|
class OrderUpsertInput:
|
||||||
|
__slots__ = (
|
||||||
|
"orderNo", "outOrderNo", "courierSource", "courierName", "courierNo",
|
||||||
|
"merchantId", "countryRegion", "factoryName", "systemName", "systemStatus",
|
||||||
|
"platformName", "platformStatus", "createdAt", "orderedAt", "confirmedAt",
|
||||||
|
"completedAt", "shipDeadline",
|
||||||
|
)
|
||||||
|
|
||||||
|
def __init__(self, **kwargs: Any) -> None:
|
||||||
|
for k in self.__slots__:
|
||||||
|
setattr(self, k, kwargs.get(k))
|
||||||
|
|
||||||
|
|
||||||
|
class GoodsUpsertInput:
|
||||||
|
__slots__ = (
|
||||||
|
"orderNo", "productionNo", "createdAt", "outOrderNo", "itemName",
|
||||||
|
"itemColor", "itemSize", "itemStyleNo", "quantity", "status",
|
||||||
|
)
|
||||||
|
|
||||||
|
def __init__(self, **kwargs: Any) -> None:
|
||||||
|
for k in self.__slots__:
|
||||||
|
setattr(self, k, kwargs.get(k))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- RIIN 映射 ----------
|
||||||
|
|
||||||
|
def map_riin_system_status(order_status: int) -> str | None:
|
||||||
|
return RIIN_SYSTEM_STATUS_MAP.get(order_status)
|
||||||
|
|
||||||
|
|
||||||
|
def map_riin_order(raw: Any) -> OrderUpsertInput:
|
||||||
|
platform_oid = raw.platform_oid
|
||||||
|
ordered_at = _parse(raw.push_time, f"RIIN order {platform_oid} pushTime")
|
||||||
|
confirmed_at = _parse(raw.distributor_push_time, f"RIIN order {platform_oid} distributorPushTime")
|
||||||
|
system_status = map_riin_system_status(raw.order_status) if raw.order_status is not None else None
|
||||||
|
is_terminal = system_status in ("已发货", "已关闭", "已取消")
|
||||||
|
completed_at: datetime | None = None
|
||||||
|
if is_terminal:
|
||||||
|
completed_at = _parse(raw.shipping_time, "") or _parse(raw.finish_time, "")
|
||||||
|
ship_deadline = _parse(raw.except_last_shipping_time, "")
|
||||||
|
courier_source = RIIN_EXPRESS_SOURCE_MAP.get(raw.express_source) if raw.express_source is not None else None
|
||||||
|
factory_name = raw.distributor_push_target_name
|
||||||
|
|
||||||
|
return OrderUpsertInput(
|
||||||
|
orderNo=platform_oid,
|
||||||
|
outOrderNo=raw.source_platform_oid,
|
||||||
|
courierSource=courier_source,
|
||||||
|
courierName=raw.delivery_courier,
|
||||||
|
courierNo=raw.courier_number,
|
||||||
|
merchantId=raw.sub_name,
|
||||||
|
countryRegion=normalize_country_region(raw.receiver_country, factory_name),
|
||||||
|
factoryName=factory_name,
|
||||||
|
systemName="RIIN",
|
||||||
|
systemStatus=system_status,
|
||||||
|
platformName=RIIN_PLATFORM_TYPE_MAP.get(raw.platform_type, str(raw.platform_type)) if raw.platform_type is not None else None,
|
||||||
|
platformStatus=RIIN_PLATFORM_STATUS_MAP.get(raw.platform_order_status, raw.platform_order_status) if raw.platform_order_status else None,
|
||||||
|
createdAt=safe_date(raw.order_create_time, f"RIIN order {platform_oid} orderCreateTime"),
|
||||||
|
orderedAt=ordered_at,
|
||||||
|
confirmedAt=confirmed_at,
|
||||||
|
completedAt=completed_at,
|
||||||
|
shipDeadline=ship_deadline,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def map_riin_goods(raw: Any, parent_order_no: str, parent_created_at: datetime) -> GoodsUpsertInput:
|
||||||
|
product_code = raw.product_code
|
||||||
|
item_name = product_code.split("-")[0] if product_code else None
|
||||||
|
return GoodsUpsertInput(
|
||||||
|
orderNo=parent_order_no,
|
||||||
|
productionNo=raw.id,
|
||||||
|
createdAt=parent_created_at,
|
||||||
|
outOrderNo=raw.platform_oid,
|
||||||
|
itemName=item_name,
|
||||||
|
itemColor=raw.color_code,
|
||||||
|
itemSize=raw.size_code,
|
||||||
|
itemStyleNo=item_name,
|
||||||
|
quantity=raw.num if raw.num is not None else 1,
|
||||||
|
status=RIIN_GOODS_STATUS_MAP.get(raw.goods_status) if raw.goods_status else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- SDS 映射 ----------
|
||||||
|
|
||||||
|
def map_sds_channel_type(channel_type: Any) -> str | None:
|
||||||
|
if not isinstance(channel_type, str):
|
||||||
|
return None
|
||||||
|
return SDS_CHANNEL_TYPE_LABEL_MAP.get(channel_type)
|
||||||
|
|
||||||
|
|
||||||
|
def map_sds_system_status(status: int) -> str | None:
|
||||||
|
return SDS_SYSTEM_STATUS_MAP.get(status)
|
||||||
|
|
||||||
|
|
||||||
|
def map_sds_order(raw: Any, channel_type_map: dict[int, str | None], pre_courier: str | None = None) -> OrderUpsertInput:
|
||||||
|
ordered_at = _parse(raw.gmt_pay, f"SDS order {raw.no} gmtPay")
|
||||||
|
system_status = map_sds_system_status(raw.status) if raw.status is not None else None
|
||||||
|
is_terminal = system_status in ("已完成", "已取消")
|
||||||
|
completed_at: datetime | None = None
|
||||||
|
if is_terminal and raw.finish_time and raw.finish_time > 0:
|
||||||
|
completed_at = _from_epoch_ms(raw.finish_time)
|
||||||
|
ship_deadline = _from_epoch_ms(raw.latest_ship_time) if raw.latest_ship_time else None
|
||||||
|
|
||||||
|
courier_source = pre_courier
|
||||||
|
if courier_source is None and raw.logistics_id is not None:
|
||||||
|
try:
|
||||||
|
lid = int(raw.logistics_id)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
lid = None
|
||||||
|
if lid is not None and lid in channel_type_map:
|
||||||
|
courier_source = channel_type_map[lid]
|
||||||
|
|
||||||
|
courier_no: str | None = None
|
||||||
|
if raw.parcel_list:
|
||||||
|
first = raw.parcel_list[0]
|
||||||
|
try:
|
||||||
|
courier_no = first["logistics"]["carriage"]["no"]
|
||||||
|
except (TypeError, KeyError, IndexError):
|
||||||
|
courier_no = None
|
||||||
|
|
||||||
|
factory_name: str | None = None
|
||||||
|
for item in raw.items or []:
|
||||||
|
fname = getattr(getattr(item, "factory", None), "name", None)
|
||||||
|
if not fname and isinstance(item, dict):
|
||||||
|
fname = (item.get("factory") or {}).get("name")
|
||||||
|
if fname:
|
||||||
|
factory_name = fname
|
||||||
|
break
|
||||||
|
|
||||||
|
area_name = raw.issuing_bay_area.name if raw.issuing_bay_area else None
|
||||||
|
return OrderUpsertInput(
|
||||||
|
orderNo=raw.no,
|
||||||
|
outOrderNo=raw.out_order_no,
|
||||||
|
courierSource=courier_source,
|
||||||
|
courierName=raw.carriage_name,
|
||||||
|
courierNo=courier_no,
|
||||||
|
merchantId=raw.merchant.name if raw.merchant else None,
|
||||||
|
countryRegion=normalize_country_region(area_name, factory_name),
|
||||||
|
factoryName=factory_name,
|
||||||
|
systemName="SDS",
|
||||||
|
systemStatus=system_status,
|
||||||
|
platformName=raw.merchant_store.platform_name if raw.merchant_store else None,
|
||||||
|
platformStatus=raw.online_order_status_name,
|
||||||
|
createdAt=safe_date(raw.gmt_created, f"SDS order {raw.no}"),
|
||||||
|
orderedAt=ordered_at,
|
||||||
|
confirmedAt=None,
|
||||||
|
completedAt=completed_at,
|
||||||
|
shipDeadline=ship_deadline,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_sds_style_no(name: str | None) -> str | None:
|
||||||
|
if not name:
|
||||||
|
return None
|
||||||
|
parts = name.split("()")
|
||||||
|
if len(parts) < 3:
|
||||||
|
return None
|
||||||
|
subs = parts[2].split("-")
|
||||||
|
if len(subs) == 2:
|
||||||
|
return subs[0] or None
|
||||||
|
if len(subs) >= 3:
|
||||||
|
return subs[1] or None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def map_sds_goods(raw: Any, parent_order_no: str, parent_created_at: datetime) -> GoodsUpsertInput:
|
||||||
|
product_name = raw.product_name
|
||||||
|
color_name = None
|
||||||
|
block = raw.product_color_block
|
||||||
|
if block is not None:
|
||||||
|
color_name = getattr(block, "color_name", None) or (block.get("colorName") if isinstance(block, dict) else None)
|
||||||
|
return GoodsUpsertInput(
|
||||||
|
orderNo=parent_order_no,
|
||||||
|
productionNo=raw.no or raw.id,
|
||||||
|
createdAt=parent_created_at,
|
||||||
|
outOrderNo=raw.out_order_item_id,
|
||||||
|
itemName=product_name,
|
||||||
|
itemColor=color_name,
|
||||||
|
itemSize=raw.product_size,
|
||||||
|
itemStyleNo=_parse_sds_style_no(product_name),
|
||||||
|
quantity=raw.num if raw.num is not None else 1,
|
||||||
|
status=SDS_GOODS_STATUS_MAP.get(raw.status) if raw.status is not None else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse(value: Any, context: str) -> datetime | None:
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
if isinstance(value, (int, float)):
|
||||||
|
return _from_epoch_ms(value)
|
||||||
|
try:
|
||||||
|
return safe_date(value, context)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://opencode.ai/config.json",
|
||||||
|
"mcp": {
|
||||||
|
"prefect": {
|
||||||
|
"type": "local",
|
||||||
|
"command": [
|
||||||
|
"/root/.local/bin/uvx",
|
||||||
|
"--from",
|
||||||
|
"prefect-mcp",
|
||||||
|
"prefect-mcp-server"
|
||||||
|
],
|
||||||
|
"enabled": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
[project]
|
||||||
|
name = "inkreach-prefect"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "ETL workflows orchestrated by Prefect"
|
||||||
|
requires-python = ">=3.12"
|
||||||
|
dependencies = [
|
||||||
|
"prefect>=3.4",
|
||||||
|
"prefect-sqlalchemy",
|
||||||
|
"psycopg2-binary",
|
||||||
|
"httpx",
|
||||||
|
"pandas",
|
||||||
|
"pydantic>=2.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[build-system]
|
||||||
|
requires = ["hatchling"]
|
||||||
|
build-backend = "hatchling.build"
|
||||||
|
|
||||||
|
[tool.hatch.build.targets.wheel]
|
||||||
|
packages = ["api"]
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
"""一键部署全部 12 个同步 flow(单进程注册所有 deployment)。
|
||||||
|
|
||||||
|
运行后,所有 flow + deployment 立即出现在 Prefect UI,并按各自 cron 调度执行:
|
||||||
|
uv run python serve_all.py
|
||||||
|
|
||||||
|
不想立即跑调度?部署后在 UI 的 Deployments 页面暂停(pause)对应 deployment 即可,
|
||||||
|
flow 仍会显示在 Flows 列表中。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from prefect import serve
|
||||||
|
|
||||||
|
from flows.riin.by_orders import riin_by_orders_sync
|
||||||
|
from flows.riin.incremental_sync import riin_incremental_sync
|
||||||
|
from flows.riin.order_sync import riin_order_sync
|
||||||
|
from flows.riin.recent_sync import riin_recent_sync
|
||||||
|
from flows.riin.refresh_pending import riin_refresh_pending
|
||||||
|
from flows.riin.verify import riin_verify
|
||||||
|
from flows.sds.incremental_sync import sds_incremental_sync
|
||||||
|
from flows.sds.order_sync import sds_order_sync
|
||||||
|
from flows.sds.recent_sync import sds_recent_sync
|
||||||
|
from flows.sds.refresh_100_prefix import sds_refresh_100_prefix
|
||||||
|
from flows.sds.refresh_pending import sds_refresh_pending
|
||||||
|
from flows.sds.verify import sds_verify
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
serve(
|
||||||
|
riin_order_sync.to_deployment(name="riin-order-sync"),
|
||||||
|
riin_incremental_sync.to_deployment(name="riin-incremental-sync", cron="0 * * * *"),
|
||||||
|
riin_recent_sync.to_deployment(name="riin-recent-sync", cron="30 * * * *"),
|
||||||
|
riin_refresh_pending.to_deployment(name="riin-refresh-pending", cron="0 */3 * * *"),
|
||||||
|
riin_by_orders_sync.to_deployment(name="riin-by-orders-sync"),
|
||||||
|
sds_order_sync.to_deployment(name="sds-order-sync"),
|
||||||
|
sds_incremental_sync.to_deployment(name="sds-incremental-sync", cron="15 * * * *"),
|
||||||
|
sds_recent_sync.to_deployment(name="sds-recent-sync", cron="45 * * * *"),
|
||||||
|
sds_refresh_pending.to_deployment(name="sds-refresh-pending", cron="0 */2 * * *"),
|
||||||
|
sds_refresh_100_prefix.to_deployment(name="sds-refresh-100-prefix", cron="30 */2 * * *"),
|
||||||
|
riin_verify.to_deployment(name="riin-verify", cron="0 6 * * *"),
|
||||||
|
sds_verify.to_deployment(name="sds-verify", cron="0 7 * * *"),
|
||||||
|
)
|
||||||
Reference in New Issue
Block a user