chore: init bj_power monorepo (dashboard/mes/wms/wms_client/workstation), clear subproject git metadata and align naming to folder names

This commit is contained in:
SunYF
2026-08-27 10:09:54 +08:00
commit 1f0ee844a4
408 changed files with 102201 additions and 0 deletions
@@ -0,0 +1,746 @@
# Redis SSOT + 事件重放 + 三层调度器 Phase 2 实施计划
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** 将 Redis SSOT、事件重放、三层调度器从当前"已实现但未接通"状态推进到设计文档要求的完整落地——Redis 成为运行时唯一真相源,调度器通过 ZPOPMAX 事件驱动,断电恢复可安全重放。
**Architecture:** 分三步走:(1) 修复关键断路,让 RedisBus/StateManager 完整运行;(2) 建立运行时写穿通道,Processor 事件处理同步更新 Redis(3) 接入 ZPOPMAX 调度循环和原子抢占 Lua 脚本。所有变更加 SSOT 开关隔离,不破坏现有执行内核。
**Tech Stack:** Go 1.23.4, go-zero, ent v0.14.5, Redis (go-redis/v9), PostgreSQL, testify
---
## 文件结构
### 必改文件
- Modify: `internal/svc/service_context.go` — 调用 RedisBus.Start()、注入 RecoveryCoordinator
- Modify: `internal/state/manager.go` — 补全 GetTaskState 接口方法
- Modify: `internal/state/redis_manager.go` — 修 ApplyJobTransitionScript 版本号、设备心跳
- Modify: `internal/state/memory_manager.go` — 完整实现 ApplyTaskState/GetTaskState
- Modify: `internal/state/keys.go` — 新增 sched:ready_jobs、equipment heartbeat key
- Modify: `internal/eventbus/bus.go` — Bus 接口增加 Start/Stop 方法(或保障 RedisBus 启动)
- Modify: `internal/processor/job_processor.go` — 写穿 StateManager、接入 sched:ready_jobs
- Modify: `internal/processor/dispatcher.go` — Task 超时检测
- Modify: `internal/processor/scheduler_adapter.go` — ZPOPMAX 调度循环适配
- Modify: `internal/scheduler/scheduler.go` — 接收 ZPOPMAX 单工件输入
- Modify: `internal/recovery/replayer.go` — 增量重放、ReplaySince
- Modify: `internal/recovery/assessor.go` — 接入 restoreActiveOrders
- Modify: `internal/robot/common.go` — 心跳信号
### 新建文件
- Create: `internal/processor/recovery_coordinator.go` — RecoveryCoordinator 实现
- Create: `internal/state/state_write_through.go` — 写穿辅助函数(Processor→Redis
- Create: `internal/scheduler/zpop.go` — ZPOPMAX 调度循环
- Create: `internal/state/redis_manager_test.go` — RedisStateManager 集成测试
- Create: `internal/processor/ssot_integration_test.go` — SSOT 写穿集成测试
### 现有测试要改
- Modify: `internal/state/manager_test.go` — 新增 GetTaskState 测试
- Modify: `internal/scheduler/scheduler_test.go` — 新增 ZPOPMAX 输入路径测试
---
## Chunk 1: 修复关键断路
### Task 1: 调用 RedisBus.Start() + Bus 接口补全
**Files:**
- Modify: `internal/eventbus/bus.go`
- Modify: `internal/svc/service_context.go`
**Depends on:**
- [ ] **Step 1: Bus 接口增加 Start/Stop 方法**
`bus.go``Bus` interface 追加:
```go
type Bus interface {
Publish(ctx context.Context, event Event) error
Subscribe(eventType EventType, handler EventHandler) (subID string, err error)
SubscribeAll(handler EventHandler) (subID string, err error)
Unsubscribe(subID string) error
Close() error
Start(ctx context.Context) error // 新增
Stop() error // 新增
}
```
LocalBus 的 Start/Stop 为空操作。
- [ ] **Step 2: 在 ServiceContext 中调用 bus.Start()**
`service_context.go``NewServiceContext` 中,bus 创建之后、订阅 EventLogWriter 之前:
```go
if err := bus.Start(context.Background()); err != nil {
slog.Error("svc: bus start failed", "error", err)
}
```
- [ ] **Step 3: 验证编译**
Run: `rtk go build ./...`
Expected: PASS
- [ ] **Step 4: 验证 LocalBus 不受影响**
Run: `rtk go test ./internal/eventbus/... ./internal/processor/...`
Expected: 现有测试全部 PASS
---
### Task 2: 修复 ApplyJobTransitionScript 版本号不一致
**Files:**
- Modify: `internal/state/redis_manager.go`
**Depends on:** Task 1
- [ ] **Step 1: 修改 Lua 脚本使版本递增**
当前 `applyJobTransitionScript` 直接使用 `ARGV[5]` 作为新版本:
```lua
-- 当前(有问题)
redis.call('HSET', jobKey, 'status', newStatus, 'version', newVersion)
-- 修复为递增
local currentVer = tonumber(redis.call('HGET', jobKey, 'version') or '0')
local expectedVer = tonumber(ARGV[4])
if currentVer ~= expectedVer then return 0 end
redis.call('HSET', jobKey, 'status', newStatus, 'version', currentVer + 1)
```
- [ ] **Step 2: 运行 StateManager 测试**
Run: `rtk go test ./internal/state/...`
Expected: 现有测试 PASSMemoryStateManager 行为不变)
- [ ] **Step 3: 记录检查点**
---
### Task 3: 补全 StateManager 接口
**Files:**
- Modify: `internal/state/manager.go`
- Modify: `internal/state/memory_manager.go`
- Modify: `internal/state/manager_test.go`
**Depends on:** Task 1
- [ ] **Step 1: 在接口中增加 GetTaskState**
```go
type StateManager interface {
// ... 现有方法 ...
GetTaskState(ctx context.Context, taskID string) (*TaskState, error) // 新增
}
```
同时定义 `TaskState` 结构体。
- [ ] **Step 2: MemoryStateManager 完整实现 ApplyTaskState + GetTaskState**
用内存 map 存储 task state,与 job state 同模式。
- [ ] **Step 3: RedisStateManager 实现 GetTaskState**
```go
func (m *RedisStateManager) GetTaskState(ctx context.Context, taskID string) (*TaskState, error) {
key := taskKey(taskID)
fields, err := m.client.HGetAll(ctx, key).Result()
// ...
}
```
- [ ] **Step 4: 写测试并验证**
```go
func TestGetTaskState(t *testing.T) {
mgr := NewMemoryStateManager()
err := mgr.ApplyTaskState(ctx, "task-1", &TaskState{Status: "RUNNING"})
require.NoError(t, err)
ts, err := mgr.GetTaskState(ctx, "task-1")
require.NoError(t, err)
assert.Equal(t, "RUNNING", ts.Status)
}
```
Run: `rtk go test ./internal/state/...`
Expected: PASS
---
## Chunk 2: 运行时 SSOT 写穿通道
### Task 4: 创建写穿辅助层
**Files:**
- Create: `internal/state/state_write_through.go`
**Depends on:** Chunk 1 完成
- [ ] **Step 1: 创建 WriteThroughHelper**
```go
package state
// WriteThroughHelper 封装 Processor→Redis 的写穿逻辑。
// 当 SSOT 禁用时,降级为仅写 MemoryStateManager。
type WriteThroughHelper struct {
mgr StateManager
enabled bool
}
func NewWriteThroughHelper(mgr StateManager, enabled bool) *WriteThroughHelper {
return &WriteThroughHelper{mgr: mgr, enabled: enabled}
}
// SyncJobState 写穿:将 JobRuntime 状态同步到 StateManager。
// enabled=false 时仅更新内存,不写 Redis。
func (h *WriteThroughHelper) SyncJobState(ctx context.Context, jobID int, status string, positionType, positionRefID string, version int) error {
return h.mgr.ApplyJobState(ctx, jobID, map[string]interface{}{
"status": status,
"position_type": positionType,
"position_ref": positionRefID,
"version": version,
})
}
```
- [ ] **Step 2: 验证编译**
Run: `rtk go build ./internal/state/...`
Expected: PASS
---
### Task 5: JobProcessor 事件处理器写穿 Redis
**Files:**
- Modify: `internal/processor/job_processor.go`
**Depends on:** Task 4
- [ ] **Step 1: 在 JobProcessor 中注入 WriteThroughHelper**
```go
type JobProcessor struct {
// ... 现有字段 ...
ssotWriter *state.WriteThroughHelper
}
func (jp *JobProcessor) SetSSOTWriter(w *state.WriteThroughHelper) {
jp.ssotWriter = w
}
```
- [ ] **Step 2: 在关键状态变更点调用写穿**
`onJobCompleted``onJobError``handleMachineTaskComplete` 等处理器中,状态变更后调用:
```go
if jp.ssotWriter != nil {
jp.ssotWriter.SyncJobState(ctx, jobID,
string(domainStatus),
string(jr.PositionType),
jr.PositionRefID,
jr.version,
)
}
```
- [ ] **Step 3: ServiceContext 接线**
`service_context.go` 中:
```go
if stateMgr != nil {
writer := state.NewWriteThroughHelper(stateMgr, c.SSOT.Enabled)
orderProcessor.SetSSOTWriter(writer)
}
```
- [ ] **Step 4: 验证编译 + 现有测试**
Run: `rtk go build ./... && rtk go test ./internal/processor/...`
Expected: build PASS, tests PASS(旧路径 SSOT.Enabled=false 不受影响)
- [ ] **Step 5: 记录检查点**
---
### Task 6: 写穿集成测试
**Files:**
- Create: `internal/processor/ssot_integration_test.go`
**Depends on:** Task 5
- [ ] **Step 1: 写测试 — 验证 Job 状态变更后 Redis 可读**
```go
func TestSSOTWriteThrough_JobCompleted(t *testing.T) {
mgr := state.NewMemoryStateManager()
writer := state.NewWriteThroughHelper(mgr, true)
// 模拟状态变更
err := writer.SyncJobState(ctx, 1, "COMPLETED", "ON_DOCK", "", 2)
require.NoError(t, err)
js, err := mgr.GetJobState(ctx, 1)
require.NoError(t, err)
assert.Equal(t, "COMPLETED", js["status"])
}
```
- [ ] **Step 2: 运行测试**
Run: `rtk go test ./internal/processor/... -run SSOT`
Expected: PASS
- [ ] **Step 3: 写测试 — 验证 SSOT.Enabled=false 时 MemoryStateManager 仍工作**
```go
func TestSSOTWriteThrough_Disabled(t *testing.T) {
mgr := state.NewMemoryStateManager()
writer := state.NewWriteThroughHelper(mgr, false) // 禁用
err := writer.SyncJobState(ctx, 1, "COMPLETED", "ON_DOCK", "", 2)
require.NoError(t, err)
// MemoryStateManager 仍能读到(本地写仍执行)
}
```
Run: `rtk go test ./internal/processor/... -run SSOT`
Expected: PASS
---
## Chunk 3: 事件驱动调度器(sched:ready_jobs + ZPOPMAX
### Task 7: 实现 sched:ready_jobs 推入逻辑
**Files:**
- Modify: `internal/state/keys.go`
- Create: `internal/scheduler/zpop.go`
**Depends on:** Chunk 2 完成
- [ ] **Step 1: 新增 Redis key 定义**
`keys.go` 追加:
```go
const (
SchedReadyJobsKey = "sched:ready_jobs" // Sorted Set: score=priority, member=jobID
)
```
- [ ] **Step 2: 在写穿层增加 PushReadyJob**
`state_write_through.go`
```go
func (h *WriteThroughHelper) PushReadyJob(ctx context.Context, jobID int, priority float64) error {
if !h.enabled || h.mgr == nil {
return nil
}
if rm, ok := h.mgr.(*RedisStateManager); ok {
return rm.client.ZAdd(ctx, SchedReadyJobsKey, redis.Z{
Score: priority,
Member: strconv.Itoa(jobID),
}).Err()
}
return nil
}
```
- [ ] **Step 3: 在 JobProcessor 状态变更点调用 PushReadyJob**
当 Job 进入 ON_BUFFER 或 WAITING_UNLOAD 时推送:
```go
if jp.ssotWriter != nil && (newStatus == constants.JobStatus_OnBuffer || newStatus == constants.JobStatus_WaitingUnload) {
jp.ssotWriter.PushReadyJob(ctx, jobID, float64(jr.Priority))
}
```
- [ ] **Step 4: 验证编译**
Run: `rtk go build ./...`
Expected: PASS
---
### Task 8: 实现 ZPOPMAX 调度循环
**Files:**
- Create: `internal/scheduler/zpop.go` (如已创建则修改)
**Depends on:** Task 7
- [ ] **Step 1: 实现 ZPOPMAX 阻塞循环**
```go
package scheduler
// ZPopLoop ZPOPMAX 阻塞调度循环。
// 设计文档 8.0 节:调度引擎阻塞在 ZPOPMAX 上,仅处理变动工件。
func ZPopLoop(ctx context.Context, rdb *redis.Client, handler func(jobID int) error) {
for {
select {
case <-ctx.Done():
return
default:
}
// BZPOPMAX 阻塞等待,超时 5s
results, err := rdb.BZPopMax(ctx, 5*time.Second, state.SchedReadyJobsKey).Result()
if err == redis.Nil {
continue
}
if err != nil {
slog.Error("zpop: failed", "error", err)
time.Sleep(time.Second)
continue
}
jobID, _ := strconv.Atoi(results.Member.(string))
if err := handler(jobID); err != nil {
// 处理失败放回队列(稍后重试)
rdb.ZAdd(ctx, state.SchedReadyJobsKey, redis.Z{
Score: results.Score - 0.1, // 降优先级
Member: results.Member,
})
}
}
}
```
- [ ] **Step 2: 在 ServiceContext 启动 ZPOP 循环**
```go
if c.SSOT.Enabled && goRedisClient != nil && sched != nil {
go scheduler.ZPopLoop(context.Background(), goRedisClient, func(jobID int) error {
return orderProcessor.ScheduleSingleJob(ctx, jobID)
})
}
```
- [ ] **Step 3: 在 JobProcessor 增加 ScheduleSingleJob**
```go
func (jp *JobProcessor) ScheduleSingleJob(ctx context.Context, jobID int) error {
jp.jobsMu.RLock()
jr, ok := jp.jobs[jobID]
jp.jobsMu.RUnlock()
if !ok {
return fmt.Errorf("job %d not found", jobID)
}
systemState := BuildSystemState(jp)
candidates := jp.sched.Schedule(jr, systemState)
for _, c := range candidates {
task := CandidateToRobotTask(c)
jp.dispatcher.Enqueue(task)
}
return nil
}
```
- [ ] **Step 4: 验证编译**
Run: `rtk go build ./...`
Expected: PASS
---
### Task 9: 整合新旧两条调度路径
**Files:**
- Modify: `internal/processor/job_processor.go`
- Modify: `internal/processor/scheduler_adapter.go`
**Depends on:** Task 8
- [ ] **Step 1: ReadyQueue 仅用于非 SSOT 路径**
`SSOT.Enabled=true` 时,`readyQueue` 返回 nil`notifyReady` 返回 nil。Job 仅通过 ZPOPMAX 驱动调度。
`SSOT.Enabled=false` 时,保持现有 ReadyQueue→ScheduleAndSubmit 路径。
- [ ] **Step 2: 验证两条路径不冲突**
Run: `rtk go test ./internal/processor/... ./internal/scheduler/...`
Expected: PASS
- [ ] **Step 3: 记录检查点**
---
## Chunk 4: 原子抢占 + 恢复增强 + 心跳超时
### Task 10: 实现原子抢占 Lua 脚本
**Files:**
- Modify: `internal/state/redis_manager.go`
**Depends on:** Chunk 3 完成
- [ ] **Step 1: 新增原子抢占 Lua 脚本**
设计文档 8.4 节:原子地将 `ON_BUFFER``IN_HANDLING`
```lua
-- claimJobScript: 原子抢占 Job
local jobKey = KEYS[1]
local expectedStatus = ARGV[1]
local newStatus = ARGV[2]
local expectedVer = tonumber(ARGV[3])
local currentStatus = redis.call('HGET', jobKey, 'status')
local currentVer = tonumber(redis.call('HGET', jobKey, 'version') or '0')
if currentStatus ~= expectedStatus then return {0, 'status_mismatch'} end
if currentVer ~= expectedVer then return {0, 'version_conflict'} end
redis.call('HSET', jobKey, 'status', newStatus, 'version', currentVer + 1)
return {1, 'ok'}
```
- [ ] **Step 2: 暴露 ClaimJob 方法**
```go
func (m *RedisStateManager) ClaimJob(ctx context.Context, jobID int, expectedStatus, newStatus string, expectedVer int) (bool, error) {
key := jobKey(jobID)
result, err := m.client.Eval(ctx, claimJobScript, []string{key},
expectedStatus, newStatus, expectedVer).Result()
// 返回 (抢占成功, error)
}
```
将此方法加入 `StateManager` 接口。
- [ ] **Step 3: 调度器在派发前原子抢占**
`scheduler_adapter.go` 中,`CandidateToRobotTask` 前调用 `ClaimJob`
- [ ] **Step 4: 测试**
Run: `rtk go test ./internal/state/...`
Expected: PASS
---
### Task 11: 创建 RecoveryCoordinator 并接入恢复流程
**Files:**
- Create: `internal/processor/recovery_coordinator.go`
- Modify: `internal/recovery/assessor.go`
- Modify: `internal/svc/service_context.go`
**Depends on:** Chunk 2 完成
- [ ] **Step 1: 定义 RecoveryCoordinator**
```go
package processor
type RecoveryCoordinator struct {
entClient *ent.Client
stateMgr state.StateManager
eventBus eventbus.Bus
}
func (rc *RecoveryCoordinator) AssessAndRecover(ctx context.Context, job *ent.Job) (*recovery.RecoveryGrade, error) {
grade, manualRequired, message := recovery.AssessOrderGrade([]*ent.Job{job})
switch {
case grade == "L1":
// 自动恢复:直接走 RestoreOrder
return grade, nil
case grade == "L2":
// 半自动:PL 状态确认后恢复
return grade, nil
case grade == "L3" || manualRequired:
// 创建 ManualAction 等待人工
// ...
}
return grade, nil
}
```
- [ ] **Step 2: 在 restoreActiveOrders 中调用 RecoveryCoordinator**
修改 `service_context.go``restoreActiveOrders()`,通过 `RecoveryCoordinator.AssessAndRecover` 替代直接 `RestoreOrder`
- [ ] **Step 3: 验证编译**
Run: `rtk go build ./...`
Expected: PASS
---
### Task 12: 设备心跳与 Task 超时
**Files:**
- Modify: `internal/state/redis_manager.go`
- Modify: `internal/state/keys.go`
- Modify: `internal/processor/dispatcher.go`
**Depends on:** Chunk 3 完成
- [ ] **Step 1: 设备心跳 — Redis TTL**
`redis_manager.go`
```go
func (m *RedisStateManager) UpdateEquipmentHeartbeat(ctx context.Context, equipID string) error {
return m.client.Set(ctx, equipHeartbeatKey(equipID), time.Now().Unix(), 5*time.Second).Err()
}
func (m *RedisStateManager) IsEquipmentOnline(ctx context.Context, equipID string) bool {
_, err := m.client.Get(ctx, equipHeartbeatKey(equipID)).Result()
return err == nil
}
```
- [ ] **Step 2: 心跳触发**
在 PLC 信号处理循环中,每次收到设备信号时调用 `UpdateEquipmentHeartbeat`
- [ ] **Step 3: Task 超时检测**
`dispatcher.go` 中增加超时检测 goroutine
```go
func (d *Dispatcher) startTimeoutWatchdog(ctx context.Context) {
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
d.checkTaskTimeouts(ctx)
}
}
}
func (d *Dispatcher) checkTaskTimeouts(ctx context.Context) {
d.activeTasksMu.Lock()
defer d.activeTasksMu.Unlock()
for taskID, task := range d.activeTasks {
if time.Since(task.EnqueueTime) > d.maxExecutionTime {
slog.Warn("task timeout", "taskID", taskID, "jobID", task.JobID)
// 标记超时,触发回退
}
}
}
```
- [ ] **Step 4: 验证编译**
Run: `rtk go build ./...`
Expected: PASS
---
## Chunk 5: 集成验证与回归
### Task 13: SSOT 端到端集成测试
**Files:**
- Create: `internal/processor/ssot_integration_test.go` (追加)
**Depends on:** Chunk 2-4 完成
- [ ] **Step 1: 测试 — 完整 Job 生命周期 SSOT 同步**
```go
func TestSSOTFullJobLifecycle(t *testing.T) {
mgr := state.NewMemoryStateManager()
writer := state.NewWriteThroughHelper(mgr, true)
// Job 创建 → ON_BUFFER → PROCESSING → WAITING_UNLOAD → COMPLETED
steps := []struct {
status string
posType constants.PositionType
posRef string
}{
{"ON_BUFFER", constants.PositionType_OnBuffer, "5"},
{"PROCESSING", constants.PositionType_OnEquipment, "3:0"},
{"WAITING_UNLOAD", constants.PositionType_OnEquipment, "3:0"},
{"COMPLETED", constants.PositionType_OnDock, "2:1"},
}
for i, s := range steps {
err := writer.SyncJobState(ctx, 1, s.status, string(s.posType), s.posRef, i+1)
require.NoError(t, err)
js, err := mgr.GetJobState(ctx, 1)
require.NoError(t, err)
assert.Equal(t, s.status, js["status"])
}
}
```
- [ ] **Step 2: 运行**
Run: `rtk go test ./internal/processor/... -run SSOT -v`
Expected: PASS
---
### Task 14: 全量回归验证
**Depends on:** Task 13
- [ ] **Step 1: 全量编译**
Run: `rtk go build ./...`
Expected: PASS
- [ ] **Step 2: 核心包测试**
Run: `rtk go test ./internal/state/... ./internal/processor/... ./internal/station/... ./internal/scheduler/... ./internal/eventbus/... ./internal/recovery/...`
Expected: PASS
- [ ] **Step 3: 全量测试(排除已知失败)**
Run: `rtk go test ./internal/...`
Expected: 无新增失败(排除 camera/db/preload/robot 已有问题)
- [ ] **Step 4: 覆盖率检查**
Run: `rtk go test -cover ./internal/state/... ./internal/processor/... ./internal/scheduler/...`
Expected: 新增代码路径有测试覆盖
- [ ] **Step 5: 人工验收清单**
- SSOT.Enabled=true + EventBus.Enabled=true 时服务正常启动
- Redis 中可查询到 Job 实时状态
- 设备心跳 TTL 正常刷新
- 调度器可从 sched:ready_jobs ZPOPMAX
- SSOT.Enabled=false 退化为旧路径,行为不变
---
## 风险点
1. **RedisBus.Start() 调用时机** — 必须在订阅者注册之后调用,否则事件丢失。处理:先 SubscribeAll,再 Start。
2. **写穿通道影响旧路径性能** — 每次状态变更多一层 Redis 调用。处理:异步写(goroutine + channel),不阻塞主调度路径。
3. **ZPOPMAX 与 ReadyQueue 双路径冲突** — 同一 Job 可能被两次调度。处理:SSOT 启用时完全禁用 ReadyQueue 路径,单一入口。
4. **原子抢占 Lua 脚本与现有 ApplyJobState 版本号竞争** — 两个脚本操作同一 Hash。处理:统一版本递增逻辑,所有写入走 Lua。
5. **增量重放幂等** — 已处理事件不可重复应用。处理:last_event_id 去重,版本号比较防御。
## 验证点
- `rtk go build ./...` 通过
- `rtk go test ./internal/state/...` PASS
- `rtk go test ./internal/processor/...` PASS
- `rtk go test ./internal/scheduler/...` PASS
- `rtk go test ./internal/eventbus/...` PASS
- `rtk go test ./internal/recovery/...` PASS
- SSOT.Enabled=false 旧路径回归通过
- Redis 中 `job:{id}` Hash 实时反映工件状态
- `sched:ready_jobs` Sorted Set 正确推送和消费
## 执行说明
- 当前目录不是 git 仓库,跳过 commit 步骤,改为每个任务结束记录检查点。
- 严格遵守项目规则:只改 `schema/*.go` 后再生成 ent;不要手改 `ent/`
- 使用 `rtk` 前缀运行所有命令。
- 所有新增和修改的代码通过 `rtk go build ./...` 后交付。