747 lines
21 KiB
Markdown
747 lines
21 KiB
Markdown
# Redis SSOT + 事件重放 + 三层调度器 Phase 2 实施计划
|
||||
|
|
|
|||
|
|
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|||
|
|
|
|||
|
|
**Goal:** 将 Redis SSOT、事件重放、三层调度器从当前"已实现但未接通"状态推进到设计文档要求的完整落地——Redis 成为运行时唯一真相源,调度器通过 ZPOPMAX 事件驱动,断电恢复可安全重放。
|
|||
|
|
|
|||
|
|
**Architecture:** 分三步走:(1) 修复关键断路,让 RedisBus/StateManager 完整运行;(2) 建立运行时写穿通道,Processor 事件处理同步更新 Redis;(3) 接入 ZPOPMAX 调度循环和原子抢占 Lua 脚本。所有变更加 SSOT 开关隔离,不破坏现有执行内核。
|
|||
|
|
|
|||
|
|
**Tech Stack:** Go 1.23.4, go-zero, ent v0.14.5, Redis (go-redis/v9), PostgreSQL, testify
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 文件结构
|
|||
|
|
|
|||
|
|
### 必改文件
|
|||
|
|
- Modify: `internal/svc/service_context.go` — 调用 RedisBus.Start()、注入 RecoveryCoordinator
|
|||
|
|
- Modify: `internal/state/manager.go` — 补全 GetTaskState 接口方法
|
|||
|
|
- Modify: `internal/state/redis_manager.go` — 修 ApplyJobTransitionScript 版本号、设备心跳
|
|||
|
|
- Modify: `internal/state/memory_manager.go` — 完整实现 ApplyTaskState/GetTaskState
|
|||
|
|
- Modify: `internal/state/keys.go` — 新增 sched:ready_jobs、equipment heartbeat key
|
|||
|
|
- Modify: `internal/eventbus/bus.go` — Bus 接口增加 Start/Stop 方法(或保障 RedisBus 启动)
|
|||
|
|
- Modify: `internal/processor/job_processor.go` — 写穿 StateManager、接入 sched:ready_jobs
|
|||
|
|
- Modify: `internal/processor/dispatcher.go` — Task 超时检测
|
|||
|
|
- Modify: `internal/processor/scheduler_adapter.go` — ZPOPMAX 调度循环适配
|
|||
|
|
- Modify: `internal/scheduler/scheduler.go` — 接收 ZPOPMAX 单工件输入
|
|||
|
|
- Modify: `internal/recovery/replayer.go` — 增量重放、ReplaySince
|
|||
|
|
- Modify: `internal/recovery/assessor.go` — 接入 restoreActiveOrders
|
|||
|
|
- Modify: `internal/robot/common.go` — 心跳信号
|
|||
|
|
|
|||
|
|
### 新建文件
|
|||
|
|
- Create: `internal/processor/recovery_coordinator.go` — RecoveryCoordinator 实现
|
|||
|
|
- Create: `internal/state/state_write_through.go` — 写穿辅助函数(Processor→Redis)
|
|||
|
|
- Create: `internal/scheduler/zpop.go` — ZPOPMAX 调度循环
|
|||
|
|
- Create: `internal/state/redis_manager_test.go` — RedisStateManager 集成测试
|
|||
|
|
- Create: `internal/processor/ssot_integration_test.go` — SSOT 写穿集成测试
|
|||
|
|
|
|||
|
|
### 现有测试要改
|
|||
|
|
- Modify: `internal/state/manager_test.go` — 新增 GetTaskState 测试
|
|||
|
|
- Modify: `internal/scheduler/scheduler_test.go` — 新增 ZPOPMAX 输入路径测试
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Chunk 1: 修复关键断路
|
|||
|
|
|
|||
|
|
### Task 1: 调用 RedisBus.Start() + Bus 接口补全
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Modify: `internal/eventbus/bus.go`
|
|||
|
|
- Modify: `internal/svc/service_context.go`
|
|||
|
|
|
|||
|
|
**Depends on:** 无
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: Bus 接口增加 Start/Stop 方法**
|
|||
|
|
|
|||
|
|
在 `bus.go` 的 `Bus` interface 追加:
|
|||
|
|
```go
|
|||
|
|
type Bus interface {
|
|||
|
|
Publish(ctx context.Context, event Event) error
|
|||
|
|
Subscribe(eventType EventType, handler EventHandler) (subID string, err error)
|
|||
|
|
SubscribeAll(handler EventHandler) (subID string, err error)
|
|||
|
|
Unsubscribe(subID string) error
|
|||
|
|
Close() error
|
|||
|
|
Start(ctx context.Context) error // 新增
|
|||
|
|
Stop() error // 新增
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
LocalBus 的 Start/Stop 为空操作。
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 在 ServiceContext 中调用 bus.Start()**
|
|||
|
|
|
|||
|
|
在 `service_context.go` 的 `NewServiceContext` 中,bus 创建之后、订阅 EventLogWriter 之前:
|
|||
|
|
```go
|
|||
|
|
if err := bus.Start(context.Background()); err != nil {
|
|||
|
|
slog.Error("svc: bus start failed", "error", err)
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 验证编译**
|
|||
|
|
|
|||
|
|
Run: `rtk go build ./...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
- [ ] **Step 4: 验证 LocalBus 不受影响**
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/eventbus/... ./internal/processor/...`
|
|||
|
|
Expected: 现有测试全部 PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 2: 修复 ApplyJobTransitionScript 版本号不一致
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Modify: `internal/state/redis_manager.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Task 1
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 修改 Lua 脚本使版本递增**
|
|||
|
|
|
|||
|
|
当前 `applyJobTransitionScript` 直接使用 `ARGV[5]` 作为新版本:
|
|||
|
|
```lua
|
|||
|
|
-- 当前(有问题)
|
|||
|
|
redis.call('HSET', jobKey, 'status', newStatus, 'version', newVersion)
|
|||
|
|
|
|||
|
|
-- 修复为递增
|
|||
|
|
local currentVer = tonumber(redis.call('HGET', jobKey, 'version') or '0')
|
|||
|
|
local expectedVer = tonumber(ARGV[4])
|
|||
|
|
if currentVer ~= expectedVer then return 0 end
|
|||
|
|
redis.call('HSET', jobKey, 'status', newStatus, 'version', currentVer + 1)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 运行 StateManager 测试**
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/state/...`
|
|||
|
|
Expected: 现有测试 PASS(MemoryStateManager 行为不变)
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 记录检查点**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 3: 补全 StateManager 接口
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Modify: `internal/state/manager.go`
|
|||
|
|
- Modify: `internal/state/memory_manager.go`
|
|||
|
|
- Modify: `internal/state/manager_test.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Task 1
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 在接口中增加 GetTaskState**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
type StateManager interface {
|
|||
|
|
// ... 现有方法 ...
|
|||
|
|
GetTaskState(ctx context.Context, taskID string) (*TaskState, error) // 新增
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
同时定义 `TaskState` 结构体。
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: MemoryStateManager 完整实现 ApplyTaskState + GetTaskState**
|
|||
|
|
|
|||
|
|
用内存 map 存储 task state,与 job state 同模式。
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: RedisStateManager 实现 GetTaskState**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
func (m *RedisStateManager) GetTaskState(ctx context.Context, taskID string) (*TaskState, error) {
|
|||
|
|
key := taskKey(taskID)
|
|||
|
|
fields, err := m.client.HGetAll(ctx, key).Result()
|
|||
|
|
// ...
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 4: 写测试并验证**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
func TestGetTaskState(t *testing.T) {
|
|||
|
|
mgr := NewMemoryStateManager()
|
|||
|
|
err := mgr.ApplyTaskState(ctx, "task-1", &TaskState{Status: "RUNNING"})
|
|||
|
|
require.NoError(t, err)
|
|||
|
|
|
|||
|
|
ts, err := mgr.GetTaskState(ctx, "task-1")
|
|||
|
|
require.NoError(t, err)
|
|||
|
|
assert.Equal(t, "RUNNING", ts.Status)
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/state/...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Chunk 2: 运行时 SSOT 写穿通道
|
|||
|
|
|
|||
|
|
### Task 4: 创建写穿辅助层
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Create: `internal/state/state_write_through.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Chunk 1 完成
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 创建 WriteThroughHelper**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
package state
|
|||
|
|
|
|||
|
|
// WriteThroughHelper 封装 Processor→Redis 的写穿逻辑。
|
|||
|
|
// 当 SSOT 禁用时,降级为仅写 MemoryStateManager。
|
|||
|
|
type WriteThroughHelper struct {
|
|||
|
|
mgr StateManager
|
|||
|
|
enabled bool
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
func NewWriteThroughHelper(mgr StateManager, enabled bool) *WriteThroughHelper {
|
|||
|
|
return &WriteThroughHelper{mgr: mgr, enabled: enabled}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
// SyncJobState 写穿:将 JobRuntime 状态同步到 StateManager。
|
|||
|
|
// enabled=false 时仅更新内存,不写 Redis。
|
|||
|
|
func (h *WriteThroughHelper) SyncJobState(ctx context.Context, jobID int, status string, positionType, positionRefID string, version int) error {
|
|||
|
|
return h.mgr.ApplyJobState(ctx, jobID, map[string]interface{}{
|
|||
|
|
"status": status,
|
|||
|
|
"position_type": positionType,
|
|||
|
|
"position_ref": positionRefID,
|
|||
|
|
"version": version,
|
|||
|
|
})
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 验证编译**
|
|||
|
|
|
|||
|
|
Run: `rtk go build ./internal/state/...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 5: JobProcessor 事件处理器写穿 Redis
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Modify: `internal/processor/job_processor.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Task 4
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 在 JobProcessor 中注入 WriteThroughHelper**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
type JobProcessor struct {
|
|||
|
|
// ... 现有字段 ...
|
|||
|
|
ssotWriter *state.WriteThroughHelper
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
func (jp *JobProcessor) SetSSOTWriter(w *state.WriteThroughHelper) {
|
|||
|
|
jp.ssotWriter = w
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 在关键状态变更点调用写穿**
|
|||
|
|
|
|||
|
|
在 `onJobCompleted`、`onJobError`、`handleMachineTaskComplete` 等处理器中,状态变更后调用:
|
|||
|
|
```go
|
|||
|
|
if jp.ssotWriter != nil {
|
|||
|
|
jp.ssotWriter.SyncJobState(ctx, jobID,
|
|||
|
|
string(domainStatus),
|
|||
|
|
string(jr.PositionType),
|
|||
|
|
jr.PositionRefID,
|
|||
|
|
jr.version,
|
|||
|
|
)
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: ServiceContext 接线**
|
|||
|
|
|
|||
|
|
在 `service_context.go` 中:
|
|||
|
|
```go
|
|||
|
|
if stateMgr != nil {
|
|||
|
|
writer := state.NewWriteThroughHelper(stateMgr, c.SSOT.Enabled)
|
|||
|
|
orderProcessor.SetSSOTWriter(writer)
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 4: 验证编译 + 现有测试**
|
|||
|
|
|
|||
|
|
Run: `rtk go build ./... && rtk go test ./internal/processor/...`
|
|||
|
|
Expected: build PASS, tests PASS(旧路径 SSOT.Enabled=false 不受影响)
|
|||
|
|
|
|||
|
|
- [ ] **Step 5: 记录检查点**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 6: 写穿集成测试
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Create: `internal/processor/ssot_integration_test.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Task 5
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 写测试 — 验证 Job 状态变更后 Redis 可读**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
func TestSSOTWriteThrough_JobCompleted(t *testing.T) {
|
|||
|
|
mgr := state.NewMemoryStateManager()
|
|||
|
|
writer := state.NewWriteThroughHelper(mgr, true)
|
|||
|
|
|
|||
|
|
// 模拟状态变更
|
|||
|
|
err := writer.SyncJobState(ctx, 1, "COMPLETED", "ON_DOCK", "", 2)
|
|||
|
|
require.NoError(t, err)
|
|||
|
|
|
|||
|
|
js, err := mgr.GetJobState(ctx, 1)
|
|||
|
|
require.NoError(t, err)
|
|||
|
|
assert.Equal(t, "COMPLETED", js["status"])
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 运行测试**
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/processor/... -run SSOT`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 写测试 — 验证 SSOT.Enabled=false 时 MemoryStateManager 仍工作**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
func TestSSOTWriteThrough_Disabled(t *testing.T) {
|
|||
|
|
mgr := state.NewMemoryStateManager()
|
|||
|
|
writer := state.NewWriteThroughHelper(mgr, false) // 禁用
|
|||
|
|
|
|||
|
|
err := writer.SyncJobState(ctx, 1, "COMPLETED", "ON_DOCK", "", 2)
|
|||
|
|
require.NoError(t, err)
|
|||
|
|
// MemoryStateManager 仍能读到(本地写仍执行)
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/processor/... -run SSOT`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Chunk 3: 事件驱动调度器(sched:ready_jobs + ZPOPMAX)
|
|||
|
|
|
|||
|
|
### Task 7: 实现 sched:ready_jobs 推入逻辑
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Modify: `internal/state/keys.go`
|
|||
|
|
- Create: `internal/scheduler/zpop.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Chunk 2 完成
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 新增 Redis key 定义**
|
|||
|
|
|
|||
|
|
在 `keys.go` 追加:
|
|||
|
|
```go
|
|||
|
|
const (
|
|||
|
|
SchedReadyJobsKey = "sched:ready_jobs" // Sorted Set: score=priority, member=jobID
|
|||
|
|
)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 在写穿层增加 PushReadyJob**
|
|||
|
|
|
|||
|
|
在 `state_write_through.go`:
|
|||
|
|
```go
|
|||
|
|
func (h *WriteThroughHelper) PushReadyJob(ctx context.Context, jobID int, priority float64) error {
|
|||
|
|
if !h.enabled || h.mgr == nil {
|
|||
|
|
return nil
|
|||
|
|
}
|
|||
|
|
if rm, ok := h.mgr.(*RedisStateManager); ok {
|
|||
|
|
return rm.client.ZAdd(ctx, SchedReadyJobsKey, redis.Z{
|
|||
|
|
Score: priority,
|
|||
|
|
Member: strconv.Itoa(jobID),
|
|||
|
|
}).Err()
|
|||
|
|
}
|
|||
|
|
return nil
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 在 JobProcessor 状态变更点调用 PushReadyJob**
|
|||
|
|
|
|||
|
|
当 Job 进入 ON_BUFFER 或 WAITING_UNLOAD 时推送:
|
|||
|
|
```go
|
|||
|
|
if jp.ssotWriter != nil && (newStatus == constants.JobStatus_OnBuffer || newStatus == constants.JobStatus_WaitingUnload) {
|
|||
|
|
jp.ssotWriter.PushReadyJob(ctx, jobID, float64(jr.Priority))
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 4: 验证编译**
|
|||
|
|
|
|||
|
|
Run: `rtk go build ./...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 8: 实现 ZPOPMAX 调度循环
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Create: `internal/scheduler/zpop.go` (如已创建则修改)
|
|||
|
|
|
|||
|
|
**Depends on:** Task 7
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 实现 ZPOPMAX 阻塞循环**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
package scheduler
|
|||
|
|
|
|||
|
|
// ZPopLoop ZPOPMAX 阻塞调度循环。
|
|||
|
|
// 设计文档 8.0 节:调度引擎阻塞在 ZPOPMAX 上,仅处理变动工件。
|
|||
|
|
func ZPopLoop(ctx context.Context, rdb *redis.Client, handler func(jobID int) error) {
|
|||
|
|
for {
|
|||
|
|
select {
|
|||
|
|
case <-ctx.Done():
|
|||
|
|
return
|
|||
|
|
default:
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
// BZPOPMAX 阻塞等待,超时 5s
|
|||
|
|
results, err := rdb.BZPopMax(ctx, 5*time.Second, state.SchedReadyJobsKey).Result()
|
|||
|
|
if err == redis.Nil {
|
|||
|
|
continue
|
|||
|
|
}
|
|||
|
|
if err != nil {
|
|||
|
|
slog.Error("zpop: failed", "error", err)
|
|||
|
|
time.Sleep(time.Second)
|
|||
|
|
continue
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
jobID, _ := strconv.Atoi(results.Member.(string))
|
|||
|
|
if err := handler(jobID); err != nil {
|
|||
|
|
// 处理失败放回队列(稍后重试)
|
|||
|
|
rdb.ZAdd(ctx, state.SchedReadyJobsKey, redis.Z{
|
|||
|
|
Score: results.Score - 0.1, // 降优先级
|
|||
|
|
Member: results.Member,
|
|||
|
|
})
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 在 ServiceContext 启动 ZPOP 循环**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
if c.SSOT.Enabled && goRedisClient != nil && sched != nil {
|
|||
|
|
go scheduler.ZPopLoop(context.Background(), goRedisClient, func(jobID int) error {
|
|||
|
|
return orderProcessor.ScheduleSingleJob(ctx, jobID)
|
|||
|
|
})
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 在 JobProcessor 增加 ScheduleSingleJob**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
func (jp *JobProcessor) ScheduleSingleJob(ctx context.Context, jobID int) error {
|
|||
|
|
jp.jobsMu.RLock()
|
|||
|
|
jr, ok := jp.jobs[jobID]
|
|||
|
|
jp.jobsMu.RUnlock()
|
|||
|
|
if !ok {
|
|||
|
|
return fmt.Errorf("job %d not found", jobID)
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
systemState := BuildSystemState(jp)
|
|||
|
|
candidates := jp.sched.Schedule(jr, systemState)
|
|||
|
|
for _, c := range candidates {
|
|||
|
|
task := CandidateToRobotTask(c)
|
|||
|
|
jp.dispatcher.Enqueue(task)
|
|||
|
|
}
|
|||
|
|
return nil
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 4: 验证编译**
|
|||
|
|
|
|||
|
|
Run: `rtk go build ./...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 9: 整合新旧两条调度路径
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Modify: `internal/processor/job_processor.go`
|
|||
|
|
- Modify: `internal/processor/scheduler_adapter.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Task 8
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: ReadyQueue 仅用于非 SSOT 路径**
|
|||
|
|
|
|||
|
|
当 `SSOT.Enabled=true` 时,`readyQueue` 返回 nil,`notifyReady` 返回 nil。Job 仅通过 ZPOPMAX 驱动调度。
|
|||
|
|
|
|||
|
|
当 `SSOT.Enabled=false` 时,保持现有 ReadyQueue→ScheduleAndSubmit 路径。
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 验证两条路径不冲突**
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/processor/... ./internal/scheduler/...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 记录检查点**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Chunk 4: 原子抢占 + 恢复增强 + 心跳超时
|
|||
|
|
|
|||
|
|
### Task 10: 实现原子抢占 Lua 脚本
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Modify: `internal/state/redis_manager.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Chunk 3 完成
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 新增原子抢占 Lua 脚本**
|
|||
|
|
|
|||
|
|
设计文档 8.4 节:原子地将 `ON_BUFFER` → `IN_HANDLING`。
|
|||
|
|
```lua
|
|||
|
|
-- claimJobScript: 原子抢占 Job
|
|||
|
|
local jobKey = KEYS[1]
|
|||
|
|
local expectedStatus = ARGV[1]
|
|||
|
|
local newStatus = ARGV[2]
|
|||
|
|
local expectedVer = tonumber(ARGV[3])
|
|||
|
|
|
|||
|
|
local currentStatus = redis.call('HGET', jobKey, 'status')
|
|||
|
|
local currentVer = tonumber(redis.call('HGET', jobKey, 'version') or '0')
|
|||
|
|
|
|||
|
|
if currentStatus ~= expectedStatus then return {0, 'status_mismatch'} end
|
|||
|
|
if currentVer ~= expectedVer then return {0, 'version_conflict'} end
|
|||
|
|
|
|||
|
|
redis.call('HSET', jobKey, 'status', newStatus, 'version', currentVer + 1)
|
|||
|
|
return {1, 'ok'}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 暴露 ClaimJob 方法**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
func (m *RedisStateManager) ClaimJob(ctx context.Context, jobID int, expectedStatus, newStatus string, expectedVer int) (bool, error) {
|
|||
|
|
key := jobKey(jobID)
|
|||
|
|
result, err := m.client.Eval(ctx, claimJobScript, []string{key},
|
|||
|
|
expectedStatus, newStatus, expectedVer).Result()
|
|||
|
|
// 返回 (抢占成功, error)
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
将此方法加入 `StateManager` 接口。
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 调度器在派发前原子抢占**
|
|||
|
|
|
|||
|
|
在 `scheduler_adapter.go` 中,`CandidateToRobotTask` 前调用 `ClaimJob`。
|
|||
|
|
|
|||
|
|
- [ ] **Step 4: 测试**
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/state/...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 11: 创建 RecoveryCoordinator 并接入恢复流程
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Create: `internal/processor/recovery_coordinator.go`
|
|||
|
|
- Modify: `internal/recovery/assessor.go`
|
|||
|
|
- Modify: `internal/svc/service_context.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Chunk 2 完成
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 定义 RecoveryCoordinator**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
package processor
|
|||
|
|
|
|||
|
|
type RecoveryCoordinator struct {
|
|||
|
|
entClient *ent.Client
|
|||
|
|
stateMgr state.StateManager
|
|||
|
|
eventBus eventbus.Bus
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
func (rc *RecoveryCoordinator) AssessAndRecover(ctx context.Context, job *ent.Job) (*recovery.RecoveryGrade, error) {
|
|||
|
|
grade, manualRequired, message := recovery.AssessOrderGrade([]*ent.Job{job})
|
|||
|
|
|
|||
|
|
switch {
|
|||
|
|
case grade == "L1":
|
|||
|
|
// 自动恢复:直接走 RestoreOrder
|
|||
|
|
return grade, nil
|
|||
|
|
case grade == "L2":
|
|||
|
|
// 半自动:PL 状态确认后恢复
|
|||
|
|
return grade, nil
|
|||
|
|
case grade == "L3" || manualRequired:
|
|||
|
|
// 创建 ManualAction 等待人工
|
|||
|
|
// ...
|
|||
|
|
}
|
|||
|
|
return grade, nil
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 在 restoreActiveOrders 中调用 RecoveryCoordinator**
|
|||
|
|
|
|||
|
|
修改 `service_context.go` 的 `restoreActiveOrders()`,通过 `RecoveryCoordinator.AssessAndRecover` 替代直接 `RestoreOrder`。
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 验证编译**
|
|||
|
|
|
|||
|
|
Run: `rtk go build ./...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 12: 设备心跳与 Task 超时
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Modify: `internal/state/redis_manager.go`
|
|||
|
|
- Modify: `internal/state/keys.go`
|
|||
|
|
- Modify: `internal/processor/dispatcher.go`
|
|||
|
|
|
|||
|
|
**Depends on:** Chunk 3 完成
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 设备心跳 — Redis TTL**
|
|||
|
|
|
|||
|
|
在 `redis_manager.go`:
|
|||
|
|
```go
|
|||
|
|
func (m *RedisStateManager) UpdateEquipmentHeartbeat(ctx context.Context, equipID string) error {
|
|||
|
|
return m.client.Set(ctx, equipHeartbeatKey(equipID), time.Now().Unix(), 5*time.Second).Err()
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
func (m *RedisStateManager) IsEquipmentOnline(ctx context.Context, equipID string) bool {
|
|||
|
|
_, err := m.client.Get(ctx, equipHeartbeatKey(equipID)).Result()
|
|||
|
|
return err == nil
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 心跳触发**
|
|||
|
|
|
|||
|
|
在 PLC 信号处理循环中,每次收到设备信号时调用 `UpdateEquipmentHeartbeat`。
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: Task 超时检测**
|
|||
|
|
|
|||
|
|
在 `dispatcher.go` 中增加超时检测 goroutine:
|
|||
|
|
```go
|
|||
|
|
func (d *Dispatcher) startTimeoutWatchdog(ctx context.Context) {
|
|||
|
|
ticker := time.NewTicker(5 * time.Second)
|
|||
|
|
defer ticker.Stop()
|
|||
|
|
for {
|
|||
|
|
select {
|
|||
|
|
case <-ctx.Done():
|
|||
|
|
return
|
|||
|
|
case <-ticker.C:
|
|||
|
|
d.checkTaskTimeouts(ctx)
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
func (d *Dispatcher) checkTaskTimeouts(ctx context.Context) {
|
|||
|
|
d.activeTasksMu.Lock()
|
|||
|
|
defer d.activeTasksMu.Unlock()
|
|||
|
|
for taskID, task := range d.activeTasks {
|
|||
|
|
if time.Since(task.EnqueueTime) > d.maxExecutionTime {
|
|||
|
|
slog.Warn("task timeout", "taskID", taskID, "jobID", task.JobID)
|
|||
|
|
// 标记超时,触发回退
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 4: 验证编译**
|
|||
|
|
|
|||
|
|
Run: `rtk go build ./...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Chunk 5: 集成验证与回归
|
|||
|
|
|
|||
|
|
### Task 13: SSOT 端到端集成测试
|
|||
|
|
|
|||
|
|
**Files:**
|
|||
|
|
- Create: `internal/processor/ssot_integration_test.go` (追加)
|
|||
|
|
|
|||
|
|
**Depends on:** Chunk 2-4 完成
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 测试 — 完整 Job 生命周期 SSOT 同步**
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
func TestSSOTFullJobLifecycle(t *testing.T) {
|
|||
|
|
mgr := state.NewMemoryStateManager()
|
|||
|
|
writer := state.NewWriteThroughHelper(mgr, true)
|
|||
|
|
|
|||
|
|
// Job 创建 → ON_BUFFER → PROCESSING → WAITING_UNLOAD → COMPLETED
|
|||
|
|
steps := []struct {
|
|||
|
|
status string
|
|||
|
|
posType constants.PositionType
|
|||
|
|
posRef string
|
|||
|
|
}{
|
|||
|
|
{"ON_BUFFER", constants.PositionType_OnBuffer, "5"},
|
|||
|
|
{"PROCESSING", constants.PositionType_OnEquipment, "3:0"},
|
|||
|
|
{"WAITING_UNLOAD", constants.PositionType_OnEquipment, "3:0"},
|
|||
|
|
{"COMPLETED", constants.PositionType_OnDock, "2:1"},
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
for i, s := range steps {
|
|||
|
|
err := writer.SyncJobState(ctx, 1, s.status, string(s.posType), s.posRef, i+1)
|
|||
|
|
require.NoError(t, err)
|
|||
|
|
|
|||
|
|
js, err := mgr.GetJobState(ctx, 1)
|
|||
|
|
require.NoError(t, err)
|
|||
|
|
assert.Equal(t, s.status, js["status"])
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 运行**
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/processor/... -run SSOT -v`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### Task 14: 全量回归验证
|
|||
|
|
|
|||
|
|
**Depends on:** Task 13
|
|||
|
|
|
|||
|
|
- [ ] **Step 1: 全量编译**
|
|||
|
|
|
|||
|
|
Run: `rtk go build ./...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
- [ ] **Step 2: 核心包测试**
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/state/... ./internal/processor/... ./internal/station/... ./internal/scheduler/... ./internal/eventbus/... ./internal/recovery/...`
|
|||
|
|
Expected: PASS
|
|||
|
|
|
|||
|
|
- [ ] **Step 3: 全量测试(排除已知失败)**
|
|||
|
|
|
|||
|
|
Run: `rtk go test ./internal/...`
|
|||
|
|
Expected: 无新增失败(排除 camera/db/preload/robot 已有问题)
|
|||
|
|
|
|||
|
|
- [ ] **Step 4: 覆盖率检查**
|
|||
|
|
|
|||
|
|
Run: `rtk go test -cover ./internal/state/... ./internal/processor/... ./internal/scheduler/...`
|
|||
|
|
Expected: 新增代码路径有测试覆盖
|
|||
|
|
|
|||
|
|
- [ ] **Step 5: 人工验收清单**
|
|||
|
|
- SSOT.Enabled=true + EventBus.Enabled=true 时服务正常启动
|
|||
|
|
- Redis 中可查询到 Job 实时状态
|
|||
|
|
- 设备心跳 TTL 正常刷新
|
|||
|
|
- 调度器可从 sched:ready_jobs ZPOPMAX
|
|||
|
|
- SSOT.Enabled=false 退化为旧路径,行为不变
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 风险点
|
|||
|
|
|
|||
|
|
1. **RedisBus.Start() 调用时机** — 必须在订阅者注册之后调用,否则事件丢失。处理:先 SubscribeAll,再 Start。
|
|||
|
|
2. **写穿通道影响旧路径性能** — 每次状态变更多一层 Redis 调用。处理:异步写(goroutine + channel),不阻塞主调度路径。
|
|||
|
|
3. **ZPOPMAX 与 ReadyQueue 双路径冲突** — 同一 Job 可能被两次调度。处理:SSOT 启用时完全禁用 ReadyQueue 路径,单一入口。
|
|||
|
|
4. **原子抢占 Lua 脚本与现有 ApplyJobState 版本号竞争** — 两个脚本操作同一 Hash。处理:统一版本递增逻辑,所有写入走 Lua。
|
|||
|
|
5. **增量重放幂等** — 已处理事件不可重复应用。处理:last_event_id 去重,版本号比较防御。
|
|||
|
|
|
|||
|
|
## 验证点
|
|||
|
|
|
|||
|
|
- `rtk go build ./...` 通过
|
|||
|
|
- `rtk go test ./internal/state/...` PASS
|
|||
|
|
- `rtk go test ./internal/processor/...` PASS
|
|||
|
|
- `rtk go test ./internal/scheduler/...` PASS
|
|||
|
|
- `rtk go test ./internal/eventbus/...` PASS
|
|||
|
|
- `rtk go test ./internal/recovery/...` PASS
|
|||
|
|
- SSOT.Enabled=false 旧路径回归通过
|
|||
|
|
- Redis 中 `job:{id}` Hash 实时反映工件状态
|
|||
|
|
- `sched:ready_jobs` Sorted Set 正确推送和消费
|
|||
|
|
|
|||
|
|
## 执行说明
|
|||
|
|
|
|||
|
|
- 当前目录不是 git 仓库,跳过 commit 步骤,改为每个任务结束记录检查点。
|
|||
|
|
- 严格遵守项目规则:只改 `schema/*.go` 后再生成 ent;不要手改 `ent/`。
|
|||
|
|
- 使用 `rtk` 前缀运行所有命令。
|
|||
|
|
- 所有新增和修改的代码通过 `rtk go build ./...` 后交付。
|