feat(engine): workforce engine core — tag protocol, pipelines, optimistic state
Phase 1 of TODO-agent-2: pure logic plus the tables it writes to. - domain/workforce/pipeline.py: Stage/PipelineKey/Outcome, the four pipelines, TRANSITIONS, next_stage, STAGE_ROLE (role+mode), resolve_pipeline with the no-reviewer fallback, task-kind defaults, stage -> coarse tasks.status, and the TaskPipelineState record (with the optimistic version). - application/engine/protocol.py: the [[TAG attr="v"]]body[[/TAG]] parser and its self-closing form, the per-role/mode allow-list, literal bodies (a bracket inside a body is not a tag), typed readers (ASSIGN/CREATE_TASK/ team changes/KNOWLEDGE/EDIT_FILE blocks) and the per-reply limits (3 tasks, 3 team changes, 3 knowledge items). - application/engine/state.py: advance() with the optimistic compare-and-set and park_dependents() so a failed dependency never deadlocks its children. - db/models_engine.py + migration a2b3c4d5e6f2: task_pipeline_state, employee_profiles, engine_turns, gate_results as side tables (tasks and employees stay locked). Linear head; alembic check reports no drift. - repositories/engine.py + 58 new tests (test_engine_protocol, test_engine_pipeline) covering every pipeline's happy path, illegal transitions, exactly-one-winner concurrent advance, dependency parking and all parser edge cases. Suite: 594 passed, ruff clean, mypy clean.
This commit is contained in:
mjalmoudy94
@@ -0,0 +1,142 @@
|
||||
# Agent 2 — Phase 1 report: engine core (protocol, state machine, pipelines)
|
||||
|
||||
**Date:** 2026-10-14 · **Branch:** `dev` (this repo's integration branch) · **Status:** done, all checks green
|
||||
|
||||
> Environment note: the TODO file assumes the ext4 worktree (`/home/mamad/work/smart-org-wt/agent2`,
|
||||
> branch `agent2/ai-workforce`). This checkout is the `dev` repo itself and already contains the
|
||||
> other agents' work (org wizard, invites, delivery infra), so the engine is being implemented here
|
||||
> directly on `dev` at the user's instruction. No locked file was touched; no `CHANGE REQUEST` is
|
||||
> needed so far.
|
||||
|
||||
## What was done
|
||||
|
||||
### 1.1 `app/domain/workforce/pipeline.py` (new, pure)
|
||||
|
||||
- `Stage` (13 values: the eight pipeline stages plus the side states `pending_user_decision`,
|
||||
`error_review`, `paused`, `cancelled`, `failed`).
|
||||
- `PIPELINES`: `plan_code_test` (default), `plan_code_review_test`, `code_test`, `analysis_only`
|
||||
(no worktree, no merge). Each is the ordered tuple of stages ending in `completed`.
|
||||
- `TRANSITIONS` — every legal stage move, with the reasoning in comments: a paused or parked task
|
||||
resumes **through triage** (no `paused_from` column exists, and re-triage is the safe route);
|
||||
`merge_request` is platform-executed and nobody acts in it; terminal stages never move.
|
||||
- `next_stage(pipeline, current, outcome)` with `Outcome` (`ok`, `revise`, `reject`, `ask_user`,
|
||||
`error`, `resolve`). `ok` walks the pipeline list one step; the others have fixed targets —
|
||||
work sent back always returns to the developer.
|
||||
- `STAGE_ROLE` → `(role, mode)` or `None` for stages with no AI turn. Manager modes are
|
||||
`triage` / `final` / `error` (plus `chat`), every other role is `work`.
|
||||
- `resolve_pipeline(manager_choice, task_kind, app_default, has_active_reviewer)`: priority
|
||||
manager > kind > app default > `plan_code_test`; a review pipeline without an active reviewer
|
||||
falls back to `plan_code_test`.
|
||||
- `TASK_KIND_DEFAULTS`: feature → `plan_code_test`/medium, bugfix → `code_test`/low,
|
||||
refactor → `plan_code_review_test`/high, docs → `code_test`/low, analysis → `analysis_only`/low.
|
||||
- `STAGE_STATUS`: stage → coarse `tasks.status` (`new`/`running`/`waiting_approval`/`done`/`failed`/
|
||||
`cancelled`) so existing APIs and the old worker keep working against one truthful column.
|
||||
- `TaskPipelineState` dataclass: pipeline, kind, risk, stage, cycle/max_cycles, branch,
|
||||
`last_diff_hash`/`same_diff_count`, `retry_count`, `waiver_reason`, parent/depends, `version`
|
||||
(optimistic lock).
|
||||
|
||||
### 1.2 `app/application/engine/protocol.py` (new)
|
||||
|
||||
- Parser for `[[TAG attr="v" ...]]body[[/TAG]]` and `[[TAG attr="v"/]]`, tolerant of whitespace,
|
||||
single- or double-quoted values.
|
||||
- A body is **literal**: an outer tag wins and the scanner moves past its close, so brackets
|
||||
inside a body (a source file full of `[[`) are never re-parsed.
|
||||
- `TagName` enum for all 28 tags; `ALLOWED_TAGS` is the per-role/mode matrix from the TODO
|
||||
(manager triage/final/error/chat, planner, developer, reviewer, tester) plus the six tool tags
|
||||
granted to working roles only — the manager never edits code.
|
||||
- Unknown tags, disallowed tags, stray closings, unclosed tags and malformed attribute fragments
|
||||
are dropped and reported as one error line each, in order, for the model to read next turn.
|
||||
- `BODY_TAGS`: only tags that carry a body must be closed; a forgotten `/` on a bodyless tag
|
||||
(`[[RUN cmd="pytest"]]`) is read as self-closing instead of failing the reply.
|
||||
- Limits: max 3 `CREATE_TASK`, **max 3 team changes across HIRE+UPDATE_EMPLOYEE+FIRE together**,
|
||||
max 3 `KNOWLEDGE` per reply. Extras are dropped and reported.
|
||||
- Typed readers: `AssignTag`, `TaskDraft`, `TeamChange`, `KnowledgeItem`, `EditBlock`
|
||||
(`<<<OLD … === … NEW>>>`), plus `edit_blocks_of` which refuses a malformed block rather than
|
||||
half-applying it. `ASSIGN` refuses an unknown pipeline name.
|
||||
|
||||
### 1.3 `app/infrastructure/db/models_engine.py` (stub filled) + migration `a2b3c4d5e6f2`
|
||||
|
||||
Four tables, all 1:1/side tables — no column was added to the locked `tasks` / `employees`:
|
||||
|
||||
- `task_pipeline_state` (PK `task_id`), with the denormalised `organization_id` so the scheduler's
|
||||
hot query stays single-table.
|
||||
- `employee_profiles` (PK `employee_id`): `ai_model_id`, `effort`, roles CSV, `hired_by`,
|
||||
`busy_task_id`.
|
||||
- `engine_turns`: id, task/run/employee, role, mode, stage, `prompt_chars`, `reply_excerpt`
|
||||
(Text, bounded on write), tags JSON, tool_calls, ok, error_kind, started/finished.
|
||||
- `gate_results`: id, task, gate, ok, detail (bounded), created.
|
||||
|
||||
Migration `alembic/versions/a2b3c4d5e6f2_ai_workforce.py`:
|
||||
`alembic upgrade head` applies cleanly and `alembic check` reports **no drift** between the models
|
||||
and the migration chain.
|
||||
|
||||
**Deviation (documented in the migration docstring):** the TODO pins
|
||||
`down_revision = "a0f0e1d2c3b4"`, which was right while each agent had its own branch. On the
|
||||
merged `dev` chain the heads already form a line (`a3b4c5d6e7f4` → `a1b2c3d4e5f1` →
|
||||
`d5e6f7a8b9c0`), so this migration chains after `d5e6f7a8b9c0`. Result: **one linear head**
|
||||
(`a2b3c4d5e6f2`), no merge revision needed.
|
||||
|
||||
### 1.4 `app/application/engine/state.py` (new)
|
||||
|
||||
- `advance(states, task_id, expected_from, to)`: checks legality in the domain first, then the
|
||||
repository's compare-and-set (`WHERE task_id AND stage AND version`) performs the move. A lost
|
||||
race raises `ConflictError` ("این کار هماکنون توسط موتور دیگری در حرکت است"), an unknown task
|
||||
raises `NotFoundError`.
|
||||
- `park_dependents(states, task_id)`: a failed or cancelled dependency moves every dependent to
|
||||
`pending_user_decision` (agybot bug M1). The function **refuses to act on a healthy task** — the
|
||||
guard is in the function, not in the caller — and leaves dependents that already wait on a human
|
||||
alone. A lost race on a dependent is skipped, not fatal.
|
||||
- `app/infrastructure/repositories/engine.py` (new): `PipelineStateRepository` (advance/patch/
|
||||
list_at_stages/list_dependents), `EmployeeProfileRepository`, `EngineTurnRepository`,
|
||||
`GateResultRepository`. `advance`/`patch` verify the version actually moved rather than trusting
|
||||
`rowcount`, so the check holds on SQLite and PostgreSQL alike.
|
||||
|
||||
### 1.5 Tests
|
||||
|
||||
- `tests/test_engine_pipeline.py` (24 tests): all four pipelines walk manager review → completed
|
||||
step by step, each step also legal; sent-back work returns to the developer; illegal transitions
|
||||
refused; the stage/role/mode map; stage → coarse status; pipeline resolution priority and the
|
||||
reviewer fallback; plus DB-backed tests for the version bump, the illegal move, **exactly one
|
||||
winner in a concurrent advance**, the 404, the dependency-parking rule (and the healthy-task
|
||||
guard), optimistic `patch`, cascade deletes and the employee profile round trip.
|
||||
- `tests/test_engine_protocol.py` (34 tests): paired/self-closing parsing, whitespace tolerance,
|
||||
nested brackets staying literal, tag order, unknown/disallowed/stray/unclosed/malformed
|
||||
rejections, the full per-role matrix, toolkit-for-working-roles-only, typed readers
|
||||
(assign/task draft/team changes/knowledge/edit blocks) and all three limits.
|
||||
|
||||
## Checks (PLAN-v2 §7.4)
|
||||
|
||||
```
|
||||
cd apps/api && .venv/bin/python -m pytest -q → 594 passed, 1 skipped (whole suite; 58 new)
|
||||
cd apps/api && .venv/bin/ruff check . → All checks passed!
|
||||
cd apps/api && .venv/bin/mypy app → Success: no issues found in 144 source files
|
||||
alembic upgrade head (scratch sqlite) → applies through a2b3c4d5e6f2, single head
|
||||
alembic check (scratch sqlite) → No new upgrade operations detected.
|
||||
```
|
||||
|
||||
The web suite was not touched (no UI in Phase 1).
|
||||
|
||||
## Deviations & notes for the reviewer
|
||||
|
||||
1. Implemented on `dev` instead of an `agent2/ai-workforce` worktree (user instruction; the
|
||||
worktree path in the TODO is a Linux path unavailable here).
|
||||
2. Migration `down_revision` chained to the merged head (above) to keep one linear chain.
|
||||
3. `task_pipeline_state.organization_id` added beyond the TODO's column list: it is denormalised
|
||||
from the task so the scheduler never joins back, and it keeps the tenant filter single-table.
|
||||
4. `resolve_pipeline` applies the no-reviewer fallback to an explicit manager choice too. A review
|
||||
stage nobody can act on is a dead end; the manager who wants a reviewer should hire one.
|
||||
5. Team-change limit counts HIRE + UPDATE_EMPLOYEE + FIRE **together** (one budget per turn), which
|
||||
is what "max 3 team changes per turn" means.
|
||||
6. New repository module `app/infrastructure/repositories/engine.py` (the engine columns live
|
||||
beside `models_engine.py`); `repositories/workforce.py` was left untouched.
|
||||
|
||||
## Open questions / CHANGE REQUESTs
|
||||
|
||||
- None for Phase 1. Expected for later phases: org automation policy behind `ModelGrantPort`
|
||||
(TODO 4.1 says to assume enabled until then) and the worker hook for memory expiry.
|
||||
|
||||
## Next
|
||||
|
||||
Phase 2 — `engine/prompt.py`, `engine/tools.py` (sandbox execution, git refusal, path safety),
|
||||
`engine/runner.py` (`run_turn` with the scripted fake model, error classification, usage events)
|
||||
and `tests/test_engine_runner.py` + `tests/engine_fakes.py`.
|
||||
Reference in New Issue
Block a user