feat(integration): prepaid billing, engine UI, credits console, direct tasks, v2 final

- Prepaid credit system: org wallet, debit on usage, chat gate, admin credit console - Engine UI: /manager, /tasks, /tasks/[id], /workforce, /dashboard, /traces pages - Direct task creation (HTTP POST /tasks, MCP tasks_create) at pending_manager_review - Pause/resume/cancel/answer actions in engine API and UI - Employee cap (7 per org default), review pipeline guard (active reviewer required) - Scheduler: wide window prevents org starvation; archived employees excluded from cap - Tenant isolation: project-id cross-org 404, engine never uses another tenant's sandbox - Order route fix: manager built with pipeline states + employee profiles - Contract extension: ModelGrantPort.get_org_policy with DefaultOrgPolicy - Engine event types in @agy/contracts (task.step, task.gate, run.log, run.tool, team.changed) - api-engine.ts typed client; api-credit.ts client - Alembic migration c4d5e6f7a8b9: prepaid credit + engine UI tables - 859 tests passed, 1 skipped; ruff/mypy/tsc/eslint/vitest all green Refs: PLAN-v2.md §9 (T1–T43), todo-v2-final.md
2026-10-14 23:19:05 +03:30
parent fef5a9deab
commit d532929189
117 changed files with 13119 additions and 640 deletions
@@ -0,0 +1,216 @@
+# Agent 2 — Final report: AI workforce engine
+
+Scope: phases 1–5 of `docs/plan/TODO-agent-2-ai-workforce.md`, plus the prepaid billing and direct-task
+changes made after them (section 10). Every fact here was checked against the code with grep or `ls`.
+
+## 1. Status
+
+- TODO boxes: **26 of 29 ticked**. Three stay open, each with its reason in the TODO:
+  - 2.3: `run.status` is not published for engine runs (run.log and run.tool are).
+  - 4.1: turns run one after another inside a tick; the per-org and global caps only bound what a tick selects.
+  - 4.6: `tests/test_engine_flow.py` lacks the max-cycles and three-conflict failure tests (section 11).
+- Every other TODO item has its code, its tests and (for phase 5) its UI in the tree.
+- CHANGE REQUESTs still open: none (section 8).
+
+## 2. Architecture
+
+```
+order (manager chat, or POST /tasks)
+  -> manager chat (ManagerTurns.chat; may emit CREATE_TASK)
+  -> task + task_pipeline_state at pending_manager_review
+  -> scheduler tick; each stage move is one turn or one gate:
+     pending_manager_review   manager triage: ASSIGN | REJECT | ASK_USER
+     pending_analysis         planner: PLAN_SPEC | SPLIT (tools)
+     pending_implementation   developer: WORK_REPORT (tools)
+                              gates after it: diff guard, no-progress, tests
+     pending_review           reviewer: REVIEW_PASS | REVIEW_CHANGES (plan_code_review_test only)
+     pending_test             tester: TEST_PASS | TEST_FAIL; green gate or waiver
+     pending_final_review     manager final: COMPLETE | REVISE (REVISE goes back, cycle + 1)
+     merge_request            commit; sync with base (retest if base moved); merge into dev; push dev
+                              -> DeliveryHooks.on_branch_pushed(branch="dev", source="engine")
+     completed                MemoryPort.on_task_finished (its failure never fails the task)
+  side states: pending_user_decision, error_review (manager error mode: RESOLVE_ERROR | CANCEL),
+               paused (no credit or a budget cap), cancelled, failed
+```
+
+Pipelines (`app/domain/workforce/pipeline.py`, `PIPELINES`):
+- `plan_code_test` (default): manager review, analysis, implementation, test, final review, merge, completed.
+- `plan_code_review_test`: as above with a review stage after implementation; falls back to `plan_code_test`
+  when no reviewer is active.
+- `code_test`: no planner stage.
+- `analysis_only`: manager review, analysis, completed. No worktree and no merge.
+
+The AI never runs git. `RepoWorkspacePort` does commit, sync, merge and push. The `RUN` tool refuses
+`git push`, `git remote` and `git config`.
+
+## 3. Tag protocol (`app/application/engine/protocol.py`, `ALLOWED_TAGS`)
+
+| role / mode | allowed tags |
+|---|---|
+| manager / triage | ASSIGN, REJECT, ASK_USER, KNOWLEDGE |
+| manager / final | COMPLETE, REVISE, KNOWLEDGE |
+| manager / error | RESOLVE_ERROR, CANCEL, KNOWLEDGE |
+| manager / chat | CREATE_TASK, PAUSE_TASK, RESUME_TASK, CANCEL_TASK, HIRE, UPDATE_EMPLOYEE, FIRE, KNOWLEDGE |
+| planner / work | PLAN_SPEC, SPLIT, ASK_USER + toolkit |
+| developer / work | WORK_REPORT, ASK_USER + toolkit |
+| reviewer / work | REVIEW_PASS, REVIEW_CHANGES + toolkit |
+| tester / work | TEST_PASS, TEST_FAIL + toolkit |
+
+Toolkit (working roles only): `READ_FILE`, `WRITE_FILE`, `EDIT_FILE`, `LIST_DIR`, `SEARCH`, `RUN`.
+
+Limits: at most 3 `CREATE_TASK` per reply, 3 team changes per turn (HIRE, UPDATE_EMPLOYEE, FIRE together),
+3 `KNOWLEDGE` per turn. A disallowed tag is dropped and reported back to the model as an error line.
+
+## 4. Endpoints added or changed
+
+Engine (`app/interfaces/http/routers/engine.py`, prefix `/api/organizations/{org_id}/engine`):
+- `GET /status`: the engine status, including the wallet balance. Every member may read it, so they can see
+  why work stopped.
+- `GET /tasks`: the task board (engine tasks with stage, pipeline, kind, cycle and assignee).
+- `GET /tasks/{task_id}`: task detail with pipeline state, turns, gates, last diff, waiver reason and merged sha.
+- `GET /turns`: the organization's newest engine turns.
+- `POST /tasks/{task_id}/pause`, `/resume`, `/cancel`, `/answer`: resume is refused while the wallet is at zero.
+  A task of another organization is a 404.
+- `GET /team`: employees with role, tier, status and busy task. Non-owners see tier only, never a model.
+- `POST /team/hire` (role + tier) and `POST /team/{employee_id}/fire`.
+
+Owner credit console (`app/interfaces/http/routers/admin_credits.py`, platform owner only):
+- `GET /api/admin/credits`: the balance of every organization.
+- `POST /api/admin/orgs/{org_id}/credit`: a positive credit with a required reason, capped per request; writes one
+  history row of kind `owner_credit`.
+- `GET /api/admin/orgs/{org_id}/credit-history`.
+
+Billing lockdown (`app/interfaces/http/routers/billing.py`):
+- Changed: `POST /api/organizations/{org_id}/billing/add-credit` is platform-owner only. Org admins are refused.
+  A negative amount claws back, but never leaves the wallet below zero. It writes one `adjustment` row.
+- Changed: `POST /api/organizations/{org_id}/billing/payments/{payment_id}/settle` is platform-owner only. A
+  succeeded settle credits the wallet through the credit ledger as one `adjustment` row. An already-final
+  payment is returned untouched.
+- Unchanged: `checkout` only creates a pending payment.
+
+Workforce (`app/interfaces/http/routers/workforce.py`, `app/application/services/workforce_service.py`):
+- Changed: `POST /api/organizations/{org_id}/tasks` creates a pipeline state at `pending_manager_review` with
+  kind `feature`, so the engine picks the task up. The MCP `tasks_create` tool calls the same `create_task`.
+- Changed: the legacy run start (`WorkforceService.start_run`, the «اجرا» button) answers 409 for a task the
+  engine owns.
+
+## 5. Tables
+
+- `task_pipeline_state` (one row per engine task): pipeline, kind, risk, stage, cycle, max_cycles, branch,
+  last_diff_hash, same_diff_count, retry_count, waiver_reason, parent_task_id, depends_on (JSON), version
+  (optimistic lock), `last_diff` (capped at 20,000 characters), `merged_sha`.
+- `employee_profiles` (one row per employee): `ai_model_id` (internal; never returned to non-owners), effort,
+  roles, `hired_by` (`user` or `manager`), `busy_task_id`.
+- `engine_turns` (one row per employee turn): task, run, employee, role, mode, stage, prompt_chars,
+  `reply_excerpt` (at most 4,000 characters), tags (JSON), tool_calls, ok, error_kind, started_at, finished_at.
+- `gate_results` (one row per gate run): gate (`tests`, `diff_guard`, `no_progress`, `green`, `lint`), ok,
+  detail (at most 8,000 characters).
+- `credit_transactions` (`app/infrastructure/db/models_billing.py`): the history of each organization's prepaid
+  wallet, with owner adjustments and usage debits. Unique on `usage_event_id`.
+- Migration of the engine tables: `alembic/versions/a2b3c4d5e6f2_ai_workforce.py`.
+
+## 6. Settings (`app/core/config.py`)
+
+Engine (`ENGINE_*`):
+
+| setting | default | meaning |
+|---|---|---|
+| `ENGINE_MAX_TOOL_CALLS_PER_TURN` | 40 | tool calls allowed in one turn |
+| `ENGINE_MAX_PROTOCOL_RETRIES` | 3 | retries after a protocol error in one turn |
+| `ENGINE_TURN_TIMEOUT_SECONDS` | 900 | wall-clock limit for one turn |
+| `ENGINE_MAX_CONCURRENT_TURNS` | 8 | global cap on selected turns per tick |
+| `ENGINE_MAX_TURNS_PER_ORG` | 3 | per-organization cap on selected turns per tick |
+| `ENGINE_TESTS_TIMEOUT_SECONDS` | 600 | timeout for the project's test command |
+| `ENGINE_MAX_MERGE_CONFLICTS` | 3 | conflicts before `error_review` |
+| `ENGINE_MAX_TASK_CREDITS` | 0.0 | per-task spend cap; 0 disables it |
+| `ENGINE_MAX_CYCLES` | 6 | implementation cycles before `error_review` |
+| `ENGINE_MAX_EMPLOYEES` | 20 | platform ceiling on employees per organization |
+| `ENGINE_TICK_SECONDS` | 10.0 | pause between worker ticks in loop mode |
+| `ENGINE_PROMPT_CHAR_CAP` | 24000 | hard cap on one turn's prompt; the oldest tool results are cut first |
+| `ENGINE_TOOL_OUTPUT_CHARS` | 8000 | tail kept from one tool output |
+| `ENGINE_REPLY_EXCERPT_CHARS` | 4000 | reply text stored in `engine_turns` |
+| `ENGINE_LOG_FLUSH_SECONDS` | 0.5 | batching window for live log events |
+| `ENGINE_LOG_MAX_LINES_PER_EVENT` | 200 | lines per `run.log` event |
+
+Billing: `BILLING_STARTING_CREDIT` (default 0.0) is the credit a new organization starts with.
+
+The per-organization employee cap is 7 by default (the org policy, section 10). `ENGINE_MAX_EMPLOYEES` is only
+the ceiling.
+
+## 7. Realtime events
+
+- `task.step` (`engine/events.py`): on every stage move. Channels: `task:` and `org:`.
+- `task.gate`: after each gate result. Channels: `task:` and `org:`.
+- `run.log` and `run.tool`: batched run output and tool calls. Channels: `run:`, `task:` and `org:`.
+- `team.changed`: on hire, fire and update. Channel: `org:`.
+- `task.created`: when the manager creates a task (CREATE_TASK) and when `create_task` runs. Channel: `task:`.
+- `run.status` is not published for engine runs (open 2.3).
+
+## 8. CHANGE REQUESTs
+
+None open. The two expected CRs have landed:
+- The org policy is on `ModelGrantPort` (`app/application/ports/model_grants.py`, `get_org_policy`).
+- The memory expiry hook runs in the billing worker pass (`app/workers/billing.py`, `MemoryRepository.expire`).
+
+## 9. Manual QA (owner, on a real host)
+
+These steps are not run by the agent. The owner runs them:
+1. Owner credits an organization at `/admin/credits`.
+2. Order work in `/manager`: the task cards appear inline, and the `/tasks` board moves live.
+3. Open a task: stepper, turn timeline, live log, gates and diff. Pause, resume and cancel (each asks to confirm).
+4. At zero credit, work pauses and resume is refused until credit is added.
+5. On `/workforce`, hire and fire by role and tier. No model name appears for a member.
+6. A pushed task shows its short sha, with a link to `/app`.
+
+## 10. Changes outside the original TODO (owner-approved)
+
+- Settle credits through the ledger: `BillingService.settle_payment` calls `credit_ledger.apply`.
+- Wallet debit on usage: every usage row debits its organization's wallet once (`UsageLedgerRepository.record`,
+  keyed on the usage row id).
+- Prepaid pause: a zero, negative or missing wallet pauses engine turns and the manager chat
+  (`Scheduler._budget_reason`, `balance <= 0`).
+- Employee cap 7 per organization by default (`DEFAULT_MAX_EMPLOYEES` in `model_grant_service.py`); the admin
+  page default is 7 too.
+- Owner credit page `/admin/credits`, with credit history.
+- Direct tasks enter the engine: kind `feature` at `pending_manager_review`. The legacy run refuses them.
+- Pause, resume, cancel and answer, in the API and in the UI (with confirm).
+- Reference count in the web realtime client (`apps/web/src/lib/ws.ts`): a channel stays subscribed while any
+  live hook still holds it. The server hears about it only when the last holder leaves, and a reconnect replays
+  only the channels still held.
+
+## 11. Known limitations and leftovers
+
+Deliberate limitations:
+- Turns run one after another inside a tick (open 4.1). The caps bound what a tick selects, not how many run at once.
+- The stored diff is capped at 20,000 characters.
+- A budget-paused task is not resumed automatically when credit is added. Resume it by hand; it is refused while
+  the credit is still zero.
+- Invoices from the billing worker are informational under prepaid. They debit nothing; the usage ledger debits
+  the wallet.
+- The manager chat is also refused at zero credit, because it costs model calls.
+
+Open TODO items:
+- 2.3: `run.status` is not published for engine runs. `engine/scheduler.py` sets the run row's status without an
+  event, so stage moves (`task.step`) are the only live signal.
+- 4.1: no concurrent turns inside a tick (see above).
+- 4.6: `tests/test_engine_flow.py` has no max-cycles test (`max_cycles` reaching `error_review`) and only one
+  merge conflict (retry), not three. Its no-progress case is only a passing gate row. The blocking no-progress
+  case is in `tests/test_engine_gates.py`. Cancel mid-turn is in `tests/test_engine_runner.py`. The dependency
+  failure rule is in `tests/test_engine_pipeline.py`.
+
+Minor leftovers, known and not fixed:
+- The billing page's payments card shows the payment adapter name (`mock`) to members.
+- Paused tasks do not highlight the stage they paused at.
+- Engine task pages do not show legacy approval cards.
+- Two concurrent settles of one payment could credit twice: `PaymentRepository.set_status` is unconditional.
+- The generic Persian word «مدل» appears in two non-owner hints.
+- Manual real-environment QA (docker, caddy, ssh/ftp) is for the owner on a real host.
+
+## 12. Deviations from the TODO wording
+
+The behaviour is the same; the names or channels differ:
+- 1.1: `TASK_KINDS` is the `TaskKind` enum plus `TASK_KIND_DEFAULTS` (`app/domain/workforce/pipeline.py`).
+- 2.1: the memory context is built in `runner.py` (`_build_prompt`, budget 2,500) and passed to `prompt.py`.
+- 3.1: `post_user_message` is `ManagerTurns.chat` (`app/application/engine/manager.py`).
+- 5.4: the live log subscribes to `task:<id>` (`apps/web/src/app/(app)/tasks/[id]/_components/live-log.tsx`),
+  not `run:<id>`. The `task:` channel also carries `run.log`.
@@ -22,7 +22,7 @@ and nothing reaches production without a human click.
 | 1 | `DockerSandboxAdapter` (hardened workspace + demo containers), `process.py` (the single hardened subprocess runner), orphan reaping, infra hidden from non-owners |
 | 2 | `LocalGitWorkspaceAdapter` — the platform's own git: per-project gitdir, worktrees per task, per-repo lock, hooks disabled, credentials only ever as a *path* |
 | 3 | Live CI/CD: pipeline runs/steps/releases/runtimes, build containers, blue/green demo deploy, live batched logs, remote polling |
-| 4 | Owner domain console: list/attach/detach/resync routes, custom FQDNs behind a setting, MCP gateway system route |
+| 4 | Owner domain console: list/attach/detach/resync routes, custom FQDNs behind a setting, MCP gateway as a read-only suggestion (`GET /suggested`; the `mcp` label is protected, so it cannot be attached) |
 | 5 | Production deploy: `asyncssh` + `aioftp` adapters with host-key TOFU, atomic symlink switch, FTP backup-first, `production_deploys` ledger |
 | 6 | The Application page (four tabs + live build console + production panel), `/admin/domains`, `/admin/sandbox`, `/projects* → /app` redirect, typed client + contracts |

@@ -98,7 +98,7 @@ All paths are under `/api`.

 | Method | Path | Notes |
 |---|---|---|
-| GET | `` | every route, app routes and system routes |
+| GET | `` | every attached app route. The MCP gateway host is a read-only suggestion (`/suggested`), not a route the owner can enable: the label `mcp` is protected |
 | POST | `` | attach a host (label, or a custom FQDN when the setting is on) |
 | DELETE | `/{domain_id}` | detach, host typed back as confirmation |
 | POST | `/sync` | idempotent re-sync from the database |
@@ -125,9 +125,12 @@ All four endpoints (ports, domains create/list/delete) became platform-owner onl
 ## 5. Tables (all in `models_delivery.py`)

 `pipeline_runs`, `pipeline_steps`, `releases`, `app_runtimes`,
-`production_deploys`, `system_routes`. Migration
+`production_deploys`. Migration
 `a3b4c5d6e7f3_delivery_infra` (`down_revision = "a0f0e1d2c3b4"`).

+The MCP gateway has no table: `GET /api/admin/domains/suggested` computes its
+suggestion from settings.
+
 ---

 ## 6. Manual QA script
@@ -157,8 +160,8 @@ Run against a real docker + caddy host with
 8. **Production, FTP**: the same against a `pure-ftpd` container; expect a
   `ftp-backup-<ts>.tar.gz` and, on rollback, that archive re-uploaded.
 9. **Domains**: as the owner attach a second label; re-sync twice and check the
-   second call reports `up_to_date: true` with no Caddy reload. Enable the MCP
-   system route; `mcp.<root>` must answer through the gateway port.
+   second call reports `up_to_date: true` with no Caddy reload. Check that
+   `GET /api/admin/domains/suggested` shows `mcp.<root>` as a suggestion (`enabled=false`).
 10. **SSRF**: attempt a production deploy to `127.0.0.1`, `10.0.0.1` and
    `169.254.169.254`. All three must be refused before any connection.
 11. **Tenant isolation**: a member of org B gets 404 on every org-A delivery and
@@ -213,3 +216,5 @@ Run against a real docker + caddy host with
  (D6), but it also means Agent 2's engine must go through `RepoWorkspacePort`
  for all of it, and a git failure surfaces to the customer as a Persian message
  rather than a git exit code.
+- **No sandbox restart.** `/admin/sandbox` lists and stops containers; there is
+  no restart action (TODO 6.3 open).