docs: v2 plan (shared plan + foundation + 3 agent TODO files)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
mjalmoudy94
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
8eb631aedf
commit
3a68b0fa07
@@ -0,0 +1,266 @@
|
||||
# Smart Org v2 — Plan (shared by all agents)
|
||||
|
||||
Date: 2026-10-13 · Base branch: `dev` · Read this file fully before your own TODO file.
|
||||
|
||||
This file is the **single source of truth** for the v2 work. Three agents work in parallel, each from
|
||||
its own TODO file:
|
||||
|
||||
| File | Agent | Area |
|
||||
|---|---|---|
|
||||
| `docs/plan/TODO-0-foundation.md` | Claude (runs FIRST, before the agents) | shared contracts, DB foundation, wiring |
|
||||
| `docs/plan/TODO-agent-1-org-platform.md` | Agent 1 | invites, org wizard, app profile, sharing, owner model grants, memory, MCP |
|
||||
| `docs/plan/TODO-agent-2-ai-workforce.md` | Agent 2 | admin (manager) AI, employees, dynamic pipelines, agent loop, test/commit/push |
|
||||
| `docs/plan/TODO-agent-3-delivery-infra.md` | Agent 3 | docker sandbox, git workspaces, live CI/CD, demo server, Caddy domains, SSH/FTP deploy |
|
||||
|
||||
---
|
||||
|
||||
## 1. Product vision (what the owner asked for)
|
||||
|
||||
1. **One platform owner** controls the whole application and its infrastructure: AI accounts, git,
|
||||
application rules, adding users, domains, sandboxes, models. Nobody else touches infra.
|
||||
2. An **invited user** lands on an **organization creation wizard**. The wizard connects the org to **our
|
||||
git (Gitea)** or **the user's own git** and asks for the application type, language, framework, and so on.
|
||||
**One organization = exactly one application.** A user can create several orgs and can **share** an
|
||||
org with other people so they help manage it.
|
||||
3. The wizard also asks (optionally) for **SSH or FTP** details to deploy the app to **production**.
|
||||
The platform can also run the app inside a **sandbox as a live demo**.
|
||||
4. The user gives orders to the **admin AI (manager)**. The manager brings in **AI employees**, built
|
||||
only from the **models the owner grants to that org**. They do the task, **test it and push it**.
|
||||
5. A **live CI/CD** mechanism shows the deployment on our local **demo sandbox server**, live.
|
||||
6. **Caddy** routes different domains to the applications.
|
||||
7. This is the **web version of `/projects/agybot`** (Telegram bot). Reuse agybot's proven ideas, not its code.
|
||||
8. Every org **improves itself** through **memories**: how its users like to manage the org, and how the
|
||||
app's logic and code work. It also exposes **MCP** so AI agents can control our app and read or push data.
|
||||
9. **Main difference from agybot:** users never touch infrastructure or AI accounts. They only give
|
||||
orders and tasks. Tasks run through **dynamic pipelines**.
|
||||
|
||||
## 2. What already exists (audited 2026-10-13, the code is the truth, not old TODO.md ticks)
|
||||
|
||||
- **Transport:** the API is WebSocket-RPC only. `app/main.py` exposes only `/health*` and `/api/ws`. Every
|
||||
handler lives in an internal FastAPI app (`app/interfaces/realtime/rpc_app.py`) that is reached
|
||||
in-process. Realtime push uses channels `org:<id>`, `task:<id>`, `run:<id>`, `thread:<id>`,
|
||||
`session:<id>`, `user:<id>`, `admin` (`app/application/ports/realtime.py`, authz in
|
||||
`app/interfaces/realtime/authz.py`).
|
||||
- **Roles:** platform roles `owner` (exactly one, enforced by a DB partial unique index), `manager` and
|
||||
`user`. Org roles are `owner > admin > manager > member > viewer`. There is no public signup. Accounts
|
||||
come from the owner (`/admin/users`) or from an org invite link.
|
||||
- **Admin console:** agy (Google OAuth) accounts with pool rotation and 429 cooldown, plus model
|
||||
catalog, routing policies, rate cards, usage and error console.
|
||||
- **Projects:** `projects` holds only name/slug/description/default_branch. There is no app profile and
|
||||
no "1 org = 1 app" rule. Gitea adapter (real) and mock. Preview domains through the Caddy adapter
|
||||
(real) and mock.
|
||||
- **Sandbox:** full CRUD API, but only Daytona/E2B adapters (untested live) and a mock that runs nothing.
|
||||
- **Workforce:** templates, employees, threads, tasks, runs. **The run engine makes ONE LLM call per
|
||||
task and stores the text.** Tool calls are recorded, never executed. No employee edits code, runs
|
||||
tests, commits or pushes.
|
||||
- **Missing entirely:** org wizard, app profile, external git, SSH/FTP deploy, CI/CD, demo server, docker
|
||||
sandbox, memory system, MCP, per-org model grants.
|
||||
- **Tests:** about 233 API tests (pytest, DB via `Base.metadata.create_all` in `tests/conftest.py`) and 64
|
||||
web tests (vitest). Five-step alembic chain ending at `d4e5f6a7b8c9`.
|
||||
|
||||
## 3. What we take from agybot (concepts only, never import it)
|
||||
|
||||
| agybot | v2 equivalent | Owner |
|
||||
|---|---|---|
|
||||
| Roles `manager, planner, programmer, reviewer, tester` (+ optional critic) | same role keys | Agent 2 |
|
||||
| `[[TAG attr="..."]]body[[/TAG]]` protocol, per-role allowed tags | same, in `app/application/engine/protocol.py` | Agent 2 |
|
||||
| Pipelines `plan_code_test` (default), `plan_code_review_test`, `code_test`, `analysis_only` | same names | Agent 2 |
|
||||
| Pipeline choice: manager's `pipeline=` > task-kind default > app default; review pipeline needs an active reviewer | same | Agent 2 |
|
||||
| Task statuses `pending_manager_review → pending_analysis → pending_implementation → pending_review → pending_test → pending_final_review → merge_request → completed` (+ `pending_user_decision`, `error_review`, `paused`, `cancelled`, `failed`) | same, with optimistic `expected_from` | Agent 2 |
|
||||
| Gates: real test run after WORK_REPORT, green gate (waiving needs a written reason), diff guard, no-progress stop (2 identical diffs), max cycles, transient-error retries, escalate model on red | same | Agent 2 |
|
||||
| Worktree per task `task/<id>`, merge into `dev`, push | same, through `RepoWorkspacePort` | Agent 3 (port), Agent 2 (use) |
|
||||
| Manager `HIRE/UPDATE_EMPLOYEE/FIRE` (max 3 changes per turn, max 20 employees), only on org-shared models | same, only on **owner-granted** models | Agent 2 |
|
||||
| Org knowledge `[[KNOWLEDGE]]` (max 3 per turn, caps 1,500 chars/key and 6,000/org), lessons with 30-day TTL, project summaries per task | memory system | Agent 1 |
|
||||
| `app_control_mcp` / `remote_dev_mcp` | MCP gateway with scoped tokens | Agent 1 |
|
||||
| `caddy_manager` (single-label hosts under root, localhost targets, validate→revert→reload) | existing `CaddyDomainRouterAdapter` | Agent 3 |
|
||||
| `ssh_manager.push_project`, `ftp_manager.deploy_project` | deploy adapters | Agent 3 |
|
||||
| `org_live_stream` (ring buffer, polled) | realtime push events (no polling) | Agents 2 & 3 |
|
||||
|
||||
**agybot bugs we must NOT repeat:** deploy publishing the wrong code silently; `git_publish` stuck
|
||||
mid-merge; observe mode applying real state changes; a failed/cancelled dependency deadlocking its
|
||||
dependents; manager chat cost not counted in the budget; git operations without the repo lock; about
|
||||
870 silent `except Exception` blocks; plaintext credentials; the AI running directly on the host;
|
||||
`git status --porcelain` hiding new dirs (always use `-uall`).
|
||||
|
||||
## 4. Architecture decisions (fixed, do not re-decide)
|
||||
|
||||
- **D1. 1 org = 1 app.** The app is the existing `projects` row (the table name stays). `projects` gets
|
||||
`UNIQUE(organization_id)`. The UI calls it "Application" (اپلیکیشن). Route `/projects` becomes `/app`.
|
||||
- **D2. App profile.** These are columns on `projects`: app_type, language, framework, runtime_version,
|
||||
package_manager, setup/build/test/start commands, app_port, healthcheck_path, git_source
|
||||
(`platform|external`), external_repo_url, wizard_completed_at. External git credentials and deploy
|
||||
targets live in their own tables, **encrypted at rest** with the existing crypto helper. They are never
|
||||
returned raw to the frontend (only `has_secret: true` and masked values).
|
||||
- **D3. Owner model grants.** `org_model_grants(org_id, ai_model_id, daily_credit_cap, is_active)`. The
|
||||
engine may only hire or run on granted models. Users never see model names; the UI shows the
|
||||
employee's role and a neutral "tier" label.
|
||||
- **D4. Execution engine = in-process agent loop.** It uses the existing `ModelProviderPort` (owner's agy
|
||||
accounts, pool rotation) with the agybot tag protocol for actions **and** tools. Tools run **inside the
|
||||
org's sandbox container** through `SandboxProviderPort`. There are no agent CLIs in v1, so AI
|
||||
credentials never leave the server and users never see them.
|
||||
- **D5. Sandbox = local Docker** (`SANDBOX_PROVIDER=docker`, the binary is configurable so podman works
|
||||
too). Per app there is one **workspace container** (employees work in it) and one **demo container**
|
||||
(runs the latest green build). Hardened: non-root, `--cap-drop ALL`, `no-new-privileges`, cpu/mem/pids
|
||||
limits, ports published on `127.0.0.1` only.
|
||||
- **D6. Git is done by the platform, never by the AI.** `RepoWorkspacePort` runs git **on the host** with
|
||||
a gitdir that is **never mounted** into the container. Git is always called with explicit
|
||||
`--git-dir/--work-tree`, hooks disabled (`core.hooksPath=/dev/null`), `-uall`, `core.quotePath=false`,
|
||||
and a per-repo lock. Remote credentials go through `GIT_ASKPASS` or a temporary key file and **never
|
||||
appear in a URL, log or exception**.
|
||||
- **D7. CI/CD is platform-native.** A push to `dev` (by the engine or seen by polling the remote) creates
|
||||
a `pipeline_run`. Steps: checkout → setup → build → test → package → deploy demo → healthcheck → route.
|
||||
Logs stream live as realtime events on `org:<id>` and a bounded tail is stored. A failed run keeps the
|
||||
previous demo running.
|
||||
- **D8. Production deploy is always human-triggered** (org owner/admin, with confirmation and audit).
|
||||
SSH: release dir + `current` symlink + post-deploy command + rollback. FTP/FTPS: upload a release, keep
|
||||
the previous one for rollback. User-supplied hosts are SSRF-checked (no private, loopback or link-local
|
||||
IPs unless the owner allowlists them). SSH host key is pinned on first successful test (TOFU).
|
||||
- **D9. Domains.** Each app gets an automatic demo host `<org-slug>.<PREVIEW_ROOT_DOMAIN>`. Only the
|
||||
**owner** can attach extra hosts. Targets are always `127.0.0.1:<allocated port>`. The existing rules in
|
||||
`app/domain/platform/rules.py` stay authoritative.
|
||||
- **D10. Memory** (`org_memories`) is the org's long-term self-knowledge: rules, preferences, lessons,
|
||||
facts and the code map, with caps, dedupe, TTL for lessons and pinning. Users can view, edit, pin and
|
||||
delete it. The engine reads it through `MemoryPort.build_context` and feeds it through
|
||||
`MemoryPort.on_task_finished`.
|
||||
- **D11. MCP gateway** is a separate HTTP entry point (`services/mcp-gateway`) using the official `mcp`
|
||||
Python SDK (streamable HTTP). Bearer tokens are hashed, scoped, org-bound and expire. Every call is
|
||||
audited and org-enforced.
|
||||
- **D12. Platform invites.** The owner can create an invite link for a **new platform account**, which
|
||||
becomes role `manager` and may create orgs. A logged-in manager with no org is sent to the wizard. Org
|
||||
invites (join an existing org) stay as they are.
|
||||
- **D13. Automation safety.** Tasks run automatically up to merge into `dev`, followed by a demo deploy.
|
||||
Nothing goes to production without a human click. No code path may flip that default.
|
||||
|
||||
## 5. Contracts between agents (created by Claude in TODO-0, then FROZEN)
|
||||
|
||||
All contracts live in `apps/api/app/application/ports/` and are `typing.Protocol`s with a
|
||||
working default implementation, so each agent can test alone. **Agents must not change a contract
|
||||
file.** If you need a change, write a `CHANGE REQUEST` in your phase report, keep coding against the
|
||||
current signature, and add a local adapter or shim in your own files.
|
||||
|
||||
```python
|
||||
# ports/repo_workspace.py — implemented by Agent 3, used by Agent 2 (and Agent 1 for test_remote)
|
||||
class RepoWorkspacePort(Protocol):
|
||||
async def ensure_workspace(self, *, project_id: str) -> WorkspaceRef: ... # clone/fetch app repo on host
|
||||
async def create_task_worktree(self, *, project_id: str, task_id: str, base: str = "dev") -> WorktreeRef: ...
|
||||
async def diff(self, *, worktree: WorktreeRef) -> DiffResult: ... # text + changed files (-uall)
|
||||
async def commit(self, *, worktree: WorktreeRef, message: str, author: str) -> str | None: ... # sha or None if clean
|
||||
async def sync_with_base(self, *, worktree: WorktreeRef) -> MergeOutcome: ... # merge base into task branch
|
||||
async def merge_into(self, *, project_id: str, branch: str, target: str = "dev") -> MergeOutcome: ...
|
||||
async def push(self, *, project_id: str, branch: str) -> PushOutcome: ...
|
||||
async def remove_worktree(self, *, worktree: WorktreeRef) -> None: ...
|
||||
async def head(self, *, project_id: str, branch: str) -> str | None: ...
|
||||
async def test_remote(self, *, url: str, credentials: RemoteCredentials) -> RemoteCheck: ...
|
||||
# WorktreeRef has: project_id, task_id, branch, host_path, sandbox_path (path inside the workspace container)
|
||||
|
||||
# ports/delivery_hooks.py — implemented by Agent 3, called by Agent 1 (app created) and Agent 2 (pushed)
|
||||
class DeliveryHooks(Protocol):
|
||||
async def on_app_created(self, *, organization_id: str, project_id: str) -> None: ...
|
||||
async def on_branch_pushed(self, *, organization_id: str, project_id: str, branch: str, sha: str, source: str) -> None: ...
|
||||
|
||||
# ports/deploy_target.py — implemented by Agent 3, called by Agent 1's wizard
|
||||
class DeployTargetTester(Protocol):
|
||||
async def test_connection(self, *, target: DeployTargetSpec) -> DeployCheck: ... # ok, message, fingerprint
|
||||
|
||||
# ports/memory.py — implemented by Agent 1, used by Agent 2
|
||||
class MemoryPort(Protocol):
|
||||
async def build_context(self, *, organization_id: str, role: str, query: str, budget_chars: int = 2500) -> str: ...
|
||||
async def record(self, *, organization_id: str, kind: str, content: str, source: str,
|
||||
evidence_task_id: str | None = None, key: str | None = None) -> str | None: ...
|
||||
async def on_task_finished(self, *, organization_id: str, task: TaskOutcome) -> None: ...
|
||||
|
||||
# ports/model_grants.py — implemented by Agent 1, used by Agent 2
|
||||
class ModelGrantPort(Protocol):
|
||||
async def granted_models(self, *, organization_id: str) -> list[GrantedModel]: ... # id, capabilities, tier, caps
|
||||
async def is_granted(self, *, organization_id: str, ai_model_id: str) -> bool: ...
|
||||
|
||||
# ports/workforce_bootstrap.py — implemented by Agent 2, called by Agent 1's wizard
|
||||
class WorkforceBootstrapPort(Protocol):
|
||||
async def bootstrap_team(self, *, organization_id: str, project_id: str) -> None: ... # creates the manager + default team
|
||||
```
|
||||
|
||||
Existing contracts that stay as they are: `SandboxProviderPort` (Agent 3 adds the docker adapter),
|
||||
`GitProviderPort`, `DomainRouterPort`, `ModelProviderPort`, `EventPublisher`/`RealtimeEvent`.
|
||||
|
||||
**Realtime event names** (use the existing `RealtimeEvent` + `get_publisher()`):
|
||||
|
||||
| Channel | Event types | Producer |
|
||||
|---|---|---|
|
||||
| `task:<id>` | `task.status`, `task.step`, `task.gate` | Agent 2 |
|
||||
| `run:<id>` | `run.log`, `run.tool`, `run.status` | Agent 2 |
|
||||
| `org:<id>` | `task.created`, `team.changed` | Agent 2 |
|
||||
| `org:<id>` | `pipeline.queued`, `pipeline.step`, `pipeline.log`, `pipeline.status`, `release.live`, `deploy.status` | Agent 3 |
|
||||
| `org:<id>` | `memory.changed`, `org.updated` | Agent 1 |
|
||||
|
||||
Log events are **batched** (at most one event per 500 ms per stream, max 200 lines per event).
|
||||
|
||||
## 6. File ownership (no two agents edit the same file)
|
||||
|
||||
| Area | Agent 1 | Agent 2 | Agent 3 |
|
||||
|---|---|---|---|
|
||||
| API routers | `app_setup.py`*, `memory.py`*, `mcp_tokens.py`*, `admin_grants.py`*, `organizations.py`, `invites.py`, `admin_users.py`, `auth.py`, `knowledge.py` | `workforce.py` | `projects.py`, `sandbox.py`, `delivery.py`*, `admin_domains.py`* |
|
||||
| Services | `org_setup_service.py`*, `memory_service.py`*, `model_grant_service.py`*, `mcp_service.py`*, `platform_users.py`, `knowledge_service.py` | `workforce_service.py`, `app/application/engine/**`* | `project_service.py`, `sandbox_service.py`, `delivery_service.py`*, `deploy_service.py`* |
|
||||
| Adapters | — | — | everything in `app/infrastructure/adapters/` (incl. `registry.py`, `mock.py`) |
|
||||
| Repositories | `identity.py`, `knowledge.py`, `memory.py`* | `workforce.py` | `projects.py`, `sandbox.py`, `delivery.py`* |
|
||||
| Domain | `identity/`, `knowledge/`, `memory/`*, `projects/catalog.py`* | `workforce/` | `platform/`, `sandbox/`, `projects/models.py` |
|
||||
| DB tables | `db/models_org.py`* | `db/models_engine.py`* | `db/models_delivery.py`* |
|
||||
| Wiring | `container_org.py`* | `container_engine.py`* | `container_delivery.py`* |
|
||||
| Workers | — | `workers/ai_runtime.py` | `workers/sandbox_gateway.py`, `workers/delivery.py`* |
|
||||
| Other API | `app/interfaces/mcp/**`*, `services/mcp-gateway/**` | — | — |
|
||||
| Web pages `apps/web/src/app/` | `onboarding/`, `accept-invite/`, `(app)/settings/`, `(app)/memory/`*, `(app)/knowledge/`, `(app)/admin/users/`, `(app)/admin/model-grants/`*, `(app)/admin/mcp/`*, `(app)/admin/organizations/` | `(app)/manager/`, `(app)/tasks/`, `(app)/workforce/`, `(app)/traces/`, `(app)/dashboard/` | `(app)/app/`*, `(app)/projects/` (delete after moving), `(app)/admin/domains/`*, `(app)/admin/sandbox/`*, `components/sandbox-panel.tsx` |
|
||||
| Web API client | `lib/api-org.ts`* | `lib/api-engine.ts`* | `lib/api-delivery.ts`* |
|
||||
| Tests | `tests/test_org_*.py`, `test_memory*.py`, `test_mcp*.py`, `test_model_grants*.py`, `test_invites.py`, `test_platform_hierarchy.py`, web `src/test/org-*.test.tsx` | `tests/test_engine_*.py`, `test_workforce*.py`, `test_workers.py`, web `src/test/engine-*.test.tsx` | `tests/test_docker_*.py`, `test_git_workspace*.py`, `test_delivery*.py`, `test_deploy*.py`, `test_project_repo.py`, `test_domain_isolation.py`, `test_platform_rules.py`, web `src/test/delivery-*.test.tsx` |
|
||||
| Alembic (one file per agent, more only if chained off your own) | `a1b2c3d4e5f1_org_platform.py` | `a2b3c4d5e6f2_ai_workforce.py` | `a3b4c5d6e7f3_delivery_infra.py` |
|
||||
|
||||
`*` = new file (Claude creates an empty stub in TODO-0, the agent fills it).
|
||||
|
||||
**Locked files (Claude only):** `db/models.py`, `container.py`, `rpc_app.py`, `workers/__main__.py`,
|
||||
`ports/*.py` contracts, `tests/conftest.py`, `pyproject.toml`, `package.json`, `pnpm-lock.yaml`,
|
||||
`apps/web/src/lib/api.ts`, `apps/web/src/lib/nav.ts`, `components/AppShell.tsx`, `alembic/env.py`, root
|
||||
`README.md`/`PLAN.md`/`TODO.md`, `docs/plan/PLAN-v2.md`, every other agent's TODO file.
|
||||
|
||||
**Append-only blocks.** `app/core/config.py` and `.env.example` each have three marked blocks
|
||||
(`# >>> agent-1` … `# <<< agent-1`, and so on). You may add lines **only inside your own block**.
|
||||
|
||||
Each new migration uses `down_revision = "a0f0e1d2c3b4"` (the foundation). Claude adds the merge
|
||||
revision when the branches are merged.
|
||||
|
||||
## 7. Working rules for every agent
|
||||
|
||||
1. Work **only** in your own git worktree and branch (given in your TODO file). Never `cd` outside it and
|
||||
never write to `/tmp`. Put scratch scripts in `./_scratch/` inside your worktree and delete them at the end.
|
||||
2. **Do NOT commit, push, merge or rebase.** Claude reviews and commits each phase.
|
||||
3. At the **end of each phase**, stop and write `docs/plan/reports/agent-<N>-phase-<K>.md` containing:
|
||||
what you did, files changed, test output summary, deviations, open questions, and `CHANGE REQUEST`s.
|
||||
Then wait.
|
||||
4. Before declaring a phase done, run **all** checks (not only your tests) and paste the summary lines:
|
||||
```bash
|
||||
cd apps/api && .venv/bin/python -m pytest -q && .venv/bin/ruff check . && .venv/bin/mypy app
|
||||
cd apps/web && pnpm test && pnpm exec tsc --noEmit && pnpm lint
|
||||
```
|
||||
Every existing test must stay green. Never delete or weaken an existing test to make it pass. If an
|
||||
existing test encodes behaviour that this plan changes on purpose, update it and explain why in the report.
|
||||
5. Every multi-tenant query is filtered by `organization_id`. Every destructive or production action has
|
||||
a confirmation and an audit event. No secret ever goes to the frontend, a log, an exception message, a
|
||||
realtime event or an LLM prompt.
|
||||
6. No silent `except Exception: pass`. Catch narrow errors, or record them through
|
||||
`app.infrastructure.error_log`.
|
||||
7. Users (non-owner) never see model names, providers, accounts, containers, ports or raw infra
|
||||
errors. Translate them to plain Persian messages.
|
||||
8. UI language is Persian and RTL. Reuse `packages/ui` components and the existing motion classes.
|
||||
9. Match the existing code style: hexagonal layers (domain has no FastAPI/SQLAlchemy imports), type
|
||||
hints everywhere, docstrings in the same tone as nearby code.
|
||||
10. Do not run `agy`, do not call real external services in tests (use fakes). Integration tests that need
|
||||
docker, network or real git remotes must be skipped unless an env flag is set.
|
||||
|
||||
## 8. Integration (Claude, after the agents)
|
||||
|
||||
1. Review each phase report and diff, run the full checks, and commit on the agent's branch.
|
||||
2. Merge the branches into `dev` one by one: Agent 3 → Agent 1 → Agent 2. Run the full tests after **each**
|
||||
merge.
|
||||
3. Add the alembic merge revision for the three heads and run `alembic upgrade head` + `alembic check`.
|
||||
4. Replace the Null/default contract implementations in `container.py` with the real ones and run the
|
||||
end-to-end smoke test:
|
||||
wizard → app created → demo domain → manager order → task runs → tests → push → pipeline → demo live
|
||||
→ production deploy (to a local SSH/FTP test server).
|
||||
5. Update `README.md`, `TODO.md` and the architecture doc, then commit.
|
||||
Reference in New Issue
Block a user