smart-org/docs/plan/reports/agent-3-final.md

# Agent 3 — Final report: Delivery & Infra (phases 1–6)

Date: 2026-10-14 · Branch: `dev` · Supersedes and summarises
`agent-3-phase-1.md` … `agent-3-phase-6.md`.

This file is the integration hand-off: what exists, what it needs on the host,
every setting and endpoint it added, the tables it owns, the change requests still
open, the manual QA script, and the things deliberately not done.

The golden property, stated once because every design decision serves it: **the AI
never touches the host, the network, or a credential.** Git runs as the platform
user with a gitdir that is never mounted into a container; code runs inside
hardened containers; user-supplied hosts pass an SSRF check before a byte leaves;
and nothing reaches production without a human click.

---

## 1. What exists

| Phase | Delivered |
|---|---|
| 1 | `DockerSandboxAdapter` (hardened workspace + demo containers), `process.py` (the single hardened subprocess runner), orphan reaping, infra hidden from non-owners |
| 2 | `LocalGitWorkspaceAdapter` — the platform's own git: per-project gitdir, worktrees per task, per-repo lock, hooks disabled, credentials only ever as a *path* |
| 3 | Live CI/CD: pipeline runs/steps/releases/runtimes, build containers, blue/green demo deploy, live batched logs, remote polling |
| 4 | Owner domain console: list/attach/detach/resync routes, custom FQDNs behind a setting, MCP gateway as a read-only suggestion (`GET /suggested`; the `mcp` label is protected, so it cannot be attached) |
| 5 | Production deploy: `asyncssh` + `aioftp` adapters with host-key TOFU, atomic symlink switch, FTP backup-first, `production_deploys` ledger |
| 6 | The Application page (four tabs + live build console + production panel), `/admin/domains`, `/admin/sandbox`, `/projects* → /app` redirect, typed client + contracts |

Phase 5's adapters are **lazy imports**. `asyncssh` and `aioftp` are dependencies,
but importing them costs real start-up time and neither is needed unless somebody
configures a deploy target. `deploy_target.py` therefore resolves them inside
`_connect()`, which is also what keeps `mypy` honest about an optional import.

---

## 2. Host requirements

| Requirement | Why | Notes |
|---|---|---|
| `docker` **or** `podman` on `PATH` (`SANDBOX_DOCKER_BIN`) | the only sanctioned way tenant code runs | the flags used are the common subset; the workspace/demo/build containers are `--read-only`, non-root, `--cap-drop ALL`, `no-new-privileges`, cpu/mem/pids capped |
| `git` ≥ 2.20 on `PATH` | the platform runs git itself (D6) | worktrees + `--git-dir`/`--work-tree` |
| `caddy` on `PATH` **and** `import /etc/caddy/agy-preview/*.caddy` in the Caddyfile (`DOMAIN_ROUTER=caddy`) | preview + custom domains | the adapter **refuses to run** if the import line is missing — silently writing a file Caddy never reads would report a success that is false |
| Caddy reload permission | saving a route reloads the live proxy | `caddy reload` needs to signal the caddy service; on a systemd host that is a polkit/sudoers entry, **not** root. The command used is `caddy reload --config <file>`, so granting the service user passwordless `caddy reload` is sufficient. |
| Writable, disk-rich dirs under `SANDBOX_WORKSPACES_ROOT` / `ARTIFACTS_ROOT` | work trees, per-run copies, artifacts, tarballs | default `./.data/...` (gitignored). Per-run copies mean disk grows with project size × runs — see Known limitations |
| `ftps` targets | explicit TLS | a plain `ftp` target is accepted and documented as unencrypted |
| `DEPLOY_IT=1` / `SANDBOX_DOCKER_IT=1` | integration tests | both skip cleanly otherwise |

Nothing else runs on the host. There is no `shell=True` anywhere in the
workstream; `app/infrastructure/adapters/process.py` is the only module that calls
`asyncio.create_subprocess_exec`.

---

## 3. Settings (agent-3 block, `app/core/config.py`)

| Setting | Default | Meaning |
|---|---|---|
| `SANDBOX_DOCKER_BIN` | `docker` | container runtime CLI (`podman` works) |
| `SANDBOX_WORKSPACES_ROOT` | `./.data/workspaces` | work trees, gitdirs, run dirs |
| `SANDBOX_DEFAULT_CPUS` / `_MEMORY_MB` | `1.0` / `1024` | container caps |
| `DEMO_PORT_START` / `DEMO_PORT_END` | `8100` / `8999` | demo port pool (inside the managed range) |
| `ARTIFACTS_ROOT` | `./.data/artifacts` | release tarballs, FTP backups |
| `DELIVERY_POLL_SECONDS` | `60` | how often remotes are polled for outside pushes |
| `DEPLOY_ALLOW_PRIVATE_HOSTS` | `false` | escape hatch for a private-network target; **off by default** |
| `GIT_WORKSPACE` | `fake` | `local` enables the real host git adapter |
| `ALLOW_CUSTOM_DOMAINS` | `false` | owner-supplied external FQDNs |
| `GIT_ALLOW_FILE_PROTOCOL` | `false` | `protocol.file.allow=never`; `true` only to test against a local bare repo |
| `SANDBOX_PROVIDER` / `DOMAIN_ROUTER` / `GIT_PROVIDER` | `mock` / `mock` / `mock` | shared with Phase 1; `docker`/`caddy` are the real values |

`.env.example` carries the same block, and the `docker` / `deploy_it` pytest
markers are registered in `pyproject.toml`.

---

## 4. Endpoints added or changed

All paths are under `/api`.

### Delivery — `/organizations/{org_id}/delivery` (tenant-scoped)

| Method | Path | Role | Notes |
|---|---|---|---|
| GET | `/runs` | member | pipeline run list |
| GET | `/runs/{run_id}` | member | run + its 8 steps + log tails |
| POST | `/runs` | org admin/owner | manual re-run of the `dev` head |
| POST | `/runs/{run_id}/cancel` | org admin/owner | |
| GET | `/releases` | member | release history |
| POST | `/releases/rollback` | org admin/owner | demo rollback, slug-confirmed |
| GET | `/runtime` | member | demo URL, live release, health |
| GET | `/app` | member | app summary for the page header |
| GET | `/git` | member | repo URL + `dev` head + last run |
| GET | `/production` | member | target summary + deploy history |
| POST | `/production/test` | org admin/owner | connection test + host-key TOFU |
| POST | `/production/deploy` | org admin/owner | **human-triggered**, `confirm: "<org-slug>"`, one at a time |
| POST | `/production/rollback` | org admin/owner | |

### Domains — `/admin/domains` (platform owner)

| Method | Path | Notes |
|---|---|---|
| GET | `` | every attached app route. The MCP gateway host is a read-only suggestion (`/suggested`), not a route the owner can enable: the label `mcp` is protected |
| POST | `` | attach a host (label, or a custom FQDN when the setting is on) |
| DELETE | `/{domain_id}` | detach, host typed back as confirmation |
| POST | `/sync` | idempotent re-sync from the database |
| GET | `/suggested` | MCP gateway suggestion |

### Sandbox — `/organizations/{org_id}/sandbox/*` (changed in Phase 1)

Mutating routes (create/start/stop workspace, run command, read/write file,
snapshot, preview URL, set limits) became **platform-owner only**. Read-only status
(templates, effective limits, workspace list, command log) stayed member-open.

### Sandbox console — `/admin/sandbox` (new, platform owner)

`GET /admin/sandbox/containers` and `POST /admin/sandbox/containers/{id}/stop`,
mounted on a separate `admin_router` inside `sandbox.py` so the tenant router
stays org-scoped.

### Projects — `/organizations/{org_id}/projects/{pid}/preview/*` (changed in Phase 4)

All four endpoints (ports, domains create/list/delete) became platform-owner only.

---

## 5. Tables (all in `models_delivery.py`)

`pipeline_runs`, `pipeline_steps`, `releases`, `app_runtimes`,
`production_deploys`. Migration
`a3b4c5d6e7f3_delivery_infra` (`down_revision = "a0f0e1d2c3b4"`).

The MCP gateway has no table: `GET /api/admin/domains/suggested` computes its
suggestion from settings.

---

## 6. Manual QA script

Run against a real docker + caddy host with
`SANDBOX_PROVIDER=docker GIT_WORKSPACE=local DOMAIN_ROUTER=caddy`.

1. **Wizard → app**: create an org, complete the wizard with `git_source=platform`.
   Expect a demo host `<org-slug>.<root>` claimed and an `app_runtimes` row with a
   `pending` status.
2. **First build**: push a commit to `dev`. Watch `org:<id>` receive
   `pipeline.queued` → 8 × `pipeline.step` → `pipeline.status`. Expect `checkout`,
   `setup`, `build`, `test`, `package`, `deploy_demo`, `healthcheck`, `route`, a
   release marked live, and the demo container on the new port.
3. **Open the demo URL** — the healthcheck step already proved the app answers.
4. **Red build**: push a commit that fails the test command. Expect `failed` with a
   Persian summary and the **previous demo still serving on the old port**.
5. **Rollback**: from the app page, roll the demo back. The release flips live,
   the container is replaced from the artifact, `release.live` fires.
6. **Outside push**: with `git_source=external`, push to the remote from another
   machine. Within `DELIVERY_POLL_SECONDS` a run appears with `trigger=poll`.
7. **Production, SSH**: `POST /production/test` against a local `sshd` container,
   then `POST /production/deploy` with `confirm: "<org-slug>"`. Expect a release
   dir, the atomic `current` symlink switch, the post-deploy hook, and pruning to
   the newest 5. Change the server's host key → the next test must **fail**
   ("کلید میزبان تغییر کرده است").
8. **Production, FTP**: the same against a `pure-ftpd` container; expect a
   `ftp-backup-<ts>.tar.gz` and, on rollback, that archive re-uploaded.
9. **Domains**: as the owner attach a second label; re-sync twice and check the
   second call reports `up_to_date: true` with no Caddy reload. Check that
   `GET /api/admin/domains/suggested` shows `mcp.<root>` as a suggestion (`enabled=false`).
10. **SSRF**: attempt a production deploy to `127.0.0.1`, `10.0.0.1` and
    `169.254.169.254`. All three must be refused before any connection.
11. **Tenant isolation**: a member of org B gets 404 on every org-A delivery and
    admin-domains path; a non-admin member gets 403 on every mutation.
12. **Secret hygiene**: set a deploy target, run a deploy, then grep the logs, the
    `pipeline_steps.log_tail` and the realtime events for the password. Nothing.

---

## 7. CHANGE REQUESTS (still open, for integration)

1. **The AI workforce (Agent 2) needs the two delivery hooks wired.**
   `container_delivery.build_delivery_hooks(session)` returns the real service;
   the wizard (Agent 1) should call `on_app_created` and the engine should call
   `on_branch_pushed` after a merge+push. Until then nothing queues runs except
   a manual `POST /delivery/runs`.
2. **The app profile is Agent 1's.** The Application page reads
   `setup/build/test/start commands`, `app_port` and `healthcheck_path` from it.
   The Git tab shows the `dev` head; the app-type badges in the header wait for
   Phase A1-2.
3. **The web needs the caller's *org role*** to gate the production buttons
   client-side. `MeResponse.organizations` already carries `OrganizationWithRole`.
   Today the server is the boundary and the UI renders optimistically.
4. **Reserved-label enforcement now lives in two places** and they must stay in
   step. The tenant path is guarded in `ProjectService.create_preview_domain`
   (`label in PROTECTED_HOST_LABELS` → `PermissionDeniedError`); the owner path is
   guarded inside `rules.validate_extra_host`. A future third entry point that
   builds a host should call one of them, not re-derive the rule.

---

## 8. Known limitations (deliberate, documented)

- **Build containers cannot write into the AI's work tree.** Each run gets a
  *copy* of the tree under `<root>/…/runs/<run_id>/` (`.git`/`node_modules`
  excluded). `RepoWorkspacePort` is frozen and has no checkout primitive, and a
  build writing `dist/` into `work/` would show up in the next task's diff as if
  an employee had changed it.
- **Disk per run.** Nothing prunes `runs/`, `artifacts/` or `ftp-backup-*` yet.
- **The healthcheck is a plain GET** with 20 × 3 s retries, no readiness protocol,
  no auth on the probe path. Any 2xx/3xx counts as healthy.
- **`snapshot` does not capture the work tree** (bind mount) — image state only.
- **FTP has no atomic switch and no host key.** The deploy is backup-first so
  there is always something to undo, but the window is not zero, and the plain
  `ftp` kind is unencrypted by definition.
- **No container-level egress policy for build containers** beyond the default
  bridge. A hermetic build would need an explicit deny and a package mirror.
- **`routes_fingerprint` compares host:port pairs.** Changing a *label* on a demo
  host is a remove+add, which the digest does catch; a change that keeps both
  host and port but edits the site block body would not.
- **The AI never commits, merges, pushes or inspects git** — that is the point
  (D6), but it also means Agent 2's engine must go through `RepoWorkspacePort`
  for all of it, and a git failure surfaces to the customer as a Persian message
  rather than a git exit code.
- **No sandbox restart.** `/admin/sandbox` lists and stops containers; there is
  no restart action (TODO 6.3 open).