feat(delivery): live CI/CD pipeline + demo runtime (v2 phase A3-3)
A push to `dev` now becomes a live demo, and a failed build never takes the demo down. - Four tables (pipeline_runs, pipeline_steps, releases, app_runtimes) and their migration; alembic upgrade/downgrade/check all clean. - DeliveryService implements DeliveryHooks: on_app_created provisions the demo port + host + first build (idempotently); on_branch_pushed queues a run for `dev` and supersedes older *queued* runs, leaving a running one alone. - The 8-step pipeline. Blue/green is real: the replacement starts on a staging port, is healthchecked, and only then does the route move — then the previous container is stopped. A failure removes the replacement, releases its port and leaves the old container serving. - The demo port ledger is the DATABASE, not the router adapters' in-memory maps: the API and the delivery worker are separate processes. - Runs are claimed with a conditional UPDATE, so two workers never build the same run. - Live logs are batched (1 event / 500 ms, 200 lines) and redacted as a whole tail — per-chunk redaction misses a secret split across pipe reads, and the *stored* tail leaked until this was fixed. - delivery worker: drain the queue, then poll remotes for pushes made outside the engine. - 7 routes; reads need membership, actions need projects.manage and are audited. Tests: 415 API (+21, 1 docker-marked skipped); ruff, mypy, alembic check clean. Report: docs/plan/reports/agent-3-phase-3.md
This commit is contained in:
mjalmoudy94
@@ -0,0 +1,136 @@
|
||||
# Agent 3 — Phase 3 report: live CI/CD + demo server
|
||||
|
||||
Date: 2026-10-14 · Branch: `dev`
|
||||
|
||||
## What was done
|
||||
|
||||
### 3.1 Tables (`models_delivery.py` + migration `a3b4c5d6e7f3`)
|
||||
|
||||
`pipeline_runs`, `pipeline_steps` (bounded 64 KB log tail), `releases`, `app_runtimes`
|
||||
(one row per project — the primary key is the project id, because D5 gives each app
|
||||
exactly one demo container). `alembic upgrade head` → `downgrade -1` → `upgrade head`
|
||||
→ `check` all clean, no autogenerate drift.
|
||||
|
||||
### 3.2 `delivery_service.py` — `DeliveryHooks`
|
||||
|
||||
- `on_app_created` allocates the demo port, claims `<org-slug>.<PREVIEW_ROOT_DOMAIN>`
|
||||
through the existing `DomainRouterPort`, creates the runtime row, ensures the
|
||||
workspace and queues the first build. **Idempotent**: a retry reserves no second
|
||||
port and creates no second run.
|
||||
- `on_branch_pushed` only acts on `dev` (a task branch is work in progress) and
|
||||
demotes older **queued** runs for the project. A run that has already started is
|
||||
left alone — cancelling it mid-flight would strand a container.
|
||||
|
||||
### The demo port ledger is the database, not the adapter
|
||||
|
||||
`PreviewDomainRepository` (the `preview_ports` table) is the source of truth, not the
|
||||
router adapters' in-memory maps. The API and the delivery worker are separate
|
||||
processes; an in-process port map would hand the same port to two projects the moment
|
||||
a second process started. Two repository methods were added for the switch:
|
||||
`set_port` (re-point a service at a new port) and `delete_port` (release a transient
|
||||
reservation).
|
||||
|
||||
### Blue/green, done properly
|
||||
|
||||
The replacement container is started on a **staging** port, never the live one:
|
||||
|
||||
1. `deploy_demo` reserves a transient staging port (`demo-staging-<run>`) and starts
|
||||
the new container on it, detached, `--rm`-less so it survives the call.
|
||||
2. `healthcheck` polls `http://127.0.0.1:<staging><healthcheck_path>` — 20 tries,
|
||||
3 s apart, any 2xx/3xx passes.
|
||||
3. `route` reconfigures Caddy to the staging port, **then** swaps the ledger
|
||||
(`delete_port` staging → `set_port demo`), updates the runtime row, and only
|
||||
**then** stops the previous container.
|
||||
|
||||
Order in step 3 is load-bearing and was found by a test: `preview_ports.port` is unique
|
||||
platform-wide, so the staging row must be released *before* the `demo` row can move
|
||||
onto the port it held.
|
||||
|
||||
A failure at any step removes the failed replacement and releases its port, leaving
|
||||
the previous container serving. `test_a_failed_build_keeps_the_previous_demo` asserts
|
||||
exactly that, including that `route` never ran.
|
||||
|
||||
### 3.3 Worker (`workers/delivery.py`)
|
||||
|
||||
`run_pass` drains the queued runs oldest-first, then polls remotes (gated by
|
||||
`DELIVERY_POLL_SECONDS`). **Runs are claimed atomically**: `PipelineRunRepository.claim`
|
||||
is a conditional `UPDATE … WHERE status='queued'`, so two worker processes can never
|
||||
build the same run. The worker survives a failing run — it records the error and
|
||||
continues.
|
||||
|
||||
### 3.4 Live logs
|
||||
|
||||
`_LogBatcher` coalesces each step's output into at most one event per 500 ms, at most
|
||||
200 lines per event, and keeps a bounded tail. Redaction happens on the **whole
|
||||
accumulated tail**, not per chunk — a secret split across two pipe reads would slip
|
||||
past a per-chunk filter, and this was a real bug the redaction test caught.
|
||||
|
||||
### 3.5 Router (`delivery.py`)
|
||||
|
||||
`GET /runs`, `GET /runs/{id}` (with steps), `GET /releases`, `GET /runtime` need only
|
||||
org membership; `POST /runs`, `POST /runs/{id}/cancel`, `POST /releases/rollback`
|
||||
require `projects.manage` and are audited.
|
||||
|
||||
## Files changed
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `app/domain/delivery/models.py` (+`__init__.py`) | new — run/step/release/runtime + status enums |
|
||||
| `app/infrastructure/db/models_delivery.py` | the four tables |
|
||||
| `alembic/versions/a3b4c5d6e7f3_delivery_infra.py` | new migration |
|
||||
| `app/infrastructure/repositories/delivery.py` | new — 4 repositories incl. `claim` |
|
||||
| `app/infrastructure/repositories/projects.py` | `set_port`, `delete_port` |
|
||||
| `app/application/services/delivery_service.py` | new — hooks, pipeline, blue/green |
|
||||
| `app/application/container_delivery.py` | real hooks + `build_delivery_service` |
|
||||
| `app/application/container.py` | re-exports `build_delivery_service` |
|
||||
| `app/workers/delivery.py` | real drain + poll loop |
|
||||
| `app/interfaces/http/routers/delivery.py` | 7 routes |
|
||||
| `app/interfaces/http/deps.py` | `get_delivery_service` |
|
||||
| `app/infrastructure/adapters/docker_sandbox.py` | `run_step` (fresh build/demo container) |
|
||||
| `tests/test_delivery.py` | new — 21 tests + 1 docker-marked |
|
||||
| `tests/test_foundation.py` | the delivery worker test now redirects the session factory |
|
||||
|
||||
## Test output
|
||||
|
||||
```
|
||||
apps/api: 415 passed, 1 skipped (was 394; +21, +1 skipped docker test)
|
||||
ruff check . -> All checks passed!
|
||||
mypy app -> Success: no issues found in 132 source files
|
||||
alembic: upgrade head / downgrade -1 / upgrade head / check -> all clean
|
||||
apps/web: 67 passed (unchanged)
|
||||
```
|
||||
|
||||
## Deviations from the TODO
|
||||
|
||||
1. **`package` runs through the same container runner** rather than a host `tar`. It
|
||||
keeps every step going through one hardened path, and a packaging failure is
|
||||
recorded but does not fail the run (the demo serves from the work tree).
|
||||
2. **A staging port per run, swapped into the `demo` ledger entry.** The TODO says
|
||||
"start the new container on a temporary second port" without saying how the port is
|
||||
chosen; a fresh reservation per run is the only option that cannot collide with a
|
||||
concurrent deploy of another project.
|
||||
3. **`healthcheck` and `route` do not call the container runtime.** They are a poll and
|
||||
a config write, so they appear in `pipeline_steps` but not in the container-call
|
||||
list. The step sequence in `pipeline_steps` is the full eight, as specified.
|
||||
4. **The worker's remote poll is gated in-process.** `DELIVERY_POLL_SECONDS` is honoured
|
||||
per worker process. With several workers each would poll on its own schedule; the
|
||||
poll is idempotent (it compares against the last *succeeded* sha) so the cost is
|
||||
duplicated network, not duplicated builds.
|
||||
|
||||
## Open questions / CHANGE REQUESTS
|
||||
|
||||
- **CHANGE REQUEST (for integration):** `build_repo_workspace(session)` returns a port
|
||||
bound to the caller's session. `DeliveryService` holds it for the lifetime of one
|
||||
request/pass, which is correct, but the worker must keep building the service per
|
||||
pass rather than caching it. Noted in the Phase 2 report as well.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- The `route` step relies on the Caddy adapter's own validate → revert → reload. Its
|
||||
`routes_fingerprint` optimisation lands in Phase 4.
|
||||
- A demo rollback moves the route but not the container, so rolling back to a sha whose
|
||||
container has already been replaced serves the *current* container's code under the
|
||||
older release label. Making the container match requires keeping one container per
|
||||
release, which the single-`app_runtimes`-row design deliberately does not do. This is
|
||||
the honest limit of "rollback = a pointer move" and is documented rather than hidden.
|
||||
- No production deploy path exists yet (Phase 5), by design.
|
||||
Reference in New Issue
Block a user