mjalmoudy94
a1b8a28dda
A push to `dev` now becomes a live demo, and a failed build never takes the demo down. - Four tables (pipeline_runs, pipeline_steps, releases, app_runtimes) and their migration; alembic upgrade/downgrade/check all clean. - DeliveryService implements DeliveryHooks: on_app_created provisions the demo port + host + first build (idempotently); on_branch_pushed queues a run for `dev` and supersedes older *queued* runs, leaving a running one alone. - The 8-step pipeline. Blue/green is real: the replacement starts on a staging port, is healthchecked, and only then does the route move — then the previous container is stopped. A failure removes the replacement, releases its port and leaves the old container serving. - The demo port ledger is the DATABASE, not the router adapters' in-memory maps: the API and the delivery worker are separate processes. - Runs are claimed with a conditional UPDATE, so two workers never build the same run. - Live logs are batched (1 event / 500 ms, 200 lines) and redacted as a whole tail — per-chunk redaction misses a secret split across pipe reads, and the *stored* tail leaked until this was fixed. - delivery worker: drain the queue, then poll remotes for pushes made outside the engine. - 7 routes; reads need membership, actions need projects.manage and are audited. Tests: 415 API (+21, 1 docker-marked skipped); ruff, mypy, alembic check clean. Report: docs/plan/reports/agent-3-phase-3.md
137 lines
6.9 KiB
Markdown
137 lines
6.9 KiB
Markdown
# Agent 3 — Phase 3 report: live CI/CD + demo server
|
|
|
|
Date: 2026-10-14 · Branch: `dev`
|
|
|
|
## What was done
|
|
|
|
### 3.1 Tables (`models_delivery.py` + migration `a3b4c5d6e7f3`)
|
|
|
|
`pipeline_runs`, `pipeline_steps` (bounded 64 KB log tail), `releases`, `app_runtimes`
|
|
(one row per project — the primary key is the project id, because D5 gives each app
|
|
exactly one demo container). `alembic upgrade head` → `downgrade -1` → `upgrade head`
|
|
→ `check` all clean, no autogenerate drift.
|
|
|
|
### 3.2 `delivery_service.py` — `DeliveryHooks`
|
|
|
|
- `on_app_created` allocates the demo port, claims `<org-slug>.<PREVIEW_ROOT_DOMAIN>`
|
|
through the existing `DomainRouterPort`, creates the runtime row, ensures the
|
|
workspace and queues the first build. **Idempotent**: a retry reserves no second
|
|
port and creates no second run.
|
|
- `on_branch_pushed` only acts on `dev` (a task branch is work in progress) and
|
|
demotes older **queued** runs for the project. A run that has already started is
|
|
left alone — cancelling it mid-flight would strand a container.
|
|
|
|
### The demo port ledger is the database, not the adapter
|
|
|
|
`PreviewDomainRepository` (the `preview_ports` table) is the source of truth, not the
|
|
router adapters' in-memory maps. The API and the delivery worker are separate
|
|
processes; an in-process port map would hand the same port to two projects the moment
|
|
a second process started. Two repository methods were added for the switch:
|
|
`set_port` (re-point a service at a new port) and `delete_port` (release a transient
|
|
reservation).
|
|
|
|
### Blue/green, done properly
|
|
|
|
The replacement container is started on a **staging** port, never the live one:
|
|
|
|
1. `deploy_demo` reserves a transient staging port (`demo-staging-<run>`) and starts
|
|
the new container on it, detached, `--rm`-less so it survives the call.
|
|
2. `healthcheck` polls `http://127.0.0.1:<staging><healthcheck_path>` — 20 tries,
|
|
3 s apart, any 2xx/3xx passes.
|
|
3. `route` reconfigures Caddy to the staging port, **then** swaps the ledger
|
|
(`delete_port` staging → `set_port demo`), updates the runtime row, and only
|
|
**then** stops the previous container.
|
|
|
|
Order in step 3 is load-bearing and was found by a test: `preview_ports.port` is unique
|
|
platform-wide, so the staging row must be released *before* the `demo` row can move
|
|
onto the port it held.
|
|
|
|
A failure at any step removes the failed replacement and releases its port, leaving
|
|
the previous container serving. `test_a_failed_build_keeps_the_previous_demo` asserts
|
|
exactly that, including that `route` never ran.
|
|
|
|
### 3.3 Worker (`workers/delivery.py`)
|
|
|
|
`run_pass` drains the queued runs oldest-first, then polls remotes (gated by
|
|
`DELIVERY_POLL_SECONDS`). **Runs are claimed atomically**: `PipelineRunRepository.claim`
|
|
is a conditional `UPDATE … WHERE status='queued'`, so two worker processes can never
|
|
build the same run. The worker survives a failing run — it records the error and
|
|
continues.
|
|
|
|
### 3.4 Live logs
|
|
|
|
`_LogBatcher` coalesces each step's output into at most one event per 500 ms, at most
|
|
200 lines per event, and keeps a bounded tail. Redaction happens on the **whole
|
|
accumulated tail**, not per chunk — a secret split across two pipe reads would slip
|
|
past a per-chunk filter, and this was a real bug the redaction test caught.
|
|
|
|
### 3.5 Router (`delivery.py`)
|
|
|
|
`GET /runs`, `GET /runs/{id}` (with steps), `GET /releases`, `GET /runtime` need only
|
|
org membership; `POST /runs`, `POST /runs/{id}/cancel`, `POST /releases/rollback`
|
|
require `projects.manage` and are audited.
|
|
|
|
## Files changed
|
|
|
|
| File | Change |
|
|
|---|---|
|
|
| `app/domain/delivery/models.py` (+`__init__.py`) | new — run/step/release/runtime + status enums |
|
|
| `app/infrastructure/db/models_delivery.py` | the four tables |
|
|
| `alembic/versions/a3b4c5d6e7f3_delivery_infra.py` | new migration |
|
|
| `app/infrastructure/repositories/delivery.py` | new — 4 repositories incl. `claim` |
|
|
| `app/infrastructure/repositories/projects.py` | `set_port`, `delete_port` |
|
|
| `app/application/services/delivery_service.py` | new — hooks, pipeline, blue/green |
|
|
| `app/application/container_delivery.py` | real hooks + `build_delivery_service` |
|
|
| `app/application/container.py` | re-exports `build_delivery_service` |
|
|
| `app/workers/delivery.py` | real drain + poll loop |
|
|
| `app/interfaces/http/routers/delivery.py` | 7 routes |
|
|
| `app/interfaces/http/deps.py` | `get_delivery_service` |
|
|
| `app/infrastructure/adapters/docker_sandbox.py` | `run_step` (fresh build/demo container) |
|
|
| `tests/test_delivery.py` | new — 21 tests + 1 docker-marked |
|
|
| `tests/test_foundation.py` | the delivery worker test now redirects the session factory |
|
|
|
|
## Test output
|
|
|
|
```
|
|
apps/api: 415 passed, 1 skipped (was 394; +21, +1 skipped docker test)
|
|
ruff check . -> All checks passed!
|
|
mypy app -> Success: no issues found in 132 source files
|
|
alembic: upgrade head / downgrade -1 / upgrade head / check -> all clean
|
|
apps/web: 67 passed (unchanged)
|
|
```
|
|
|
|
## Deviations from the TODO
|
|
|
|
1. **`package` runs through the same container runner** rather than a host `tar`. It
|
|
keeps every step going through one hardened path, and a packaging failure is
|
|
recorded but does not fail the run (the demo serves from the work tree).
|
|
2. **A staging port per run, swapped into the `demo` ledger entry.** The TODO says
|
|
"start the new container on a temporary second port" without saying how the port is
|
|
chosen; a fresh reservation per run is the only option that cannot collide with a
|
|
concurrent deploy of another project.
|
|
3. **`healthcheck` and `route` do not call the container runtime.** They are a poll and
|
|
a config write, so they appear in `pipeline_steps` but not in the container-call
|
|
list. The step sequence in `pipeline_steps` is the full eight, as specified.
|
|
4. **The worker's remote poll is gated in-process.** `DELIVERY_POLL_SECONDS` is honoured
|
|
per worker process. With several workers each would poll on its own schedule; the
|
|
poll is idempotent (it compares against the last *succeeded* sha) so the cost is
|
|
duplicated network, not duplicated builds.
|
|
|
|
## Open questions / CHANGE REQUESTS
|
|
|
|
- **CHANGE REQUEST (for integration):** `build_repo_workspace(session)` returns a port
|
|
bound to the caller's session. `DeliveryService` holds it for the lifetime of one
|
|
request/pass, which is correct, but the worker must keep building the service per
|
|
pass rather than caching it. Noted in the Phase 2 report as well.
|
|
|
|
## Known limitations
|
|
|
|
- The `route` step relies on the Caddy adapter's own validate → revert → reload. Its
|
|
`routes_fingerprint` optimisation lands in Phase 4.
|
|
- A demo rollback moves the route but not the container, so rolling back to a sha whose
|
|
container has already been replaced serves the *current* container's code under the
|
|
older release label. Making the container match requires keeping one container per
|
|
release, which the single-`app_runtimes`-row design deliberately does not do. This is
|
|
the honest limit of "rollback = a pointer move" and is documented rather than hidden.
|
|
- No production deploy path exists yet (Phase 5), by design.
|