From aaee3329317a75247c6c77d11ab40a5f3c920ad5 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 23:57:07 +0000 Subject: [PATCH 01/26] docs: add warm dev loop proposal (mxbuild --serve + reload_model) Draft proposal covering two scenarios: a Docker-free `mxcli dev` warm run loop to replace the docker inner loop, and an iPad split-screen preview workflow (Claude Code Web + browser, preview-before-commit). Built on measured mxbuild --serve incremental builds (~0.8s warm vs ~30-60s one-shot) and the runtime reload_model hot-reload primitive (~0.1s, no restart), with restartRequired branching page/microflow from entity/DDL changes. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_mxcli_dev_warm_loop.md | 246 ++++++++++++++++++ 1 file changed, 246 insertions(+) create mode 100644 docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md new file mode 100644 index 000000000..d49277401 --- /dev/null +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -0,0 +1,246 @@ +--- +title: warm dev loop — Docker-free run and iPad split-screen preview +status: draft +date: 2026-07-17 +--- + +# Proposal: Warm dev loop (`mxbuild --serve` + `reload_model`) + +**Status:** Draft +**Date:** 2026-07-17 +**Relates to:** `PROPOSAL_check_mxbuild_gap_heuristics.md` (the static-check gate that +runs *before* this loop ever builds), `.claude/skills/mendix/docker-workflow.md` +(the loop this replaces for inner-loop iteration). + +## Problem Statement + +The current way to run and test a Mendix app with mxcli is `mxcli docker build` / +`mxcli docker run`. It has two costs that dominate the edit→test loop: + +1. **Full build every time.** The one-shot `mxbuild` CLI recompiles the whole + deployment on each invocation. Measured on a one-entity/one-page/one-microflow + app: **~30–60 s per build** (Java compile + full model export + package). +2. **Docker.** The runtime and database run in containers, adding image pulls and + container lifecycle to every cycle — and Docker is **unavailable** in important + contexts: Claude Code on the web, and iPad. In those environments the current + loop cannot run at all. + +Two primitives, both verified to exist and work in Mendix **11.6.3+**, make a far +tighter loop possible: + +- **`mxbuild --serve`** — an HTTP build server (default port 6543) that keeps the + model loaded and rebuilds **incrementally**, skipping unchanged work (including + Java compilation when no Java changed). Each build returns `restartRequired`. +- **`reload_model`** — a runtime admin-port action that swaps the model into the + **running** JVM in-place, draining in-flight actions first (near-zero-downtime). + No process restart. + +This proposal wires those two primitives into mxcli to serve **two scenarios**: + +- **Scenario A — a fast, Docker-free `mxcli dev` loop** to replace `mxcli docker` + for inner-loop iteration. +- **Scenario B — an iPad split-screen workflow**: Claude Code on the web in one + pane, a browser preview in the other, prompt → build → test, all on the iPad, + **without committing to the repo** before testing. + +## BSON Structure + +Not applicable — this is a build/runtime orchestration feature. It touches no +Mendix document serialization. It drives two already-existing interfaces: the +mxbuild HTTP build API and the runtime M2EE admin API. + +## Measured evidence + +All numbers below were measured directly (Mendix 11.6.3, Linux x86-64, JDK 21) on +a one-entity/one-page/one-microflow app during the investigation that motivated +this proposal. + +| Step | Time | Notes | +|------|------|-------| +| `mxbuild` one-shot CLI (full) | **~30–60 s** | current loop; no incremental mode | +| `mxbuild --serve`, cold (first build) | ~13.7 s | loads model into the server | +| `mxbuild --serve`, warm, **no change** | ~1.1 s | model cached | +| `mxbuild --serve`, warm, **microflow change** | **~0.8 s** | Java **not** recompiled | +| `mxbuild --serve`, warm, **entity/attribute change** | ~7.3 s | `restartRequired: true` | +| `reload_model` (hot, no restart) | **~0.07–0.27 s** | `"reload": true`, JVM pid unchanged | +| **Full warm loop** (exec → serve build → reload) | **~1 s** | microflow/page change, no restart | + +`restartRequired` from the serve API cleanly separates the two change classes: + +- **page / microflow change** → `restartRequired: false` → `reload_model` (~0.1 s) +- **entity / domain change** → `restartRequired: true` → `execute_ddl_commands` + + runtime restart (the DDL gate is real and unavoidable — the runtime refuses a + stale schema) + +`mxbuild --serve` and `reload_model` are present in both **11.6.3** and **11.12.1** +(verified). The 11.12 CLI adds no build-skip flags (only `--export-secrets`), so the +incremental path is `--serve`, not a new one-shot flag. + +## Proposed CLI + +### Scenario A: `mxcli dev` — Docker-free warm run loop + +A long-lived local dev supervisor. Boots (or reuses) the standalone runtime + +Postgres, starts `mxbuild --serve`, and on each model change runs +`serve build (Deploy)` → branch on `restartRequired` → `reload_model` or +DDL+restart. + +```bash +# start a warm dev session (no Docker) +mxcli dev -p app.mpr +# → downloads/caches mxbuild + runtime (once), starts Postgres, +# boots the runtime, starts `mxbuild --serve`, prints the local URL, +# and holds everything warm. + +# apply a change and hot-reload (from another shell, or driven by the agent) +mxcli dev reload -p app.mpr # serve-build + restartRequired-aware reload +mxcli dev exec change.mdl -p app.mpr # exec MDL, then reload in one step +mxcli dev status # runtime status, warm-build server health +mxcli dev stop +``` + +`mxcli dev` **subsumes the inner loop of `mxcli docker run`** while keeping +`mxcli docker` for produced-artifact / container-parity builds. + +Optionally a `--watch` mode: watch the `.mpr` (v2 `mprcontents/`) for changes and +auto-run the reload cycle. + +### Scenario B: `mxcli dev serve` + a preview container (iPad) + +The iPad has two panes: **Claude Code on the web** (prompt + edit) and **Safari** +(the running app). The catch, established during investigation: the Claude Code web +sandbox is **egress-only and cannot expose a port** (all reverse tunnels are blocked +by a 443-only + TLS-intercepting egress policy). So the *running* app must live in a +container that has real public ingress — a **Codespace** (public port forwarding → +`https://-8080.app.github.dev`) is the natural fit. + +Two containers, coupled **without a commit** (the explicit requirement — preview WIP +before committing): + +``` +┌─────────────────────────┐ push built deployment ┌──────────────────────────┐ +│ DEV: Claude Code Web │ ── (443, authenticated, no git) ──▶ │ PREVIEW: Codespace │ +│ edits app.mpr │ │ runtime + mxbuild --serve │ +│ mxbuild --serve (build) │ │ Postgres │ +│ │ ◀── agent verifies via public URL ── │ :8080 forwarded PUBLIC │ +└─────────────────────────┘ (curl / Playwright) └──────────────────────────┘ + Safari tab ▲ (iPad) +``` + +- **DEV** (Claude Code Web): prompt Claude → `mxcli` edits `app.mpr` → + `mxbuild --serve` produces the Deploy artifact → `mxcli dev push` uploads it to the + Preview container over 443. +- **PREVIEW** (Codespace): receives the artifact, runs `reload_model` (or DDL+restart + on `restartRequired`), serves the public URL. Booted by a `devcontainer.json` + + `postStart` script that reproduces the standalone-boot recipe. +- **iPad UX:** prompt in the Claude pane → ~1–2 s → refresh the Safari tab → see the + change. Nothing is committed until the user is happy. + +Proposed commands: + +```bash +# in the Preview container (Codespace), started by the devcontainer: +mxcli dev serve -p app.mpr --intake-port 9000 --intake-secret $TOKEN +# → boots runtime + Postgres + mxbuild --serve, forwards 8080 public, +# exposes an authenticated /deploy intake for artifact pushes. + +# in the DEV container (Claude Code Web): +mxcli dev push --to https://-9000.app.github.dev --secret $TOKEN -p app.mpr +# → mxbuild --serve (Deploy) locally, then upload the deployment delta; +# the Preview container reloads and returns the live status. +``` + +### Coupling: options considered + +| Option | Preview WIP w/o commit? | Notes | +|--------|-------------------------|-------| +| **A. Push built deployment artifact over 443** (recommended) | ✅ | robust; runtime in a legit host; ~1–2 s + network | +| B. Push MDL/model, build in the Preview container | ✅ | keeps a synced `.mpr` in the Preview container; heavier transfer for MPR v2 | +| C. Git branch, Codespace pulls | ❌ | requires a commit — **rejected**, violates the core requirement | +| D. Live tunnel (app stays in DEV, Codespace relays) | ✅ | instant, but fragile (chisel over two proxies), ties the app to the ephemeral sandbox, and Codespace-as-relay is an AUP gray area — **not recommended** | + +## Implementation Plan + +Reuse the existing `cmd/mxcli/docker/` plumbing wherever possible — it already +downloads/caches mxbuild + runtime and speaks the M2EE admin API. + +### Files to modify/create + +| File | Change | +|------|--------| +| `cmd/mxcli/dev.go` | New `mxcli dev` command tree (`dev`, `dev reload`, `dev exec`, `dev serve`, `dev push`, `dev status`, `dev stop`). | +| `cmd/mxcli/docker/mxserve.go` | New: `mxbuild --serve` client — start the daemon, `POST /build {target:Deploy}`, parse `status` + `restartRequired`. | +| `cmd/mxcli/docker/admin.go` | Extract the M2EE admin client (auth header `X-M2EE-Authentication: base64(pass)`, actions `update_appcontainer_configuration`, `update_configuration`, `start`, `execute_ddl_commands`, **`reload_model`**, `runtime_status`) out of `docker/oql.go` into a reusable client. | +| `cmd/mxcli/docker/localboot.go` | New: standalone runtime boot (see the config-set below), so `dev` can boot without Docker. | +| `cmd/mxcli/docker/download.go` | Reuse `DownloadMxBuild` / `DownloadRuntime` (no change). | +| `cmd/mxcli/docker/mxenv.go` | Reuse `PrepareMxCommand` (LD_PRELOAD FreeType fix) for the `mx`/`mxbuild` children (no change). | +| `.devcontainer/preview/devcontainer.json` + `boot.sh` | New: Codespace preview container — `forwardPorts: [8080]` public, Postgres, `postStart` → `mxcli dev serve`. | +| `.claude/skills/mendix/docker-workflow.md` | Add a "warm dev loop" section pointing at `mxcli dev`. | +| `cmd/mxcli/dev_test.go` | Tests for the serve client and the `restartRequired` branch logic (mock the serve/admin HTTP endpoints). | + +### The standalone-boot config set (discovered, must be encoded) + +The standalone runtime launcher (`com.mendix.container.boot.Main`, driven via the +admin API) requires a specific config that Studio Pro / the buildpack normally +supply. `mxcli dev` must set all of it or `start` fails: + +- Env: `MX_INSTALL_PATH`, `M2EE_ADMIN_PASS`, `M2EE_ADMIN_PORT`, and the FreeType + `LD_PRELOAD` on the JVM. +- Ensure the `data/{files,tmp,model-upload}` dirs exist under the deployment. +- `update_configuration`: `BasePath` (deployment dir), `RuntimePath` + (`/runtime`), `DatabaseType/Host/Name/UserName/Password` + (`DatabaseHost` includes the port, e.g. `127.0.0.1:5432`), `DTAPMode`, and + **`MicroflowConstants`** (design-time default values are *not* auto-applied + standalone — missing a constant surfaces at runtime as HTTP 530 → login bounce). +- Boot sequence: `update_appcontainer_configuration` (runtime port) → + `update_configuration` → `start` → on `"database has to be updated"`: + `execute_ddl_commands` → `start`. +- Reload cycle: `mxbuild --serve` Deploy build → if `restartRequired` then + `execute_ddl_commands` (if schema changed) + restart, else `reload_model`. + +## Version Compatibility + +- `mxbuild --serve` and `reload_model`: **≥ 11.6.3** (verified on 11.6.3 and + 11.12.1). No new flag needed on 11.12 — the incremental path is `--serve` in both. +- Register the feature in `sdk/versions/mendix-11.yaml` (and 10 if we verify the + admin action name `reload_model` there) with the correct `min_version`, and add a + `checkFeature()` pre-check with an actionable hint. +- **Production-representative testing note:** running `DTAPMode=P` requires the app's + security level to be `CHECKEVERYTHING` (Production) or `start` refuses + (`result:11`). `mxcli dev` should default to `D` for the inner loop and expose a + `--dtap P` flag; when `P` is set, surface the security-level requirement up front. + +## Test Plan + +- `cmd/mxcli/docker/mxserve_test.go` — mock the serve HTTP endpoint; assert request + shape (`{target:Deploy, projectFilePath}`) and `restartRequired` parsing. +- `cmd/mxcli/docker/admin_test.go` — mock the admin endpoint; assert the boot + sequence and the `reload_model` vs DDL+restart branch. +- Integration (gated, needs mxbuild + runtime + Postgres, like the existing docker + tests): boot a fixture app, do a microflow change → assert warm reload with pid + unchanged; do an entity change → assert `restartRequired` → DDL path. +- `mdl-examples/` — a small fixture app the dev-loop tests drive. + +## Open Questions + +1. **Process lifecycle.** Long-lived `mxbuild --serve` and the runtime JVM need + supervision (restart on crash, port cleanup). PID files under `.mxcli/`? A + `dev status`/`dev stop`? (In the sandbox, background JVMs were reaped on idle — + the Codespace/preview host must keep them alive; document the idle-stop caveat.) +2. **Artifact transfer for Scenario B.** Push the whole Deploy `model/` each time, or + compute a delta? MPR v2 is a folder; the *deployment* `model/` is more compact, + but still worth measuring. What's the auth model for the intake endpoint (bearer + secret over the public forward)? +3. **Codespaces AUP.** Running your own app for preview is intended use; the + Codespace-as-relay variant (Option D) is a gray area. The proposal recommends A + (artifact push) precisely to stay clearly on the "previewing your own project" + side. Confirm this framing. +4. **Warm-serve memory.** `mxbuild --serve` holds the model in memory; quantify for + large apps and document guidance. +5. **`reload_model` scope.** Verified for microflow changes (hot) and flagged + correctly for entity changes (`restartRequired`). Not yet exercised: styling-only + (`update_styling` exists as a separate admin action) and multi-unit changes. + Worth a coverage pass before shipping. +6. **Relationship to `check`.** The instant `mxcli check` gate (see + `PROPOSAL_check_mxbuild_gap_heuristics.md`) should run *before* every warm build so + most errors never reach even the ~0.8 s build. `mxcli dev` should call it first. From 673710c901d5e5366207a2c01bbd7ecd78367488 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 00:04:35 +0000 Subject: [PATCH 02/26] docs: add prototype-to-production workflow to warm dev loop proposal Frame the iPad scenario as one leg of the full journey: Claude design prototype -> Mendix model via mxcli/MDL -> production security (ALTER PROJECT SECURITY LEVEL PRODUCTION, per-role grants) -> live warm-loop testing under real security, all on iPad without Studio Pro. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_mxcli_dev_warm_loop.md | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index d49277401..39ff5f4d5 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -43,6 +43,40 @@ This proposal wires those two primitives into mxcli to serve **two scenarios**: pane, a browser preview in the other, prompt → build → test, all on the iPad, **without committing to the repo** before testing. +## End-to-end workflow: Claude design prototype → secured production Mendix app (on iPad) + +Scenario B is one leg of a larger journey this loop is meant to unlock — taking an +idea all the way to a **secured, production-grade Mendix app entirely on an iPad**, +with no desktop Studio Pro in the loop: + +1. **Prototype in Claude.** A Claude design Artifact (a clickable HTML/React mockup) + captures the app's screens, data, and flows — fast and throwaway, no Mendix yet. +2. **Translate to Mendix in Claude Code (web).** In the same iPad session, mxcli/MDL + turns the prototype into a real Mendix model: `create entity` / `create page` / + `create microflow` / navigation. The prototype's structure becomes actual `.mpr` + documents — a running Mendix app, not a mockup. +3. **Secure it for production.** Raise the app from prototype to production security + and make it pass the real Mendix consistency rules — the exact path proven during + the investigation behind this proposal: + - `ALTER PROJECT SECURITY LEVEL PRODUCTION` — required before `DTAPMode=P` will + even boot (otherwise `start` returns `result:11 — security must be + CHECKEVERYTHING`). + - `grant` / `revoke` page, entity, and microflow access per module role; define + user roles and their mappings; wire the login and anonymous navigation profiles. + - `mxcli check` (instant) plus the warm `mxbuild` build surface the + production-only errors up front (e.g. *"page needs an allowed role"*, which we + hit and fixed with a single `grant`) instead of discovering them at deploy time. +4. **Test each change live, on the iPad.** The warm loop (§Proposed CLI) previews + every edit in ~1 s in the Safari pane, under **real production security** (login + enforced, anonymous blocked) — so the author iterates prototype → model → secured + app → test without ever leaving the tablet. + +The deliverable is not a throwaway preview but a **secured, production-representative +Mendix app** — the same artifact you would deploy. The warm dev loop is what makes +step 4 fast enough for this to be a genuine authoring experience rather than a batch +process, and Scenario B is what makes it work on a device that has neither Docker nor +Studio Pro. + ## BSON Structure Not applicable — this is a build/runtime orchestration feature. It touches no From a8404b4d04bbab1983b7967a0c098c4da5664e9c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 02:14:47 +0000 Subject: [PATCH 03/26] docs: add hot-reload scope + DB metamodel-catalog root cause to warm-loop proposal Verified reload_model reloads model store + microflows + i18n only, not the datastorage/entity layer. The runtime reconciles its metamodel catalog (mendixsystem$entity/entityidentifier/attribute/association) at startup, so entity/view-entity/association changes need a restart. Confirmed directly: a view entity added via reload_model never appears in mendixsystem$entity. Adds the two-signal decision matrix (restartRequired x get_ddl_commands) and the key nuance that OQL view entities need restart WITHOUT DDL (query-time, not materialized). Notes the standalone boot lacks preview_execute_oql for functional OQL verification. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_mxcli_dev_warm_loop.md | 63 +++++++++++++++++-- 1 file changed, 57 insertions(+), 6 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 39ff5f4d5..404f78703 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -110,6 +110,50 @@ this proposal. (verified). The 11.12 CLI adds no build-skip flags (only `--export-secrets`), so the incremental path is `--serve`, not a new one-shot flag. +## Hot-reload scope: what `reload_model` can and cannot do (verified) + +`reload_model` reloads exactly three subsystems — the model store +(`ProjectModelStoreLoader.load`), the microflow engine +(`MicroflowEngineModule.reload`), and translations (`I18NProcessor.reload`) — after +draining in-flight actions. It does **not** touch the datastorage / entity layer. + +The runtime maintains a **metamodel catalog inside the app database** — +`mendixsystem$entity`, `mendixsystem$entityidentifier`, `mendixsystem$attribute`, +`mendixsystem$association` — and reconciles it **at startup**, not during a reload. +This was verified directly: a view entity added via `reload_model` (`TaskView`) +returned `Success` but **never appeared in `mendixsystem$entity`**, while the base +`Task` (present at the last startup) has its row (`entity_name` → `table_name` → a +stable identifier GUID). So any change that needs a catalog row — a new/changed +entity, **including OQL view entities**, and associations in +`mendixsystem$association` (e.g. a view entity referencing a persistent entity) — +requires a **restart**, because that is when the catalog is reconciled. + +This makes the apply-decision **two-dimensional**: `restartRequired` (from the serve +build) and `get_ddl_commands` (from the runtime) are **independent** signals. + +| Change class | `restartRequired` | `get_ddl_commands` | Apply via | +|---|---|---|---| +| microflow / page / text | `false` | 0 | `reload_model` (~0.1 s, hot) | +| **OQL view entity** | **`true`** | **0** | **restart** — catalog sync, **no DDL** | +| persistable entity / attribute | `true` | > 0 | `execute_ddl_commands` → restart | +| association (adds FK) | `true` | > 0 | `execute_ddl_commands` → restart | + +The loop must branch on **both** signals: `restartRequired == false` → `reload_model`; +`restartRequired == true` → restart, running `execute_ddl_commands` **only if** +`get_ddl_commands > 0`. Treating "restart" as implying "DDL" is wrong — **OQL view +entities are the counterexample**: they need a restart (to write the +`mendixsystem$entity` catalog row) but no DDL (they are query-time, not materialized — +confirmed: no view/table for `TaskView` in Postgres). + +**Caveat (not fully closed):** `reload_model` *returns* `Success` for a view-entity +change (it doesn't reject it), and I could not run the functional tiebreaker — +querying `TaskView` post-reload — because the standalone hand-boot does not register +the `preview_execute_oql` admin action (it needs the dev-preview flag `mxcli docker` +sets). The conclusion rests on the authoritative `restartRequired` signal plus the +catalog evidence, which agree. Enabling the dev-preview flag on the standalone boot +(so the agent can verify via OQL) is itself a small item worth doing — see Open +Questions. + ## Proposed CLI ### Scenario A: `mxcli dev` — Docker-free warm run loop @@ -229,8 +273,10 @@ supply. `mxcli dev` must set all of it or `start` fails: - Boot sequence: `update_appcontainer_configuration` (runtime port) → `update_configuration` → `start` → on `"database has to be updated"`: `execute_ddl_commands` → `start`. -- Reload cycle: `mxbuild --serve` Deploy build → if `restartRequired` then - `execute_ddl_commands` (if schema changed) + restart, else `reload_model`. +- Reload cycle (two independent signals — see § Hot-reload scope): `mxbuild --serve` + Deploy build → if `restartRequired == false` then `reload_model`; else restart, + running `execute_ddl_commands` first **only if** `get_ddl_commands > 0`. Do **not** + couple "restart" to "DDL" — OQL view entities need a restart with zero DDL. ## Version Compatibility @@ -271,10 +317,15 @@ supply. `mxcli dev` must set all of it or `start` fails: side. Confirm this framing. 4. **Warm-serve memory.** `mxbuild --serve` holds the model in memory; quantify for large apps and document guidance. -5. **`reload_model` scope.** Verified for microflow changes (hot) and flagged - correctly for entity changes (`restartRequired`). Not yet exercised: styling-only - (`update_styling` exists as a separate admin action) and multi-unit changes. - Worth a coverage pass before shipping. +5. **`reload_model` scope (largely resolved — see § Hot-reload scope).** Verified: + microflow changes hot-reload; entity/view-entity/association changes flip + `restartRequired` because the runtime reconciles its DB metamodel catalog + (`mendixsystem$entity` / `entityidentifier` / `attribute` / `association`) at + startup, not on reload. Open sub-items: (a) enable the `preview_execute_oql` + dev flag on the standalone boot so the loop can *functionally* verify a change via + OQL (needed to close the view-entity caveat and to power agentic verification); + (b) exercise styling-only changes (`update_styling` is a distinct admin action — + likely a third hot path); (c) multi-unit and mixed changes. 6. **Relationship to `check`.** The instant `mxcli check` gate (see `PROPOSAL_check_mxbuild_gap_heuristics.md`) should run *before* every warm build so most errors never reach even the ~0.8 s build. `mxcli dev` should call it first. From 4a799cdfb36bbfacef48872accaa3e6dd74e6c33 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 02:41:11 +0000 Subject: [PATCH 04/26] docs: add tunnel-hub scaling (self-registration + admin overview) to warm-loop proposal One static hub fronting many dynamic dev containers, verified via chisel: - chisel fan-in (one server, many clients on distinct ports; --authfile isolation) - why chisel works where ngrok/cloudflared/localtunnel didn't (go install past the gate, single-443 multiplex, system trust store already holds the MITM CA) - mxcli tunnel-hub (static) + mxcli dev --hub (self-registering dynamic side) - registration flow over chisel's authfile-reload + client-declared-remote primitives - admin overview page: fleet list + per-container pending-change summaries - Mendix fleet wrinkles (subdomain ApplicationRootUrl, cookie/DB isolation, slots) - reclassify coupling Option D (live tunnel) from "not recommended" to verified/preferred for the multi-container dev loop, artifact-push kept as durable fallback Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_mxcli_dev_warm_loop.md | 95 ++++++++++++++++++- 1 file changed, 94 insertions(+), 1 deletion(-) diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 404f78703..011414800 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -235,7 +235,81 @@ mxcli dev push --to https://-9000.app.github.dev --secret $TOKEN -p app.mp | **A. Push built deployment artifact over 443** (recommended) | ✅ | robust; runtime in a legit host; ~1–2 s + network | | B. Push MDL/model, build in the Preview container | ✅ | keeps a synced `.mpr` in the Preview container; heavier transfer for MPR v2 | | C. Git branch, Codespace pulls | ❌ | requires a commit — **rejected**, violates the core requirement | -| D. Live tunnel (app stays in DEV, Codespace relays) | ✅ | instant, but fragile (chisel over two proxies), ties the app to the ephemeral sandbox, and Codespace-as-relay is an AUP gray area — **not recommended** | +| D. Live tunnel (app stays in DEV, static relay forwards) | ✅ | **viable — chisel verified** (see § Scaling). Cleanest dev loop: hot-reload stays local, no artifact sync, and the relay is a generic static component. Caveat: the DEV runtime is ephemeral. **Preferred for the multi-container dev loop; A remains the durable/persistent-preview fallback.** | + +## Scaling: tunnel hub with self-registration and admin overview + +The two-container model generalizes to **one static hub fronting many dynamic dev +containers** — parallel agents on different features, or the several apps of one +distributed solution. Verified: a single `chisel server --reverse` fans in to multiple +clients, each on its own server-side port (`server:8001 → container A`, +`server:8002 → container B`), and `--authfile` (auto-reloaded on change) isolates them +per-agent. + +**Why chisel and not the others** (all tested this session): it installs via +`go install` (through `proxy.golang.org`, bypassing the GitHub gate), multiplexes +everything over a **single 443 connection** (no second data port — the wall that killed +localtunnel and cloudflared), and its TLS **passes the sandbox's mandatory MITM** +because it uses the system trust store, which already contains the agent CA (ngrok +failed only because it pins its own roots). The one hop not provable without a live +host: the WebSocket upgrade through Codespaces' `app.github.dev` proxy — a **VPS relay +removes that unknown** and is cleaner AUP-wise for a pure relay. + +### `mxcli tunnel-hub` (static) + `mxcli dev --hub` (dynamic) + +- **`mxcli tunnel-hub`** — the always-on static container. Embeds a chisel server + (`github.com/jpillora/chisel/server` as a library), a host-routing layer + (Caddy/Traefik or built-in), a **registration API**, and an **admin overview page**. + Runs once on a VPS (preferred) or a Codespace. It never changes when an app changes. +- **`mxcli dev --hub `** — the dynamic side self-registers on startup. + +Self-registration flow — chisel has **no native registration API**; this thin control +plane adds one on top of its two primitives (the auto-reloading `--authfile` and the +client-declared reverse remote): + +``` +mxcli dev --hub https://hub.example.com (on container startup) + → POST /register {app, feature, agent} → hub allocates {port, subdomain, token} + appends users.json (chisel reloads) + writes vhost subdomain→port (proxy reloads) + ← {port, subdomain, token} + → chisel client HUB R::localhost:8080 --auth + → set Mendix ApplicationRootUrl = https://.example.com + → heartbeat POST /status {health, last_reload, change_summary} (periodic) +on stop / TTL expiry → POST /deregister (frees the slot) +``` + +### Admin overview page + +The hub serves an authenticated dashboard — the single place to see and reach every +in-flight preview: + +- **One row per dev container:** agent/feature name, app, **clickable public URL**, + runtime health (from the heartbeat), last hot-reload time, uptime. +- **Pending changes per container:** each `mxcli dev` reports its delta — from + `mxcli diff-local` against the committed baseline, or the running list of MDL + statements applied this session — and the hub aggregates them, so a reviewer sees + *what each agent has changed* without opening each session. +- **Use cases:** a PM/reviewer clicks through parallel feature previews; comparing two + agents' takes on the same screen side by side; spotting a container that has drifted + or broken; a distributed-solution operator seeing all constituent apps and their + states at once. + +The change list makes the hub more than a router — it is a **live index of work in +progress across the fleet**, which is exactly the "keep overview" need. + +### Mendix wrinkles at fleet scale + +- **Subdomain, not path**, per container — `ApplicationRootUrl` = the container's own + subdomain. Path-prefix routing fights Mendix's SPA/XAS/cookies; subdomains give each + app a clean origin and free **cookie/session isolation** between agents. +- **DB per container** — each dev container needs its own Postgres/schema + demo data, + or agents overwrite each other. (Sandbox model: each has its own local Postgres.) +- **Slot assignment** — the hub owns the `{port, subdomain, token}` map via the + registration API; `--authfile` pins each agent to its allowed remote. +- **Distributed solution** — constituent apps register as separate containers on + separate subdomains and integrate via their public routes (published REST/OData), + faithfully previewing the real multi-app topology. ## Implementation Plan @@ -254,6 +328,11 @@ downloads/caches mxbuild + runtime and speaks the M2EE admin API. | `cmd/mxcli/docker/mxenv.go` | Reuse `PrepareMxCommand` (LD_PRELOAD FreeType fix) for the `mx`/`mxbuild` children (no change). | | `.devcontainer/preview/devcontainer.json` + `boot.sh` | New: Codespace preview container — `forwardPorts: [8080]` public, Postgres, `postStart` → `mxcli dev serve`. | | `.claude/skills/mendix/docker-workflow.md` | Add a "warm dev loop" section pointing at `mxcli dev`. | +| `cmd/mxcli/tunnelhub/server.go` | New: `mxcli tunnel-hub` — embedded chisel server (`chisel/server` lib), host routing, slot allocation, `--authfile` writer. | +| `cmd/mxcli/tunnelhub/register.go` | New: registration API (`/register`, `/status` heartbeat, `/deregister`); owns the `{port, subdomain, token}` map + TTL cleanup. | +| `cmd/mxcli/tunnelhub/admin.go` | New: authenticated admin overview page — fleet list (URL, health, last reload) + per-container change summaries. | +| `cmd/mxcli/dev.go` | Add `--hub `: self-register on startup, run the chisel client, set `ApplicationRootUrl` from the assigned subdomain, push `change_summary` heartbeats, deregister on stop. | +| `cmd/mxcli/tunnelhub/*_test.go` | Tests: slot allocation/isolation, authfile + vhost reload, register/heartbeat/deregister lifecycle, admin-page rendering. | | `cmd/mxcli/dev_test.go` | Tests for the serve client and the `restartRequired` branch logic (mock the serve/admin HTTP endpoints). | ### The standalone-boot config set (discovered, must be encoded) @@ -329,3 +408,17 @@ supply. `mxcli dev` must set all of it or `start` fails: 6. **Relationship to `check`.** The instant `mxcli check` gate (see `PROPOSAL_check_mxbuild_gap_heuristics.md`) should run *before* every warm build so most errors never reach even the ~0.8 s build. `mxcli dev` should call it first. +7. **Tunnel-hub auth & multi-tenancy.** The hub fronts multiple previews on one host — + who may view the admin page, and how are per-container URLs protected (per-subdomain + auth, shared secret, SSO)? Registration tokens must be scoped so one agent cannot + claim another's slot (`--authfile` allowed-remote regex per user). +8. **Change-summary source & cadence.** `mxcli diff-local` (git baseline) vs a running + MDL-statement log — which is the better per-container "changes" feed for the admin + page, and how often to push it (every reload vs periodic)? +9. **WebSocket through `app.github.dev` unverified.** chisel's reverse tunnel is proven + through the sandbox egress + MITM, but not through a live Codespace forward. Verify + on a real Codespace, or default the hub to a VPS (also the cleaner AUP posture for a + pure relay). +10. **Fleet lifecycle.** TTL/heartbeat-timeout cleanup of dead containers, port/subdomain + reclamation, and what the admin page shows for a container whose sandbox was reaped + (the ephemeral-runtime reality observed this session). From 2c4c1fccc12eeeb0a7e55234f3735887f8677dc0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 02:48:51 +0000 Subject: [PATCH 05/26] docs: add provisioning (new project -> running app, SessionStart bootstrap) How a user goes from a new Claude Code Web project to a running testable app: - entry points: mxcli new (new app) / clone + mxcli init (existing) - gap identified: init scaffolds a devcontainer + .claude tooling but no SessionStart hook; Claude Code Web reaps Postgres/JVM on idle, so it needs an idempotent `mxcli dev up` wired to SessionStart - dev up = setup mxcli/mxbuild/mxruntime (all exist) + Postgres + boot + serve - first-run vs warm cost table; heavy costs are one-time and image-cacheable - adds impl rows (dev up, init SessionStart hook) and an open question on image pre-baking + the SessionStart contract Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_mxcli_dev_warm_loop.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 011414800..a048bd9d9 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -237,6 +237,58 @@ mxcli dev push --to https://-9000.app.github.dev --secret $TOKEN -p app.mp | C. Git branch, Codespace pulls | ❌ | requires a commit — **rejected**, violates the core requirement | | D. Live tunnel (app stays in DEV, static relay forwards) | ✅ | **viable — chisel verified** (see § Scaling). Cleanest dev loop: hot-reload stays local, no artifact sync, and the relay is a generic static component. Caveat: the DEV runtime is ephemeral. **Preferred for the multi-container dev loop; A remains the durable/persistent-preview fallback.** | +## Provisioning: from new Claude Code Web project to a running testable app + +Two entry points, both on tooling that already exists: + +- **New app:** `mxcli new MyApp --version ` — downloads MxBuild, runs + `mx create-project`, scaffolds `.claude/` tooling + a devcontainer, and installs the + Linux mxcli binary. Optionally seed the blank model from a Claude design prototype + (mxcli/MDL `create entity` / `page` / `microflow`) per § End-to-end workflow. +- **Existing app:** point the Claude Code Web project at a repo containing the `.mpr`, + then `mxcli init` — adds `.claude/` skills/commands/CLAUDE.md and a devcontainer if + missing. + +### Initializing the Claude Code Web session + +`mxcli init` / `new` today scaffold a **devcontainer + `.claude/` tooling** — enough for +Codespaces and local dev containers, but Claude Code Web needs one more piece: a +**SessionStart hook** (or devcontainer `postStart`) that idempotently brings the runtime +up on **every** session, because background processes (Postgres, the JVM) are **reaped +on idle** — observed repeatedly during this investigation. Proposed: `mxcli init` also +emits an `mxcli dev up` bootstrap, wired to SessionStart, that idempotently: + +1. **Ensure mxcli** — prefer a prebuilt binary via `mxcli setup mxcli` (fast) over a + from-source build (needs antlr4 + Go, ~70 s). +2. **Ensure MxBuild + runtime cached** — `mxcli setup mxbuild -p app.mpr` and + `mxcli setup mxruntime -p app.mpr` (both already exist). One-time ~700 MB / ~30–40 s; + **bake into the devcontainer image** so the first session is instant. +3. **Start Postgres + create the app DB** — re-run every session (survives reaping). +4. **Boot the runtime + `mxbuild --serve`** — the standalone-boot recipe + (BasePath / RuntimePath / MicroflowConstants / data-dirs → `start` → DDL if needed), + leaving a warm serve daemon. + +End state: the session comes up **ready to prompt → build → test**. Then: + +- `mxcli dev` — local testable app at `localhost:8080` (the ~1 s warm loop). +- `mxcli dev --hub ` — externally testable: self-registers with the hub and gets a + public subdomain (§ Scaling). + +### First-run vs warm + +| Phase | One-time (first session / image build) | Per session (after reap) | +|-------|----------------------------------------|--------------------------| +| mxcli binary | `setup mxcli` download (or ~70 s build) | cached | +| MxBuild + runtime | ~700 MB, ~30–40 s | cached | +| Postgres + DB | — | start + createdb (~seconds) | +| runtime boot | — | ~40 s cold / seconds warm | + +The heavy costs are one-time and **image-cacheable**; steady-state session startup is +just Postgres + a runtime boot. The only real gap is the **SessionStart bootstrap** — +mxcli already has every ingredient (`new`, `init`, `setup mx*`, the standalone boot); +this proposal wires them into one idempotent `mxcli dev up` that Claude Code Web runs on +session start. + ## Scaling: tunnel hub with self-registration and admin overview The two-container model generalizes to **one static hub fronting many dynamic dev @@ -332,6 +384,8 @@ downloads/caches mxbuild + runtime and speaks the M2EE admin API. | `cmd/mxcli/tunnelhub/register.go` | New: registration API (`/register`, `/status` heartbeat, `/deregister`); owns the `{port, subdomain, token}` map + TTL cleanup. | | `cmd/mxcli/tunnelhub/admin.go` | New: authenticated admin overview page — fleet list (URL, health, last reload) + per-container change summaries. | | `cmd/mxcli/dev.go` | Add `--hub `: self-register on startup, run the chisel client, set `ApplicationRootUrl` from the assigned subdomain, push `change_summary` heartbeats, deregister on stop. | +| `cmd/mxcli/dev.go` (`dev up`) | Idempotent bootstrap for provisioning: `setup mxcli`/`mxbuild`/`mxruntime` if missing, start Postgres + create DB, boot runtime + `mxbuild --serve`. Safe to re-run every session (survives reaping). | +| `cmd/mxcli/init.go` | Emit a **SessionStart hook** (Claude Code Web) and/or devcontainer `postStart` that runs `mxcli dev up`; keep the existing devcontainer + `.claude/` scaffolding. | | `cmd/mxcli/tunnelhub/*_test.go` | Tests: slot allocation/isolation, authfile + vhost reload, register/heartbeat/deregister lifecycle, admin-page rendering. | | `cmd/mxcli/dev_test.go` | Tests for the serve client and the `restartRequired` branch logic (mock the serve/admin HTTP endpoints). | @@ -422,3 +476,8 @@ supply. `mxcli dev` must set all of it or `start` fails: 10. **Fleet lifecycle.** TTL/heartbeat-timeout cleanup of dead containers, port/subdomain reclamation, and what the admin page shows for a container whose sandbox was reaped (the ephemeral-runtime reality observed this session). +11. **Provisioning baseline.** Best mechanism to pre-bake MxBuild + runtime (~700 MB) + into the Claude Code Web image so first session is instant, vs downloading on first + `mxcli dev up`. And the exact SessionStart hook contract on Claude Code Web (does it + run on every resume, with what timeout?) — determines whether `dev up` must be fully + idempotent and how long it may take. From 27aede7c23d7a22609fe35421026eaa150da0ec3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 03:27:45 +0000 Subject: [PATCH 06/26] docs: make prompt template the primary web/iPad provisioning path Restructure Provisioning into three entry points (prompt template primary for web/iPad, repo template, local CLI). Add the prompt-template subsection with: - the draft bootstrap prompt - rule 1: committing the config is mandatory (survives reaping; produces the repo-template end state without a maintained template) - rule 2: mxcli delivery is an environment-layer concern (github gate blocks release curl; go install works only for a public module) - proven via chisel Adds impl rows (new --emit-template, shipped prompt template, public-module/ env-preinstall delivery) and open question 12 on mxcli delivery into a gated empty repo. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_mxcli_dev_warm_loop.md | 62 ++++++++++++++++--- 1 file changed, 54 insertions(+), 8 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index a048bd9d9..17e5a0490 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -239,15 +239,52 @@ mxcli dev push --to https://-9000.app.github.dev --secret $TOKEN -p app.mp ## Provisioning: from new Claude Code Web project to a running testable app -Two entry points, both on tooling that already exists: +Three entry points, ordered by how little you need to start with: + +- **Prompt template (primary for web/iPad — starts from a *truly empty* repo).** Open an + empty repo in Claude Code Web and paste a fixed bootstrap prompt; the agent provisions + the project. No local CLI and no pre-built template repo to maintain — the agent runs + *current* mxcli and can seed the model from a Claude design prototype in the same + prompt. Detailed below. +- **Repo template ("Use this template" on GitHub).** A `mendix-mxcli-starter` marked as a + GitHub template → a new repo with the config already committed. Deterministic, but needs + per-version upkeep; a good fixed, reviewed baseline. Could be generated by + `mxcli new --emit-template` + CI per Mendix version so it is never hand-maintained. +- **Local CLI.** `mxcli new MyApp --version ` (new app: downloads MxBuild, runs + `mx create-project`, scaffolds `.claude/` + a devcontainer, installs the Linux mxcli + binary) or `mxcli init` on an existing repo with an `.mpr`. Requires a machine with the + CLI — **not available on an iPad**. + +### Prompt template (the web/iPad-native path) + +The agent turns an empty repo into a self-sufficient Mendix project in one session: -- **New app:** `mxcli new MyApp --version ` — downloads MxBuild, runs - `mx create-project`, scaffolds `.claude/` tooling + a devcontainer, and installs the - Linux mxcli binary. Optionally seed the blank model from a Claude design prototype - (mxcli/MDL `create entity` / `page` / `microflow`) per § End-to-end workflow. -- **Existing app:** point the Claude Code Web project at a repo containing the `.mpr`, - then `mxcli init` — adds `.claude/` skills/commands/CLAUDE.md and a devcontainer if - missing. +``` +This is an empty repo. Provision it as a Mendix app developed with mxcli: +1. Ensure `mxcli` is available (should be pre-installed by the environment; else + `go install github.com/mendixlabs/mxcli/cmd/mxcli@latest`). +2. `mxcli new App --version 11.6.3` at the repo root (or `mxcli init` if an .mpr exists). +3. Add `.claude/settings.json` with a SessionStart hook running `./mxcli dev up -p App.mpr`, + then COMMIT .mpr + .devcontainer/ + .claude/ so future sessions self-bootstrap. +4. `mxcli dev up -p App.mpr` — cache MxBuild+runtime, start Postgres, boot the runtime. +5. Confirm HTTP 200 locally and report. +(Optional) Seed the domain model, pages, and microflows from this prototype: . +``` + +Two rules make this robust: + +- **Committing the config (step 3) is mandatory.** The prompt is a *one-time seed*; its + output must be committed (`.mpr` + `.devcontainer/` + `.claude/` SessionStart hook) so + that after idle **reaping** the next session self-bootstraps from files, not from + re-prompting. Steady state is then file-driven and deterministic — the prompt template + *produces* the repo-template end state without a maintained template. +- **mxcli delivery is an *environment*-layer concern, not the prompt's.** Step 1 is the + fragile part: GitHub is gated in web sessions (a release-binary `curl` may be blocked), + and `go install` via `proxy.golang.org` works **only if mxcli is a public Go module** + (the mechanism was proven for chisel this session). Robust fix: the Claude Code Web + **environment setup script / image pre-installs mxcli** (and pre-caches MxBuild+runtime), + so the prompt only does logic (`new` → commit → `dev up`), never delivery. Publishing + mxcli as a public Go module is the fallback. ### Initializing the Claude Code Web session @@ -386,6 +423,9 @@ downloads/caches mxbuild + runtime and speaks the M2EE admin API. | `cmd/mxcli/dev.go` | Add `--hub `: self-register on startup, run the chisel client, set `ApplicationRootUrl` from the assigned subdomain, push `change_summary` heartbeats, deregister on stop. | | `cmd/mxcli/dev.go` (`dev up`) | Idempotent bootstrap for provisioning: `setup mxcli`/`mxbuild`/`mxruntime` if missing, start Postgres + create DB, boot runtime + `mxbuild --serve`. Safe to re-run every session (survives reaping). | | `cmd/mxcli/init.go` | Emit a **SessionStart hook** (Claude Code Web) and/or devcontainer `postStart` that runs `mxcli dev up`; keep the existing devcontainer + `.claude/` scaffolding. | +| `cmd/mxcli/new.go` | Add `--emit-template`: write a GitHub-template-ready repo (config + starter `.mpr`), for CI-per-version publishing of `mendix-mxcli-starter`. | +| `docs-site/.../bootstrap-prompt.md` | Ship the canonical **prompt template** (the web/iPad seed prompt) as documented, copy-pasteable text. | +| release / go.mod | Ensure mxcli is deliverable into a gated web session — **public Go module** (`go install`) and/or an **environment setup-script** that pre-installs it; do not rely on a GitHub release `curl` (may be gate-blocked). | | `cmd/mxcli/tunnelhub/*_test.go` | Tests: slot allocation/isolation, authfile + vhost reload, register/heartbeat/deregister lifecycle, admin-page rendering. | | `cmd/mxcli/dev_test.go` | Tests for the serve client and the `restartRequired` branch logic (mock the serve/admin HTTP endpoints). | @@ -481,3 +521,9 @@ supply. `mxcli dev` must set all of it or `start` fails: `mxcli dev up`. And the exact SessionStart hook contract on Claude Code Web (does it run on every resume, with what timeout?) — determines whether `dev up` must be fully idempotent and how long it may take. +12. **mxcli delivery into a gated empty repo (the prompt-template blocker).** GitHub is + gated in web sessions, so a release-binary `curl` may be blocked; `go install` via + `proxy.golang.org` only works if mxcli is a **public** module. Decide the canonical + delivery: publish a public Go module, pre-install mxcli in the environment image, or + both. This determines whether the prompt-template path works unassisted or must lean + on the environment layer. From a7f41e4d68b1b1ea9ca59fc146171477171df6bd Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 03:38:08 +0000 Subject: [PATCH 07/26] docs: add delivery-slices section to warm-loop proposal Keep one proposal but show how to ship it incrementally as four independent, value-ordered slices: (1) warm local loop, (2) provisioning, (3) single-app external preview, (4) tunnel hub. Recommends 1->2 first (solo dev, no external parts), 3 for single-app preview, 4 as a separately-reviewed scale-out. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_mxcli_dev_warm_loop.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 17e5a0490..9865ef09a 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -451,6 +451,29 @@ supply. `mxcli dev` must set all of it or `start` fails: running `execute_ddl_commands` first **only if** `get_ddl_commands > 0`. Do **not** couple "restart" to "DDL" — OQL view entities need a restart with zero DDL. +## Delivery slices + +This stays one proposal, but it should **ship as four independent slices** — each +valuable on its own, each a candidate for its own PR. Ordered by value-to-risk; each +builds on the previous. + +| # | Slice | Delivers | Depends on | Size / risk | +|---|-------|----------|------------|-------------| +| 1 | **Warm local loop** — `mxcli dev` (serve daemon + M2EE admin client + `restartRequired` × `get_ddl_commands` branching) | Docker-free ~1 s edit→test loop, locally | nothing new | medium / low — highest bang-for-buck | +| 2 | **Provisioning** — `mxcli dev up` bootstrap + `mxcli init` SessionStart hook + prompt template | a fresh Claude Code Web session comes up testable; iPad-native start | slice 1 | small / low | +| 3 | **Single-app external preview** — `mxcli dev serve` + chisel client → one static relay + `ApplicationRootUrl` wiring | a shareable live preview URL (the iPad two-container flow) | slice 1 | medium / medium — the `app.github.dev` WebSocket hop is unverified (a VPS relay avoids it) | +| 4 | **Tunnel hub** — `mxcli tunnel-hub` + `mxcli dev --hub` + registration API + admin overview | many dev containers behind one ingress; fleet overview + per-container change lists | slice 3 | large / higher — a product in its own right, with a multi-tenant auth surface | + +Recommended sequencing: **1 → 2** delivers the complete solo dev experience (Scenario A +plus provisioning) with no external moving parts, and can ship first. **3** adds external +preview for a single app. **4** is the scale-out and deserves its own design/security +review — build it only once 1–3 are proven. Slices 1–2 stand alone if the tunnel/hub work +is deferred indefinitely. + +Cross-cut: the instant `mxcli check` gate (`PROPOSAL_check_mxbuild_gap_heuristics.md`) +should front slice 1's build on every iteration, so most errors never reach even the +~0.8 s warm build. + ## Version Compatibility - `mxbuild --serve` and `reload_model`: **≥ 11.6.3** (verified on 11.6.3 and From 88252c5407d98bb4df8db308789f7b28450d7b52 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 03:50:50 +0000 Subject: [PATCH 08/26] docs: regenerate proposals README index (add warm-loop + 4 others) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran scripts/gen_proposals_readme.py. Indexes 5 previously-missing active proposals: mxcli_dev_warm_loop, check_mxbuild_gap_heuristics, ai_capability_dataset, architecture_graph_visualization, playwright_session_reuse. Side effect, flagged: PROPOSAL_marketplace_modules (status: shipping) and UNIFIED_SCHEMA_REGISTRY (status: descoped) drop from the index because their statuses are non-canonical; the generator only buckets canonical statuses. They remain on disk and re-index once their statuses are normalised (maintainer's call — not changed here). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- docs/11-proposals/README.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index 0ad4dd8cc..7f24ebe14 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -26,10 +26,10 @@ for display in this README): -## Active Proposals (78) +## Active Proposals (83) -### In Progress (partial) (8) +### In Progress (partial) (7) | Proposal | Status | Summary | |----------|--------|---------| @@ -38,11 +38,10 @@ for display in this README): | [Implement `mxcli structure` Command](mxcli-structure-proposal.md) | Partial | mxcli is a Go CLI tool for working with Mendix projects. | | [Multi-Version Support: Consolidated Architecture & Status](MULTI_VERSION_SUPPORT.md) | Partial | Mendix projects vary along three versioning axes, and mxcli must handle all of them correctly: | | [mxcli Linter - Implementation Proposal (Revised)](PROPOSAL_mxcli_linter.md) | Partial | Add an extensible linting framework to mxcli that leverages the existing SQLite-based catalog system. | -| [mxcli marketplace — Download & Manage Marketplace Modules](PROPOSAL_marketplace_modules.md) | Partial | After four rounds of spiking (scripts/auth-discovery-spike.sh), the | | [Navigation Support in MDL](navigation-support.md) | Partial | Every Mendix project has exactly one navigation$NavigationDocument — a project-level singleton containing navigation profiles (Responsive, P | | [Podman Support as Docker Alternative](PROPOSAL_podman_support.md) | Partial | Docker Desktop requires a paid subscription for larger organizations. | -### Proposed (36) +### Proposed (35) | Proposal | Status | Summary | |----------|--------|---------| @@ -77,22 +76,24 @@ for display in this README): | [SHOW/DESCRIBE Scheduled Events](show-describe-scheduled-events.md) | Proposed | Document type: ScheduledEvents$ScheduledEvent | | [SHOW/DESCRIBE Support for Missing Pluggable (React) Widgets](show-describe-pluggable-widgets.md) | Proposed | Scope: Improve DESCRIBE PAGE/SNIPPET output for pluggable widgets that currently fall through to generic formatting | | [Unified mxcli & MDL Documentation Site](proposal-documentation-site.md) | Proposed | mxcli has extensive documentation — 119 markdown files, a complete language specification, architecture docs, examples, and user guides — bu | -| [Unified Schema Registry](UNIFIED_SCHEMA_REGISTRY.md) | Descoped | Serialization core retired into engalar's roundtrip codec; survives as thin inspection/migration features over `gen/*` — see [backend strategy § Version handling](PROPOSAL_backend_strategy.md#version-handling--the-schema-registry-overlap-investigation-2026-06-05) | | [Version-Aware MDL and BSON Serialization](version-aware-mdl.md) | Proposed | The Mendix metamodel evolves across versions. | | [VS Code MDL Visualizations](PROPOSAL_vscode_visualizations.md) | Proposed | Add visual diagram previews to the VS Code MDL extension, enabling users to see entity-relationship diagrams, microflow flowcharts, page wir | | [VS Code Search — Quick Pick + Workspace Symbol](PROPOSAL_vscode_search.md) | Proposed | Full-text search exists in mxcli (mxcli search) but is only accessible via the terminal. | | [Workflow Improvements: ALTER WORKFLOW + Cross-References](PROPOSAL_workflow_improvements.md) | Proposed | Workflow support in mxcli has full CREATE/DESCRIBE/DROP/SHOW coverage with 13 activity types and BSON round-trip fidelity. | -### Draft (33) +### Draft (38) | Proposal | Status | Summary | |----------|--------|---------| | [Agent Document Type Support in MDL](PROPOSAL_agent_document_support.md) | Draft | Mendix 11.9 introduces Agents as a first-class concept for building agentic AI applications. | +| [Architecture Graph Visualization (communities, layers, god-nodes)](PROPOSAL_architecture_graph_visualization.md) | Draft | Issue: TBD (file before implementation) | | [Association Mapping in IMPORT](PROPOSAL_import_associations.md) | Draft | Parent: PROPOSAL_mxcli_sql.md (Phase 3 extension) | | [Backend Strategy — adopt engalar's modelsdk base + multi-backend (MCP first)](PROPOSAL_backend_strategy.md) | Draft | - Adopt engalar's modelsdk foundation as the base rather than merging 1109 | | [Bulk Change Custom Widget Properties](PROPOSAL_bulk_widget_property_updates.md) | Draft | Custom widgets (pluggable widgets) in Mendix have complex nested property structures. | | [Bulk External Action Support from OData Contracts](PROPOSAL_external_actions_bulk_create.md) | Draft | Issue #143 requests importing all entities and actions from a consumed OData service. | +| [check heuristics for constructs MxBuild rejects](PROPOSAL_check_mxbuild_gap_heuristics.md) | Draft | Relates to: PROPOSAL_expression_type_checking.md (shares the ModelResolver | | [Claude Code Prompt: Sprotty-based Domain Model Visualization PoC](proposal_sprotty_visualization.md) | Draft | I'm building a VS Code extension that visualizes Mendix Domain Models using Sprotty. | +| [Deterministic AI-Capability Dataset & Report Generator](PROPOSAL_ai_capability_dataset.md) | Draft | We publish an AI Capabilities report (ai-capabilities-report-.html) that | | [Entity Positioning in ALTER ENTITY](PROPOSAL_entity_position.md) | Draft | Entity positions in the domain model canvas can only be set during create entity via the @position annotation. | | [Expression Type Checking for mxcli check](PROPOSAL_expression_type_checking.md) | Draft | mxcli check is currently a syntactic validator only. | | [Extend UPDATE WIDGETS to Built-in Widgets via Schema Registry](PROPOSAL_update_builtin_widget_properties.md) | Draft | update widgets currently only works for pluggable widgets (ComboBox, DataGrid2, etc.) — it cannot modify properties on built-in widgets like | @@ -115,11 +116,13 @@ for display in this README): | [mxcli auth — Mendix Platform Authentication](PROPOSAL_platform_auth.md) | Draft | A growing set of mxcli features need to talk to Mendix platform APIs on behalf of the user: | | [mxcli catalog — Mendix Catalog Integration](PROPOSAL_catalog_integration.md) | Draft | ⚠️ TERMINOLOGY NOTE: This proposal covers the external Mendix Catalog service at catalog.mendix.com (CLI: mxcli catalog search), which is se | | [mxcli Playground](mxcli-playground.md) | Draft | A public GitHub repository (mendixlabs/mxcli-playground) containing a ready-to-use Mendix project pre-configured with mxcli, Claude Code ski | +| [Playwright Session Reuse and Lifecycle Control](PROPOSAL_playwright_session_reuse.md) | Draft | Builds on proposal-playwright-cli.md, which | | [Project Brain — Persistent Knowledge and Session Scaffolding for Long-Term AI Collaboration](PROPOSAL_project_brain.md) | Draft | mxcli is developed with significant AI involvement, yet the project's knowledge infrastructure is built for static human reading rather than | | [RENAME with Reference Refactoring](PROPOSAL_rename_refactoring.md) | Draft | Renaming entities, microflows, pages, and modules is one of the most common refactoring operations. | | [Replace Generated Playwright Tests with playwright-cli](proposal-playwright-cli.md) | Draft | The current approach (documented in proposal-playwright-testing.md) has Claude Code generate TypeScript test files (.spec.ts), then run them | | [Self-Describing Syntax Feature Registry](syntax-feature-registry.md) | Draft | Branch: research/recursive-help-discovery | | [Version-Aware Agent Support](PROPOSAL_version_aware_agent_support.md) | Draft | Three use cases require mxcli to be version-aware at the MDL level: | +| [warm dev loop — Docker-free run and iPad split-screen preview](PROPOSAL_mxcli_dev_warm_loop.md) | Draft | Relates to: PROPOSAL_check_mxbuild_gap_heuristics.md (the static-check gate that | ## Archived (24) From 2bfb306edf687ca3a7b349a6785f039305794bc3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 03:53:11 +0000 Subject: [PATCH 09/26] docs: normalise two non-canonical proposal statuses, reindex README Lint category-2 fix (maintainer-approved) so both re-appear in the index: - PROPOSAL_marketplace_modules: status shipping -> partial (active); tidy date - UNIFIED_SCHEMA_REGISTRY: status descoped -> abandoned; git mv to archive/ (terminal status belongs in archive/; retired into engalar's codec) Regenerated README via scripts/gen_proposals_readme.py: marketplace now under In Progress, UNIFIED under Archived, warm-loop still indexed. No proposals dropped from the index. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- docs/11-proposals/PROPOSAL_marketplace_modules.md | 4 ++-- docs/11-proposals/README.md | 8 +++++--- .../11-proposals/{ => archive}/UNIFIED_SCHEMA_REGISTRY.md | 3 ++- 3 files changed, 9 insertions(+), 6 deletions(-) rename docs/11-proposals/{ => archive}/UNIFIED_SCHEMA_REGISTRY.md (99%) diff --git a/docs/11-proposals/PROPOSAL_marketplace_modules.md b/docs/11-proposals/PROPOSAL_marketplace_modules.md index 8d6897156..75c7d43d7 100644 --- a/docs/11-proposals/PROPOSAL_marketplace_modules.md +++ b/docs/11-proposals/PROPOSAL_marketplace_modules.md @@ -1,7 +1,7 @@ --- title: mxcli marketplace — Download & Manage Marketplace Modules -status: shipping -date: 2026-03-23 (initial), revised 2026-04-16 (spike), 2026-06-05 (download unblocked) +status: partial +date: 2026-06-05 --- # Proposal: `mxcli marketplace` — Download & Manage Marketplace Modules diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index 7f24ebe14..0cabcc190 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -26,10 +26,10 @@ for display in this README): -## Active Proposals (83) +## Active Proposals (82) -### In Progress (partial) (7) +### In Progress (partial) (8) | Proposal | Status | Summary | |----------|--------|---------| @@ -38,6 +38,7 @@ for display in this README): | [Implement `mxcli structure` Command](mxcli-structure-proposal.md) | Partial | mxcli is a Go CLI tool for working with Mendix projects. | | [Multi-Version Support: Consolidated Architecture & Status](MULTI_VERSION_SUPPORT.md) | Partial | Mendix projects vary along three versioning axes, and mxcli must handle all of them correctly: | | [mxcli Linter - Implementation Proposal (Revised)](PROPOSAL_mxcli_linter.md) | Partial | Add an extensible linting framework to mxcli that leverages the existing SQLite-based catalog system. | +| [mxcli marketplace — Download & Manage Marketplace Modules](PROPOSAL_marketplace_modules.md) | Partial | Unblocking path #1 below has happened: the content API now returns a | | [Navigation Support in MDL](navigation-support.md) | Partial | Every Mendix project has exactly one navigation$NavigationDocument — a project-level singleton containing navigation profiles (Responsive, P | | [Podman Support as Docker Alternative](PROPOSAL_podman_support.md) | Partial | Docker Desktop requires a paid subscription for larger organizations. | @@ -125,13 +126,14 @@ for display in this README): | [warm dev loop — Docker-free run and iPad split-screen preview](PROPOSAL_mxcli_dev_warm_loop.md) | Draft | Relates to: PROPOSAL_check_mxbuild_gap_heuristics.md (the static-check gate that | -## Archived (24) +## Archived (25) Terminal-status proposals. See [archive/](archive/). | Proposal | Status | Summary | |----------|--------|---------| +| [Unified Schema Registry](archive/UNIFIED_SCHEMA_REGISTRY.md) | Abandoned | Mendix project metadata (BSON document schemas, widget property structures, MDL keyword | | [Architecture Improvements for Agentic Development and Reduced Merge Conflicts](archive/PROPOSAL_agentic_architecture_improvements.md) | Done | Two related friction points have been identified: | | [Augment Widget Templates from .mpk at Runtime](archive/PROPOSAL_mpk_widget_augmentation.md) | Done | When mxcli creates pluggable widgets (ComboBox, DataGrid2, etc.), it uses static JSON templates extracted from a Mendix 11.6.0 project. | | [Business Events Support in mxcli](archive/PROPOSAL_business_events_support.md) | Done | Business Events is a Mendix feature for asynchronous event-driven integration, allowing applications to publish and subscribe to business ev | diff --git a/docs/11-proposals/UNIFIED_SCHEMA_REGISTRY.md b/docs/11-proposals/archive/UNIFIED_SCHEMA_REGISTRY.md similarity index 99% rename from docs/11-proposals/UNIFIED_SCHEMA_REGISTRY.md rename to docs/11-proposals/archive/UNIFIED_SCHEMA_REGISTRY.md index cfa2befac..28d19a891 100644 --- a/docs/11-proposals/UNIFIED_SCHEMA_REGISTRY.md +++ b/docs/11-proposals/archive/UNIFIED_SCHEMA_REGISTRY.md @@ -1,6 +1,7 @@ --- title: Unified Schema Registry -status: descoped +status: abandoned +date: 2026-06-05 --- # Proposal: Unified Schema Registry From 13f01686c5d02053a38724a3ac31101f0d8eb761 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 04:20:25 +0000 Subject: [PATCH 10/26] docs: capture diagnostics-catalog research + commit extraction script Document the first-part research (mxcli check improvements) as a proposal: - where Mendix's diagnostics catalog lives (Mendix.Modeler.Texts.dll PO files) - structure: CODE__SUMMARY msgids, {param} templates, severity via CE/CW/CI prefix, LOCATION + MX_DOCS_CAT tags - findings: 1318 codes (11.12.1), static-checkability tiering, mxcli already reimplements ~77-81, mx check is same-floor as a build (Go check is the only speedup) - approach: versioned data asset, port by tier/frequency, harvest-against-oracle - verbatim examples (small fair-use sample) Commit scripts/extract-diagnostics-catalog.sh (the methodology). Its OUTPUT is Mendix's proprietary corpus and is NOT committed (see IP open question). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_check_diagnostics_catalog.md | 181 ++++++++++++++++++ docs/11-proposals/README.md | 5 +- scripts/extract-diagnostics-catalog.sh | 100 ++++++++++ 3 files changed, 284 insertions(+), 2 deletions(-) create mode 100644 docs/11-proposals/PROPOSAL_check_diagnostics_catalog.md create mode 100755 scripts/extract-diagnostics-catalog.sh diff --git a/docs/11-proposals/PROPOSAL_check_diagnostics_catalog.md b/docs/11-proposals/PROPOSAL_check_diagnostics_catalog.md new file mode 100644 index 000000000..05a87de61 --- /dev/null +++ b/docs/11-proposals/PROPOSAL_check_diagnostics_catalog.md @@ -0,0 +1,181 @@ +--- +title: mxcli check — mine Mendix's diagnostics catalog to close the check↔mxbuild gap +status: draft +date: 2026-07-18 +--- + +# Proposal: mine Mendix's diagnostics catalog for `mxcli check` + +**Status:** Draft +**Date:** 2026-07-18 +**Relates to:** `PROPOSAL_check_mxbuild_gap_heuristics.md` (one *tactical* slice of this — +6 specific constructs), `PROPOSAL_expression_type_checking.md` (Tier B enabler), +`PROPOSAL_mxcli_dev_warm_loop.md` (this catalog fronts the warm loop's build). + +## Problem Statement + +`mxcli check` reports "Check passed" for many constructs that the real MxBuild +(`docker build` / `mx check`) then rejects. Each miss costs a full build round-trip +(~30–60 s) to discover — and in the warm-dev-loop world (`PROPOSAL_mxcli_dev_warm_loop.md`) +it's the difference between an instant gate and a ~0.8 s build. The gap is currently +closed **reactively**, one rule at a time, whenever someone hits a build error. + +Mendix already ships the complete list of things it will reject: a **diagnostics catalog** +embedded in `Mendix.Modeler.Texts.dll`. This proposal mines it into a **target list** so +`check`/`lint` coverage becomes a measured, prioritised program instead of a reactive one. + +## BSON Structure + +Not applicable — this is a static-analysis/tooling proposal. It reads Mendix's build-tool +assemblies (design-time), not app `.mpr` documents. + +## Research findings (verified this session) + +**Where the catalog lives.** `Mendix.Modeler.Texts.dll` embeds GNU-gettext `.po` resources +named `problem-descriptions_.po` (en-US primary, plus deprecated/test variants and +5 translations). They *are* the catalog. + +**Format — already structured, not free text.** +- `msgid` = `CODE__SUMMARY`, e.g. `CE0001__ASSOCIATION_BETWEEN_PERSISTABLE_...`. +- `msgstr` = a **parameterised template** with named `{PLACEHOLDER}` tokens (and + pluralisation tokens like `{S_IF_PLURAL}`), not runtime-assembled fragments. +- **Severity is encoded in the prefix**: `CE` = Consistency Error, `CW` = Warning, + `CI` = Info. No inference needed. +- Each entry carries a `# LOCATION: .cs` comment and an optional + `# NOTE: MX_DOCS_CAT:` docs-deep-link tag. + +**Scale (Mendix 11.12.1, primary en-US PO):** **1318 codes** — 1265 `CE` + 53 `CW` +(`CI` codes live mostly in the deprecated/test POs); **727/1318 are parameterised**. +Present identically in 11.6.3 and 11.12.1. + +**Static-checkability tiering** (heuristic, by message + source subsystem): + +| Tier | Share | Nature | `mxcli check` reach | +|------|-------|--------|---------------------| +| A | ~60% (791) | structural / reference / required-property / naming | direct — model already parsed | +| B | ~26% (338) | expression / type-inference / XPath | needs a type-inference pass (`PROPOSAL_expression_type_checking.md`) | +| C | ~9% (120) | cross-document / datasource / client-compat | whole-model graph analysis | +| D | ~5% (69) | runtime / platform / tooling state | **not** statically checkable — skip | + +Subsystem split (by source file): Microflows 364, Domain model 275, Pages/Widgets 248, +Integration/REST/OData 129, Security 29, Workflows 15, Navigation 7. + +**mxcli already does ~6% of this by hand.** The source references **~77–81 distinct +`CE`/`CW` codes** across ~24 `validate_*.go` files and ~30 lint rules — proof the approach +works; the catalog just turns "port as we hit them" into an ordered backlog of ~1240 left. + +**There is no cheaper Mendix-native shortcut.** `mx check` (consistency only, no build) +measured **~22 s** — *slower* than an incremental build — because both cold-start the .NET +model engine + load the full `.mpr`. So mxcli's Go-native `check` is the **only** path under +the ~18–60 s floor. That is the entire justification for porting rules into Go rather than +shelling out to `mx check`. + +## Extraction methodology + +Reproducible, ~1 min, no Studio Pro. The **script is ours; its output is Mendix's corpus** +(see § Open questions on IP). Committed as `scripts/extract-diagnostics-catalog.sh`: + +```bash +# 1. stream just the one assembly from the Mendix CDN (no full mxbuild download) +curl -s "https://cdn.mendix.com/runtime/mxbuild-${VERSION}.tar.gz" \ + | tar -xz --occurrence=1 modeler/Mendix.Modeler.Texts.dll + +# 2. extract the embedded PO resources (needs mono-utils → monodis) +monodis --manifest Mendix.Modeler.Texts.dll # lists the .po resources +monodis --mresources Mendix.Modeler.Texts.dll # writes each resource to a file + +# 3. parse problem-descriptions_en-US.po into structured rows: +# code, prefix→severity, slug, message-template, {params}, source .cs, MX_DOCS_CAT +# then tier each by static-checkability. (Python parser in the script.) +``` + +`msgid` → `(prefix, number, slug)`; prefix → severity; `{...}` tokens → parameter list; +`# LOCATION:`/`# NOTE:` comments → source file + docs category. Output: `catalog.csv` / +`catalog.json` (local only). Re-run per Mendix version to refresh. + +## Representative examples (verbatim, small fair-use sample) + +``` +CE0001 error An association between a persistable entity and a non-persistable entity + must start in the non-persistable entity and have owner 'Default'. +CE0002 error Owner must be 'Default' for self-referential associations. +CE0006 error An XPath constraint on an access rule on {DESCRIPTION} is not allowed, + because it is not persistable. +CE0190 error Scheduled events of type Legacy are removed; right click for possible + remedies. [# NOTE: MX_DOCS_CAT:ScheduledEventMigration] +CE0529 error The selected page '{PAGE_NAME}' contains {A_IF_SINGULAR}required + parameter{S_IF_PLURAL} and can not be used as home page. +CE0572 error Widget{S_IF_PLURAL} {WIDGETS} {IS_OR_ARE} not supported in React client. +CW0001 warn Property 'Sort order' of the {CONTAINER} has no effect when a page is + used for selecting. +CW0040 warn Action activity that has a side effect on the client is not recommended + here because the microflow is used as a data source for + {CONTAINER_TYPE_AND_NAME}. +CI1561 info Path parameter 'x' does not appear in the path ''. +``` + +Plus the production-security rule this session actually hit and fixed with one `grant`: +*"At least one allowed role must be selected if the page is used from navigation, a +button, or a URL."* + +## Proposed approach + +1. **Ship the catalog as a versioned data asset** — `sdk/versions/diagnostics-{version}.json`, + produced by the extraction script per Mendix version. Lets `check`/`lint` emit + **Studio-Pro-identical codes + messages + severities**, a UX win even before new rules. + (Subject to the IP decision below — may ship code IDs + our own message text rather than + Mendix's verbatim strings.) +2. **Port by tier then frequency.** Tier A first (cheapest, highest hit-rate), Tier B behind + the type-inference work, Tier C case-by-case, Tier D never. Route `CE`→`check` (hard), + `CW`→`lint` (advisory) — severity is free from the prefix. +3. **Harvest-against-the-oracle loop.** Use `mx check` (~22 s, the full 1318-code truth) as + an **offline oracle** over a corpus of deliberately-broken projects; diff its findings + against `mxcli check`; the misses are the ranked backlog; port; re-run to confirm parity. + This is the build tool for the porting effort, and reuses the "run mxbuild over broken + projects and harvest output" fallback from the very first spike as a *deliberate* tool. +4. `PROPOSAL_check_mxbuild_gap_heuristics.md` is the first tactical instance; this proposal + is the strategic program it fits into. + +## Implementation Plan + +| File | Change | +|------|--------| +| `scripts/extract-diagnostics-catalog.sh` | New: the reproducible extractor (stream DLL → `monodis --mresources` → parse+tier). Committed here. | +| `sdk/versions/diagnostics-{version}.json` | Generated catalog data asset per version (IP decision pending). | +| `mdl/linter/rules/`, `mdl/executor/validate_*.go` | Port Tier-A codes, prioritised by frequency; cite the `CE`/`CW` code in each. | +| `mdl/executor/…` (oracle harness) | Offline `mx check` vs `mxcli check` parity report over a broken-project corpus → ranked backlog. | +| `mdl-examples/check-gap/` | Broken-project fixtures feeding the oracle harness. | + +## Version Compatibility + +- Codes are stable across 11.6.3 / 11.12.1; the extractor is version-parameterised, so the + catalog refreshes per Mendix release. Register generated assets alongside the existing + `sdk/versions/mendix-{9,10,11}.yaml`. +- New/removed codes between versions are themselves a useful signal (what Studio Pro started + or stopped checking). + +## Test Plan + +- Extractor: unit-test the PO parser against a small committed fixture PO (our own, not + Mendix's), asserting code/severity/params/location parsing. +- Oracle harness (gated, needs mxbuild): for each broken fixture, assert `mxcli check` now + reports the same code `mx check` does — parity regression per ported rule. +- Each ported rule: a `mdl-examples/check-gap/` fixture + a `validate_*`/lint test. + +## Open Questions + +1. **IP / licensing (blocking the data-asset shape).** The message *text* is Mendix's + proprietary copyrighted content extracted from their DLL. Committing all 1318 verbatim + into the repo is redistribution. Options: (a) ship only **code IDs + severity + source + + our own paraphrased messages**; (b) generate the full catalog **locally at build time** + from the user's own licensed mxbuild (never committed); (c) confirm redistribution is + permitted. Default: **(b)** — commit the extractor, not the corpus. +2. **`monodis` dependency.** The extractor needs `mono-utils`. Alternatives: `ilspycmd` + (dotnet tool) or a small pure-Go PE manifest-resource reader to drop the mono dep. +3. **Tiering is heuristic.** The A/B/C/D split is keyword+subsystem-based; the real + per-rule trigger predicate lives in Mendix's `.cs` (the `LOCATION` breadcrumb) and must + be reverse-engineered per rule. The catalog gives the *target + message*, not the + predicate — sizing must account for that. +4. **Overlap with `PROPOSAL_check_mxbuild_gap_heuristics.md`.** Fold that proposal's 6 cases + in as the first harvested batch, or keep it separate and cross-link? (Recommend: keep it + as the tactical pilot; this proposal owns the methodology + backlog.) diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index 0cabcc190..2b0df5589 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -26,7 +26,7 @@ for display in this README): -## Active Proposals (82) +## Active Proposals (83) ### In Progress (partial) (8) @@ -82,7 +82,7 @@ for display in this README): | [VS Code Search — Quick Pick + Workspace Symbol](PROPOSAL_vscode_search.md) | Proposed | Full-text search exists in mxcli (mxcli search) but is only accessible via the terminal. | | [Workflow Improvements: ALTER WORKFLOW + Cross-References](PROPOSAL_workflow_improvements.md) | Proposed | Workflow support in mxcli has full CREATE/DESCRIBE/DROP/SHOW coverage with 13 activity types and BSON round-trip fidelity. | -### Draft (38) +### Draft (39) | Proposal | Status | Summary | |----------|--------|---------| @@ -116,6 +116,7 @@ for display in this README): | [Multi-Project Tree View](PROPOSAL_multi_project_tree.md) | Draft | Mendix solutions increasingly span multiple applications: a frontend and a backend, or a microservices landscape (product catalog, orders, f | | [mxcli auth — Mendix Platform Authentication](PROPOSAL_platform_auth.md) | Draft | A growing set of mxcli features need to talk to Mendix platform APIs on behalf of the user: | | [mxcli catalog — Mendix Catalog Integration](PROPOSAL_catalog_integration.md) | Draft | ⚠️ TERMINOLOGY NOTE: This proposal covers the external Mendix Catalog service at catalog.mendix.com (CLI: mxcli catalog search), which is se | +| [mxcli check — mine Mendix's diagnostics catalog to close the check↔mxbuild gap](PROPOSAL_check_diagnostics_catalog.md) | Draft | Relates to: PROPOSAL_check_mxbuild_gap_heuristics.md (one tactical slice of this — | | [mxcli Playground](mxcli-playground.md) | Draft | A public GitHub repository (mendixlabs/mxcli-playground) containing a ready-to-use Mendix project pre-configured with mxcli, Claude Code ski | | [Playwright Session Reuse and Lifecycle Control](PROPOSAL_playwright_session_reuse.md) | Draft | Builds on proposal-playwright-cli.md, which | | [Project Brain — Persistent Knowledge and Session Scaffolding for Long-Term AI Collaboration](PROPOSAL_project_brain.md) | Draft | mxcli is developed with significant AI involvement, yet the project's knowledge infrastructure is built for static human reading rather than | diff --git a/scripts/extract-diagnostics-catalog.sh b/scripts/extract-diagnostics-catalog.sh new file mode 100755 index 000000000..4d5285dc6 --- /dev/null +++ b/scripts/extract-diagnostics-catalog.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# +# extract-diagnostics-catalog.sh — mine Mendix's consistency-diagnostics catalog +# (CE/CW/CI codes) from Mendix.Modeler.Texts.dll into structured, tiered data. +# +# The SCRIPT is ours; its OUTPUT is Mendix's proprietary message corpus — do NOT +# commit the generated catalog.{csv,json}. See PROPOSAL_check_diagnostics_catalog.md. +# +# Usage: scripts/extract-diagnostics-catalog.sh [VERSION] [LOCALE] [OUTDIR] +# Example: scripts/extract-diagnostics-catalog.sh 11.12.1 en-US /tmp/diagcat +# Requires: curl, tar, monodis (apt-get install -y mono-utils), python3 +set -euo pipefail + +VERSION="${1:-11.12.1}" +LOCALE="${2:-en-US}" +OUTDIR="${3:-./diag-catalog-$VERSION}" +DLL="modeler/Mendix.Modeler.Texts.dll" +URL="https://cdn.mendix.com/runtime/mxbuild-${VERSION}.tar.gz" + +command -v monodis >/dev/null || { echo "ERROR: monodis not found (apt-get install -y mono-utils)" >&2; exit 1; } +mkdir -p "$OUTDIR"; cd "$OUTDIR" + +echo "[1/3] streaming $DLL from $URL ..." +# tar --occurrence=1 exits after the one member, closing the pipe early; curl then +# gets a (harmless) SIGPIPE/write-failure. Tolerate it and verify by file existence. +set +o pipefail +curl -fsSL "$URL" 2>/dev/null | tar -xz --occurrence=1 "$DLL" || true +set -o pipefail +[ -f "$DLL" ] || { echo "ERROR: failed to extract $DLL from $URL" >&2; exit 1; } + +echo "[2/3] extracting embedded PO resources via monodis ..." +mkdir -p po && ( cd po && monodis --mresources "../$DLL" >/dev/null 2>&1 ) +PO="$(ls po/*problem-descriptions_${LOCALE}.po | grep -vE 'deprecated|test' | head -1)" +[ -n "$PO" ] || { echo "ERROR: primary $LOCALE PO not found" >&2; exit 1; } + +echo "[3/3] parsing + tiering $PO ..." +python3 - "$PO" "$VERSION" <<'PY' +import re, json, csv, sys, collections +po, version = sys.argv[1], sys.argv[2] +SEV = {"CE": "error", "CW": "warning", "CI": "info"} +# static-checkability tiers (heuristic: message text + source subsystem) +D = ['version control','marketplace','app store','environment','deployment','license', + 'certificate','migration','team server','commit','branch','merge','governance'] +C = ['data source','react client','not supported in','offline','native','reachable', + 'end event','unreachable','navigation','home page','loop'] +B = ['xpath','expression','type of','of type','cannot be converted','incompatible', + 'return type','data type','parameter','must return','regular expression','template'] +def tier(blob): + if any(k in blob for k in D): return 'D' + if any(k in blob for k in C): return 'C' + if any(k in blob for k in B): return 'B' + return 'A' + +code_re = re.compile(r'^(C[EWI])(\d+)__(.*)$') +rows, loc = [], [] +lines = open(po, encoding='utf-8').read().splitlines() +i = 0 +while i < len(lines): + ln = lines[i] + if ln.startswith('# LOCATION:'): + loc.append(re.sub(r'\s*\(in project.*', '', ln.split('LOCATION:', 1)[1]).strip()) + elif ln.startswith('# NOTE:') and 'MX_DOCS_CAT' in ln: + docs = ln.split('MX_DOCS_CAT', 1)[1].lstrip(':(colon) ').strip() + elif ln.startswith('msgid "'): + mid = ln[7:-1] + j = i + 1 + while j < len(lines) and not lines[j].startswith('msgstr'): j += 1 + msg = '' + if j < len(lines): + msg = lines[j][8:].rstrip('"') + k = j + 1 + while k < len(lines) and lines[k].startswith('"'): + msg += lines[k].strip().strip('"'); k += 1 + m = code_re.match(mid) + if m: + params = sorted(set(re.findall(r'\{([A-Z0-9_]+)\}', msg))) + rows.append(dict(code=m.group(1)+m.group(2), severity=SEV[m.group(1)], + slug=m.group(3), message=msg, params=params, + locations=sorted(set(loc)), + docs_cat=locals().get('docs'), + tier=tier((m.group(3) + ' ' + msg + ' ' + ' '.join(loc)).lower()))) + loc, docs = [], None + i += 1 + +json.dump({"version": version, "codes": rows}, open("catalog.json", "w"), indent=1) +with open("catalog.csv", "w", newline='') as f: + w = csv.writer(f); w.writerow(["code","severity","tier","message","params","source","docs_cat"]) + for r in rows: + w.writerow([r['code'], r['severity'], r['tier'], r['message'], + ';'.join(r['params']), ';'.join(r['locations']), r['docs_cat'] or '']) + +sev = collections.Counter(r['severity'] for r in rows) +tiers = collections.Counter(r['tier'] for r in rows) +print(f" codes: {len(rows)} ({dict(sev)})") +print(f" tiers: {dict(sorted(tiers.items()))} (A=structural B=expr/type C=cross-doc D=runtime)") +print(f" parameterised: {sum(1 for r in rows if r['params'])}/{len(rows)}") +print(f" wrote catalog.json + catalog.csv (LOCAL ONLY — Mendix's corpus, do not commit)") +PY +echo "Done: $OUTDIR/catalog.{json,csv}" From 0a67c477fc7bbe392abdf4a07a0175d3089d1f70 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 04:44:23 +0000 Subject: [PATCH 11/26] feat(docker): mxbuild --serve client (warm incremental build) for run --local MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First increment of the warm dev loop (PROPOSAL_mxcli_dev_warm_loop.md, slice 1): a Go client for `mxbuild --serve`. StartServe launches the build server (reusing resolveJDK21 + PrepareMxCommand's FreeType LD_PRELOAD), Build POSTs /build and parses {status, restartRequired} — the signal for reload_model vs restart — and Stop shuts it down. - mxserve.go: ServeServer (StartServe/Build/Stop), BuildRequest/BuildResult. - mxserve_test.go: 5 unit tests over the HTTP client (mocked endpoint). - mxserve_integration_test.go: gated real end-to-end; measured cold 21.8s -> warm 1.6s incremental against the spike app. Naming: this backs the chosen `mxcli run --local` command (wiring in a later increment). Next: warm the roundtrip test harness (replace cold `mx check`). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/docker/mxserve.go | 239 +++++++++++++++++++ cmd/mxcli/docker/mxserve_integration_test.go | 56 +++++ cmd/mxcli/docker/mxserve_test.go | 123 ++++++++++ 3 files changed, 418 insertions(+) create mode 100644 cmd/mxcli/docker/mxserve.go create mode 100644 cmd/mxcli/docker/mxserve_integration_test.go create mode 100644 cmd/mxcli/docker/mxserve_test.go diff --git a/cmd/mxcli/docker/mxserve.go b/cmd/mxcli/docker/mxserve.go new file mode 100644 index 000000000..0bf24a6fd --- /dev/null +++ b/cmd/mxcli/docker/mxserve.go @@ -0,0 +1,239 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + "os/exec" + "path/filepath" + "sync" + "syscall" + "time" +) + +// mxbuild --serve is an incremental build server: it keeps the model loaded and +// rebuilds only what changed, so a warm build is ~1s vs ~30-60s for a one-shot +// `mxbuild` invocation. It speaks HTTP: POST /build with a JSON request, and the +// response carries `status` plus `restartRequired` (the signal for whether a +// runtime restart is needed vs a hot `reload_model`). See +// docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md. + +// ServeOptions configures StartServe. +type ServeOptions struct { + // MxBuildPath is the mxbuild binary. When empty it is resolved from Version + // (CachedMxBuildPath) or the newest cached mxbuild (AnyCachedMxBuildPath). + MxBuildPath string + // Version is the Mendix version used to resolve mxbuild when MxBuildPath is empty. + Version string + // JavaHome is the JDK 21 home. When empty it is resolved via resolveJDK21(). + JavaHome string + // Host to bind the serve HTTP API (default 127.0.0.1). + Host string + // Port for the serve HTTP API (default 6543). + Port int + // ReadyTimeout bounds how long StartServe waits for the API (default 90s). + ReadyTimeout time.Duration +} + +// ServeTarget is the build target for a serve request. +type ServeTarget string + +const ( + // TargetDeploy writes the deployment structure directly into the project's + // deployment dir (no .mda) — the form the runtime reads for reload_model. + TargetDeploy ServeTarget = "Deploy" + // TargetPackage produces an .mda at MdaFilePath. + TargetPackage ServeTarget = "Package" +) + +// BuildRequest mirrors the mxbuild serve /build request schema. +type BuildRequest struct { + Target ServeTarget `json:"target"` + ProjectFilePath string `json:"projectFilePath"` + MdaFilePath string `json:"mdaFilePath,omitempty"` + UseLooseVersionCheck bool `json:"useLooseVersionCheck,omitempty"` +} + +// BuildResult is the parsed serve /build response. Raw holds the full body for +// callers that need startup_metrics or other fields. +type BuildResult struct { + Status string `json:"status"` + RestartRequired bool `json:"restartRequired"` + Message string `json:"message"` + Raw json.RawMessage `json:"-"` +} + +// OK reports whether the build succeeded. +func (r *BuildResult) OK() bool { return r.Status == "Success" } + +// ServeServer wraps a long-lived `mxbuild --serve` process and its build API. +type ServeServer struct { + Host string + Port int + cmd *exec.Cmd + log *syncBuffer +} + +// StartServe launches `mxbuild --serve` and blocks until the build API responds. +// Call Stop() to shut it down. The first Build() loads the model (cold, ~10-15s); +// subsequent builds are incremental (~1s). +func StartServe(opts ServeOptions) (*ServeServer, error) { + mxbuildPath := opts.MxBuildPath + if mxbuildPath == "" && opts.Version != "" { + mxbuildPath = CachedMxBuildPath(opts.Version) + } + if mxbuildPath == "" { + mxbuildPath = AnyCachedMxBuildPath() + } + if mxbuildPath == "" { + return nil, fmt.Errorf("mxbuild not found; run 'mxcli setup mxbuild -p ' or pass ServeOptions.MxBuildPath") + } + + javaHome := opts.JavaHome + if javaHome == "" { + jh, err := resolveJDK21() + if err != nil { + return nil, err + } + javaHome = jh + } + javaExe := filepath.Join(javaHome, "bin", "java") + + host := opts.Host + if host == "" { + host = "127.0.0.1" + } + port := opts.Port + if port == 0 { + port = 6543 + } + + cmd := exec.Command(mxbuildPath, "--serve", + fmt.Sprintf("--host=%s", host), + fmt.Sprintf("--port=%d", port), + fmt.Sprintf("--java-home=%s", javaHome), + fmt.Sprintf("--java-exe-path=%s", javaExe), + ) + PrepareMxCommand(cmd) // FreeType LD_PRELOAD workaround + log := &syncBuffer{} + cmd.Stdout = log + cmd.Stderr = log + + if err := cmd.Start(); err != nil { + return nil, fmt.Errorf("starting mxbuild --serve: %w", err) + } + + s := &ServeServer{Host: host, Port: port, cmd: cmd, log: log} + + readyTimeout := opts.ReadyTimeout + if readyTimeout == 0 { + readyTimeout = 90 * time.Second + } + if err := s.waitReady(readyTimeout); err != nil { + _ = s.Stop() + return nil, fmt.Errorf("mxbuild --serve did not become ready: %w\n--- mxbuild output ---\n%s", err, s.log.String()) + } + return s, nil +} + +func (s *ServeServer) baseURL() string { return fmt.Sprintf("http://%s:%d", s.Host, s.Port) } + +// waitReady polls the build endpoint until it responds (an empty POST yields a +// JSON validation error, which still proves the HTTP server is up). +func (s *ServeServer) waitReady(timeout time.Duration) error { + deadline := time.Now().Add(timeout) + client := &http.Client{Timeout: 3 * time.Second} + for time.Now().Before(deadline) { + resp, err := client.Post(s.baseURL()+"/build", "application/json", bytes.NewReader([]byte("{}"))) + if err == nil { + resp.Body.Close() + return nil + } + if !s.alive() { + return fmt.Errorf("mxbuild --serve exited during startup") + } + time.Sleep(500 * time.Millisecond) + } + return fmt.Errorf("timed out after %s", timeout) +} + +// Build sends a build request to the warm server and parses the result. The +// caller inspects RestartRequired to decide reload_model vs restart. +func (s *ServeServer) Build(req BuildRequest) (*BuildResult, error) { + if req.Target == "" { + req.Target = TargetDeploy + } + bodyBytes, err := json.Marshal(req) + if err != nil { + return nil, fmt.Errorf("marshaling build request: %w", err) + } + // Cold builds can take ~60s; keep a generous timeout. + client := &http.Client{Timeout: 10 * time.Minute} + resp, err := client.Post(s.baseURL()+"/build", "application/json", bytes.NewReader(bodyBytes)) + if err != nil { + return nil, fmt.Errorf("serve build request to %s: %w", s.baseURL(), err) + } + defer resp.Body.Close() + + raw, err := io.ReadAll(resp.Body) + if err != nil { + return nil, fmt.Errorf("reading serve response: %w", err) + } + var res BuildResult + if err := json.Unmarshal(raw, &res); err != nil { + return nil, fmt.Errorf("decoding serve response (HTTP %d): %w -- body: %s", resp.StatusCode, err, string(raw)) + } + res.Raw = raw + return &res, nil +} + +// alive reports whether the serve process is still running (Linux: signal 0). +func (s *ServeServer) alive() bool { + if s.cmd == nil || s.cmd.Process == nil { + return false + } + return s.cmd.Process.Signal(syscall.Signal(0)) == nil +} + +// Log returns the captured mxbuild --serve output (for diagnostics). +func (s *ServeServer) Log() string { return s.log.String() } + +// Stop terminates the serve process (SIGTERM, then SIGKILL after a grace period). +func (s *ServeServer) Stop() error { + if s.cmd == nil || s.cmd.Process == nil { + return nil + } + _ = s.cmd.Process.Signal(syscall.SIGTERM) + done := make(chan error, 1) + go func() { done <- s.cmd.Wait() }() + select { + case <-done: + case <-time.After(5 * time.Second): + _ = s.cmd.Process.Kill() + <-done + } + return nil +} + +// syncBuffer is a goroutine-safe bytes.Buffer for capturing subprocess output +// while it is read concurrently for diagnostics. +type syncBuffer struct { + mu sync.Mutex + b bytes.Buffer +} + +func (w *syncBuffer) Write(p []byte) (int, error) { + w.mu.Lock() + defer w.mu.Unlock() + return w.b.Write(p) +} + +func (w *syncBuffer) String() string { + w.mu.Lock() + defer w.mu.Unlock() + return w.b.String() +} diff --git a/cmd/mxcli/docker/mxserve_integration_test.go b/cmd/mxcli/docker/mxserve_integration_test.go new file mode 100644 index 000000000..2c7f3ee3f --- /dev/null +++ b/cmd/mxcli/docker/mxserve_integration_test.go @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "testing" + "time" +) + +// TestServeIntegration exercises the real StartServe -> Build -> Stop path +// against an actual `mxbuild --serve`. Skipped unless mxbuild is cached and a +// test .mpr is available (set MXSERVE_TEST_MPR, or it defaults to the spike app). +func TestServeIntegration(t *testing.T) { + if testing.Short() { + t.Skip("skipping serve integration test in -short mode") + } + if AnyCachedMxBuildPath() == "" { + t.Skip("no mxbuild cached; run 'mxcli setup mxbuild -p '") + } + mpr := os.Getenv("MXSERVE_TEST_MPR") + if mpr == "" { + mpr = "/tmp/spikeapp/SpikeApp.mpr" + } + if _, err := os.Stat(mpr); err != nil { + t.Skipf("no test .mpr at %s; set MXSERVE_TEST_MPR", mpr) + } + + srv, err := StartServe(ServeOptions{Port: 6555}) + if err != nil { + t.Fatalf("StartServe: %v", err) + } + defer srv.Stop() + + t0 := time.Now() + cold, err := srv.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: mpr}) + if err != nil { + t.Fatalf("cold build: %v\n--- serve log ---\n%s", err, srv.Log()) + } + if !cold.OK() { + t.Fatalf("cold build not OK: status=%s msg=%s", cold.Status, cold.Message) + } + coldDur := time.Since(t0) + + t1 := time.Now() + warm, err := srv.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: mpr}) + if err != nil { + t.Fatalf("warm build: %v", err) + } + if !warm.OK() { + t.Fatalf("warm build not OK: status=%s", warm.Status) + } + warmDur := time.Since(t1) + + t.Logf("cold=%s warm=%s (incremental serve should make warm << cold)", coldDur, warmDur) +} diff --git a/cmd/mxcli/docker/mxserve_test.go b/cmd/mxcli/docker/mxserve_test.go new file mode 100644 index 000000000..d0ef5d967 --- /dev/null +++ b/cmd/mxcli/docker/mxserve_test.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "testing" +) + +// newTestServe returns a ServeServer wired to an httptest server, so the HTTP +// client (Build) can be tested without spawning mxbuild. +func newTestServe(t *testing.T, handler http.HandlerFunc) *ServeServer { + t.Helper() + ts := httptest.NewServer(handler) + t.Cleanup(ts.Close) + u, err := url.Parse(ts.URL) + if err != nil { + t.Fatalf("parse test URL: %v", err) + } + host, portStr, err := net.SplitHostPort(u.Host) + if err != nil { + t.Fatalf("split host:port: %v", err) + } + port, _ := strconv.Atoi(portStr) + return &ServeServer{Host: host, Port: port} +} + +func TestServeBuild_DeployRestartRequired(t *testing.T) { + s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/build" { + t.Errorf("path = %q, want /build", r.URL.Path) + } + if r.Method != http.MethodPost { + t.Errorf("method = %q, want POST", r.Method) + } + var req BuildRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Errorf("decode request: %v", err) + } + if req.Target != TargetDeploy { + t.Errorf("target = %q, want Deploy (default)", req.Target) + } + if req.ProjectFilePath != "/x/App.mpr" { + t.Errorf("projectFilePath = %q", req.ProjectFilePath) + } + _, _ = w.Write([]byte(`{"restartRequired": true, "status": "Success"}`)) + }) + + res, err := s.Build(BuildRequest{ProjectFilePath: "/x/App.mpr"}) // Target empty -> Deploy + if err != nil { + t.Fatalf("Build: %v", err) + } + if !res.OK() { + t.Errorf("OK() = false, status = %q", res.Status) + } + if !res.RestartRequired { + t.Error("RestartRequired = false, want true (domain/view-entity change)") + } +} + +func TestServeBuild_HotReloadable(t *testing.T) { + s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`{"restartRequired": false, "status": "Success"}`)) + }) + res, err := s.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: "/x/App.mpr"}) + if err != nil { + t.Fatalf("Build: %v", err) + } + if !res.OK() { + t.Errorf("OK() = false, status = %q", res.Status) + } + if res.RestartRequired { + t.Error("RestartRequired = true, want false (microflow/page change -> reload_model)") + } +} + +func TestServeBuild_Failure(t *testing.T) { + s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusBadRequest) + _, _ = w.Write([]byte(`{"status": "Failure", "message": "Specified 'target' is invalid."}`)) + }) + res, err := s.Build(BuildRequest{Target: "Bogus", ProjectFilePath: "/x/App.mpr"}) + if err != nil { + t.Fatalf("Build should parse the failure envelope, got transport error: %v", err) + } + if res.OK() { + t.Error("OK() = true, want false") + } + if res.Message == "" { + t.Error("Message empty, want the failure message") + } +} + +func TestServeBuild_PackageTargetSendsMdaPath(t *testing.T) { + s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { + var req BuildRequest + _ = json.NewDecoder(r.Body).Decode(&req) + if req.Target != TargetPackage { + t.Errorf("target = %q, want Package", req.Target) + } + if req.MdaFilePath != "/out/app.mda" { + t.Errorf("mdaFilePath = %q, want /out/app.mda", req.MdaFilePath) + } + _, _ = w.Write([]byte(`{"restartRequired": true, "status": "Success"}`)) + }) + if _, err := s.Build(BuildRequest{Target: TargetPackage, ProjectFilePath: "/x/App.mpr", MdaFilePath: "/out/app.mda"}); err != nil { + t.Fatalf("Build: %v", err) + } +} + +func TestServeBuild_BadJSONBody(t *testing.T) { + s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`not json`)) + }) + if _, err := s.Build(BuildRequest{ProjectFilePath: "/x/App.mpr"}); err == nil { + t.Error("expected an error decoding a non-JSON body") + } +} From 111bbba7628eaf84cdeaf250b8c696d6495086af Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 05:06:53 +0000 Subject: [PATCH 12/26] feat(docker): preflight the mxbuild runtime/ dir in StartServe A serve (or one-shot) build fails Java compilation with a cryptic "package com.mendix.* does not exist" when the mxbuild cache is missing its sibling runtime/ dir (the Mendix API libraries on the javac classpath). verifyMxBuildCache fails fast in StartServe with an actionable message instead. Found while verifying --serve on 11.12.1 (cold 10.2s -> warm 1.1s once the runtime/ dir was present). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/docker/mxserve.go | 20 ++++++++++++++++++++ cmd/mxcli/docker/mxserve_test.go | 28 ++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/cmd/mxcli/docker/mxserve.go b/cmd/mxcli/docker/mxserve.go index 0bf24a6fd..23f235e83 100644 --- a/cmd/mxcli/docker/mxserve.go +++ b/cmd/mxcli/docker/mxserve.go @@ -8,6 +8,7 @@ import ( "fmt" "io" "net/http" + "os" "os/exec" "path/filepath" "sync" @@ -78,6 +79,22 @@ type ServeServer struct { log *syncBuffer } +// verifyMxBuildCache checks the mxbuild cache is complete: the sibling runtime/ +// directory (which holds the Mendix API libraries on the javac classpath) must +// exist next to modeler/. Without it, Java compilation fails with a cryptic +// "package com.mendix.* does not exist" — in both serve and one-shot builds. A +// normal `mxcli setup mxbuild` / DownloadMxBuild extracts both dirs; this guards +// against a partially-populated cache. mxbuildPath is /modeler/mxbuild, +// so the runtime dir is /runtime. +func verifyMxBuildCache(mxbuildPath string) error { + runtimeDir := filepath.Join(filepath.Dir(filepath.Dir(mxbuildPath)), "runtime") + if fi, err := os.Stat(runtimeDir); err != nil || !fi.IsDir() { + return fmt.Errorf("mxbuild cache incomplete: missing runtime directory %s "+ + "(Java compilation needs the Mendix runtime libraries) -- re-run 'mxcli setup mxbuild -p '", runtimeDir) + } + return nil +} + // StartServe launches `mxbuild --serve` and blocks until the build API responds. // Call Stop() to shut it down. The first Build() loads the model (cold, ~10-15s); // subsequent builds are incremental (~1s). @@ -92,6 +109,9 @@ func StartServe(opts ServeOptions) (*ServeServer, error) { if mxbuildPath == "" { return nil, fmt.Errorf("mxbuild not found; run 'mxcli setup mxbuild -p ' or pass ServeOptions.MxBuildPath") } + if err := verifyMxBuildCache(mxbuildPath); err != nil { + return nil, err + } javaHome := opts.JavaHome if javaHome == "" { diff --git a/cmd/mxcli/docker/mxserve_test.go b/cmd/mxcli/docker/mxserve_test.go index d0ef5d967..3c0f774a6 100644 --- a/cmd/mxcli/docker/mxserve_test.go +++ b/cmd/mxcli/docker/mxserve_test.go @@ -8,6 +8,8 @@ import ( "net/http" "net/http/httptest" "net/url" + "os" + "path/filepath" "strconv" "testing" ) @@ -113,6 +115,32 @@ func TestServeBuild_PackageTargetSendsMdaPath(t *testing.T) { } } +func TestVerifyMxBuildCache(t *testing.T) { + // layout: /modeler/mxbuild and /runtime + cache := t.TempDir() + modeler := filepath.Join(cache, "modeler") + if err := os.MkdirAll(modeler, 0o755); err != nil { + t.Fatal(err) + } + mxbuild := filepath.Join(modeler, "mxbuild") + if err := os.WriteFile(mxbuild, []byte("#!/bin/sh\n"), 0o755); err != nil { + t.Fatal(err) + } + + // runtime/ missing -> error + if err := verifyMxBuildCache(mxbuild); err == nil { + t.Error("expected error when runtime/ dir is missing") + } + + // runtime/ present -> ok + if err := os.MkdirAll(filepath.Join(cache, "runtime"), 0o755); err != nil { + t.Fatal(err) + } + if err := verifyMxBuildCache(mxbuild); err != nil { + t.Errorf("expected no error when runtime/ present, got %v", err) + } +} + func TestServeBuild_BadJSONBody(t *testing.T) { s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte(`not json`)) From e92de16eeb6d1ce41f17cf4e492632017cb92380 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 05:23:06 +0000 Subject: [PATCH 13/26] feat(docker): add RuntimeController for the local warm-dev loop RuntimeController wraps the M2EE admin API for the standalone `run --local` loop (increment 3 of the warm-dev-loop proposal). It is the novel, fully unit-testable core of that command, separate from process launch and cobra wiring which follow. - ApplyBuild maps a serve build's restartRequired flag to the apply action: false -> hot reload_model (page/microflow/text change), true -> relaunch the runtime (entity/view/association change, which the runtime reconciles in its metamodel catalog only at startup) then run the start cycle. - Start models the DB-aware start sequence discovered this session: start -> if the runtime reports the schema is out of date -> execute_ddl_commands -> start. DDL is discovered during the start cycle, not by a pre-check on the running (old-model) runtime. - DecideApply and needsDBUpdate are pure and documented in one place. This differs from docker's Reload(): that path warns on pending DDL and tells the user to restart; the controller actually executes the DDL and drives the restart, off the serve restartRequired signal. Tests use an httptest mock admin that records the action sequence, asserting: reload vs restart branch, the clean vs out-of-date start cycle, DDL failure abort, status parsing, and the pure-function cases. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/docker/runtime_controller.go | 153 +++++++++++++ cmd/mxcli/docker/runtime_controller_test.go | 242 ++++++++++++++++++++ 2 files changed, 395 insertions(+) create mode 100644 cmd/mxcli/docker/runtime_controller.go create mode 100644 cmd/mxcli/docker/runtime_controller_test.go diff --git a/cmd/mxcli/docker/runtime_controller.go b/cmd/mxcli/docker/runtime_controller.go new file mode 100644 index 000000000..23b3f3b8d --- /dev/null +++ b/cmd/mxcli/docker/runtime_controller.go @@ -0,0 +1,153 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "strings" +) + +// RuntimeController drives the Mendix runtime admin (M2EE) API for the local dev +// loop: hot-reload the model, run the DB-aware start cycle, and apply a serve +// build result via the restartRequired branch. It orchestrates admin actions; +// the actual runtime process launch/relaunch is the caller's responsibility (a +// domain/view/association change needs a fresh runtime start — see +// docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md § Hot-reload scope). +type RuntimeController struct { + opts M2EEOptions +} + +// NewRuntimeController returns a controller for the given admin API connection. +func NewRuntimeController(opts M2EEOptions) *RuntimeController { + return &RuntimeController{opts: opts} +} + +// ApplyAction is the action taken for a build result. +type ApplyAction int + +const ( + // ActionReload: hot reload_model (no restart) — page/microflow/text change. + ActionReload ApplyAction = iota + // ActionRestart: the runtime must be relaunched — entity/view/association + // change (the runtime reconciles its metamodel catalog only at startup). + ActionRestart +) + +func (a ApplyAction) String() string { + if a == ActionReload { + return "reload" + } + return "restart" +} + +// DecideApply maps a build's restartRequired flag to the apply action. Kept +// separate so the decision is trivially testable and documented in one place. +func DecideApply(restartRequired bool) ApplyAction { + if restartRequired { + return ActionRestart + } + return ActionReload +} + +// ReloadModel hot-reloads the model into the running runtime (model store + +// microflow engine + i18n), draining in-flight actions first. No process +// restart, no DDL. Use only when the build reported restartRequired=false. +func (c *RuntimeController) ReloadModel() error { + resp, err := CallM2EE(c.opts, "reload_model", nil) + if err != nil { + return err + } + if msg := resp.M2EEError(); msg != "" { + return fmt.Errorf("reload_model failed: %s", msg) + } + return nil +} + +// Start runs the runtime start sequence, handling an empty or out-of-date +// database: start -> if the runtime reports the schema must change -> +// execute_ddl_commands -> start. Returns the final start response. +func (c *RuntimeController) Start() (*M2EEResponse, error) { + resp, err := CallM2EE(c.opts, "start", nil) + if err != nil { + return nil, err + } + if needsDBUpdate(resp) { + ddl, err := CallM2EE(c.opts, "execute_ddl_commands", nil) + if err != nil { + return nil, err + } + if msg := ddl.M2EEError(); msg != "" { + return nil, fmt.Errorf("execute_ddl_commands failed: %s", msg) + } + resp, err = CallM2EE(c.opts, "start", nil) + if err != nil { + return nil, err + } + } + if msg := resp.M2EEError(); msg != "" { + return resp, fmt.Errorf("start failed: %s", msg) + } + return resp, nil +} + +// RuntimeStatus returns the runtime status string (e.g. "running", "starting"). +func (c *RuntimeController) RuntimeStatus() (string, error) { + resp, err := CallM2EE(c.opts, "runtime_status", nil) + if err != nil { + return "", err + } + fb := resp.Feedback() + if fb == nil { + return "", nil + } + status, _ := fb["status"].(string) + return status, nil +} + +// ApplyBuild applies a serve build result to the running runtime: +// - restartRequired=false -> reload_model (hot, done here). +// - restartRequired=true -> relaunch (via the caller's restart func) then run +// the DB-aware Start cycle. +// +// restart may be nil when the caller drives the relaunch itself; in that case a +// restart-required build only returns ActionRestart (no admin calls are made). +func (c *RuntimeController) ApplyBuild(build *BuildResult, restart func() error) (ApplyAction, error) { + if build == nil { + return ActionReload, fmt.Errorf("nil build result") + } + action := DecideApply(build.RestartRequired) + if action == ActionReload { + return action, c.ReloadModel() + } + if restart == nil { + return action, nil + } + if err := restart(); err != nil { + return action, fmt.Errorf("restarting runtime: %w", err) + } + if _, err := c.Start(); err != nil { + return action, err + } + return action, nil +} + +// needsDBUpdate reports whether a start response indicates the database schema +// must be updated before the runtime can serve (result 3 / "database has to be +// updated" / a synchronizationreason in the feedback). +func needsDBUpdate(resp *M2EEResponse) bool { + if resp == nil { + return false + } + if resp.Result == 3 { + return true + } + if strings.Contains(strings.ToLower(resp.Message), "database has to be updated") { + return true + } + if fb := resp.Feedback(); fb != nil { + if _, ok := fb["synchronizationreason"]; ok { + return true + } + } + return false +} diff --git a/cmd/mxcli/docker/runtime_controller_test.go b/cmd/mxcli/docker/runtime_controller_test.go new file mode 100644 index 000000000..d2e20cfa2 --- /dev/null +++ b/cmd/mxcli/docker/runtime_controller_test.go @@ -0,0 +1,242 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "testing" +) + +// mockAdmin is a fake M2EE admin API. It records the sequence of actions it +// received and returns a scripted response per action. A response func may +// return a different value on each call (e.g. start-then-DDL-then-start). +type mockAdmin struct { + calls []string + handlers map[string]func(call int) M2EEResponse + seen map[string]int +} + +func newMockAdmin(t *testing.T, handlers map[string]func(call int) M2EEResponse) (*mockAdmin, M2EEOptions) { + t.Helper() + m := &mockAdmin{handlers: handlers, seen: map[string]int{}} + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req struct { + Action string `json:"action"` + } + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Errorf("decode admin request: %v", err) + } + m.calls = append(m.calls, req.Action) + h, ok := m.handlers[req.Action] + if !ok { + // Default: success with empty feedback. + _ = json.NewEncoder(w).Encode(M2EEResponse{}) + return + } + resp := h(m.seen[req.Action]) + m.seen[req.Action]++ + _ = json.NewEncoder(w).Encode(resp) + })) + t.Cleanup(ts.Close) + + u, err := url.Parse(ts.URL) + if err != nil { + t.Fatalf("parse test URL: %v", err) + } + host, portStr, err := net.SplitHostPort(u.Host) + if err != nil { + t.Fatalf("split host:port: %v", err) + } + port, _ := strconv.Atoi(portStr) + // Direct + Token bypasses docker-exec and .env resolution. + return m, M2EEOptions{Host: host, Port: port, Token: "test", Direct: true} +} + +func ok(int) M2EEResponse { return M2EEResponse{} } + +func TestDecideApply(t *testing.T) { + if got := DecideApply(false); got != ActionReload { + t.Errorf("DecideApply(false) = %v, want ActionReload", got) + } + if got := DecideApply(true); got != ActionRestart { + t.Errorf("DecideApply(true) = %v, want ActionRestart", got) + } + if ActionReload.String() != "reload" || ActionRestart.String() != "restart" { + t.Errorf("String() = %q/%q", ActionReload.String(), ActionRestart.String()) + } +} + +func TestReloadModel(t *testing.T) { + m, opts := newMockAdmin(t, map[string]func(int) M2EEResponse{"reload_model": ok}) + if err := NewRuntimeController(opts).ReloadModel(); err != nil { + t.Fatalf("ReloadModel: %v", err) + } + if len(m.calls) != 1 || m.calls[0] != "reload_model" { + t.Errorf("calls = %v, want [reload_model]", m.calls) + } +} + +func TestReloadModel_Error(t *testing.T) { + m, opts := newMockAdmin(t, map[string]func(int) M2EEResponse{ + "reload_model": func(int) M2EEResponse { return M2EEResponse{Result: 1, Message: "boom"} }, + }) + if err := NewRuntimeController(opts).ReloadModel(); err == nil { + t.Error("expected error from a non-zero result") + } + if len(m.calls) != 1 { + t.Errorf("calls = %v", m.calls) + } +} + +func TestStart_CleanDatabase(t *testing.T) { + m, opts := newMockAdmin(t, map[string]func(int) M2EEResponse{"start": ok}) + if _, err := NewRuntimeController(opts).Start(); err != nil { + t.Fatalf("Start: %v", err) + } + if len(m.calls) != 1 || m.calls[0] != "start" { + t.Errorf("calls = %v, want a single [start]", m.calls) + } +} + +func TestStart_OutOfDateDatabase(t *testing.T) { + // First start reports the schema must change (result 3); after DDL, start succeeds. + m, opts := newMockAdmin(t, map[string]func(int) M2EEResponse{ + "start": func(call int) M2EEResponse { + if call == 0 { + return M2EEResponse{Result: 3, Message: "database has to be updated"} + } + return M2EEResponse{} + }, + "execute_ddl_commands": ok, + }) + if _, err := NewRuntimeController(opts).Start(); err != nil { + t.Fatalf("Start: %v", err) + } + want := []string{"start", "execute_ddl_commands", "start"} + if len(m.calls) != 3 || m.calls[0] != want[0] || m.calls[1] != want[1] || m.calls[2] != want[2] { + t.Errorf("calls = %v, want %v", m.calls, want) + } +} + +func TestStart_DDLFails(t *testing.T) { + m, opts := newMockAdmin(t, map[string]func(int) M2EEResponse{ + "start": func(int) M2EEResponse { return M2EEResponse{Result: 3} }, + "execute_ddl_commands": func(int) M2EEResponse { return M2EEResponse{Result: 1, Message: "ddl boom"} }, + }) + if _, err := NewRuntimeController(opts).Start(); err == nil { + t.Error("expected error when execute_ddl_commands fails") + } + // start once, ddl once, then abort (no second start). + if len(m.calls) != 2 { + t.Errorf("calls = %v, want [start execute_ddl_commands]", m.calls) + } +} + +func TestRuntimeStatus(t *testing.T) { + m, opts := newMockAdmin(t, map[string]func(int) M2EEResponse{ + "runtime_status": func(int) M2EEResponse { + return M2EEResponse{RawFeedback: json.RawMessage(`{"status":"running"}`)} + }, + }) + status, err := NewRuntimeController(opts).RuntimeStatus() + if err != nil { + t.Fatalf("RuntimeStatus: %v", err) + } + if status != "running" { + t.Errorf("status = %q, want running", status) + } + if len(m.calls) != 1 || m.calls[0] != "runtime_status" { + t.Errorf("calls = %v", m.calls) + } +} + +func TestApplyBuild_HotReload(t *testing.T) { + m, opts := newMockAdmin(t, map[string]func(int) M2EEResponse{"reload_model": ok}) + restartCalled := false + action, err := NewRuntimeController(opts).ApplyBuild( + &BuildResult{Status: "Success", RestartRequired: false}, + func() error { restartCalled = true; return nil }, + ) + if err != nil { + t.Fatalf("ApplyBuild: %v", err) + } + if action != ActionReload { + t.Errorf("action = %v, want ActionReload", action) + } + if restartCalled { + t.Error("restart callback should not run for a hot-reloadable build") + } + if len(m.calls) != 1 || m.calls[0] != "reload_model" { + t.Errorf("calls = %v, want [reload_model]", m.calls) + } +} + +func TestApplyBuild_Restart(t *testing.T) { + m, opts := newMockAdmin(t, map[string]func(int) M2EEResponse{"start": ok}) + restartCalled := false + action, err := NewRuntimeController(opts).ApplyBuild( + &BuildResult{Status: "Success", RestartRequired: true}, + func() error { restartCalled = true; return nil }, + ) + if err != nil { + t.Fatalf("ApplyBuild: %v", err) + } + if action != ActionRestart { + t.Errorf("action = %v, want ActionRestart", action) + } + if !restartCalled { + t.Error("restart callback must run for a restart-required build") + } + // After relaunch, ApplyBuild runs the Start cycle (clean DB -> single start). + if len(m.calls) != 1 || m.calls[0] != "start" { + t.Errorf("calls = %v, want [start]", m.calls) + } +} + +func TestApplyBuild_RestartNilCallback(t *testing.T) { + // restart=nil means the caller drives relaunch; ApplyBuild makes no admin calls. + m, opts := newMockAdmin(t, nil) + action, err := NewRuntimeController(opts).ApplyBuild( + &BuildResult{Status: "Success", RestartRequired: true}, nil) + if err != nil { + t.Fatalf("ApplyBuild: %v", err) + } + if action != ActionRestart { + t.Errorf("action = %v, want ActionRestart", action) + } + if len(m.calls) != 0 { + t.Errorf("calls = %v, want none", m.calls) + } +} + +func TestApplyBuild_NilBuild(t *testing.T) { + _, opts := newMockAdmin(t, nil) + if _, err := NewRuntimeController(opts).ApplyBuild(nil, nil); err == nil { + t.Error("expected error for a nil build result") + } +} + +func TestNeedsDBUpdate(t *testing.T) { + cases := []struct { + name string + resp *M2EEResponse + want bool + }{ + {"nil", nil, false}, + {"clean", &M2EEResponse{}, false}, + {"result3", &M2EEResponse{Result: 3}, true}, + {"message", &M2EEResponse{Message: "The database has to be updated first"}, true}, + {"feedback", &M2EEResponse{RawFeedback: json.RawMessage(`{"synchronizationreason":"x"}`)}, true}, + {"other-error", &M2EEResponse{Result: 1, Message: "unrelated"}, false}, + } + for _, tc := range cases { + if got := needsDBUpdate(tc.resp); got != tc.want { + t.Errorf("%s: needsDBUpdate = %v, want %v", tc.name, got, tc.want) + } + } +} From 11b30fd93e75c02003f096eaee9dd4836a75382d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 05:27:14 +0000 Subject: [PATCH 14/26] feat(docker): add standalone runtime boot for the local warm loop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit localboot.go boots a Mendix runtime as a plain JVM (no Docker) for `mxcli run --local`, using the boot recipe reverse-engineered this session: launch runtimelauncher.jar with the M2EE/MX_INSTALL_PATH env, configure over the admin API (update_appcontainer_configuration + update_configuration), then run the RuntimeController start cycle. Key detail: a standalone runtime does NOT apply design-time constant defaults, so a missing constant 530s the app. mxbuild writes the resolved constants to /model/config.json; readDeploymentConstants lifts them and feeds them back as MicroflowConstants — so booting any mxbuild-generated app works without hand-listing its constants. LocalRuntime also exposes Restart() (re-launch + re-apply config, since config is per-process) for use as the RuntimeController.ApplyBuild restart callback, and Controller()/AppURL()/HealthOK()/Stop(). The pure config/env/path builders and the deployment-constants reader are unit-tested; process launch is left to an integration path (mxbuild + JDK). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/docker/localboot.go | 346 +++++++++++++++++++++++++++++ cmd/mxcli/docker/localboot_test.go | 188 ++++++++++++++++ 2 files changed, 534 insertions(+) create mode 100644 cmd/mxcli/docker/localboot.go create mode 100644 cmd/mxcli/docker/localboot_test.go diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go new file mode 100644 index 000000000..dc7cdc832 --- /dev/null +++ b/cmd/mxcli/docker/localboot.go @@ -0,0 +1,346 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "os/exec" + "path/filepath" + "syscall" + "time" +) + +// localboot.go boots a Mendix runtime as a plain JVM process (no Docker), for the +// warm local dev loop (`mxcli run --local`). The recipe was reverse-engineered +// this session and is documented in docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md: +// +// java -jar /runtime/launcher/runtimelauncher.jar (cwd = /runtime) +// env: M2EE_ADMIN_PASS, M2EE_ADMIN_PORT, M2EE_ADMIN_LISTEN_ADDRESSES, +// MX_INSTALL_PATH=, MX_LOG_LEVEL, plus the FreeType LD_PRELOAD fix. +// then over the M2EE admin API: +// update_appcontainer_configuration (runtime_port) +// update_configuration (BasePath, RuntimePath, DB, MicroflowConstants) +// start -> [database has to be updated -> execute_ddl_commands -> start] +// +// The design-time constant defaults are NOT auto-applied to a standalone runtime; +// they must be passed as MicroflowConstants or the app 530s. mxbuild writes them, +// already resolved, to /model/config.json — readDeploymentConstants +// lifts them from there. + +// DBConfig is the external Postgres the standalone runtime connects to. +type DBConfig struct { + Type string // e.g. "PostgreSQL" + Host string // "host:port", e.g. "127.0.0.1:5432" + Name string + User string + Password string +} + +// LocalRuntimeOptions configures StartLocalRuntime. +type LocalRuntimeOptions struct { + // DeployDir is the deployment directory (the runtime's BasePath). The mxbuild + // serve Deploy target writes the model/web structure here. + DeployDir string + // InstallPath is the mxbuild cache root (MX_INSTALL_PATH); its runtime/ child + // holds the launcher and the runtime libraries. + InstallPath string + // JavaHome is the JDK home used to launch the runtime. + JavaHome string + // AdminPort is the M2EE admin API port (default 8090). + AdminPort int + // AppPort is the HTTP port the app serves on (default 8080). + AppPort int + // AdminPass is the M2EE admin password (required). + AdminPass string + // ListenAddr binds both the admin API and the app (default 127.0.0.1). + ListenAddr string + // DTAPMode is D/A/T/P (default "D"). + DTAPMode string + // DB is the database the runtime connects to. + DB DBConfig + // ReadyTimeout bounds how long StartLocalRuntime waits for the admin API + // (default 90s). + ReadyTimeout time.Duration + // Stdout/Stderr receive progress messages (default os.Stdout/os.Stderr). + Stdout io.Writer + Stderr io.Writer +} + +// LocalRuntime is a booted standalone runtime process plus its admin connection. +type LocalRuntime struct { + opts LocalRuntimeOptions + cmd *exec.Cmd + log *syncBuffer + m2ee M2EEOptions + ctrl *RuntimeController +} + +func (o *LocalRuntimeOptions) applyDefaults() { + if o.AdminPort == 0 { + o.AdminPort = 8090 + } + if o.AppPort == 0 { + o.AppPort = 8080 + } + if o.ListenAddr == "" { + o.ListenAddr = "127.0.0.1" + } + if o.DTAPMode == "" { + o.DTAPMode = "D" + } + if o.ReadyTimeout == 0 { + o.ReadyTimeout = 90 * time.Second + } + if o.Stdout == nil { + o.Stdout = os.Stdout + } + if o.Stderr == nil { + o.Stderr = os.Stderr + } +} + +// runtimeDir is /runtime. +func (o *LocalRuntimeOptions) runtimeDir() string { return filepath.Join(o.InstallPath, "runtime") } + +// launcherJar is the runtime launcher jar. +func (o *LocalRuntimeOptions) launcherJar() string { + return filepath.Join(o.runtimeDir(), "launcher", "runtimelauncher.jar") +} + +// localRuntimeEnv builds the environment for the runtime JVM, layered on the +// current process environment. PrepareMxCommand later adds the FreeType fix. +func localRuntimeEnv(o LocalRuntimeOptions) []string { + return append(os.Environ(), + "M2EE_ADMIN_PASS="+o.AdminPass, + fmt.Sprintf("M2EE_ADMIN_PORT=%d", o.AdminPort), + "M2EE_ADMIN_LISTEN_ADDRESSES="+o.ListenAddr, + "MX_INSTALL_PATH="+o.InstallPath, + "MX_LOG_LEVEL=i", + ) +} + +// appContainerParams is the update_appcontainer_configuration payload (which port +// and address the app itself listens on). +func appContainerParams(o LocalRuntimeOptions) map[string]any { + return map[string]any{ + "runtime_port": o.AppPort, + "runtime_listen_addresses": o.ListenAddr, + } +} + +// runtimeConfigParams is the update_configuration payload. constants are the +// app's MicroflowConstants (name -> resolved default); pass an empty map for an +// app with no constants. +func runtimeConfigParams(o LocalRuntimeOptions, constants map[string]string) map[string]any { + if constants == nil { + constants = map[string]string{} + } + return map[string]any{ + "BasePath": o.DeployDir, + "RuntimePath": o.runtimeDir(), + "DTAPMode": o.DTAPMode, + "DatabaseType": o.DB.Type, + "DatabaseHost": o.DB.Host, + "DatabaseName": o.DB.Name, + "DatabaseUserName": o.DB.User, + "DatabasePassword": o.DB.Password, + "MicroflowConstants": constants, + } +} + +// deploymentConfig mirrors the parts of /model/config.json mxbuild +// writes: a pre-resolved Constants map (name -> default value as a string). +type deploymentConfig struct { + Constants map[string]string `json:"Constants"` +} + +// readDeploymentConstants lifts the resolved constant defaults mxbuild wrote to +// /model/config.json. A standalone runtime does not apply design-time +// defaults itself, so these must be fed back in via update_configuration. Missing +// file / no constants yields an empty map (not an error): an app may have none. +func readDeploymentConstants(deployDir string) (map[string]string, error) { + path := filepath.Join(deployDir, "model", "config.json") + data, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return map[string]string{}, nil + } + return nil, fmt.Errorf("reading %s: %w", path, err) + } + var cfg deploymentConfig + if err := json.Unmarshal(data, &cfg); err != nil { + return nil, fmt.Errorf("parsing %s: %w", path, err) + } + if cfg.Constants == nil { + return map[string]string{}, nil + } + return cfg.Constants, nil +} + +// ensureDataDirs creates the data/{files,tmp,model-upload} directories the +// runtime expects under the deployment dir. m2ee normally creates these; a bare +// serve Deploy / unzipped .mda does not. +func ensureDataDirs(deployDir string) error { + for _, sub := range []string{"files", "tmp", "model-upload"} { + if err := os.MkdirAll(filepath.Join(deployDir, "data", sub), 0o755); err != nil { + return err + } + } + return nil +} + +// StartLocalRuntime boots the runtime process, configures it over the admin API, +// and runs the DB-aware start cycle. On success the app is serving on AppPort. +// Call Stop() to shut it down. +func StartLocalRuntime(opts LocalRuntimeOptions) (*LocalRuntime, error) { + opts.applyDefaults() + if opts.AdminPass == "" { + return nil, fmt.Errorf("AdminPass is required") + } + if opts.InstallPath == "" { + return nil, fmt.Errorf("InstallPath is required") + } + if _, err := os.Stat(opts.launcherJar()); err != nil { + return nil, fmt.Errorf("runtime launcher not found at %s (incomplete mxbuild cache?): %w", opts.launcherJar(), err) + } + if err := ensureDataDirs(opts.DeployDir); err != nil { + return nil, fmt.Errorf("creating runtime data directories: %w", err) + } + + rt := &LocalRuntime{ + opts: opts, + m2ee: M2EEOptions{ + Host: opts.ListenAddr, + Port: opts.AdminPort, + Token: opts.AdminPass, + Direct: true, + Timeout: 150 * time.Second, + }, + } + rt.ctrl = NewRuntimeController(rt.m2ee) + + if err := rt.spawnAndConfigure(); err != nil { + return nil, err + } + if _, err := rt.ctrl.Start(); err != nil { + _ = rt.Stop() + return nil, fmt.Errorf("starting runtime: %w\n--- runtime output ---\n%s", err, rt.log.String()) + } + fmt.Fprintf(opts.Stdout, "Runtime started; app serving at %s\n", rt.AppURL()) + return rt, nil +} + +// spawnAndConfigure launches (or relaunches) the JVM and applies the admin +// configuration up to but not including start. It is used both for the initial +// boot and for a restart (config is per-process and must be re-applied). +func (rt *LocalRuntime) spawnAndConfigure() error { + javaExe := filepath.Join(rt.opts.JavaHome, "bin", "java") + cmd := exec.Command(javaExe, "-jar", rt.opts.launcherJar(), rt.opts.DeployDir) + cmd.Dir = rt.opts.runtimeDir() + cmd.Env = localRuntimeEnv(rt.opts) + PrepareMxCommand(cmd) // FreeType LD_PRELOAD workaround, layered on cmd.Env + log := &syncBuffer{} + cmd.Stdout = log + cmd.Stderr = log + if err := cmd.Start(); err != nil { + return fmt.Errorf("launching runtime JVM: %w", err) + } + rt.cmd = cmd + rt.log = log + + if err := rt.waitAdminReady(rt.opts.ReadyTimeout); err != nil { + _ = rt.Stop() + return fmt.Errorf("runtime admin API did not come up: %w\n--- runtime output ---\n%s", err, log.String()) + } + + if _, err := CallM2EE(rt.m2ee, "update_appcontainer_configuration", appContainerParams(rt.opts)); err != nil { + return fmt.Errorf("update_appcontainer_configuration: %w", err) + } + constants, err := readDeploymentConstants(rt.opts.DeployDir) + if err != nil { + return err + } + if _, err := CallM2EE(rt.m2ee, "update_configuration", runtimeConfigParams(rt.opts, constants)); err != nil { + return fmt.Errorf("update_configuration: %w", err) + } + return nil +} + +// waitAdminReady polls runtime_status until the admin API responds or times out. +func (rt *LocalRuntime) waitAdminReady(timeout time.Duration) error { + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + if _, err := CallM2EE(rt.m2ee, "runtime_status", nil); err == nil { + return nil + } + if !rt.alive() { + return fmt.Errorf("runtime process exited during startup") + } + time.Sleep(500 * time.Millisecond) + } + return fmt.Errorf("timed out after %s", timeout) +} + +// Controller returns the runtime controller for applying serve build results. +func (rt *LocalRuntime) Controller() *RuntimeController { return rt.ctrl } + +// Restart relaunches the JVM and re-applies configuration (but does not start — +// use it as the ApplyBuild restart callback, which runs Start afterwards). +func (rt *LocalRuntime) Restart() error { + _ = rt.stopProcess() + return rt.spawnAndConfigure() +} + +// AppURL is the base URL the app serves on. +func (rt *LocalRuntime) AppURL() string { + return fmt.Sprintf("http://%s:%d/", rt.opts.ListenAddr, rt.opts.AppPort) +} + +// HealthOK reports whether the app answers an HTTP request (any status < 500). +func (rt *LocalRuntime) HealthOK() bool { + client := &http.Client{Timeout: 5 * time.Second} + resp, err := client.Get(rt.AppURL()) + if err != nil { + return false + } + defer resp.Body.Close() + return resp.StatusCode < 500 +} + +// Log returns the captured runtime output (for diagnostics). +func (rt *LocalRuntime) Log() string { return rt.log.String() } + +func (rt *LocalRuntime) alive() bool { + if rt.cmd == nil || rt.cmd.Process == nil { + return false + } + return rt.cmd.Process.Signal(syscall.Signal(0)) == nil +} + +// Stop shuts the runtime down gracefully via the admin API, then terminates the +// process (SIGTERM, SIGKILL after a grace period). +func (rt *LocalRuntime) Stop() error { + _, _ = CallM2EE(rt.m2ee, "shutdown", nil) // best-effort graceful stop + return rt.stopProcess() +} + +func (rt *LocalRuntime) stopProcess() error { + if rt.cmd == nil || rt.cmd.Process == nil { + return nil + } + _ = rt.cmd.Process.Signal(syscall.SIGTERM) + done := make(chan error, 1) + go func() { done <- rt.cmd.Wait() }() + select { + case <-done: + case <-time.After(8 * time.Second): + _ = rt.cmd.Process.Kill() + <-done + } + rt.cmd = nil + return nil +} diff --git a/cmd/mxcli/docker/localboot_test.go b/cmd/mxcli/docker/localboot_test.go new file mode 100644 index 000000000..712d68bda --- /dev/null +++ b/cmd/mxcli/docker/localboot_test.go @@ -0,0 +1,188 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "path/filepath" + "testing" + "time" +) + +func testLocalOpts() LocalRuntimeOptions { + o := LocalRuntimeOptions{ + DeployDir: "/app/deployment", + InstallPath: "/cache/11.12.1", + JavaHome: "/jdk21", + AdminPass: "secret", + DB: DBConfig{ + Type: "PostgreSQL", Host: "127.0.0.1:5432", + Name: "appdb", User: "mendix", Password: "mendix", + }, + } + o.applyDefaults() + return o +} + +func TestApplyDefaults(t *testing.T) { + var o LocalRuntimeOptions + o.applyDefaults() + if o.AdminPort != 8090 || o.AppPort != 8080 { + t.Errorf("ports = %d/%d, want 8090/8080", o.AdminPort, o.AppPort) + } + if o.ListenAddr != "127.0.0.1" { + t.Errorf("ListenAddr = %q", o.ListenAddr) + } + if o.DTAPMode != "D" { + t.Errorf("DTAPMode = %q, want D", o.DTAPMode) + } + if o.ReadyTimeout != 90*time.Second { + t.Errorf("ReadyTimeout = %v", o.ReadyTimeout) + } + if o.Stdout == nil || o.Stderr == nil { + t.Error("Stdout/Stderr should default to non-nil") + } +} + +func TestPathHelpers(t *testing.T) { + o := testLocalOpts() + if got := o.runtimeDir(); got != filepath.FromSlash("/cache/11.12.1/runtime") { + t.Errorf("runtimeDir = %q", got) + } + if got := o.launcherJar(); got != filepath.FromSlash("/cache/11.12.1/runtime/launcher/runtimelauncher.jar") { + t.Errorf("launcherJar = %q", got) + } +} + +func TestLocalRuntimeEnv(t *testing.T) { + o := testLocalOpts() + env := localRuntimeEnv(o) + want := map[string]bool{ + "M2EE_ADMIN_PASS=secret": false, + "M2EE_ADMIN_PORT=8090": false, + "M2EE_ADMIN_LISTEN_ADDRESSES=127.0.0.1": false, + "MX_INSTALL_PATH=/cache/11.12.1": false, + "MX_LOG_LEVEL=i": false, + } + for _, e := range env { + if _, ok := want[e]; ok { + want[e] = true + } + } + for k, seen := range want { + if !seen { + t.Errorf("env missing %q", k) + } + } +} + +func TestAppContainerParams(t *testing.T) { + p := appContainerParams(testLocalOpts()) + if p["runtime_port"] != 8080 { + t.Errorf("runtime_port = %v, want 8080", p["runtime_port"]) + } + if p["runtime_listen_addresses"] != "127.0.0.1" { + t.Errorf("runtime_listen_addresses = %v", p["runtime_listen_addresses"]) + } +} + +func TestRuntimeConfigParams(t *testing.T) { + o := testLocalOpts() + consts := map[string]string{"Mod.Key": "val"} + p := runtimeConfigParams(o, consts) + checks := map[string]any{ + "BasePath": "/app/deployment", + "RuntimePath": filepath.FromSlash("/cache/11.12.1/runtime"), + "DTAPMode": "D", + "DatabaseType": "PostgreSQL", + "DatabaseHost": "127.0.0.1:5432", + "DatabaseName": "appdb", + "DatabaseUserName": "mendix", + "DatabasePassword": "mendix", + } + for k, want := range checks { + if p[k] != want { + t.Errorf("%s = %v, want %v", k, p[k], want) + } + } + mc, ok := p["MicroflowConstants"].(map[string]string) + if !ok || mc["Mod.Key"] != "val" { + t.Errorf("MicroflowConstants = %v", p["MicroflowConstants"]) + } +} + +func TestRuntimeConfigParams_NilConstants(t *testing.T) { + p := runtimeConfigParams(testLocalOpts(), nil) + mc, ok := p["MicroflowConstants"].(map[string]string) + if !ok || len(mc) != 0 { + t.Errorf("MicroflowConstants should be an empty (non-nil) map, got %v", p["MicroflowConstants"]) + } +} + +func TestReadDeploymentConstants(t *testing.T) { + dir := t.TempDir() + modelDir := filepath.Join(dir, "model") + if err := os.MkdirAll(modelDir, 0o755); err != nil { + t.Fatal(err) + } + cfg := `{"Configuration":{"DatabaseType":"HSQLDB"},"Constants":{"A.X":"1","B.Y":"two"},"AdminPassword":"1"}` + if err := os.WriteFile(filepath.Join(modelDir, "config.json"), []byte(cfg), 0o644); err != nil { + t.Fatal(err) + } + got, err := readDeploymentConstants(dir) + if err != nil { + t.Fatalf("readDeploymentConstants: %v", err) + } + if got["A.X"] != "1" || got["B.Y"] != "two" || len(got) != 2 { + t.Errorf("constants = %v", got) + } +} + +func TestReadDeploymentConstants_Missing(t *testing.T) { + // No config.json -> empty map, no error (an app may legitimately have none). + got, err := readDeploymentConstants(t.TempDir()) + if err != nil { + t.Fatalf("expected no error for a missing config.json, got %v", err) + } + if len(got) != 0 { + t.Errorf("constants = %v, want empty", got) + } +} + +func TestReadDeploymentConstants_BadJSON(t *testing.T) { + dir := t.TempDir() + modelDir := filepath.Join(dir, "model") + _ = os.MkdirAll(modelDir, 0o755) + _ = os.WriteFile(filepath.Join(modelDir, "config.json"), []byte("{not json"), 0o644) + if _, err := readDeploymentConstants(dir); err == nil { + t.Error("expected an error for malformed config.json") + } +} + +func TestEnsureDataDirs(t *testing.T) { + dir := t.TempDir() + if err := ensureDataDirs(dir); err != nil { + t.Fatalf("ensureDataDirs: %v", err) + } + for _, sub := range []string{"files", "tmp", "model-upload"} { + p := filepath.Join(dir, "data", sub) + if fi, err := os.Stat(p); err != nil || !fi.IsDir() { + t.Errorf("missing data dir %s", p) + } + } +} + +func TestStartLocalRuntime_Validation(t *testing.T) { + // Missing AdminPass. + if _, err := StartLocalRuntime(LocalRuntimeOptions{InstallPath: "/x"}); err == nil { + t.Error("expected error for missing AdminPass") + } + // Missing InstallPath. + if _, err := StartLocalRuntime(LocalRuntimeOptions{AdminPass: "p"}); err == nil { + t.Error("expected error for missing InstallPath") + } + // Launcher jar not present. + if _, err := StartLocalRuntime(LocalRuntimeOptions{AdminPass: "p", InstallPath: t.TempDir()}); err == nil { + t.Error("expected error when the launcher jar is absent") + } +} From cca93d0501b0ce904f6df58c1b9943e4f9016a28 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 05:40:31 +0000 Subject: [PATCH 15/26] feat(run): add `mxcli run --local` warm dev loop Wires the serve client and standalone runtime boot into a single command: a warm, Docker-free local dev loop. First build is cold (~10-15s); after that a model change rebuilds incrementally via mxbuild --serve and is hot-applied, with the serve build's restartRequired flag choosing reload vs restart. RunLocal (cmd/mxcli/docker/runlocal.go): - detects the project version, ensures mxbuild + runtime are cached, and links the runtime into the mxbuild cache so serve's javac resolves the Mendix API (avoids "package com.mendix.* does not exist"); - reuses an already-cached runtime instead of re-downloading (resolveRuntimeInstall); - checks the database is reachable (Postgres; devcontainer-friendly defaults, db name derived from the project) with an actionable error; - starts serve, does the first Deploy build, boots the runtime, and stays up; - with --watch, polls the project (v1 and v2 layouts, deployment/ excluded) and rebuilds + hot-applies each change via RuntimeController.ApplyBuild. Also resolves JDK 21 in StartLocalRuntime when JavaHome is unset. cmd/mxcli/cmd_run.go adds the top-level `run` command with --local/--watch and --app-port/--admin-port/--serve-port/--db-* flags. Verified end-to-end against a blank Mendix 11.12.1 app on a clean Postgres: boot -> HTTP 200; a microflow edit applies via reload in ~1.1s (no restart); an entity add applies via restart in ~9s, runs DDL, and returns HTTP 200. Pure helpers (defaults, db-name derivation, runtime linking, change signal) are unit-tested; process/JVM orchestration is exercised by the manual E2E above. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/cmd_run.go | 95 ++++++++ cmd/mxcli/docker/localboot.go | 9 + cmd/mxcli/docker/runlocal.go | 366 ++++++++++++++++++++++++++++++ cmd/mxcli/docker/runlocal_test.go | 139 ++++++++++++ 4 files changed, 609 insertions(+) create mode 100644 cmd/mxcli/cmd_run.go create mode 100644 cmd/mxcli/docker/runlocal.go create mode 100644 cmd/mxcli/docker/runlocal_test.go diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go new file mode 100644 index 000000000..8881e8d4f --- /dev/null +++ b/cmd/mxcli/cmd_run.go @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "os" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/spf13/cobra" +) + +var runCmd = &cobra.Command{ + Use: "run", + Short: "Run a Mendix app locally in a warm dev loop", + Long: `Run a Mendix app with a warm, Docker-free dev loop (--local). + +'mxcli run --local' keeps a mxbuild --serve process and a standalone Mendix +runtime hot. The first build is cold (~10-15s); after that a model change +rebuilds incrementally (~1s) and is applied without a full restart: + + - page / microflow / text change -> hot reload_model (no restart) + - entity / view / association -> runtime restart (metamodel is + reconciled only at startup) + +The serve build reports which is needed, so the right action is chosen +automatically. With --watch, mxcli rebuilds and hot-applies on every change. + +Requirements: + - Mendix 11.x project (JDK 21; version-aware JDK selection is a follow-up) + - A reachable PostgreSQL (the devcontainer provides one); the database must + already exist. Defaults: 127.0.0.1:5432, user 'mendix', db from the project + name. Override with --db-host/--db-name/--db-user/--db-password. + +Examples: + mxcli run --local -p app.mpr + mxcli run --local -p app.mpr --watch + mxcli run --local -p app.mpr --app-port 8081 --db-name myapp +`, + Run: func(cmd *cobra.Command, args []string) { + local, _ := cmd.Flags().GetBool("local") + if !local { + fmt.Fprintln(os.Stderr, "Error: only --local is supported for now (use 'mxcli docker run' for the container workflow)") + os.Exit(1) + } + projectPath, _ := cmd.Flags().GetString("project") + if projectPath == "" { + fmt.Fprintln(os.Stderr, "Error: --project (-p) is required") + os.Exit(1) + } + + watch, _ := cmd.Flags().GetBool("watch") + appPort, _ := cmd.Flags().GetInt("app-port") + adminPort, _ := cmd.Flags().GetInt("admin-port") + servePort, _ := cmd.Flags().GetInt("serve-port") + dbHost, _ := cmd.Flags().GetString("db-host") + dbName, _ := cmd.Flags().GetString("db-name") + dbUser, _ := cmd.Flags().GetString("db-user") + dbPassword, _ := cmd.Flags().GetString("db-password") + + opts := docker.LocalRunOptions{ + ProjectPath: projectPath, + AppPort: appPort, + AdminPort: adminPort, + ServePort: servePort, + Watch: watch, + DB: docker.DBConfig{ + Host: dbHost, + Name: dbName, + User: dbUser, + Password: dbPassword, + }, + Stdout: os.Stdout, + Stderr: os.Stderr, + } + + if err := docker.RunLocal(opts); err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + }, +} + +func init() { + runCmd.Flags().Bool("local", false, "Run locally without Docker (warm serve + standalone runtime)") + runCmd.Flags().Bool("watch", false, "Rebuild and hot-apply on every project change") + runCmd.Flags().Int("app-port", 0, "HTTP port for the app (default 8080)") + runCmd.Flags().Int("admin-port", 0, "M2EE admin API port (default 8090)") + runCmd.Flags().Int("serve-port", 0, "mxbuild --serve port (default 6543)") + runCmd.Flags().String("db-host", "", "Database host:port (default 127.0.0.1:5432)") + runCmd.Flags().String("db-name", "", "Database name (default derived from the project name)") + runCmd.Flags().String("db-user", "", "Database user (default mendix)") + runCmd.Flags().String("db-password", "", "Database password (default mendix)") + rootCmd.AddCommand(runCmd) +} diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index dc7cdc832..276f0e481 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -204,6 +204,15 @@ func StartLocalRuntime(opts LocalRuntimeOptions) (*LocalRuntime, error) { if opts.InstallPath == "" { return nil, fmt.Errorf("InstallPath is required") } + if opts.JavaHome == "" { + // Mendix 11 needs JDK 21. Version-aware selection (9/10) is a follow-up; + // the local loop targets 11.x for now. + jh, err := resolveJDK21() + if err != nil { + return nil, err + } + opts.JavaHome = jh + } if _, err := os.Stat(opts.launcherJar()); err != nil { return nil, fmt.Errorf("runtime launcher not found at %s (incomplete mxbuild cache?): %w", opts.launcherJar(), err) } diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go new file mode 100644 index 000000000..9593454f1 --- /dev/null +++ b/cmd/mxcli/docker/runlocal.go @@ -0,0 +1,366 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "io" + "io/fs" + "net" + "os" + "os/signal" + "path/filepath" + "strings" + "syscall" + "time" + + "github.com/mendixlabs/mxcli/sdk/mpr" +) + +// runlocal.go is the `mxcli run --local` orchestrator: a warm, Docker-free dev +// loop. It keeps a mxbuild --serve process and a standalone runtime hot, so a +// model change rebuilds incrementally (~1s) and is applied by a hot reload_model +// (page/microflow/text) or a runtime restart (entity/view/association) — the +// serve build's restartRequired flag decides which. See +// docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md. + +// LocalRunOptions configures RunLocal. +type LocalRunOptions struct { + // ProjectPath is the .mpr file. + ProjectPath string + // DeployDir is where the serve Deploy target writes (default /deployment). + DeployDir string + // MxBuildPath overrides mxbuild resolution (optional). + MxBuildPath string + // AppPort / AdminPort / ServePort default to 8080 / 8090 / 6543. + AppPort int + AdminPort int + ServePort int + // AdminPass is the M2EE admin password (default is a fixed local-dev value). + AdminPass string + // DB is the Postgres the runtime connects to (devcontainer defaults applied). + DB DBConfig + // Watch keeps running, rebuilding+applying on every project change. + Watch bool + // PollInterval is how often Watch checks for changes (default 1s). + PollInterval time.Duration + Stdout io.Writer + Stderr io.Writer +} + +// defaultLocalAdminPass is the admin password for a local dev runtime. The admin +// API binds to 127.0.0.1 only, so a fixed value is acceptable for local use. +const defaultLocalAdminPass = "mxcli-local-dev" + +func (o *LocalRunOptions) applyDefaults() { + if o.DeployDir == "" { + o.DeployDir = filepath.Join(filepath.Dir(o.ProjectPath), "deployment") + } + if o.AppPort == 0 { + o.AppPort = 8080 + } + if o.AdminPort == 0 { + o.AdminPort = 8090 + } + if o.ServePort == 0 { + o.ServePort = 6543 + } + if o.AdminPass == "" { + o.AdminPass = defaultLocalAdminPass + } + if o.PollInterval == 0 { + o.PollInterval = time.Second + } + if o.DB.Type == "" { + o.DB.Type = "PostgreSQL" + } + if o.DB.Host == "" { + o.DB.Host = "127.0.0.1:5432" + } + if o.DB.User == "" { + o.DB.User = "mendix" + } + if o.DB.Password == "" { + o.DB.Password = "mendix" + } + if o.DB.Name == "" { + o.DB.Name = deriveDBName(o.ProjectPath) + } + if o.Stdout == nil { + o.Stdout = os.Stdout + } + if o.Stderr == nil { + o.Stderr = os.Stderr + } +} + +// deriveDBName turns a project file name into a safe Postgres database name: +// lowercased, non-alphanumerics collapsed to underscores, leading digit prefixed. +func deriveDBName(projectPath string) string { + base := strings.TrimSuffix(filepath.Base(projectPath), filepath.Ext(projectPath)) + var b strings.Builder + for _, r := range strings.ToLower(base) { + if (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') { + b.WriteRune(r) + } else { + b.WriteByte('_') + } + } + name := strings.Trim(b.String(), "_") + if name == "" { + return "mxlocal" + } + if name[0] >= '0' && name[0] <= '9' { + name = "db_" + name + } + return name +} + +// ensureMxBuildRuntimeSibling makes the downloaded runtime available as a +// runtime/ sibling of the mxbuild cache's modeler/ dir. mxbuild's Deploy/serve +// javac step resolves the Mendix API from there; without it compilation fails +// with "package com.mendix.* does not exist". It is a no-op if the sibling +// already exists (symlink or real dir). Mirrors ensurePADFiles' link pattern. +func ensureMxBuildRuntimeSibling(version string, w io.Writer) error { + mxbuildDir, err := MxBuildCacheDir(version) + if err != nil { + return err + } + dst := filepath.Join(mxbuildDir, "runtime") + if _, err := os.Stat(dst); err == nil { + return nil // already present (bundled or linked previously) + } + runtimeDir, err := RuntimeCacheDir(version) + if err != nil { + return err + } + src := filepath.Join(runtimeDir, "runtime") + if _, err := os.Stat(src); err != nil { + return fmt.Errorf("runtime not found at %s (run 'mxcli setup mxbuild' / DownloadRuntime first): %w", src, err) + } + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + return err + } + if err := os.Symlink(src, dst); err != nil { + return fmt.Errorf("linking runtime into mxbuild cache: %w", err) + } + fmt.Fprintf(w, " Linked runtime into mxbuild cache: %s -> %s\n", dst, src) + return nil +} + +// pingTCP dials host (host:port) to check reachability within timeout. +func pingTCP(hostPort string, timeout time.Duration) error { + conn, err := net.DialTimeout("tcp", hostPort, timeout) + if err != nil { + return err + } + _ = conn.Close() + return nil +} + +// projectMTime returns the newest mtime under the project directory, ignoring +// the deployment dir and VCS metadata. It is the change signal for Watch: it +// covers both MPR v1 (single .mpr file) and v2 (metadata + mprcontents/). +func projectMTime(projectDir, deployDir string) time.Time { + var newest time.Time + deployAbs, _ := filepath.Abs(deployDir) + _ = filepath.WalkDir(projectDir, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return nil + } + if d.IsDir() { + name := d.Name() + if name == ".git" || name == "node_modules" || name == "deployment" { + return fs.SkipDir + } + if abs, _ := filepath.Abs(path); abs == deployAbs { + return fs.SkipDir + } + return nil + } + if info, err := d.Info(); err == nil && info.ModTime().After(newest) { + newest = info.ModTime() + } + return nil + }) + return newest +} + +// RunLocal boots the warm local dev loop: resolve tooling, start mxbuild --serve +// and a standalone runtime, do the first build+apply, then (with Watch) rebuild +// and hot-apply on every project change until interrupted. +func RunLocal(opts LocalRunOptions) error { + opts.applyDefaults() + w, stderr := opts.Stdout, opts.Stderr + + // 1. Detect the project's Mendix version. + fmt.Fprintln(w, "Detecting project version...") + reader, err := mpr.Open(opts.ProjectPath) + if err != nil { + return fmt.Errorf("opening project: %w", err) + } + pv := reader.ProjectVersion() + reader.Close() + version := pv.ProductVersion + fmt.Fprintf(w, " Mendix version: %s\n", version) + + // 2. Ensure mxbuild + runtime are cached, and linked for the serve javac step. + fmt.Fprintln(w, "Ensuring MxBuild and runtime are available...") + if _, err := DownloadMxBuild(version, w); err != nil { + return fmt.Errorf("setting up mxbuild: %w", err) + } + installPath, err := resolveRuntimeInstall(version, w) + if err != nil { + return fmt.Errorf("setting up runtime: %w", err) + } + + // 3. Check the database is reachable (we don't provision it here). + if err := pingTCP(opts.DB.Host, 3*time.Second); err != nil { + return fmt.Errorf("database not reachable at %s: %w\n"+ + " Start Postgres and create the '%s' database (user %q), or pass --db-* flags.", + opts.DB.Host, err, opts.DB.Name, opts.DB.User) + } + + // 4. Start the warm build server. + fmt.Fprintln(w, "Starting mxbuild --serve...") + serve, err := StartServe(ServeOptions{ + Version: version, + Host: "127.0.0.1", + Port: opts.ServePort, + }) + if err != nil { + return fmt.Errorf("starting mxbuild serve: %w", err) + } + defer serve.Stop() + + // 5. First build (cold — loads the model). + fmt.Fprintln(w, "Building (first build is cold, ~10-15s)...") + build, err := serve.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: opts.ProjectPath}) + if err != nil { + return fmt.Errorf("initial build: %w", err) + } + if !build.OK() { + return fmt.Errorf("initial build failed: %s\n%s", build.Message, string(build.Raw)) + } + + // 6. Boot the runtime against the fresh deployment. + rt, err := StartLocalRuntime(LocalRuntimeOptions{ + DeployDir: opts.DeployDir, + InstallPath: installPath, + // JavaHome left empty: StartLocalRuntime resolves JDK 21. + AppPort: opts.AppPort, + AdminPort: opts.AdminPort, + AdminPass: opts.AdminPass, + DB: opts.DB, + Stdout: w, + Stderr: stderr, + }) + if err != nil { + return err + } + defer rt.Stop() + + fmt.Fprintf(w, "\nApp is running at %s\n", rt.AppURL()) + + // 7. Stay up until interrupted. With --watch, rebuild + hot-apply on every + // project change; otherwise just keep the runtime serving. + if opts.Watch { + return watchAndApply(opts, serve, rt) + } + fmt.Fprintln(w, "(run with --watch to rebuild and hot-apply on changes; Ctrl-C to stop)") + waitForInterrupt() + fmt.Fprintln(w, "\nShutting down...") + return nil +} + +// resolveRuntimeInstall returns the directory to use as MX_INSTALL_PATH (its +// runtime/ child holds the launcher and Mendix libraries), downloading the +// runtime only when nothing usable is cached. It also makes the runtime visible +// to mxbuild's serve javac step as a runtime/ sibling of modeler/. +// +// Preference order: +// 1. the dedicated runtime cache (~/.mxcli/runtime/{v}), if it has the launcher; +// 2. a runtime/ already inside the mxbuild cache (bundled or previously linked); +// 3. download the runtime, then link it into the mxbuild cache. +func resolveRuntimeInstall(version string, w io.Writer) (string, error) { + if p := CachedRuntimePath(version); p != "" { + if err := ensureMxBuildRuntimeSibling(version, w); err != nil { + return "", err + } + return p, nil + } + mxbuildDir, err := MxBuildCacheDir(version) + if err != nil { + return "", err + } + launcher := filepath.Join(mxbuildDir, "runtime", "launcher", "runtimelauncher.jar") + if fi, err := os.Stat(launcher); err == nil && !fi.IsDir() { + // The mxbuild cache already carries a usable runtime/; use it directly. + return mxbuildDir, nil + } + if _, err := DownloadRuntime(version, w); err != nil { + return "", err + } + if err := ensureMxBuildRuntimeSibling(version, w); err != nil { + return "", err + } + return CachedRuntimePath(version), nil +} + +// waitForInterrupt blocks until the process receives SIGINT or SIGTERM. +func waitForInterrupt() { + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + defer signal.Stop(sigCh) + <-sigCh +} + +// watchAndApply polls the project for changes and applies each rebuild until the +// user interrupts (Ctrl-C). StartLocalRuntime already resolved the JVM; here we +// only rebuild via serve and let the RuntimeController decide reload vs restart. +func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime) error { + w := opts.Stdout + projectDir := filepath.Dir(opts.ProjectPath) + + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + defer signal.Stop(sigCh) + + fmt.Fprintln(w, "Watching for changes (Ctrl-C to stop)...") + last := projectMTime(projectDir, opts.DeployDir) + ticker := time.NewTicker(opts.PollInterval) + defer ticker.Stop() + + for { + select { + case <-sigCh: + fmt.Fprintln(w, "\nShutting down...") + return nil + case <-ticker.C: + now := projectMTime(projectDir, opts.DeployDir) + if !now.After(last) { + continue + } + last = now + fmt.Fprintln(w, "Change detected, rebuilding...") + start := time.Now() + build, err := serve.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: opts.ProjectPath}) + if err != nil { + fmt.Fprintf(opts.Stderr, " build error: %v\n", err) + continue + } + if !build.OK() { + fmt.Fprintf(opts.Stderr, " build failed: %s\n", build.Message) + continue + } + action, err := rt.Controller().ApplyBuild(build, rt.Restart) + if err != nil { + fmt.Fprintf(opts.Stderr, " apply (%s) failed: %v\n", action, err) + continue + } + // Refresh last after the apply so edits made during the build are caught. + last = projectMTime(projectDir, opts.DeployDir) + fmt.Fprintf(w, " applied via %s in %s -> %s\n", action, time.Since(start).Round(time.Millisecond), rt.AppURL()) + } + } +} diff --git a/cmd/mxcli/docker/runlocal_test.go b/cmd/mxcli/docker/runlocal_test.go new file mode 100644 index 000000000..49d908edb --- /dev/null +++ b/cmd/mxcli/docker/runlocal_test.go @@ -0,0 +1,139 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "io" + "os" + "path/filepath" + "testing" + "time" +) + +func TestDeriveDBName(t *testing.T) { + cases := map[string]string{ + "/path/App1112.mpr": "app1112", + "/path/My App.mpr": "my_app", + "/x/Sales-CRM.mpr": "sales_crm", + "/x/123Numbers.mpr": "db_123numbers", + "/x/__weird__.mpr": "weird", + "/x/.mpr": "mxlocal", + } + for in, want := range cases { + if got := deriveDBName(in); got != want { + t.Errorf("deriveDBName(%q) = %q, want %q", in, got, want) + } + } +} + +func TestLocalRunOptions_Defaults(t *testing.T) { + o := LocalRunOptions{ProjectPath: "/proj/App1112.mpr"} + o.applyDefaults() + if o.DeployDir != filepath.FromSlash("/proj/deployment") { + t.Errorf("DeployDir = %q", o.DeployDir) + } + if o.AppPort != 8080 || o.AdminPort != 8090 || o.ServePort != 6543 { + t.Errorf("ports = %d/%d/%d", o.AppPort, o.AdminPort, o.ServePort) + } + if o.AdminPass != defaultLocalAdminPass { + t.Errorf("AdminPass = %q", o.AdminPass) + } + if o.DB.Type != "PostgreSQL" || o.DB.Host != "127.0.0.1:5432" || o.DB.User != "mendix" || o.DB.Password != "mendix" { + t.Errorf("DB defaults = %+v", o.DB) + } + if o.DB.Name != "app1112" { + t.Errorf("DB.Name = %q, want app1112", o.DB.Name) + } + if o.PollInterval != time.Second { + t.Errorf("PollInterval = %v", o.PollInterval) + } +} + +func TestLocalRunOptions_DefaultsRespectOverrides(t *testing.T) { + o := LocalRunOptions{ + ProjectPath: "/proj/App.mpr", + AppPort: 9000, + DB: DBConfig{Host: "db:5432", Name: "custom", User: "u", Password: "p"}, + } + o.applyDefaults() + if o.AppPort != 9000 { + t.Errorf("AppPort override lost: %d", o.AppPort) + } + if o.DB.Host != "db:5432" || o.DB.Name != "custom" || o.DB.User != "u" || o.DB.Password != "p" { + t.Errorf("DB overrides lost: %+v", o.DB) + } +} + +func TestEnsureMxBuildRuntimeSibling(t *testing.T) { + // Point the cache roots at a temp HOME so we don't touch the real cache. + home := t.TempDir() + t.Setenv("HOME", home) + + version := "99.99.99" + // Build the runtime cache with a runtime/ dir. + runtimeCache, _ := RuntimeCacheDir(version) + realRuntime := filepath.Join(runtimeCache, "runtime") + if err := os.MkdirAll(realRuntime, 0o755); err != nil { + t.Fatal(err) + } + // mxbuild cache exists (modeler/) but has no runtime/ sibling yet. + mxbuildCache, _ := MxBuildCacheDir(version) + if err := os.MkdirAll(filepath.Join(mxbuildCache, "modeler"), 0o755); err != nil { + t.Fatal(err) + } + + if err := ensureMxBuildRuntimeSibling(version, io.Discard); err != nil { + t.Fatalf("ensureMxBuildRuntimeSibling: %v", err) + } + link := filepath.Join(mxbuildCache, "runtime") + if _, err := os.Stat(link); err != nil { + t.Errorf("runtime sibling not created: %v", err) + } + // Idempotent second call. + if err := ensureMxBuildRuntimeSibling(version, io.Discard); err != nil { + t.Errorf("second call should be a no-op, got %v", err) + } +} + +func TestEnsureMxBuildRuntimeSibling_MissingSource(t *testing.T) { + home := t.TempDir() + t.Setenv("HOME", home) + version := "99.99.98" + mxbuildCache, _ := MxBuildCacheDir(version) + _ = os.MkdirAll(filepath.Join(mxbuildCache, "modeler"), 0o755) + // No runtime cache -> error. + if err := ensureMxBuildRuntimeSibling(version, io.Discard); err == nil { + t.Error("expected error when the runtime source is absent") + } +} + +func TestProjectMTime(t *testing.T) { + dir := t.TempDir() + deploy := filepath.Join(dir, "deployment") + _ = os.MkdirAll(deploy, 0o755) + _ = os.MkdirAll(filepath.Join(dir, ".git"), 0o755) + + mpr := filepath.Join(dir, "App.mpr") + if err := os.WriteFile(mpr, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + base := projectMTime(dir, deploy) + if base.IsZero() { + t.Fatal("expected a non-zero mtime") + } + + // A change under deployment/ must NOT advance the signal. + future := time.Now().Add(time.Hour) + deployFile := filepath.Join(deploy, "model.mdp") + _ = os.WriteFile(deployFile, []byte("y"), 0o644) + _ = os.Chtimes(deployFile, future, future) + if got := projectMTime(dir, deploy); got.After(base) { + t.Error("deployment/ changes should be ignored by the watch signal") + } + + // A change to the project file MUST advance the signal. + _ = os.Chtimes(mpr, future, future) + if got := projectMTime(dir, deploy); !got.After(base) { + t.Error("project file change should advance the watch signal") + } +} From fa949289e5e0104a3b57bd5520787c655456107f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 13:29:24 +0000 Subject: [PATCH 16/26] docs(run-local): document the warm local dev loop Adds user-facing docs for `mxcli run --local`: - docs-site page tools/run-local.md (+ SUMMARY.md TOC entry); - synced skill .claude/skills/mendix/run-local.md (how-to, apply-decision table, Postgres prereq, the edit -> hot-apply loop, validation checklist); - CLAUDE.md implementation-status entry. All three state the current limitation honestly: the runtime/model/API work and the app answers HTTP 200, but the browser client renders blank because the web client bundle (web/dist/index.js) is produced by a rollup step the serve path does not run. Records the finding and the next step in the proposal. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/run-local.md | 98 +++++++++++++++++++ CLAUDE.md | 1 + docs-site/src/SUMMARY.md | 1 + docs-site/src/tools/run-local.md | 88 +++++++++++++++++ .../PROPOSAL_mxcli_dev_warm_loop.md | 22 +++++ 5 files changed, 210 insertions(+) create mode 100644 .claude/skills/mendix/run-local.md create mode 100644 docs-site/src/tools/run-local.md diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md new file mode 100644 index 000000000..b3b052a66 --- /dev/null +++ b/.claude/skills/mendix/run-local.md @@ -0,0 +1,98 @@ +# Warm Local Dev Loop — `mxcli run --local` + +## Overview + +`mxcli run --local` runs a Mendix app **without Docker**, keeping a +`mxbuild --serve` process and a standalone runtime hot so model changes apply in +~1 s instead of a ~30–60 s rebuild-and-restart. Use it as the fast inner loop when +iterating on an app; use `mxcli docker run` when you need the fully-rendered browser +client (see the limitation below). + +## When to Use This Skill + +Use this when: +- You want the fastest edit → running-app loop for a Mendix 11.x project. +- You're driving the model programmatically (`mxcli exec`/MDL) and want each change + live immediately. +- You're doing runtime/model/API-level or headless verification. + +Prefer `mxcli docker run` when: +- You need the app's pages to **render in a browser** (see limitation). +- The project is Mendix 9/10 (JDK 11/17 — not yet supported by `run --local`). + +## Usage + +```bash +# boot once and keep serving (Ctrl-C to stop) +mxcli run --local -p app.mpr + +# boot and hot-apply on every project change +mxcli run --local -p app.mpr --watch +``` + +## How apply is chosen + +Every warm rebuild reports whether a restart is required; `run --local` applies the +cheapest action automatically: + +| Change | Apply | Cost | +|--------|-------|------| +| page / microflow / nanoflow / text | hot `reload_model` (no restart) | ~1 s | +| entity / view entity / association | runtime restart + DDL | ~9 s | + +Structural changes need a restart because the runtime reconciles its entity/ +association catalog only at startup; behavioural changes are hot-reloaded. + +## Prerequisites + +- **Mendix 11.x** project (runtime launches under **JDK 21**). +- A reachable **PostgreSQL** with the database already created. In the devcontainer: + + ```bash + createdb -h 127.0.0.1 -U mendix "$(basename app.mpr .mpr | tr '[:upper:]' '[:lower:]')" + ``` + + Defaults: `127.0.0.1:5432`, user `mendix`, db derived from the project name. + Override with `--db-host/--db-name/--db-user/--db-password`. If the database is + unreachable the command stops with an actionable message (it does not provision it). + +## The intended loop + +```bash +# terminal 1: keep the app hot +mxcli run --local -p app.mpr --watch + +# terminal 2 (or an agent): edit the model — the change hot-applies automatically +mxcli exec add-page.mdl -p app.mpr +``` + +`--watch` observes the project directory (MPR v1 and v2 layouts), ignoring +`deployment/`, `.git`, and `node_modules`. + +## Flags + +| Flag | Default | Purpose | +|------|---------|---------| +| `--local` | — | Required; run without Docker | +| `--watch` | off | Rebuild + hot-apply on each change | +| `--app-port` / `--admin-port` / `--serve-port` | 8080 / 8090 / 6543 | Ports | +| `--db-host` / `--db-name` / `--db-user` / `--db-password` | 127.0.0.1:5432 / derived / mendix / mendix | Database | + +## Limitation — browser pages render blank (for now) + +The runtime, model, and admin API work end to end (the app answers HTTP 200 and +reload/restart apply correctly), **but the browser client does not yet render**: the +web client bundle (`web/dist/index.js`) is produced by a rollup step the serve/ +standalone path does not currently run, so `index.html` 404s on its main bundle. + +So `run --local` is currently for **runtime/model/API iteration and headless +checks**, not visual page-design work. For a rendered browser client, use +`mxcli docker run`. Closing this gap is tracked follow-up work. + +## Validation checklist + +- [ ] Project is Mendix 11.x. +- [ ] Postgres is running and the target database exists. +- [ ] `curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/` returns `200`. +- [ ] With `--watch`, editing a microflow logs `applied via reload`; adding an entity + logs `applied via restart` and creates the table in Postgres. diff --git a/CLAUDE.md b/CLAUDE.md index f61eb2b55..1cb9058aa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -523,6 +523,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) +- Warm local dev loop (`mxcli run --local [--watch]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Boots Mendix 11.x apps to HTTP 200; browser client bundle (`web/dist/`) not yet produced by the serve path — see `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - OQL query execution against running runtime (`mxcli oql`) - Business event services (SHOW/DESCRIBE/CREATE/DROP) - Project settings (SHOW/DESCRIBE/ALTER) diff --git a/docs-site/src/SUMMARY.md b/docs-site/src/SUMMARY.md index 674ed3932..4e5b3b62d 100644 --- a/docs-site/src/SUMMARY.md +++ b/docs-site/src/SUMMARY.md @@ -151,6 +151,7 @@ - [IMPORT FROM](tools/import-from.md) - [Credential Management](tools/credentials.md) - [Database Connector Generation](tools/connector-generation.md) +- [Local Dev Loop](tools/run-local.md) - [Docker Integration](tools/docker.md) - [mxcli docker build](tools/docker-build.md) - [mxcli docker check](tools/docker-check.md) diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md new file mode 100644 index 000000000..7037a30f1 --- /dev/null +++ b/docs-site/src/tools/run-local.md @@ -0,0 +1,88 @@ +# mxcli run --local + +`mxcli run --local` runs a Mendix app in a **warm, Docker-free dev loop**. It keeps +a `mxbuild --serve` process and a standalone Mendix runtime hot, so after the first +(cold) build a model change rebuilds incrementally and is hot-applied — without a +full rebuild-image-and-restart cycle. + +```bash +mxcli run --local -p app.mpr +mxcli run --local -p app.mpr --watch +``` + +## Why + +The Docker path (`mxcli docker run`) rebuilds a full deployment package and restarts +the container on every change (~30–60 s). `run --local` instead: + +- keeps the model loaded in `mxbuild --serve` — a warm rebuild is ~1 s; +- keeps the runtime process up and applies each change over the M2EE admin API; +- chooses the cheapest apply automatically from the build's `restartRequired` flag: + +| Change | Apply | Cost | +|--------|-------|------| +| page / microflow / nanoflow / text | hot `reload_model` (no restart) | ~1 s | +| entity / view entity / association | runtime restart + DDL | ~9 s | + +The metamodel catalog (entities/associations) is reconciled only at runtime startup, +so structural changes need a restart; behavioural changes do not. + +## What it does + +1. Detects the project's Mendix version. +2. Ensures MxBuild and the runtime are cached (downloads once, reused after). +3. Checks the database is reachable (it does **not** provision it — see below). +4. Starts `mxbuild --serve` and does the first (cold) build into `deployment/`. +5. Boots a standalone runtime against that deployment and serves it. +6. With `--watch`, rebuilds and hot-applies on every project change until `Ctrl-C`. + +## Requirements + +- **Mendix 11.x** project. The runtime is launched under **JDK 21**; version-aware + JDK selection for Mendix 9/10 is a follow-up. +- A reachable **PostgreSQL**, with the database already created. The devcontainer + provides one. Defaults: `127.0.0.1:5432`, user `mendix`, database derived from the + project file name (`App1112.mpr` → `app1112`). If the DB is unreachable, `run + --local` stops with an actionable message rather than booting. + +```bash +# devcontainer Postgres, one-time DB creation +createdb -h 127.0.0.1 -U mendix app1112 +``` + +## Flags + +| Flag | Default | Purpose | +|------|---------|---------| +| `--local` | — | Required; run without Docker | +| `--watch` | off | Rebuild + hot-apply on every project change | +| `--app-port` | 8080 | App HTTP port | +| `--admin-port` | 8090 | M2EE admin API port | +| `--serve-port` | 6543 | `mxbuild --serve` port | +| `--db-host` | 127.0.0.1:5432 | Database `host:port` | +| `--db-name` | derived from project | Database name | +| `--db-user` / `--db-password` | mendix / mendix | Database credentials | + +## The change signal + +`--watch` watches the project directory (both MPR v1 single-file and v2 +`mprcontents/` layouts), ignoring `deployment/`, `.git`, and `node_modules`. The +intended loop is: an agent (or you) edits the model with `mxcli exec`/MDL, and the +running `run --local` picks the change up and hot-applies it. + +## Current limitation — browser page rendering + +The runtime, model, and admin API work end to end (the app answers HTTP 200, and +model reload/restart is applied correctly). **However, the browser client does not +yet render**: the `mxbuild` web client bundle (`web/dist/index.js`) is produced by a +separate rollup step that the serve/standalone path does not currently run, so +`index.html` 404s on its main bundle and the page is blank. + +This means `run --local` is presently useful for **runtime/model/API-level +iteration and headless verification**, but **not yet for visual page-design +iteration**. Closing the client-bundle gap (running mxbuild's bundled rollup tooling +after the deploy build) is tracked as follow-up work. Until then, use `mxcli docker +run` when you need the fully-rendered client in a browser. + +See also: [PROPOSAL_mxcli_dev_warm_loop](../../../docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md), +[mxcli docker run](docker-run.md), [Playwright Testing](playwright.md). diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 9865ef09a..acd7a5383 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -154,6 +154,28 @@ catalog evidence, which agree. Enabling the dev-preview flag on the standalone b (so the agent can verify via OQL) is itself a small item worth doing — see Open Questions. +## Implementation status (slice 1 — `mxcli run --local`) + +Shipped and verified end to end against a blank Mendix 11.12.1 app on a clean +Postgres: boot → HTTP 200; a microflow edit applies via `reload_model` in ~1.1 s (no +restart); an entity add applies via restart in ~9 s, runs `execute_ddl_commands`, and +returns HTTP 200 with the table present in Postgres. Pieces: +`docker.RuntimeController` (reload-vs-restart + DDL-aware start), `docker.LocalRuntime` +(standalone boot; constants lifted from `deployment/model/config.json`), +`docker.RunLocal` (+ `--watch`), and the `run --local` command. + +**Known gap — browser client bundle not produced (blocks visual page iteration).** +The runtime/model/admin API work, but a browser loads blank: `index.html` requests +`web/dist/index.js`, which neither the serve `Deploy` target nor a one-shot `mxbuild` +produces here. The client bundle is built by a **rollup** step (`web/rollup.config.mjs`, +using mxbuild's bundled `modeler/tools/node` tooling) that the serve/standalone path +does not run — `web/index.js` (the rollup *input*) is present but the bundled +`web/dist/index.js` output is missing. Until this is closed, `run --local` is for +runtime/model/API iteration and headless checks; use `mxcli docker run` for a rendered +client. **Next step:** invoke mxbuild's rollup client build after the deploy build (or +find the serve option that emits `dist/`), then wire Playwright screenshotting for the +pixel-perfect page loop. + ## Proposed CLI ### Scenario A: `mxcli dev` — Docker-free warm run loop From ada5a4f2809e31e293c619f90b1e8ba0b03f6e05 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 13:44:00 +0000 Subject: [PATCH 17/26] feat(run-local): bundle the browser client so pages render The serve Deploy target writes the web client *source* (web/index.js, web/pages/*, web/widgets/*, web/rollup.config.mjs) but not the rollup *bundle* (web/dist/index.js) that index.html loads, so a standalone-served app 404'd on its client bundle and rendered blank. docker.BuildWebClient runs mxbuild's own bundled rollup runner (modeler/tools/node/rollup-runner.mjs, NODE_ENV=production, one-shot) over deployment/web to produce web/dist/. resolveNodeTooling locates the bundled node + runner from the mxbuild path, preferring the GOOS/GOARCH platform dir with a glob fallback. RunLocal calls it after the initial build and after each --watch rebuild. Verified end to end: `mxcli run --local` on a blank 11.12.1 app now serves /dist/index.js (HTTP 200), and driving the devcontainer Chromium with Playwright renders the Mendix homepage fully. Docs (site page, skill, CLAUDE.md, proposal) updated: pages now render; the remaining item is making the --watch client rebuild incremental (a full rollup bundle is ~6-7s today) via rollup's watch-mode companion bundler. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/run-local.md | 24 +-- CLAUDE.md | 2 +- cmd/mxcli/docker/runlocal.go | 23 ++- cmd/mxcli/docker/webclient.go | 147 ++++++++++++++++++ cmd/mxcli/docker/webclient_test.go | 94 +++++++++++ docs-site/src/tools/run-local.md | 30 ++-- .../PROPOSAL_mxcli_dev_warm_loop.md | 28 ++-- 7 files changed, 309 insertions(+), 39 deletions(-) create mode 100644 cmd/mxcli/docker/webclient.go create mode 100644 cmd/mxcli/docker/webclient_test.go diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index b3b052a66..d161cc223 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -14,11 +14,12 @@ Use this when: - You want the fastest edit → running-app loop for a Mendix 11.x project. - You're driving the model programmatically (`mxcli exec`/MDL) and want each change live immediately. -- You're doing runtime/model/API-level or headless verification. +- You're iterating on **page design** (the app renders in a real browser) or doing + runtime/model/API/headless verification. Prefer `mxcli docker run` when: -- You need the app's pages to **render in a browser** (see limitation). - The project is Mendix 9/10 (JDK 11/17 — not yet supported by `run --local`). +- You want a container-parity deployment rather than a standalone runtime. ## Usage @@ -78,21 +79,22 @@ mxcli exec add-page.mdl -p app.mpr | `--app-port` / `--admin-port` / `--serve-port` | 8080 / 8090 / 6543 | Ports | | `--db-host` / `--db-name` / `--db-user` / `--db-password` | 127.0.0.1:5432 / derived / mendix / mendix | Database | -## Limitation — browser pages render blank (for now) +## Pages render in the browser -The runtime, model, and admin API work end to end (the app answers HTTP 200 and -reload/restart apply correctly), **but the browser client does not yet render**: the -web client bundle (`web/dist/index.js`) is produced by a rollup step the serve/ -standalone path does not currently run, so `index.html` 404s on its main bundle. +`run --local` bundles the browser client (`web/dist/`) with mxbuild's rollup tooling +after the deploy build, so the app renders in a real browser (verified with +Playwright + the devcontainer's Chromium). Pair it with Playwright screenshots for a +pixel-perfect page loop: edit → auto-rebuild → re-screenshot. -So `run --local` is currently for **runtime/model/API iteration and headless -checks**, not visual page-design work. For a rendered browser client, use -`mxcli docker run`. Closing this gap is tracked follow-up work. +**Watch cost:** in `--watch` the client bundle is rebuilt on every change (~6–7 s full +bundle today). Behavioural changes (microflows) don't strictly need it; making the +client rebuild incremental/conditional is a tracked optimization. ## Validation checklist - [ ] Project is Mendix 11.x. - [ ] Postgres is running and the target database exists. -- [ ] `curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/` returns `200`. +- [ ] `curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/` returns `200`, + and `.../dist/index.js` also returns `200` (client bundle served). - [ ] With `--watch`, editing a microflow logs `applied via reload`; adding an entity logs `applied via restart` and creates the table in Postgres. diff --git a/CLAUDE.md b/CLAUDE.md index 1cb9058aa..62e70cf1c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -523,7 +523,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) -- Warm local dev loop (`mxcli run --local [--watch]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Boots Mendix 11.x apps to HTTP 200; browser client bundle (`web/dist/`) not yet produced by the serve path — see `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` +- Warm local dev loop (`mxcli run --local [--watch]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser (verified with Playwright). `--watch` re-bundles the client each change (~6-7s full bundle; incremental is a follow-up) — see `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - OQL query execution against running runtime (`mxcli oql`) - Business event services (SHOW/DESCRIBE/CREATE/DROP) - Project settings (SHOW/DESCRIBE/ALTER) diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index 9593454f1..986cf9681 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -206,7 +206,8 @@ func RunLocal(opts LocalRunOptions) error { // 2. Ensure mxbuild + runtime are cached, and linked for the serve javac step. fmt.Fprintln(w, "Ensuring MxBuild and runtime are available...") - if _, err := DownloadMxBuild(version, w); err != nil { + mxbuildPath, err := DownloadMxBuild(version, w) + if err != nil { return fmt.Errorf("setting up mxbuild: %w", err) } installPath, err := resolveRuntimeInstall(version, w) @@ -243,6 +244,14 @@ func RunLocal(opts LocalRunOptions) error { return fmt.Errorf("initial build failed: %s\n%s", build.Message, string(build.Raw)) } + // 5b. Bundle the browser client (web/dist). The serve Deploy target writes the + // client source but not the rollup bundle, so without this the app 404s on + // /dist/index.js and renders blank. + fmt.Fprintln(w, "Bundling web client...") + if err := BuildWebClient(WebClientOptions{DeployDir: opts.DeployDir, MxBuildPath: mxbuildPath, Stdout: w}); err != nil { + return fmt.Errorf("bundling web client: %w", err) + } + // 6. Boot the runtime against the fresh deployment. rt, err := StartLocalRuntime(LocalRuntimeOptions{ DeployDir: opts.DeployDir, @@ -265,7 +274,7 @@ func RunLocal(opts LocalRunOptions) error { // 7. Stay up until interrupted. With --watch, rebuild + hot-apply on every // project change; otherwise just keep the runtime serving. if opts.Watch { - return watchAndApply(opts, serve, rt) + return watchAndApply(opts, serve, rt, mxbuildPath) } fmt.Fprintln(w, "(run with --watch to rebuild and hot-apply on changes; Ctrl-C to stop)") waitForInterrupt() @@ -318,7 +327,7 @@ func waitForInterrupt() { // watchAndApply polls the project for changes and applies each rebuild until the // user interrupts (Ctrl-C). StartLocalRuntime already resolved the JVM; here we // only rebuild via serve and let the RuntimeController decide reload vs restart. -func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime) error { +func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, mxbuildPath string) error { w := opts.Stdout projectDir := filepath.Dir(opts.ProjectPath) @@ -353,6 +362,14 @@ func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime) e fmt.Fprintf(opts.Stderr, " build failed: %s\n", build.Message) continue } + // Rebuild the browser client bundle so page/widget edits are reflected + // (the serve Deploy target writes client source but not the rollup + // bundle). This is a full ~7s bundle today; an incremental watch-mode + // companion bundler is the optimization follow-up. + if err := BuildWebClient(WebClientOptions{DeployDir: opts.DeployDir, MxBuildPath: mxbuildPath, Stdout: w}); err != nil { + fmt.Fprintf(opts.Stderr, " web client build failed: %v\n", err) + continue + } action, err := rt.Controller().ApplyBuild(build, rt.Restart) if err != nil { fmt.Fprintf(opts.Stderr, " apply (%s) failed: %v\n", action, err) diff --git a/cmd/mxcli/docker/webclient.go b/cmd/mxcli/docker/webclient.go new file mode 100644 index 000000000..0d56cd035 --- /dev/null +++ b/cmd/mxcli/docker/webclient.go @@ -0,0 +1,147 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "runtime" + "time" +) + +// webclient.go builds the browser client bundle (web/dist/index.js) for a +// deployment. mxbuild's serve Deploy target writes the client *source* +// (web/index.js, web/pages/*, web/widgets/*) and a self-contained +// web/rollup.config.mjs, but it does NOT run the rollup step that bundles them +// into web/dist/. A standalone-served app therefore 404s on /dist/index.js and +// renders blank. This runs mxbuild's own bundled rollup runner to close that gap. +// +// The runner (modeler/tools/node/rollup-runner.mjs) with NODE_ENV=production does +// a one-shot rollup() + bundle.write(config.output) into web/dist. It resolves +// `rollup` from tools/node/node_modules (relative to the runner) and loads +// rollup.config.mjs from its working directory (the deployment web dir). + +// WebClientOptions configures BuildWebClient. +type WebClientOptions struct { + // DeployDir is the deployment directory; its web/ child holds the client + // source and rollup.config.mjs, and receives web/dist/. + DeployDir string + // MxBuildPath is /modeler/mxbuild; the node tooling is resolved from + // its sibling tools/node directory. + MxBuildPath string + // Timeout bounds the bundle build (default 5m). + Timeout time.Duration + // Stdout receives a short progress line (default discarded). + Stdout io.Writer +} + +// resolveNodeTooling returns the bundled node binary and rollup-runner.mjs paths +// from an mxbuild binary path (/modeler/mxbuild -> /modeler/tools/node). +func resolveNodeTooling(mxbuildPath string) (nodeBin, runner string, err error) { + toolsNode := filepath.Join(filepath.Dir(mxbuildPath), "tools", "node") + runner = filepath.Join(toolsNode, "rollup-runner.mjs") + if _, err := os.Stat(runner); err != nil { + return "", "", fmt.Errorf("rollup runner not found at %s (incomplete mxbuild?): %w", runner, err) + } + nodeBin = findNodeBinary(toolsNode) + if nodeBin == "" { + return "", "", fmt.Errorf("bundled node binary not found under %s", toolsNode) + } + return nodeBin, runner, nil +} + +// findNodeBinary locates mxbuild's bundled node under tools/node//. +// It prefers the GOOS/GOARCH-matched directory, then falls back to any platform +// dir containing a node binary (so a cache for one arch still resolves cleanly). +func findNodeBinary(toolsNode string) string { + exe := "node" + if runtime.GOOS == "windows" { + exe = "node.exe" + } + archAlias := map[string]string{"amd64": "x64", "arm64": "arm64", "386": "x86"} + arch := archAlias[runtime.GOARCH] + if arch == "" { + arch = runtime.GOARCH + } + osAlias := map[string]string{"darwin": "darwin", "linux": "linux", "windows": "win"} + goos := osAlias[runtime.GOOS] + if goos == "" { + goos = runtime.GOOS + } + // Preferred exact match, then common alternates, then a glob fallback. + candidates := []string{ + filepath.Join(toolsNode, goos+"-"+arch, exe), + filepath.Join(toolsNode, exe), + } + for _, c := range candidates { + if fi, err := os.Stat(c); err == nil && !fi.IsDir() { + return c + } + } + matches, _ := filepath.Glob(filepath.Join(toolsNode, "*", exe)) + for _, m := range matches { + if fi, err := os.Stat(m); err == nil && !fi.IsDir() { + return m + } + } + return "" +} + +// BuildWebClient bundles the deployment's browser client into web/dist. It is a +// no-op-safe prerequisite for serving a rendered app; call it after each serve +// Deploy build. Returns an error if the web dir, tooling, or rollup build fails. +func BuildWebClient(opts WebClientOptions) error { + w := opts.Stdout + if w == nil { + w = io.Discard + } + webDir := filepath.Join(opts.DeployDir, "web") + if fi, err := os.Stat(filepath.Join(webDir, "rollup.config.mjs")); err != nil || fi.IsDir() { + return fmt.Errorf("no rollup.config.mjs in %s (run a serve Deploy build first)", webDir) + } + nodeBin, runner, err := resolveNodeTooling(opts.MxBuildPath) + if err != nil { + return err + } + timeout := opts.Timeout + if timeout == 0 { + timeout = 5 * time.Minute + } + + start := time.Now() + cmd := exec.Command(nodeBin, runner) + cmd.Dir = webDir + cmd.Env = append(os.Environ(), + "NODE_ENV=production", + "MX_WEB_CLIENT_BUILD_LOG="+filepath.Join(opts.DeployDir, "log", "web-client-build.log"), + ) + log := &syncBuffer{} + cmd.Stdout = log + cmd.Stderr = log + + done := make(chan error, 1) + if err := cmd.Start(); err != nil { + return fmt.Errorf("launching web client build: %w", err) + } + go func() { done <- cmd.Wait() }() + select { + case err := <-done: + if err != nil { + return fmt.Errorf("web client build failed: %w\n%s", err, log.String()) + } + case <-time.After(timeout): + _ = cmd.Process.Kill() + <-done + return fmt.Errorf("web client build timed out after %s", timeout) + } + + dist := filepath.Join(webDir, "dist", "index.js") + if _, err := os.Stat(dist); err != nil { + return fmt.Errorf("web client build reported success but %s is missing:\n%s", dist, log.String()) + } + fmt.Fprintf(w, " Web client bundled in %s\n", time.Since(start).Round(time.Millisecond)) + return nil +} diff --git a/cmd/mxcli/docker/webclient_test.go b/cmd/mxcli/docker/webclient_test.go new file mode 100644 index 000000000..9c1d849e0 --- /dev/null +++ b/cmd/mxcli/docker/webclient_test.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "path/filepath" + "runtime" + "testing" +) + +func TestResolveNodeTooling(t *testing.T) { + cache := t.TempDir() + toolsNode := filepath.Join(cache, "modeler", "tools", "node") + // platform dir matching this host (that's what findNodeBinary prefers) + exe := "node" + if runtime.GOOS == "windows" { + exe = "node.exe" + } + archAlias := map[string]string{"amd64": "x64", "arm64": "arm64", "386": "x86"} + arch := archAlias[runtime.GOARCH] + if arch == "" { + arch = runtime.GOARCH + } + osAlias := map[string]string{"darwin": "darwin", "linux": "linux", "windows": "win"} + goos := osAlias[runtime.GOOS] + if goos == "" { + goos = runtime.GOOS + } + platDir := filepath.Join(toolsNode, goos+"-"+arch) + if err := os.MkdirAll(platDir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(platDir, exe), []byte("#!/bin/sh\n"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(toolsNode, "rollup-runner.mjs"), []byte("// runner\n"), 0o644); err != nil { + t.Fatal(err) + } + + mxbuildPath := filepath.Join(cache, "modeler", "mxbuild") + nodeBin, runner, err := resolveNodeTooling(mxbuildPath) + if err != nil { + t.Fatalf("resolveNodeTooling: %v", err) + } + if nodeBin != filepath.Join(platDir, exe) { + t.Errorf("nodeBin = %q", nodeBin) + } + if runner != filepath.Join(toolsNode, "rollup-runner.mjs") { + t.Errorf("runner = %q", runner) + } +} + +func TestResolveNodeTooling_GlobFallback(t *testing.T) { + // A cache for a different platform: findNodeBinary falls back to any */node. + cache := t.TempDir() + toolsNode := filepath.Join(cache, "modeler", "tools", "node") + other := filepath.Join(toolsNode, "some-other-plat") + if err := os.MkdirAll(other, 0o755); err != nil { + t.Fatal(err) + } + exe := "node" + if runtime.GOOS == "windows" { + exe = "node.exe" + } + _ = os.WriteFile(filepath.Join(other, exe), []byte("x"), 0o755) + _ = os.WriteFile(filepath.Join(toolsNode, "rollup-runner.mjs"), []byte("x"), 0o644) + + nodeBin, _, err := resolveNodeTooling(filepath.Join(cache, "modeler", "mxbuild")) + if err != nil { + t.Fatalf("resolveNodeTooling: %v", err) + } + if nodeBin != filepath.Join(other, exe) { + t.Errorf("nodeBin = %q, want glob fallback %q", nodeBin, filepath.Join(other, exe)) + } +} + +func TestResolveNodeTooling_MissingRunner(t *testing.T) { + cache := t.TempDir() + _ = os.MkdirAll(filepath.Join(cache, "modeler", "tools", "node"), 0o755) + if _, _, err := resolveNodeTooling(filepath.Join(cache, "modeler", "mxbuild")); err == nil { + t.Error("expected error when rollup-runner.mjs is absent") + } +} + +func TestBuildWebClient_NoRollupConfig(t *testing.T) { + // web/ without rollup.config.mjs -> clear error (serve build not run yet). + dep := t.TempDir() + _ = os.MkdirAll(filepath.Join(dep, "web"), 0o755) + err := BuildWebClient(WebClientOptions{DeployDir: dep, MxBuildPath: "/x/modeler/mxbuild"}) + if err == nil { + t.Error("expected error when rollup.config.mjs is missing") + } +} diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index 7037a30f1..f264a4722 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -33,8 +33,12 @@ so structural changes need a restart; behavioural changes do not. 2. Ensures MxBuild and the runtime are cached (downloads once, reused after). 3. Checks the database is reachable (it does **not** provision it — see below). 4. Starts `mxbuild --serve` and does the first (cold) build into `deployment/`. -5. Boots a standalone runtime against that deployment and serves it. -6. With `--watch`, rebuilds and hot-applies on every project change until `Ctrl-C`. +5. Bundles the browser client (`web/dist/`) with mxbuild's rollup tooling — the + serve Deploy target writes client *source* but not the bundle, so this step is + what makes pages render. +6. Boots a standalone runtime against that deployment and serves it. +7. With `--watch`, rebuilds, re-bundles the client, and hot-applies on every + project change until `Ctrl-C`. ## Requirements @@ -70,19 +74,19 @@ createdb -h 127.0.0.1 -U mendix app1112 intended loop is: an agent (or you) edits the model with `mxcli exec`/MDL, and the running `run --local` picks the change up and hot-applies it. -## Current limitation — browser page rendering +## Pages render in the browser -The runtime, model, and admin API work end to end (the app answers HTTP 200, and -model reload/restart is applied correctly). **However, the browser client does not -yet render**: the `mxbuild` web client bundle (`web/dist/index.js`) is produced by a -separate rollup step that the serve/standalone path does not currently run, so -`index.html` 404s on its main bundle and the page is blank. +`run --local` bundles the browser client (`web/dist/`) after the deploy build, so +the app renders in a real browser — verified by driving the pre-installed Chromium +with Playwright against a booted app (the Mendix homepage renders fully). This makes +`run --local` usable for **visual page-design iteration**, not just headless checks. -This means `run --local` is presently useful for **runtime/model/API-level -iteration and headless verification**, but **not yet for visual page-design -iteration**. Closing the client-bundle gap (running mxbuild's bundled rollup tooling -after the deploy build) is tracked as follow-up work. Until then, use `mxcli docker -run` when you need the fully-rendered client in a browser. +**Watch cost.** In `--watch`, the client bundle is rebuilt on every change — a full +rollup bundle (~6–7 s) today. Behavioural changes (microflows) don't actually need +it, and page changes could use an incremental watch-mode bundler; making the client +rebuild incremental/conditional is a tracked optimization. For pixel-perfect page +work, pair this with Playwright screenshots (see below): edit → auto-rebuild → +re-screenshot. See also: [PROPOSAL_mxcli_dev_warm_loop](../../../docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md), [mxcli docker run](docker-run.md), [Playwright Testing](playwright.md). diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index acd7a5383..3984b8db8 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -164,17 +164,23 @@ returns HTTP 200 with the table present in Postgres. Pieces: (standalone boot; constants lifted from `deployment/model/config.json`), `docker.RunLocal` (+ `--watch`), and the `run --local` command. -**Known gap — browser client bundle not produced (blocks visual page iteration).** -The runtime/model/admin API work, but a browser loads blank: `index.html` requests -`web/dist/index.js`, which neither the serve `Deploy` target nor a one-shot `mxbuild` -produces here. The client bundle is built by a **rollup** step (`web/rollup.config.mjs`, -using mxbuild's bundled `modeler/tools/node` tooling) that the serve/standalone path -does not run — `web/index.js` (the rollup *input*) is present but the bundled -`web/dist/index.js` output is missing. Until this is closed, `run --local` is for -runtime/model/API iteration and headless checks; use `mxcli docker run` for a rendered -client. **Next step:** invoke mxbuild's rollup client build after the deploy build (or -find the serve option that emits `dist/`), then wire Playwright screenshotting for the -pixel-perfect page loop. +**Browser client bundle — closed.** Initially a blocker: a browser loaded blank +because `index.html` requests `web/dist/index.js`, which neither the serve `Deploy` +target nor a one-shot `mxbuild` produces. The client bundle is built by a **rollup** +step (`web/rollup.config.mjs`, using mxbuild's bundled `modeler/tools/node` tooling) +that the serve/standalone path doesn't run — `web/index.js` (the rollup *input*) is +written but `web/dist/index.js` is not. Fixed by `docker.BuildWebClient`, which runs +mxbuild's `rollup-runner.mjs` (production, one-shot) over `deployment/web` after the +deploy build. Verified: driving the devcontainer's Chromium with Playwright renders +the Mendix homepage fully. `web/dist/index.js` now serves HTTP 200 out of the box. + +**Remaining optimization.** In `--watch` the client bundle is rebuilt in full on +every change (~6–7 s). Behavioural changes (microflows) don't need it, and page +changes could use rollup's incremental watch mode (the runner's non-production branch +speaks the `modern-web-bundler-protocol` over stdout — a long-lived companion bundler, +mirroring `mxbuild --serve`). Making the client rebuild incremental/conditional is the +next optimization; then wire Playwright screenshotting into the loop for the +pixel-perfect page workflow. ## Proposed CLI From cf6a0dafeb0b1346dd66fce913f9635c3bf5bf57 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 14:55:34 +0000 Subject: [PATCH 18/26] feat(run-local): incremental client bundler + Playwright screenshots Completes the pixel-perfect page loop for `mxcli run --local --watch`. Incremental web client bundler (docker.WebClientWatcher): - keeps mxbuild's rollup runner hot in watch mode (client-side mirror of `mxbuild --serve`), parsing its modern-web-bundler-protocol stdout to count successful bundles; a page/widget edit re-bundles in ~3-4s vs ~7s cold. - runs with CHOKIDAR_USEPOLLING (+INTERVAL): inotify is silent on container overlay filesystems, so without polling change detection took tens of seconds; polling drops it to ~1s. - the watch loop re-bundles only when the edit touched web/ client source (mtime gate); microflow/entity edits skip the bundle and just hot-reload. WaitForRebuild settles out cleanly if the touched file isn't a rollup input, so it never hangs. Non-watch mode keeps the deterministic one-shot bundle. Playwright screenshots (docker.CaptureScreenshot, --screenshot): - captures a PNG after boot and each applied change via Playwright's built-in `screenshot` command (Chromium from PLAYWRIGHT_BROWSERS_PATH; no playwright-cli dependency). --screenshot-path / --screenshot-url configure output and page. Change signal fix: watch model source (.mpr + mprcontents/) only, not the whole project dir. The build rewrites theme-cache/, .mendix-cache/, deployment/ every run and screenshots land in .mxcli/; a whole-dir watcher self-triggered an infinite rebuild loop. Verified E2E on a blank 11.12.1 app: page edit -> incremental re-bundle -> reload -> auto-screenshot showing the live change (~4.7s); microflow edit -> reload, no re-bundle (~4s); no self-trigger loop while idle. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/run-local.md | 26 +- CLAUDE.md | 2 +- cmd/mxcli/cmd_run.go | 19 +- cmd/mxcli/docker/runlocal.go | 158 ++++++++-- cmd/mxcli/docker/runlocal_test.go | 80 ++++- cmd/mxcli/docker/screenshot.go | 110 +++++++ cmd/mxcli/docker/screenshot_test.go | 51 ++++ cmd/mxcli/docker/webclient_watch.go | 289 ++++++++++++++++++ cmd/mxcli/docker/webclient_watch_test.go | 112 +++++++ docs-site/src/tools/run-local.md | 50 ++- .../PROPOSAL_mxcli_dev_warm_loop.md | 30 +- 11 files changed, 847 insertions(+), 80 deletions(-) create mode 100644 cmd/mxcli/docker/screenshot.go create mode 100644 cmd/mxcli/docker/screenshot_test.go create mode 100644 cmd/mxcli/docker/webclient_watch.go create mode 100644 cmd/mxcli/docker/webclient_watch_test.go diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index d161cc223..808106082 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -76,6 +76,8 @@ mxcli exec add-page.mdl -p app.mpr |------|---------|---------| | `--local` | — | Required; run without Docker | | `--watch` | off | Rebuild + hot-apply on each change | +| `--screenshot` | off | Playwright PNG after boot + each change | +| `--screenshot-path` / `--screenshot-url` | `.mxcli/run-local.png` / app root | Screenshot output / page | | `--app-port` / `--admin-port` / `--serve-port` | 8080 / 8090 / 6543 | Ports | | `--db-host` / `--db-name` / `--db-user` / `--db-password` | 127.0.0.1:5432 / derived / mendix / mendix | Database | @@ -83,12 +85,26 @@ mxcli exec add-page.mdl -p app.mpr `run --local` bundles the browser client (`web/dist/`) with mxbuild's rollup tooling after the deploy build, so the app renders in a real browser (verified with -Playwright + the devcontainer's Chromium). Pair it with Playwright screenshots for a -pixel-perfect page loop: edit → auto-rebuild → re-screenshot. +Playwright + the devcontainer's Chromium). -**Watch cost:** in `--watch` the client bundle is rebuilt on every change (~6–7 s full -bundle today). Behavioural changes (microflows) don't strictly need it; making the -client rebuild incremental/conditional is a tracked optimization. +- **`--watch`** keeps a long-lived incremental bundler hot (the client-side mirror of + `mxbuild --serve`): a page/widget edit re-bundles in ~3–4 s; a microflow/entity edit + skips the bundle and just hot-reloads. It uses `CHOKIDAR_USEPOLLING` because inotify + is silent on container filesystems. +- Without `--watch`, a single one-shot bundle (~7 s) runs before boot. + +## Pixel-perfect page loop + +`--screenshot` captures a PNG (default `/.mxcli/run-local.png`) after boot +and after each applied change, via Playwright's built-in `screenshot` command +(Chromium from `PLAYWRIGHT_BROWSERS_PATH`): + +```bash +mxcli run --local -p app.mpr --watch --screenshot +# edit a page -> auto rebuild -> re-bundle -> reload -> fresh screenshot +``` + +Use `--screenshot-url` to shoot a specific page (deep link). ## Validation checklist diff --git a/CLAUDE.md b/CLAUDE.md index 62e70cf1c..379991fc4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -523,7 +523,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) -- Warm local dev loop (`mxcli run --local [--watch]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser (verified with Playwright). `--watch` re-bundles the client each change (~6-7s full bundle; incremental is a follow-up) — see `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` +- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - OQL query execution against running runtime (`mxcli oql`) - Business event services (SHOW/DESCRIBE/CREATE/DROP) - Project settings (SHOW/DESCRIBE/ALTER) diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index 8881e8d4f..33030ddfc 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -57,13 +57,19 @@ Examples: dbName, _ := cmd.Flags().GetString("db-name") dbUser, _ := cmd.Flags().GetString("db-user") dbPassword, _ := cmd.Flags().GetString("db-password") + screenshot, _ := cmd.Flags().GetBool("screenshot") + screenshotPath, _ := cmd.Flags().GetString("screenshot-path") + screenshotURL, _ := cmd.Flags().GetString("screenshot-url") opts := docker.LocalRunOptions{ - ProjectPath: projectPath, - AppPort: appPort, - AdminPort: adminPort, - ServePort: servePort, - Watch: watch, + ProjectPath: projectPath, + AppPort: appPort, + AdminPort: adminPort, + ServePort: servePort, + Watch: watch, + Screenshot: screenshot, + ScreenshotPath: screenshotPath, + ScreenshotURL: screenshotURL, DB: docker.DBConfig{ Host: dbHost, Name: dbName, @@ -91,5 +97,8 @@ func init() { runCmd.Flags().String("db-name", "", "Database name (default derived from the project name)") runCmd.Flags().String("db-user", "", "Database user (default mendix)") runCmd.Flags().String("db-password", "", "Database password (default mendix)") + runCmd.Flags().Bool("screenshot", false, "Capture a Playwright screenshot after boot and each applied change") + runCmd.Flags().String("screenshot-path", "", "Screenshot output PNG (default /.mxcli/run-local.png)") + runCmd.Flags().String("screenshot-url", "", "Page URL to screenshot (default the app root)") rootCmd.AddCommand(runCmd) } diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index 986cf9681..88855ef83 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -44,8 +44,15 @@ type LocalRunOptions struct { Watch bool // PollInterval is how often Watch checks for changes (default 1s). PollInterval time.Duration - Stdout io.Writer - Stderr io.Writer + // Screenshot, when set, captures a PNG of the app after boot and after each + // applied change (requires the Playwright CLI + a browser). + Screenshot bool + // ScreenshotPath is where the PNG is written (default /.mxcli/run-local.png). + ScreenshotPath string + // ScreenshotURL overrides the page shot (default the app root). + ScreenshotURL string + Stdout io.Writer + Stderr io.Writer } // defaultLocalAdminPass is the admin password for a local dev runtime. The admin @@ -86,6 +93,9 @@ func (o *LocalRunOptions) applyDefaults() { if o.DB.Name == "" { o.DB.Name = deriveDBName(o.ProjectPath) } + if o.ScreenshotPath == "" { + o.ScreenshotPath = filepath.Join(filepath.Dir(o.ProjectPath), ".mxcli", "run-local.png") + } if o.Stdout == nil { o.Stdout = os.Stdout } @@ -158,22 +168,19 @@ func pingTCP(hostPort string, timeout time.Duration) error { return nil } -// projectMTime returns the newest mtime under the project directory, ignoring -// the deployment dir and VCS metadata. It is the change signal for Watch: it -// covers both MPR v1 (single .mpr file) and v2 (metadata + mprcontents/). -func projectMTime(projectDir, deployDir string) time.Time { +// webClientSourceMTime returns the newest mtime of the browser-client *source* +// under /web, excluding the rollup output (dist/) and the build log. +// A page/widget/theme edit bumps it (and needs a client re-bundle); a +// microflow/entity-only edit does not. Zero time if web/ is absent. +func webClientSourceMTime(deployDir string) time.Time { + webDir := filepath.Join(deployDir, "web") var newest time.Time - deployAbs, _ := filepath.Abs(deployDir) - _ = filepath.WalkDir(projectDir, func(path string, d fs.DirEntry, err error) error { + _ = filepath.WalkDir(webDir, func(path string, d fs.DirEntry, err error) error { if err != nil { return nil } if d.IsDir() { - name := d.Name() - if name == ".git" || name == "node_modules" || name == "deployment" { - return fs.SkipDir - } - if abs, _ := filepath.Abs(path); abs == deployAbs { + if d.Name() == "dist" { return fs.SkipDir } return nil @@ -186,6 +193,32 @@ func projectMTime(projectDir, deployDir string) time.Time { return newest } +// projectSourceMTime returns the newest mtime of the Mendix model *source*: the +// .mpr file (v1 stores everything here) plus the mprcontents/ document tree (v2). +// It is the change signal for Watch. Watching only the model source — not the +// whole project dir — makes it immune to build-output churn: the serve/mxbuild +// build rewrites theme-cache/, .mendix-cache/, and deployment/ on every run, and +// screenshots land in .mxcli/, none of which must re-trigger the watcher. +func projectSourceMTime(projectPath string) time.Time { + var newest time.Time + if fi, err := os.Stat(projectPath); err == nil { + newest = fi.ModTime() + } + mprcontents := filepath.Join(filepath.Dir(projectPath), "mprcontents") + _ = filepath.WalkDir(mprcontents, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return nil + } + if !d.IsDir() { + if info, err := d.Info(); err == nil && info.ModTime().After(newest) { + newest = info.ModTime() + } + } + return nil + }) + return newest +} + // RunLocal boots the warm local dev loop: resolve tooling, start mxbuild --serve // and a standalone runtime, do the first build+apply, then (with Watch) rebuild // and hot-apply on every project change until interrupted. @@ -246,10 +279,22 @@ func RunLocal(opts LocalRunOptions) error { // 5b. Bundle the browser client (web/dist). The serve Deploy target writes the // client source but not the rollup bundle, so without this the app 404s on - // /dist/index.js and renders blank. - fmt.Fprintln(w, "Bundling web client...") - if err := BuildWebClient(WebClientOptions{DeployDir: opts.DeployDir, MxBuildPath: mxbuildPath, Stdout: w}); err != nil { - return fmt.Errorf("bundling web client: %w", err) + // /dist/index.js and renders blank. In --watch we keep an incremental bundler + // hot (warm rollup graph + polling file detection -> ~3-4s re-bundles); + // otherwise a single one-shot build (~7s cold) suffices. + var watcher *WebClientWatcher + if opts.Watch { + fmt.Fprintln(w, "Starting incremental web client bundler...") + watcher, err = StartWebClientWatch(WebClientOptions{DeployDir: opts.DeployDir, MxBuildPath: mxbuildPath, Stdout: w}) + if err != nil { + return fmt.Errorf("starting web client bundler: %w", err) + } + defer watcher.Stop() + } else { + fmt.Fprintln(w, "Bundling web client...") + if err := BuildWebClient(WebClientOptions{DeployDir: opts.DeployDir, MxBuildPath: mxbuildPath, Stdout: w}); err != nil { + return fmt.Errorf("bundling web client: %w", err) + } } // 6. Boot the runtime against the fresh deployment. @@ -270,11 +315,12 @@ func RunLocal(opts LocalRunOptions) error { defer rt.Stop() fmt.Fprintf(w, "\nApp is running at %s\n", rt.AppURL()) + maybeScreenshot(opts, rt) // 7. Stay up until interrupted. With --watch, rebuild + hot-apply on every // project change; otherwise just keep the runtime serving. if opts.Watch { - return watchAndApply(opts, serve, rt, mxbuildPath) + return watchAndApply(opts, serve, rt, watcher) } fmt.Fprintln(w, "(run with --watch to rebuild and hot-apply on changes; Ctrl-C to stop)") waitForInterrupt() @@ -282,6 +328,33 @@ func RunLocal(opts LocalRunOptions) error { return nil } +// maybeScreenshot captures the app (best-effort) when --screenshot is set. A +// failure is reported but never aborts the loop — the app is still running. +func maybeScreenshot(opts LocalRunOptions, rt *LocalRuntime) { + if !opts.Screenshot { + return + } + url := opts.ScreenshotURL + if url == "" { + url = rt.AppURL() + } + if err := os.MkdirAll(filepath.Dir(opts.ScreenshotPath), 0o755); err != nil { + fmt.Fprintf(opts.Stderr, " screenshot skipped: %v\n", err) + return + } + if err := CaptureScreenshot(ScreenshotOptions{ + URL: url, + OutPath: opts.ScreenshotPath, + WaitMs: 4000, + FullPage: true, + Viewport: "1280,800", + }); err != nil { + fmt.Fprintf(opts.Stderr, " screenshot skipped: %v\n", err) + return + } + fmt.Fprintf(opts.Stdout, " screenshot -> %s\n", opts.ScreenshotPath) +} + // resolveRuntimeInstall returns the directory to use as MX_INSTALL_PATH (its // runtime/ child holds the launcher and Mendix libraries), downloading the // runtime only when nothing usable is cached. It also makes the runtime visible @@ -327,16 +400,15 @@ func waitForInterrupt() { // watchAndApply polls the project for changes and applies each rebuild until the // user interrupts (Ctrl-C). StartLocalRuntime already resolved the JVM; here we // only rebuild via serve and let the RuntimeController decide reload vs restart. -func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, mxbuildPath string) error { +func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, watcher *WebClientWatcher) error { w := opts.Stdout - projectDir := filepath.Dir(opts.ProjectPath) sigCh := make(chan os.Signal, 1) signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) defer signal.Stop(sigCh) fmt.Fprintln(w, "Watching for changes (Ctrl-C to stop)...") - last := projectMTime(projectDir, opts.DeployDir) + last := projectSourceMTime(opts.ProjectPath) ticker := time.NewTicker(opts.PollInterval) defer ticker.Stop() @@ -346,13 +418,20 @@ func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, m fmt.Fprintln(w, "\nShutting down...") return nil case <-ticker.C: - now := projectMTime(projectDir, opts.DeployDir) + now := projectSourceMTime(opts.ProjectPath) if !now.After(last) { continue } last = now fmt.Fprintln(w, "Change detected, rebuilding...") start := time.Now() + + // Capture the client-bundle generation and the web-source mtime before + // the serve build, so we can tell whether the change plausibly touched + // client source and, if so, wait for the incremental re-bundle. + genBefore := watcher.Generation() + webBefore := webClientSourceMTime(opts.DeployDir) + build, err := serve.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: opts.ProjectPath}) if err != nil { fmt.Fprintf(opts.Stderr, " build error: %v\n", err) @@ -362,22 +441,37 @@ func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, m fmt.Fprintf(opts.Stderr, " build failed: %s\n", build.Message) continue } - // Rebuild the browser client bundle so page/widget edits are reflected - // (the serve Deploy target writes client source but not the rollup - // bundle). This is a full ~7s bundle today; an incremental watch-mode - // companion bundler is the optimization follow-up. - if err := BuildWebClient(WebClientOptions{DeployDir: opts.DeployDir, MxBuildPath: mxbuildPath, Stdout: w}); err != nil { - fmt.Fprintf(opts.Stderr, " web client build failed: %v\n", err) - continue + // If the serve build touched web/ source, wait (briefly) for the + // incremental bundler to re-bundle. WaitForRebuild settles out cleanly if + // no rebuild materializes (the touched file isn't a rollup input — e.g. a + // microflow edit that rewrites a web metadata file but no page/widget), so + // this never hangs. A pure model change skips the wait entirely. + bundled := false + if webClientSourceMTime(opts.DeployDir).After(webBefore) { + // Detection is a reliable ~1s with polling, so a 2.5s settle is ample + // margin to catch a rebuild that's going to start, while keeping the + // no-rebuild case (a model edit that only grazed web/) snappy. + bundled, err = watcher.WaitForRebuild(genBefore, 2500*time.Millisecond, 90*time.Second) + if err != nil { + fmt.Fprintf(opts.Stderr, " web client rebuild failed: %v\n", err) + continue + } } + action, err := rt.Controller().ApplyBuild(build, rt.Restart) if err != nil { fmt.Fprintf(opts.Stderr, " apply (%s) failed: %v\n", action, err) continue } - // Refresh last after the apply so edits made during the build are caught. - last = projectMTime(projectDir, opts.DeployDir) - fmt.Fprintf(w, " applied via %s in %s -> %s\n", action, time.Since(start).Round(time.Millisecond), rt.AppURL()) + client := "" + if bundled { + client = ", client re-bundled" + } + fmt.Fprintf(w, " applied via %s in %s%s -> %s\n", action, time.Since(start).Round(time.Millisecond), client, rt.AppURL()) + maybeScreenshot(opts, rt) + // Refresh the baseline AFTER the apply so an edit made mid-build is + // still caught on the next tick. + last = projectSourceMTime(opts.ProjectPath) } } } diff --git a/cmd/mxcli/docker/runlocal_test.go b/cmd/mxcli/docker/runlocal_test.go index 49d908edb..f6da049f8 100644 --- a/cmd/mxcli/docker/runlocal_test.go +++ b/cmd/mxcli/docker/runlocal_test.go @@ -107,33 +107,83 @@ func TestEnsureMxBuildRuntimeSibling_MissingSource(t *testing.T) { } } -func TestProjectMTime(t *testing.T) { +func TestProjectSourceMTime(t *testing.T) { dir := t.TempDir() - deploy := filepath.Join(dir, "deployment") - _ = os.MkdirAll(deploy, 0o755) - _ = os.MkdirAll(filepath.Join(dir, ".git"), 0o755) + // Build-output/cache dirs the serve/mxbuild build churns — must be ignored. + for _, d := range []string{"deployment", "theme-cache", ".mendix-cache", ".mxcli"} { + _ = os.MkdirAll(filepath.Join(dir, d), 0o755) + } + mprcontents := filepath.Join(dir, "mprcontents", "ab") + _ = os.MkdirAll(mprcontents, 0o755) mpr := filepath.Join(dir, "App.mpr") if err := os.WriteFile(mpr, []byte("x"), 0o644); err != nil { t.Fatal(err) } - base := projectMTime(dir, deploy) + doc := filepath.Join(mprcontents, "doc.mxunit") + _ = os.WriteFile(doc, []byte("d"), 0o644) + + base := projectSourceMTime(mpr) if base.IsZero() { - t.Fatal("expected a non-zero mtime") + t.Fatal("expected a non-zero source mtime") } - // A change under deployment/ must NOT advance the signal. + // Churn in every build-output/cache dir must NOT advance the signal. future := time.Now().Add(time.Hour) - deployFile := filepath.Join(deploy, "model.mdp") - _ = os.WriteFile(deployFile, []byte("y"), 0o644) - _ = os.Chtimes(deployFile, future, future) - if got := projectMTime(dir, deploy); got.After(base) { - t.Error("deployment/ changes should be ignored by the watch signal") + for _, f := range []string{ + filepath.Join(dir, "deployment", "model.mdp"), + filepath.Join(dir, "theme-cache", "theme.compiled.css"), + filepath.Join(dir, ".mendix-cache", "x"), + filepath.Join(dir, ".mxcli", "run-local.png"), + } { + _ = os.WriteFile(f, []byte("y"), 0o644) + _ = os.Chtimes(f, future, future) + } + if projectSourceMTime(mpr).After(base) { + t.Error("build-output/cache churn must not advance the source signal") } - // A change to the project file MUST advance the signal. + // An edit to the .mpr MUST advance the signal. _ = os.Chtimes(mpr, future, future) - if got := projectMTime(dir, deploy); !got.After(base) { - t.Error("project file change should advance the watch signal") + if !projectSourceMTime(mpr).After(base) { + t.Error(".mpr change should advance the signal") + } + + // An edit under mprcontents/ (v2 documents) MUST advance the signal. + base2 := projectSourceMTime(mpr) + future2 := time.Now().Add(2 * time.Hour) + _ = os.Chtimes(doc, future2, future2) + if !projectSourceMTime(mpr).After(base2) { + t.Error("mprcontents/ change should advance the signal") + } +} + +func TestWebClientSourceMTime_ExcludesDist(t *testing.T) { + dep := t.TempDir() + web := filepath.Join(dep, "web") + dist := filepath.Join(web, "dist") + _ = os.MkdirAll(filepath.Join(web, "pages"), 0o755) + _ = os.MkdirAll(dist, 0o755) + + src := filepath.Join(web, "pages", "Home.js") + _ = os.WriteFile(src, []byte("x"), 0o644) + base := webClientSourceMTime(dep) + if base.IsZero() { + t.Fatal("expected a non-zero web source mtime") + } + + // A newer file under dist/ must NOT advance the source mtime. + future := time.Now().Add(time.Hour) + df := filepath.Join(dist, "index.js") + _ = os.WriteFile(df, []byte("y"), 0o644) + _ = os.Chtimes(df, future, future) + if webClientSourceMTime(dep).After(base) { + t.Error("dist/ changes must be excluded from the web source mtime") + } + + // A newer source file MUST advance it. + _ = os.Chtimes(src, future, future) + if !webClientSourceMTime(dep).After(base) { + t.Error("a client source change should advance the web source mtime") } } diff --git a/cmd/mxcli/docker/screenshot.go b/cmd/mxcli/docker/screenshot.go new file mode 100644 index 000000000..d86fe193e --- /dev/null +++ b/cmd/mxcli/docker/screenshot.go @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "os" + "os/exec" + "strconv" + "time" +) + +// screenshot.go captures a screenshot of the running app with Playwright's +// built-in `screenshot` subcommand. This is the standard `playwright` CLI (from +// @playwright/test), not the `playwright-cli` session tool used by +// `mxcli playwright` — it needs no live session and resolves Chromium from +// PLAYWRIGHT_BROWSERS_PATH, so it works in the devcontainer out of the box. +// +// Paired with the warm loop it gives a pixel-perfect page cycle: edit -> serve +// rebuild -> incremental client bundle -> hot apply -> re-screenshot. + +// ScreenshotOptions configures CaptureScreenshot. +type ScreenshotOptions struct { + URL string // page to shoot (default the app root) + OutPath string // output PNG path (required) + Selector string // wait for this selector before shooting (optional) + WaitMs int // fixed wait before shooting, ms (default 4000) + FullPage bool // capture the full scrollable page + Viewport string // "W,H" (e.g. "1280,800") + Timeout time.Duration // overall command timeout (default 90s) +} + +// resolvePlaywright returns the command + leading args to invoke the Playwright +// CLI: the `playwright` binary if on PATH, else `npx playwright`. The bool is +// false when neither is available. +func resolvePlaywright() (string, []string, bool) { + if p, err := exec.LookPath("playwright"); err == nil { + return p, nil, true + } + if p, err := exec.LookPath("npx"); err == nil { + return p, []string{"playwright"}, true + } + return "", nil, false +} + +// screenshotArgs builds the `screenshot` subcommand arguments (excluding the CLI +// prefix). Kept separate so it is unit-testable without invoking Playwright. +func screenshotArgs(opts ScreenshotOptions) []string { + args := []string{"screenshot"} + if opts.Selector != "" { + args = append(args, "--wait-for-selector", opts.Selector) + } else { + wait := opts.WaitMs + if wait == 0 { + wait = 4000 + } + args = append(args, "--wait-for-timeout", strconv.Itoa(wait)) + } + if opts.FullPage { + args = append(args, "--full-page") + } + if opts.Viewport != "" { + args = append(args, "--viewport-size", opts.Viewport) + } + return append(args, opts.URL, opts.OutPath) +} + +// CaptureScreenshot shoots opts.URL to opts.OutPath. Returns a clear error when +// the Playwright CLI is unavailable so the caller can degrade gracefully. +func CaptureScreenshot(opts ScreenshotOptions) error { + if opts.OutPath == "" { + return fmt.Errorf("screenshot output path is required") + } + bin, prefix, ok := resolvePlaywright() + if !ok { + return fmt.Errorf("playwright CLI not found (install with: npm i -g playwright); " + + "Chromium is resolved from PLAYWRIGHT_BROWSERS_PATH") + } + timeout := opts.Timeout + if timeout == 0 { + timeout = 90 * time.Second + } + + args := append(append([]string{}, prefix...), screenshotArgs(opts)...) + cmd := exec.Command(bin, args...) + cmd.Env = os.Environ() // carries PLAYWRIGHT_BROWSERS_PATH + out := &syncBuffer{} + cmd.Stdout = out + cmd.Stderr = out + + done := make(chan error, 1) + if err := cmd.Start(); err != nil { + return fmt.Errorf("launching playwright: %w", err) + } + go func() { done <- cmd.Wait() }() + select { + case err := <-done: + if err != nil { + return fmt.Errorf("playwright screenshot failed: %w\n%s", err, out.String()) + } + case <-time.After(timeout): + _ = cmd.Process.Kill() + <-done + return fmt.Errorf("playwright screenshot timed out after %s", timeout) + } + if _, err := os.Stat(opts.OutPath); err != nil { + return fmt.Errorf("playwright reported success but %s is missing:\n%s", opts.OutPath, out.String()) + } + return nil +} diff --git a/cmd/mxcli/docker/screenshot_test.go b/cmd/mxcli/docker/screenshot_test.go new file mode 100644 index 000000000..4c61b3b9a --- /dev/null +++ b/cmd/mxcli/docker/screenshot_test.go @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "strings" + "testing" +) + +func TestScreenshotArgs_TimeoutDefault(t *testing.T) { + args := screenshotArgs(ScreenshotOptions{URL: "http://x:8080/", OutPath: "/o.png"}) + joined := strings.Join(args, " ") + if !strings.HasPrefix(joined, "screenshot ") { + t.Errorf("args should start with the screenshot subcommand: %v", args) + } + if !strings.Contains(joined, "--wait-for-timeout 4000") { + t.Errorf("expected default 4000ms wait: %v", args) + } + // URL and out path are the trailing positional args, in order. + if args[len(args)-2] != "http://x:8080/" || args[len(args)-1] != "/o.png" { + t.Errorf("positional args wrong: %v", args) + } +} + +func TestScreenshotArgs_SelectorWins(t *testing.T) { + args := screenshotArgs(ScreenshotOptions{URL: "u", OutPath: "o", Selector: ".mx-name-x", WaitMs: 9000}) + joined := strings.Join(args, " ") + if !strings.Contains(joined, "--wait-for-selector .mx-name-x") { + t.Errorf("expected selector wait: %v", args) + } + if strings.Contains(joined, "--wait-for-timeout") { + t.Errorf("selector should replace the timeout wait: %v", args) + } +} + +func TestScreenshotArgs_FullPageAndViewport(t *testing.T) { + args := screenshotArgs(ScreenshotOptions{URL: "u", OutPath: "o", FullPage: true, Viewport: "1280,800"}) + joined := strings.Join(args, " ") + if !strings.Contains(joined, "--full-page") { + t.Errorf("expected --full-page: %v", args) + } + if !strings.Contains(joined, "--viewport-size 1280,800") { + t.Errorf("expected viewport: %v", args) + } +} + +func TestCaptureScreenshot_NoOutPath(t *testing.T) { + if err := CaptureScreenshot(ScreenshotOptions{URL: "u"}); err == nil { + t.Error("expected error when OutPath is empty") + } +} diff --git a/cmd/mxcli/docker/webclient_watch.go b/cmd/mxcli/docker/webclient_watch.go new file mode 100644 index 000000000..9476f8903 --- /dev/null +++ b/cmd/mxcli/docker/webclient_watch.go @@ -0,0 +1,289 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "bufio" + "encoding/json" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "sync" + "syscall" + "time" +) + +// webclient_watch.go runs mxbuild's rollup client bundler as a long-lived +// incremental watcher — the client-side mirror of `mxbuild --serve`. The runner +// (rollup-runner.mjs, non-production branch) keeps rollup's module graph loaded +// and rebuilds only what changed, so after the serve Deploy target rewrites the +// web client source, re-bundling web/dist is ~3-4s instead of a ~7s cold build. +// +// The runner emits status lines on stdout using the modern-web-bundler protocol: +// +// {"protocol":"mx-modern-web-bundler","type":"status","payload":{"kind":"start"}} +// {"protocol":"mx-modern-web-bundler","type":"status","payload":{"kind":"success"}} +// {"protocol":"mx-modern-web-bundler","type":"status","payload":{"kind":"error","error":{"message":"..."}}} +// +// WebClientWatcher parses those, counting successful bundles as a generation so +// callers can wait for the rebuild triggered by a given source change. +// +// Container filesystems: rollup's chokidar file watcher relies on inotify, which +// does not fire on overlay/bind-mount filesystems (devcontainers). Without a fix +// change detection takes tens of seconds; CHOKIDAR_USEPOLLING makes it ~1s. + +const bundlerProtocolName = "mx-modern-web-bundler" + +// bundlerStatus is a parsed protocol status line. +type bundlerStatus struct { + kind string // "start" | "success" | "error" + message string // populated for kind == "error" +} + +// parseBundlerStatus extracts a status from one runner stdout line. ok is false +// for non-protocol lines (plain logging), which are ignored. +func parseBundlerStatus(line string) (bundlerStatus, bool) { + var m struct { + Protocol string `json:"protocol"` + Type string `json:"type"` + Payload struct { + Kind string `json:"kind"` + Error *struct { + Message string `json:"message"` + } `json:"error"` + } `json:"payload"` + } + if err := json.Unmarshal([]byte(line), &m); err != nil { + return bundlerStatus{}, false + } + if m.Protocol != bundlerProtocolName || m.Type != "status" || m.Payload.Kind == "" { + return bundlerStatus{}, false + } + s := bundlerStatus{kind: m.Payload.Kind} + if m.Payload.Error != nil { + s.message = m.Payload.Error.Message + } + return s, true +} + +// WebClientWatcher is a running incremental rollup bundler for a deployment's +// web/ directory. +type WebClientWatcher struct { + cmd *exec.Cmd + log *syncBuffer + + mu sync.Mutex + gen int // number of successful bundles + building bool // a bundle is currently in progress + lastErr string // message of the most recent failed bundle + exited bool // the runner process has exited + updated chan struct{} // closed+replaced on every state change (broadcast) +} + +// applyStatus folds a status line into the watcher state and broadcasts. +func (wc *WebClientWatcher) applyStatus(s bundlerStatus) { + wc.mu.Lock() + switch s.kind { + case "start": + wc.building = true + wc.lastErr = "" + case "success": + wc.building = false + wc.gen++ + case "error": + wc.building = false + wc.lastErr = s.message + if wc.lastErr == "" { + wc.lastErr = "unknown bundler error" + } + } + wc.broadcastLocked() + wc.mu.Unlock() +} + +// broadcastLocked wakes all waiters. Caller holds wc.mu. +func (wc *WebClientWatcher) broadcastLocked() { + close(wc.updated) + wc.updated = make(chan struct{}) +} + +func (wc *WebClientWatcher) snapshot() (gen int, building, exited bool, lastErr string, ch chan struct{}) { + wc.mu.Lock() + defer wc.mu.Unlock() + return wc.gen, wc.building, wc.exited, wc.lastErr, wc.updated +} + +// Generation returns the number of successful bundles so far. Capture it before +// triggering a source change, then pass it to WaitForBundle. +func (wc *WebClientWatcher) Generation() int { + wc.mu.Lock() + defer wc.mu.Unlock() + return wc.gen +} + +// StartWebClientWatch launches the incremental bundler and blocks until the first +// bundle completes (so web/dist exists before the app boots). +func StartWebClientWatch(opts WebClientOptions) (*WebClientWatcher, error) { + webDir := filepath.Join(opts.DeployDir, "web") + if fi, err := os.Stat(filepath.Join(webDir, "rollup.config.mjs")); err != nil || fi.IsDir() { + return nil, fmt.Errorf("no rollup.config.mjs in %s (run a serve Deploy build first)", webDir) + } + nodeBin, runner, err := resolveNodeTooling(opts.MxBuildPath) + if err != nil { + return nil, err + } + + cmd := exec.Command(nodeBin, runner) + cmd.Dir = webDir + // NODE_ENV must be a defined string (the config's esbuild plugin injects it as + // a `define`), but NOT "production" — that would take the runner's one-shot + // branch instead of watch(). "development" gives incremental watch mode. + // CHOKIDAR_USEPOLLING makes file-change detection work on container overlay + // filesystems where inotify is silent (otherwise detection takes ~tens of s). + cmd.Env = append(os.Environ(), + "NODE_ENV=development", + "CHOKIDAR_USEPOLLING=true", + "CHOKIDAR_INTERVAL=300", + "MX_WEB_CLIENT_BUILD_LOG="+filepath.Join(opts.DeployDir, "log", "web-client-build.log"), + ) + log := &syncBuffer{} + stdout, err := cmd.StdoutPipe() + if err != nil { + return nil, fmt.Errorf("web client watcher stdout pipe: %w", err) + } + cmd.Stderr = log + + wc := &WebClientWatcher{cmd: cmd, log: log, updated: make(chan struct{})} + if err := cmd.Start(); err != nil { + return nil, fmt.Errorf("launching web client watcher: %w", err) + } + go wc.readLoop(stdout) + go func() { + _ = cmd.Wait() + wc.mu.Lock() + wc.exited = true + wc.broadcastLocked() + wc.mu.Unlock() + }() + + timeout := opts.Timeout + if timeout == 0 { + timeout = 5 * time.Minute + } + if err := wc.waitForFirstBuild(timeout); err != nil { + _ = wc.Stop() + return nil, err + } + dist := filepath.Join(webDir, "dist", "index.js") + if _, err := os.Stat(dist); err != nil { + _ = wc.Stop() + return nil, fmt.Errorf("web client watcher reported success but %s is missing:\n%s", dist, wc.log.String()) + } + return wc, nil +} + +// readLoop parses the runner's stdout and folds each status into state. It also +// tees the raw lines into the log buffer for diagnostics. +func (wc *WebClientWatcher) readLoop(stdout io.Reader) { + sc := bufio.NewScanner(stdout) + sc.Buffer(make([]byte, 0, 64*1024), 1024*1024) + for sc.Scan() { + line := sc.Text() + _, _ = wc.log.Write([]byte(line + "\n")) + if s, ok := parseBundlerStatus(line); ok { + wc.applyStatus(s) + } + } +} + +// waitForFirstBuild blocks until the first successful bundle, or an error/exit. +func (wc *WebClientWatcher) waitForFirstBuild(timeout time.Duration) error { + deadline := time.Now().Add(timeout) + for { + gen, _, exited, lastErr, ch := wc.snapshot() + if gen >= 1 { + return nil + } + if lastErr != "" { + return fmt.Errorf("initial web client build failed: %s", lastErr) + } + if exited { + return fmt.Errorf("web client watcher exited before the first build:\n%s", wc.log.String()) + } + wait := time.Until(deadline) + if wait <= 0 { + return fmt.Errorf("web client watcher timed out after %s", timeout) + } + select { + case <-ch: + case <-time.After(wait): + } + } +} + +// WaitForRebuild waits for the incremental bundle triggered by a source change. +// It returns rebuilt=true once a successful bundle beyond sinceGen lands. If no +// bundle starts within settle, it returns rebuilt=false — the change didn't +// affect rollup's module graph (e.g. it touched a web/ file that isn't a bundle +// input), so there is nothing to wait for. This never hangs: settle bounds the +// "did a rebuild even start" wait, and buildTimeout bounds one that has. +// +// Reliable because file-change detection is fast (CHOKIDAR_USEPOLLING ~1s), so a +// rebuild that is going to happen starts well within a few-second settle window. +func (wc *WebClientWatcher) WaitForRebuild(sinceGen int, settle, buildTimeout time.Duration) (bool, error) { + settleDeadline := time.Now().Add(settle) + sawBuild := false + for { + gen, building, exited, lastErr, ch := wc.snapshot() + if gen > sinceGen { + return true, nil + } + if building { + sawBuild = true + } + if sawBuild && !building && lastErr != "" { + return false, fmt.Errorf("web client build failed: %s", lastErr) + } + if exited { + return false, fmt.Errorf("web client watcher exited:\n%s", wc.log.String()) + } + var wait time.Duration + if sawBuild || building { + wait = time.Until(settleDeadline.Add(buildTimeout)) + } else { + wait = time.Until(settleDeadline) + } + if wait <= 0 { + return false, nil // no rebuild materialized + } + select { + case <-ch: + case <-time.After(wait): + if !sawBuild && !building { + return false, nil + } + } + } +} + +// Log returns the captured watcher output. +func (wc *WebClientWatcher) Log() string { return wc.log.String() } + +// Stop terminates the watcher process. +func (wc *WebClientWatcher) Stop() error { + if wc.cmd == nil || wc.cmd.Process == nil { + return nil + } + _ = wc.cmd.Process.Signal(syscall.SIGTERM) + done := make(chan error, 1) + go func() { done <- wc.cmd.Wait() }() + select { + case <-done: + case <-time.After(5 * time.Second): + _ = wc.cmd.Process.Kill() + <-done + } + return nil +} diff --git a/cmd/mxcli/docker/webclient_watch_test.go b/cmd/mxcli/docker/webclient_watch_test.go new file mode 100644 index 000000000..e9a02cfca --- /dev/null +++ b/cmd/mxcli/docker/webclient_watch_test.go @@ -0,0 +1,112 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "testing" + "time" +) + +func TestParseBundlerStatus(t *testing.T) { + cases := []struct { + line string + wantKind string + wantOK bool + }{ + {`{"protocol":"mx-modern-web-bundler","type":"status","payload":{"kind":"start"}}`, "start", true}, + {`{"protocol":"mx-modern-web-bundler","type":"status","payload":{"kind":"success"}}`, "success", true}, + {`{"protocol":"mx-modern-web-bundler","type":"status","payload":{"kind":"error","error":{"message":"boom"}}}`, "error", true}, + {`Initially, there are 40 chunks`, "", false}, // plain log line + {`{"protocol":"other","type":"status","payload":{"kind":"x"}}`, "", false}, // wrong protocol + {`{"protocol":"mx-modern-web-bundler","type":"command","payload":{"kind":"x"}}`, "", false}, // not status + {`not json`, "", false}, + } + for _, tc := range cases { + s, ok := parseBundlerStatus(tc.line) + if ok != tc.wantOK || s.kind != tc.wantKind { + t.Errorf("parseBundlerStatus(%q) = (%q,%v), want (%q,%v)", tc.line, s.kind, ok, tc.wantKind, tc.wantOK) + } + } + s, _ := parseBundlerStatus(`{"protocol":"mx-modern-web-bundler","type":"status","payload":{"kind":"error","error":{"message":"boom"}}}`) + if s.message != "boom" { + t.Errorf("error message = %q, want boom", s.message) + } +} + +// newTestWatcher returns a watcher with no process, drivable via applyStatus. +func newTestWatcher() *WebClientWatcher { + return &WebClientWatcher{log: &syncBuffer{}, updated: make(chan struct{})} +} + +func TestApplyStatus_Generation(t *testing.T) { + wc := newTestWatcher() + if wc.Generation() != 0 { + t.Fatal("initial generation should be 0") + } + wc.applyStatus(bundlerStatus{kind: "start"}) + wc.applyStatus(bundlerStatus{kind: "success"}) + if wc.Generation() != 1 { + t.Errorf("generation = %d, want 1", wc.Generation()) + } + wc.applyStatus(bundlerStatus{kind: "start"}) + wc.applyStatus(bundlerStatus{kind: "success"}) + if wc.Generation() != 2 { + t.Errorf("generation = %d, want 2", wc.Generation()) + } +} + +func TestWaitForRebuild_Success(t *testing.T) { + wc := newTestWatcher() + gen := wc.Generation() + go func() { + time.Sleep(20 * time.Millisecond) + wc.applyStatus(bundlerStatus{kind: "start"}) + time.Sleep(20 * time.Millisecond) + wc.applyStatus(bundlerStatus{kind: "success"}) + }() + rebuilt, err := wc.WaitForRebuild(gen, time.Second, 5*time.Second) + if err != nil || !rebuilt { + t.Fatalf("WaitForRebuild = (%v,%v), want (true,nil)", rebuilt, err) + } +} + +func TestWaitForRebuild_AlreadyAdvanced(t *testing.T) { + wc := newTestWatcher() + gen := wc.Generation() + wc.applyStatus(bundlerStatus{kind: "start"}) + wc.applyStatus(bundlerStatus{kind: "success"}) + rebuilt, err := wc.WaitForRebuild(gen, time.Second, 5*time.Second) + if err != nil || !rebuilt { + t.Errorf("expected immediate (true,nil), got (%v,%v)", rebuilt, err) + } +} + +func TestWaitForRebuild_SettlesOutNoRebuild(t *testing.T) { + wc := newTestWatcher() + start := time.Now() + // No bundle ever starts (the change didn't hit rollup's graph) -> false, no hang. + rebuilt, err := wc.WaitForRebuild(wc.Generation(), 150*time.Millisecond, 5*time.Second) + if err != nil { + t.Fatalf("WaitForRebuild: %v", err) + } + if rebuilt { + t.Error("expected rebuilt=false when no bundle starts") + } + if time.Since(start) > 2*time.Second { + t.Error("should settle out near the settle window, not hang") + } +} + +func TestWaitForRebuild_Error(t *testing.T) { + wc := newTestWatcher() + gen := wc.Generation() + go func() { + time.Sleep(10 * time.Millisecond) + wc.applyStatus(bundlerStatus{kind: "start"}) + time.Sleep(10 * time.Millisecond) + wc.applyStatus(bundlerStatus{kind: "error", message: "syntax error in widget"}) + }() + if _, err := wc.WaitForRebuild(gen, time.Second, 5*time.Second); err == nil { + t.Fatal("expected an error when the bundle fails") + } +} diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index f264a4722..ec8b64bb8 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -66,27 +66,47 @@ createdb -h 127.0.0.1 -U mendix app1112 | `--db-host` | 127.0.0.1:5432 | Database `host:port` | | `--db-name` | derived from project | Database name | | `--db-user` / `--db-password` | mendix / mendix | Database credentials | +| `--screenshot` | off | Capture a Playwright PNG after boot and each applied change | +| `--screenshot-path` | `/.mxcli/run-local.png` | Screenshot output PNG | +| `--screenshot-url` | app root | Page URL to screenshot | ## The change signal -`--watch` watches the project directory (both MPR v1 single-file and v2 -`mprcontents/` layouts), ignoring `deployment/`, `.git`, and `node_modules`. The -intended loop is: an agent (or you) edits the model with `mxcli exec`/MDL, and the -running `run --local` picks the change up and hot-applies it. +`--watch` watches the model **source** — the `.mpr` file and the `mprcontents/` +document tree (v2) — not the whole project dir. This is deliberate: the serve/mxbuild +build rewrites `deployment/`, `theme-cache/`, and `.mendix-cache/` on every run, and +screenshots land in `.mxcli/`; watching only the source keeps that build-output churn +from re-triggering the loop. The intended cycle: an agent (or you) edits the model +with `mxcli exec`/MDL, and the running `run --local` picks it up and hot-applies it. ## Pages render in the browser -`run --local` bundles the browser client (`web/dist/`) after the deploy build, so -the app renders in a real browser — verified by driving the pre-installed Chromium -with Playwright against a booted app (the Mendix homepage renders fully). This makes -`run --local` usable for **visual page-design iteration**, not just headless checks. - -**Watch cost.** In `--watch`, the client bundle is rebuilt on every change — a full -rollup bundle (~6–7 s) today. Behavioural changes (microflows) don't actually need -it, and page changes could use an incremental watch-mode bundler; making the client -rebuild incremental/conditional is a tracked optimization. For pixel-perfect page -work, pair this with Playwright screenshots (see below): edit → auto-rebuild → -re-screenshot. +`run --local` bundles the browser client (`web/dist/`) so the app renders in a real +browser — verified by driving the pre-installed Chromium with Playwright (the Mendix +homepage renders fully). This makes it usable for **visual page-design iteration**, +not just headless checks. + +- **Non-`--watch`**: a one-shot rollup bundle after the deploy build (~7 s cold). +- **`--watch`**: a long-lived incremental bundler stays hot (the client-side mirror + of `mxbuild --serve`), so a page/widget edit re-bundles in ~3–4 s. It runs with + `CHOKIDAR_USEPOLLING` because inotify does not fire on container overlay + filesystems — without it, change detection takes tens of seconds. The loop + re-bundles **only when the edit touched client source**: a microflow/entity edit + skips the bundle and just hot-reloads. + +## Pixel-perfect page loop + +Pass `--screenshot` and each applied change is captured to a PNG (default +`/.mxcli/run-local.png`) using Playwright's built-in `screenshot` +command (Chromium from `PLAYWRIGHT_BROWSERS_PATH` — no `playwright-cli` needed): + +```bash +mxcli run --local -p app.mpr --watch --screenshot +# edit a page with mxcli exec/MDL -> auto rebuild -> re-bundle -> reload -> new PNG +``` + +Point `--screenshot-url` at a deep link to shoot a specific page. Screenshotting a +failure or a specific state (auth, navigation) is a natural next step. See also: [PROPOSAL_mxcli_dev_warm_loop](../../../docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md), [mxcli docker run](docker-run.md), [Playwright Testing](playwright.md). diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 3984b8db8..148387c06 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -174,13 +174,29 @@ mxbuild's `rollup-runner.mjs` (production, one-shot) over `deployment/web` after deploy build. Verified: driving the devcontainer's Chromium with Playwright renders the Mendix homepage fully. `web/dist/index.js` now serves HTTP 200 out of the box. -**Remaining optimization.** In `--watch` the client bundle is rebuilt in full on -every change (~6–7 s). Behavioural changes (microflows) don't need it, and page -changes could use rollup's incremental watch mode (the runner's non-production branch -speaks the `modern-web-bundler-protocol` over stdout — a long-lived companion bundler, -mirroring `mxbuild --serve`). Making the client rebuild incremental/conditional is the -next optimization; then wire Playwright screenshotting into the loop for the -pixel-perfect page workflow. +**Incremental client bundler + screenshots — done.** `--watch` now keeps a +long-lived incremental bundler hot (`docker.WebClientWatcher`) — the client-side +mirror of `mxbuild --serve`, running the runner's watch branch and parsing its +`modern-web-bundler-protocol` stdout to count successful bundles. A page/widget edit +re-bundles in ~3–4 s (vs ~7 s cold); the loop re-bundles **only when the edit touched +web/ client source** (mtime gate), so a microflow/entity edit skips it and just +hot-reloads. `WaitForRebuild` settles out cleanly if the touched file isn't a rollup +input, so it never hangs. + +Two findings from implementation: +- **inotify is silent on container overlay filesystems** — the rollup watcher's + chokidar detected changes only after tens of seconds until `CHOKIDAR_USEPOLLING` + (+`CHOKIDAR_INTERVAL`) was set, which drops detection to ~1 s. (Thanks to the polling + tip.) +- **The change signal must watch model source, not the project dir.** The build + rewrites `theme-cache/`, `.mendix-cache/`, and `deployment/` every run, and + screenshots land in `.mxcli/`; an initial whole-dir watcher self-triggered an + infinite rebuild loop. `projectSourceMTime` watches only `.mpr` + `mprcontents/`. + +`--screenshot` captures a PNG after boot and each applied change via Playwright's +built-in `screenshot` command (Chromium from `PLAYWRIGHT_BROWSERS_PATH`; no +`playwright-cli` dependency), completing the pixel-perfect page loop. Deep-link / +authenticated-state screenshots are the natural next step. ## Proposed CLI From f186511f302a618e7edbfad01014ea12787dc03e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 15:15:07 +0000 Subject: [PATCH 19/26] feat(run-local): authenticated + deep-link screenshots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extends --screenshot for the pixel-perfect page loop on secured apps and specific pages: - --screenshot-url now accepts a page path resolved against the app root (e.g. /p/customers), or a full URL as-is, so a specific page under edit is shot rather than the app root (resolveScreenshotURL). - --screenshot-user/--screenshot-password log in once through the Mendix login form and reuse the session for every screenshot, so pages behind login render authenticated. docker.LoginAndSaveStorage runs a headless Playwright script (driving #usernameInput/#passwordInput/#loginButton) and writes a Playwright storage state; CaptureScreenshot loads it via `screenshot --load-storage`. Login is best-effort: an anonymous app with no form proceeds unauthenticated. The login script is run through the same Playwright install, resolved from the `playwright` CLI's package dir (playwright-core via require.resolve) with a system-node/mxbuild-node fallback — no hardcoded paths. It is written as .cjs so Node treats it as CommonJS. Verified E2E on the 11.12.1 app: login produces a storage state with the Mendix session cookies, and the screenshot renders the authenticated page. Docs (site page, skill, CLAUDE.md, proposal) updated. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/run-local.md | 9 +- CLAUDE.md | 2 +- cmd/mxcli/cmd_run.go | 24 +-- cmd/mxcli/docker/runlocal.go | 49 +++++- cmd/mxcli/docker/runlocal_test.go | 16 ++ cmd/mxcli/docker/screenshot.go | 4 + cmd/mxcli/docker/screenshot_login.go | 151 ++++++++++++++++++ cmd/mxcli/docker/screenshot_test.go | 8 + docs-site/src/tools/run-local.md | 20 ++- .../PROPOSAL_mxcli_dev_warm_loop.md | 14 +- 10 files changed, 272 insertions(+), 25 deletions(-) create mode 100644 cmd/mxcli/docker/screenshot_login.go diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index 808106082..21fc778b1 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -77,7 +77,8 @@ mxcli exec add-page.mdl -p app.mpr | `--local` | — | Required; run without Docker | | `--watch` | off | Rebuild + hot-apply on each change | | `--screenshot` | off | Playwright PNG after boot + each change | -| `--screenshot-path` / `--screenshot-url` | `.mxcli/run-local.png` / app root | Screenshot output / page | +| `--screenshot-path` / `--screenshot-url` | `.mxcli/run-local.png` / app root | Screenshot output / page (URL or `/path`) | +| `--screenshot-user` / `--screenshot-password` | — | Log in once, reuse session (pages behind login) | | `--app-port` / `--admin-port` / `--serve-port` | 8080 / 8090 / 6543 | Ports | | `--db-host` / `--db-name` / `--db-user` / `--db-password` | 127.0.0.1:5432 / derived / mendix / mendix | Database | @@ -104,7 +105,11 @@ mxcli run --local -p app.mpr --watch --screenshot # edit a page -> auto rebuild -> re-bundle -> reload -> fresh screenshot ``` -Use `--screenshot-url` to shoot a specific page (deep link). +- `--screenshot-url /p/customers` shoots a specific page (bare path resolved against + the app root; a full URL is used as-is). +- `--screenshot-user`/`--screenshot-password` log in once (Mendix form auth) and + reuse the session, so pages behind login render authenticated. Best-effort: an + anonymous app with no login form proceeds unauthenticated. ## Validation checklist diff --git a/CLAUDE.md b/CLAUDE.md index 379991fc4..386a2a226 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -523,7 +523,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) -- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` +- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - OQL query execution against running runtime (`mxcli oql`) - Business event services (SHOW/DESCRIBE/CREATE/DROP) - Project settings (SHOW/DESCRIBE/ALTER) diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index 33030ddfc..482cd1ebd 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -60,16 +60,20 @@ Examples: screenshot, _ := cmd.Flags().GetBool("screenshot") screenshotPath, _ := cmd.Flags().GetString("screenshot-path") screenshotURL, _ := cmd.Flags().GetString("screenshot-url") + screenshotUser, _ := cmd.Flags().GetString("screenshot-user") + screenshotPassword, _ := cmd.Flags().GetString("screenshot-password") opts := docker.LocalRunOptions{ - ProjectPath: projectPath, - AppPort: appPort, - AdminPort: adminPort, - ServePort: servePort, - Watch: watch, - Screenshot: screenshot, - ScreenshotPath: screenshotPath, - ScreenshotURL: screenshotURL, + ProjectPath: projectPath, + AppPort: appPort, + AdminPort: adminPort, + ServePort: servePort, + Watch: watch, + Screenshot: screenshot, + ScreenshotPath: screenshotPath, + ScreenshotURL: screenshotURL, + ScreenshotUser: screenshotUser, + ScreenshotPassword: screenshotPassword, DB: docker.DBConfig{ Host: dbHost, Name: dbName, @@ -99,6 +103,8 @@ func init() { runCmd.Flags().String("db-password", "", "Database password (default mendix)") runCmd.Flags().Bool("screenshot", false, "Capture a Playwright screenshot after boot and each applied change") runCmd.Flags().String("screenshot-path", "", "Screenshot output PNG (default /.mxcli/run-local.png)") - runCmd.Flags().String("screenshot-url", "", "Page URL to screenshot (default the app root)") + runCmd.Flags().String("screenshot-url", "", "Page to screenshot: a full URL or a path relative to the app root, e.g. /p/customers (default the app root)") + runCmd.Flags().String("screenshot-user", "", "Log in with this user before screenshotting (for pages behind login)") + runCmd.Flags().String("screenshot-password", "", "Password for --screenshot-user") rootCmd.AddCommand(runCmd) } diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index 88855ef83..c55005c06 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -49,10 +49,18 @@ type LocalRunOptions struct { Screenshot bool // ScreenshotPath is where the PNG is written (default /.mxcli/run-local.png). ScreenshotPath string - // ScreenshotURL overrides the page shot (default the app root). + // ScreenshotURL overrides the page shot: a full http(s) URL, or a path + // relative to the app root (e.g. "/p/customers"). Default the app root. ScreenshotURL string - Stdout io.Writer - Stderr io.Writer + // ScreenshotUser/Password log in once (Mendix form auth) and reuse the session + // for every screenshot, so pages behind login render authenticated. + ScreenshotUser string + ScreenshotPassword string + // screenshotStorage is the resolved Playwright storage-state file (from login); + // internal, set during boot. + screenshotStorage string + Stdout io.Writer + Stderr io.Writer } // defaultLocalAdminPass is the admin password for a local dev runtime. The admin @@ -315,6 +323,21 @@ func RunLocal(opts LocalRunOptions) error { defer rt.Stop() fmt.Fprintf(w, "\nApp is running at %s\n", rt.AppURL()) + + // 6b. If screenshot auth was requested, log in once and reuse the session for + // every screenshot (pages behind login render authenticated). + if opts.Screenshot && opts.ScreenshotUser != "" { + storage := filepath.Join(filepath.Dir(opts.ScreenshotPath), "run-local-storage.json") + fmt.Fprintf(w, "Logging in as %q for authenticated screenshots...\n", opts.ScreenshotUser) + if err := LoginAndSaveStorage(LoginOptions{ + AppURL: rt.AppURL(), Username: opts.ScreenshotUser, Password: opts.ScreenshotPassword, + StoragePath: storage, MxBuildPath: mxbuildPath, + }); err != nil { + fmt.Fprintf(stderr, " screenshot login failed (continuing unauthenticated): %v\n", err) + } else { + opts.screenshotStorage = storage + } + } maybeScreenshot(opts, rt) // 7. Stay up until interrupted. With --watch, rebuild + hot-apply on every @@ -334,20 +357,17 @@ func maybeScreenshot(opts LocalRunOptions, rt *LocalRuntime) { if !opts.Screenshot { return } - url := opts.ScreenshotURL - if url == "" { - url = rt.AppURL() - } if err := os.MkdirAll(filepath.Dir(opts.ScreenshotPath), 0o755); err != nil { fmt.Fprintf(opts.Stderr, " screenshot skipped: %v\n", err) return } if err := CaptureScreenshot(ScreenshotOptions{ - URL: url, + URL: resolveScreenshotURL(rt.AppURL(), opts.ScreenshotURL), OutPath: opts.ScreenshotPath, WaitMs: 4000, FullPage: true, Viewport: "1280,800", + Storage: opts.screenshotStorage, }); err != nil { fmt.Fprintf(opts.Stderr, " screenshot skipped: %v\n", err) return @@ -355,6 +375,19 @@ func maybeScreenshot(opts LocalRunOptions, rt *LocalRuntime) { fmt.Fprintf(opts.Stdout, " screenshot -> %s\n", opts.ScreenshotPath) } +// resolveScreenshotURL resolves the --screenshot-url value against the app root: +// empty -> the root; a full http(s) URL -> as-is; anything else -> treated as a +// path relative to the app root (so "/p/customers" deep-links a specific page). +func resolveScreenshotURL(appURL, urlOrPath string) string { + if urlOrPath == "" { + return appURL + } + if strings.HasPrefix(urlOrPath, "http://") || strings.HasPrefix(urlOrPath, "https://") { + return urlOrPath + } + return strings.TrimRight(appURL, "/") + "/" + strings.TrimLeft(urlOrPath, "/") +} + // resolveRuntimeInstall returns the directory to use as MX_INSTALL_PATH (its // runtime/ child holds the launcher and Mendix libraries), downloading the // runtime only when nothing usable is cached. It also makes the runtime visible diff --git a/cmd/mxcli/docker/runlocal_test.go b/cmd/mxcli/docker/runlocal_test.go index f6da049f8..2ad002728 100644 --- a/cmd/mxcli/docker/runlocal_test.go +++ b/cmd/mxcli/docker/runlocal_test.go @@ -187,3 +187,19 @@ func TestWebClientSourceMTime_ExcludesDist(t *testing.T) { t.Error("a client source change should advance the web source mtime") } } + +func TestResolveScreenshotURL(t *testing.T) { + app := "http://127.0.0.1:8080/" + cases := map[string]string{ + "": app, + "/p/customers": "http://127.0.0.1:8080/p/customers", + "p/customers": "http://127.0.0.1:8080/p/customers", + "http://host:9000/x": "http://host:9000/x", + "https://example.com/a": "https://example.com/a", + } + for in, want := range cases { + if got := resolveScreenshotURL(app, in); got != want { + t.Errorf("resolveScreenshotURL(%q) = %q, want %q", in, got, want) + } + } +} diff --git a/cmd/mxcli/docker/screenshot.go b/cmd/mxcli/docker/screenshot.go index d86fe193e..581bacdaa 100644 --- a/cmd/mxcli/docker/screenshot.go +++ b/cmd/mxcli/docker/screenshot.go @@ -27,6 +27,7 @@ type ScreenshotOptions struct { WaitMs int // fixed wait before shooting, ms (default 4000) FullPage bool // capture the full scrollable page Viewport string // "W,H" (e.g. "1280,800") + Storage string // Playwright storage-state JSON to load (for authed pages) Timeout time.Duration // overall command timeout (default 90s) } @@ -62,6 +63,9 @@ func screenshotArgs(opts ScreenshotOptions) []string { if opts.Viewport != "" { args = append(args, "--viewport-size", opts.Viewport) } + if opts.Storage != "" { + args = append(args, "--load-storage", opts.Storage) + } return append(args, opts.URL, opts.OutPath) } diff --git a/cmd/mxcli/docker/screenshot_login.go b/cmd/mxcli/docker/screenshot_login.go new file mode 100644 index 000000000..56c44114b --- /dev/null +++ b/cmd/mxcli/docker/screenshot_login.go @@ -0,0 +1,151 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "time" +) + +// screenshot_login.go logs into a secured Mendix app once and saves a Playwright +// storage state (cookies + local storage), so subsequent screenshots of pages +// behind login use `playwright screenshot --load-storage`. The `playwright` CLI +// has no scriptable login, so this drives a tiny headless script via the same +// Playwright install (resolved from the CLI's package dir — no hardcoded paths). +// +// The Mendix Atlas login page exposes stable selectors: #usernameInput, +// #passwordInput, #loginButton. If no login form appears (anonymous/public app), +// the script still saves storage and proceeds. + +// resolvePlaywrightPkgDir returns the directory of the installed `playwright` +// package (the parent for requiring playwright-core), or "" if not found. +func resolvePlaywrightPkgDir() string { + bin, err := exec.LookPath("playwright") + if err != nil { + return "" + } + real, err := filepath.EvalSymlinks(bin) + if err != nil { + real = bin + } + // real is /cli.js; the package dir is its directory. + dir := filepath.Dir(real) + if _, err := os.Stat(filepath.Join(dir, "package.json")); err != nil { + return "" + } + return dir +} + +// resolveNodeForScript returns a node binary to run the login script: the system +// node if on PATH, else mxbuild's bundled node resolved from mxbuildPath. +func resolveNodeForScript(mxbuildPath string) string { + if p, err := exec.LookPath("node"); err == nil { + return p + } + if mxbuildPath != "" { + if nodeBin, _, err := resolveNodeTooling(mxbuildPath); err == nil { + return nodeBin + } + } + return "" +} + +// loginScript is the headless login+save-storage script. It requires +// playwright-core from the resolved package dir (arg 1). Remaining args: +// appURL, username, password, storagePath. +const loginScript = ` +const pkgDir = process.argv[2]; +const [appURL, username, password, storagePath] = process.argv.slice(3); +const { chromium } = require(require.resolve("playwright-core", { paths: [pkgDir] })); +(async () => { + const b = await chromium.launch(); + const ctx = await b.newContext(); + const p = await ctx.newPage(); + await p.goto(appURL, { waitUntil: "load", timeout: 30000 }); + try { + await p.waitForSelector("#usernameInput", { timeout: 8000 }); + await p.fill("#usernameInput", username); + await p.fill("#passwordInput", password); + await Promise.all([ + p.waitForNavigation({ waitUntil: "load", timeout: 20000 }).catch(() => {}), + p.click("#loginButton"), + ]); + await p.waitForTimeout(2500); + } catch (e) { + // No login form within the timeout: anonymous or already authenticated. + process.stderr.write("login: no login form detected (" + e.message.split("\n")[0] + ")\n"); + } + await ctx.storageState({ path: storagePath }); + await b.close(); +})().catch((e) => { process.stderr.write(String(e) + "\n"); process.exit(1); }); +` + +// LoginOptions configures LoginAndSaveStorage. +type LoginOptions struct { + AppURL string + Username string + Password string + StoragePath string // where to write the Playwright storage state JSON + MxBuildPath string // fallback node source + Timeout time.Duration // default 60s +} + +// LoginAndSaveStorage logs into the app and writes a Playwright storage-state +// file to opts.StoragePath. That file is then passed to CaptureScreenshot via +// StoragePath (screenshot --load-storage) to shoot pages behind login. +func LoginAndSaveStorage(opts LoginOptions) error { + if opts.StoragePath == "" { + return fmt.Errorf("storage path is required") + } + pkgDir := resolvePlaywrightPkgDir() + if pkgDir == "" { + return fmt.Errorf("playwright package not found (install with: npm i -g playwright)") + } + node := resolveNodeForScript(opts.MxBuildPath) + if node == "" { + return fmt.Errorf("node not found to run the login script") + } + if err := os.MkdirAll(filepath.Dir(opts.StoragePath), 0o755); err != nil { + return err + } + + // .cjs so Node treats it as CommonJS (the script uses require()). + scriptPath := filepath.Join(filepath.Dir(opts.StoragePath), ".mxcli-login.cjs") + if err := os.WriteFile(scriptPath, []byte(loginScript), 0o644); err != nil { + return fmt.Errorf("writing login script: %w", err) + } + defer os.Remove(scriptPath) + + timeout := opts.Timeout + if timeout == 0 { + timeout = 60 * time.Second + } + cmd := exec.Command(node, scriptPath, pkgDir, opts.AppURL, opts.Username, opts.Password, opts.StoragePath) + cmd.Env = os.Environ() // PLAYWRIGHT_BROWSERS_PATH resolves Chromium + out := &syncBuffer{} + cmd.Stdout = out + cmd.Stderr = out + + done := make(chan error, 1) + if err := cmd.Start(); err != nil { + return fmt.Errorf("launching login script: %w", err) + } + go func() { done <- cmd.Wait() }() + select { + case err := <-done: + if err != nil { + return fmt.Errorf("login failed: %w\n%s", err, out.String()) + } + case <-time.After(timeout): + _ = cmd.Process.Kill() + <-done + return fmt.Errorf("login timed out after %s", timeout) + } + if _, err := os.Stat(opts.StoragePath); err != nil { + return fmt.Errorf("login reported success but %s is missing:\n%s", opts.StoragePath, out.String()) + } + return nil +} diff --git a/cmd/mxcli/docker/screenshot_test.go b/cmd/mxcli/docker/screenshot_test.go index 4c61b3b9a..92ec24ddf 100644 --- a/cmd/mxcli/docker/screenshot_test.go +++ b/cmd/mxcli/docker/screenshot_test.go @@ -49,3 +49,11 @@ func TestCaptureScreenshot_NoOutPath(t *testing.T) { t.Error("expected error when OutPath is empty") } } + +func TestScreenshotArgs_LoadStorage(t *testing.T) { + args := screenshotArgs(ScreenshotOptions{URL: "u", OutPath: "o", Storage: "/s.json"}) + joined := strings.Join(args, " ") + if !strings.Contains(joined, "--load-storage /s.json") { + t.Errorf("expected --load-storage: %v", args) + } +} diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index ec8b64bb8..6603c36c3 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -68,7 +68,8 @@ createdb -h 127.0.0.1 -U mendix app1112 | `--db-user` / `--db-password` | mendix / mendix | Database credentials | | `--screenshot` | off | Capture a Playwright PNG after boot and each applied change | | `--screenshot-path` | `/.mxcli/run-local.png` | Screenshot output PNG | -| `--screenshot-url` | app root | Page URL to screenshot | +| `--screenshot-url` | app root | Page to shoot: full URL, or a path relative to the app root (e.g. `/p/customers`) | +| `--screenshot-user` / `--screenshot-password` | — | Log in once (Mendix form auth) and reuse the session, so pages behind login render authenticated | ## The change signal @@ -105,8 +106,21 @@ mxcli run --local -p app.mpr --watch --screenshot # edit a page with mxcli exec/MDL -> auto rebuild -> re-bundle -> reload -> new PNG ``` -Point `--screenshot-url` at a deep link to shoot a specific page. Screenshotting a -failure or a specific state (auth, navigation) is a natural next step. +**Deep links.** `--screenshot-url /p/customers` shoots a specific page instead of the +app root (a bare path is resolved against the app URL; a full `http(s)://…` is used +as-is). + +**Pages behind login.** `--screenshot-user`/`--screenshot-password` log in once via +the Mendix login form (Playwright drives `#usernameInput`/`#passwordInput`/ +`#loginButton`), save the session as a Playwright storage state, and reuse it for +every screenshot — so authenticated pages render. Login is best-effort: if no login +form appears (anonymous app) it proceeds unauthenticated. + +```bash +mxcli run --local -p app.mpr --watch --screenshot \ + --screenshot-user demo_admin --screenshot-password '' \ + --screenshot-url /p/customer_overview +``` See also: [PROPOSAL_mxcli_dev_warm_loop](../../../docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md), [mxcli docker run](docker-run.md), [Playwright Testing](playwright.md). diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 148387c06..19a81aef2 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -195,8 +195,18 @@ Two findings from implementation: `--screenshot` captures a PNG after boot and each applied change via Playwright's built-in `screenshot` command (Chromium from `PLAYWRIGHT_BROWSERS_PATH`; no -`playwright-cli` dependency), completing the pixel-perfect page loop. Deep-link / -authenticated-state screenshots are the natural next step. +`playwright-cli` dependency), completing the pixel-perfect page loop. + +**Deep-link + authenticated screenshots — done.** `--screenshot-url` accepts a page +path (resolved against the app root) or a full URL, so a specific page under edit is +shot rather than the app root. `--screenshot-user`/`--screenshot-password` log in +once through the Mendix login form (a headless Playwright script driving +`#usernameInput`/`#passwordInput`/`#loginButton`, run via the same Playwright install +resolved from the CLI's package dir — no hardcoded paths), save the session as a +Playwright storage state, and reuse it for every screenshot via +`screenshot --load-storage`. Login is best-effort: an anonymous app with no form +proceeds unauthenticated. Verified E2E: the saved storage state carries the Mendix +session cookies and the screenshot renders the authenticated page. ## Proposed CLI From 9461e5adeb3f6dcfe59f9be980fcc31dd5159c0e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 15:27:01 +0000 Subject: [PATCH 20/26] feat(run-local): multi-page screenshot sets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --screenshot-url is now repeatable: more than one target produces a screenshot set — one PNG per page after every applied change — for a visual-regression sheet across an app's key pages. Each PNG is named from the page slug (run-local-p-customers.png, run-local-home.png; the app root -> -home). ScreenshotURLs replaces the single ScreenshotURL; maybeScreenshot loops the targets, deriving a distinct output name per page (screenshotOutName/slugifyPage) when there is more than one, and keeps the single-file default otherwise. Auth (storage state) and deep-link resolution apply to every page in the set. Verified E2E: `--screenshot-url / --screenshot-url /index.html` produced two distinct PNGs in one change cycle. Docs (site, skill, CLAUDE.md, proposal) updated. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/run-local.md | 3 +- CLAUDE.md | 2 +- cmd/mxcli/cmd_run.go | 6 +- cmd/mxcli/docker/runlocal.go | 93 ++++++++++++++++--- cmd/mxcli/docker/runlocal_test.go | 39 ++++++++ docs-site/src/tools/run-local.md | 11 ++- .../PROPOSAL_mxcli_dev_warm_loop.md | 6 ++ 7 files changed, 139 insertions(+), 21 deletions(-) diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index 21fc778b1..4d284a3d8 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -106,7 +106,8 @@ mxcli run --local -p app.mpr --watch --screenshot ``` - `--screenshot-url /p/customers` shoots a specific page (bare path resolved against - the app root; a full URL is used as-is). + the app root; a full URL is used as-is). Repeat it for a multi-page set — each page + gets its own PNG (`run-local-p-customers.png`, `run-local-home.png`). - `--screenshot-user`/`--screenshot-password` log in once (Mendix form auth) and reuse the session, so pages behind login render authenticated. Best-effort: an anonymous app with no login form proceeds unauthenticated. diff --git a/CLAUDE.md b/CLAUDE.md index 386a2a226..dcaf6cf2c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -523,7 +523,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) -- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` +- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - OQL query execution against running runtime (`mxcli oql`) - Business event services (SHOW/DESCRIBE/CREATE/DROP) - Project settings (SHOW/DESCRIBE/ALTER) diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index 482cd1ebd..ecfe987d5 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -59,7 +59,7 @@ Examples: dbPassword, _ := cmd.Flags().GetString("db-password") screenshot, _ := cmd.Flags().GetBool("screenshot") screenshotPath, _ := cmd.Flags().GetString("screenshot-path") - screenshotURL, _ := cmd.Flags().GetString("screenshot-url") + screenshotURLs, _ := cmd.Flags().GetStringArray("screenshot-url") screenshotUser, _ := cmd.Flags().GetString("screenshot-user") screenshotPassword, _ := cmd.Flags().GetString("screenshot-password") @@ -71,7 +71,7 @@ Examples: Watch: watch, Screenshot: screenshot, ScreenshotPath: screenshotPath, - ScreenshotURL: screenshotURL, + ScreenshotURLs: screenshotURLs, ScreenshotUser: screenshotUser, ScreenshotPassword: screenshotPassword, DB: docker.DBConfig{ @@ -103,7 +103,7 @@ func init() { runCmd.Flags().String("db-password", "", "Database password (default mendix)") runCmd.Flags().Bool("screenshot", false, "Capture a Playwright screenshot after boot and each applied change") runCmd.Flags().String("screenshot-path", "", "Screenshot output PNG (default /.mxcli/run-local.png)") - runCmd.Flags().String("screenshot-url", "", "Page to screenshot: a full URL or a path relative to the app root, e.g. /p/customers (default the app root)") + runCmd.Flags().StringArray("screenshot-url", nil, "Page to screenshot: a full URL or a path relative to the app root, e.g. /p/customers (default the app root). Repeat for a multi-page set.") runCmd.Flags().String("screenshot-user", "", "Log in with this user before screenshotting (for pages behind login)") runCmd.Flags().String("screenshot-password", "", "Password for --screenshot-user") rootCmd.AddCommand(runCmd) diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index c55005c06..fbf721716 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -48,10 +48,13 @@ type LocalRunOptions struct { // applied change (requires the Playwright CLI + a browser). Screenshot bool // ScreenshotPath is where the PNG is written (default /.mxcli/run-local.png). + // With multiple ScreenshotURLs, it is the base name and each page gets a + // per-page suffix (run-local-.png). ScreenshotPath string - // ScreenshotURL overrides the page shot: a full http(s) URL, or a path - // relative to the app root (e.g. "/p/customers"). Default the app root. - ScreenshotURL string + // ScreenshotURLs are the pages to shoot each change: full http(s) URLs, or + // paths relative to the app root (e.g. "/p/customers"). Empty -> the app root. + // More than one produces a screenshot set (one PNG per page). + ScreenshotURLs []string // ScreenshotUser/Password log in once (Mendix form auth) and reuse the session // for every screenshot, so pages behind login render authenticated. ScreenshotUser string @@ -361,21 +364,33 @@ func maybeScreenshot(opts LocalRunOptions, rt *LocalRuntime) { fmt.Fprintf(opts.Stderr, " screenshot skipped: %v\n", err) return } - if err := CaptureScreenshot(ScreenshotOptions{ - URL: resolveScreenshotURL(rt.AppURL(), opts.ScreenshotURL), - OutPath: opts.ScreenshotPath, - WaitMs: 4000, - FullPage: true, - Viewport: "1280,800", - Storage: opts.screenshotStorage, - }); err != nil { - fmt.Fprintf(opts.Stderr, " screenshot skipped: %v\n", err) - return + // Empty list -> the app root; multiple pages -> a screenshot set (one PNG each). + targets := opts.ScreenshotURLs + if len(targets) == 0 { + targets = []string{""} + } + multi := len(targets) > 1 + for _, t := range targets { + out := opts.ScreenshotPath + if multi { + out = screenshotOutName(opts.ScreenshotPath, t) + } + if err := CaptureScreenshot(ScreenshotOptions{ + URL: resolveScreenshotURL(rt.AppURL(), t), + OutPath: out, + WaitMs: 4000, + FullPage: true, + Viewport: "1280,800", + Storage: opts.screenshotStorage, + }); err != nil { + fmt.Fprintf(opts.Stderr, " screenshot skipped (%s): %v\n", pageLabel(t), err) + continue + } + fmt.Fprintf(opts.Stdout, " screenshot -> %s\n", out) } - fmt.Fprintf(opts.Stdout, " screenshot -> %s\n", opts.ScreenshotPath) } -// resolveScreenshotURL resolves the --screenshot-url value against the app root: +// resolveScreenshotURL resolves a --screenshot-url value against the app root: // empty -> the root; a full http(s) URL -> as-is; anything else -> treated as a // path relative to the app root (so "/p/customers" deep-links a specific page). func resolveScreenshotURL(appURL, urlOrPath string) string { @@ -388,6 +403,54 @@ func resolveScreenshotURL(appURL, urlOrPath string) string { return strings.TrimRight(appURL, "/") + "/" + strings.TrimLeft(urlOrPath, "/") } +// pageLabel is a short human label for a screenshot target ("home" for the root). +func pageLabel(urlOrPath string) string { + if urlOrPath == "" { + return "home" + } + return urlOrPath +} + +// screenshotOutName derives a per-page output path from the base path and a +// target, so a screenshot set writes distinct files: run-local.png + +// "/p/customers" -> run-local-p-customers.png; the app root -> run-local-home.png. +func screenshotOutName(basePath, urlOrPath string) string { + ext := filepath.Ext(basePath) // ".png" + stem := strings.TrimSuffix(basePath, ext) // ".../run-local" + slug := slugifyPage(urlOrPath) + return stem + "-" + slug + ext +} + +// slugifyPage turns a page URL/path into a filesystem-safe slug. +func slugifyPage(urlOrPath string) string { + s := urlOrPath + // Drop scheme://host for full URLs, keep the path. + if i := strings.Index(s, "://"); i >= 0 { + if j := strings.IndexByte(s[i+3:], '/'); j >= 0 { + s = s[i+3+j:] + } else { + s = "" + } + } + var b strings.Builder + for _, r := range strings.ToLower(s) { + if (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') { + b.WriteRune(r) + } else { + b.WriteByte('-') + } + } + slug := strings.Trim(b.String(), "-") + // collapse runs of '-' + for strings.Contains(slug, "--") { + slug = strings.ReplaceAll(slug, "--", "-") + } + if slug == "" { + return "home" + } + return slug +} + // resolveRuntimeInstall returns the directory to use as MX_INSTALL_PATH (its // runtime/ child holds the launcher and Mendix libraries), downloading the // runtime only when nothing usable is cached. It also makes the runtime visible diff --git a/cmd/mxcli/docker/runlocal_test.go b/cmd/mxcli/docker/runlocal_test.go index 2ad002728..3d7035cf6 100644 --- a/cmd/mxcli/docker/runlocal_test.go +++ b/cmd/mxcli/docker/runlocal_test.go @@ -203,3 +203,42 @@ func TestResolveScreenshotURL(t *testing.T) { } } } + +func TestSlugifyPage(t *testing.T) { + cases := map[string]string{ + "": "home", + "/": "home", + "/p/customers": "p-customers", + "p/Customer_Overview": "p-customer-overview", + "http://127.0.0.1:8080/": "home", + "http://h:8080/p/orders": "p-orders", + "/p/a//b": "p-a-b", + } + for in, want := range cases { + if got := slugifyPage(in); got != want { + t.Errorf("slugifyPage(%q) = %q, want %q", in, got, want) + } + } +} + +func TestScreenshotOutName(t *testing.T) { + base := filepath.FromSlash("/x/.mxcli/run-local.png") + cases := map[string]string{ + "/p/customers": filepath.FromSlash("/x/.mxcli/run-local-p-customers.png"), + "": filepath.FromSlash("/x/.mxcli/run-local-home.png"), + } + for in, want := range cases { + if got := screenshotOutName(base, in); got != want { + t.Errorf("screenshotOutName(%q) = %q, want %q", in, got, want) + } + } +} + +func TestPageLabel(t *testing.T) { + if pageLabel("") != "home" { + t.Errorf("pageLabel(\"\") = %q, want home", pageLabel("")) + } + if pageLabel("/p/x") != "/p/x" { + t.Errorf("pageLabel(/p/x) = %q", pageLabel("/p/x")) + } +} diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index 6603c36c3..942464368 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -68,7 +68,7 @@ createdb -h 127.0.0.1 -U mendix app1112 | `--db-user` / `--db-password` | mendix / mendix | Database credentials | | `--screenshot` | off | Capture a Playwright PNG after boot and each applied change | | `--screenshot-path` | `/.mxcli/run-local.png` | Screenshot output PNG | -| `--screenshot-url` | app root | Page to shoot: full URL, or a path relative to the app root (e.g. `/p/customers`) | +| `--screenshot-url` | app root | Page to shoot: full URL, or a path relative to the app root (e.g. `/p/customers`). Repeat for a multi-page set. | | `--screenshot-user` / `--screenshot-password` | — | Log in once (Mendix form auth) and reuse the session, so pages behind login render authenticated | ## The change signal @@ -110,6 +110,15 @@ mxcli run --local -p app.mpr --watch --screenshot app root (a bare path is resolved against the app URL; a full `http(s)://…` is used as-is). +**Multi-page sets.** Repeat `--screenshot-url` to shoot several pages after every +change — a visual-regression sheet. Each page gets its own PNG, named from the page +(`run-local-p-customers.png`, `run-local-home.png`): + +```bash +mxcli run --local -p app.mpr --watch --screenshot \ + --screenshot-url / --screenshot-url /p/customers --screenshot-url /p/orders +``` + **Pages behind login.** `--screenshot-user`/`--screenshot-password` log in once via the Mendix login form (Playwright drives `#usernameInput`/`#passwordInput`/ `#loginButton`), save the session as a Playwright storage state, and reuse it for diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 19a81aef2..1054c42cf 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -208,6 +208,12 @@ Playwright storage state, and reuse it for every screenshot via proceeds unauthenticated. Verified E2E: the saved storage state carries the Mendix session cookies and the screenshot renders the authenticated page. +**Multi-page screenshot sets — done.** `--screenshot-url` is repeatable; more than one +target produces a screenshot *set* — one PNG per page, named from the page slug +(`run-local-p-customers.png`, `run-local-home.png`) — so every change yields a +visual-regression sheet across the key pages. Verified E2E: two targets produced two +distinct PNGs in one change cycle. + ## Proposed CLI ### Scenario A: `mxcli dev` — Docker-free warm run loop From 5de6b5422c64ff942281a654f3adea8b11bc89f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 16:08:07 +0000 Subject: [PATCH 21/26] feat(run-local): --ensure-db provisions the local database (slice 2) Starts slice 2 (provisioning) of the warm-loop proposal: a fresh session should come up testable without a manual createdb. run --local previously only checked DB reachability and errored; --ensure-db now provisions it. docker.EnsureDatabase (PostgreSQL): no-op if the app can already connect; otherwise, for a local host, starts the Postgres service if the port is down, then creates the app role and database if missing via a local superuser (sudo -u postgres psql). Remote hosts are only verified, never provisioned. Identifiers are validated against a strict pg-identifier regex and the role password is single-quote-escaped before any DDL, so names/passwords can't inject. Also documents the slice-1 command-name drift in the proposal: the warm loop shipped as `mxcli run --local [--watch]`, not the envisioned `mxcli dev` tree (reload/exec/status/stop folded into --watch), and marks slice 1 shipped + slice 2's DB-provisioning piece done in the slices table. Verified E2E: with Postgres stopped and the database dropped, `run --local --ensure-db` started Postgres, created the database, booted the runtime, and the runtime synced 88 tables into the fresh DB. Pure helpers (splitHostPort, isLocalHost, quoteSQLString, identifier validation) are unit-tested. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/run-local.md | 18 +- CLAUDE.md | 2 +- cmd/mxcli/cmd_run.go | 3 + cmd/mxcli/docker/ensuredb.go | 210 ++++++++++++++++++ cmd/mxcli/docker/ensuredb_test.go | 58 +++++ cmd/mxcli/docker/runlocal.go | 16 +- docs-site/src/tools/run-local.md | 22 +- .../PROPOSAL_mxcli_dev_warm_loop.md | 12 +- 8 files changed, 318 insertions(+), 23 deletions(-) create mode 100644 cmd/mxcli/docker/ensuredb.go create mode 100644 cmd/mxcli/docker/ensuredb_test.go diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index 4d284a3d8..275e66177 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -47,15 +47,16 @@ association catalog only at startup; behavioural changes are hot-reloaded. ## Prerequisites - **Mendix 11.x** project (runtime launches under **JDK 21**). -- A reachable **PostgreSQL** with the database already created. In the devcontainer: +- A **PostgreSQL** database (defaults: `127.0.0.1:5432`, user `mendix`, db derived + from the project name; override with `--db-host/--db-name/--db-user/--db-password`). + - **`--ensure-db`** provisions it for a fresh session: starts local Postgres if the + port is down and creates the role + database if missing (local superuser via + `sudo -u postgres`). Remote hosts are only checked, not provisioned. + - Without `--ensure-db`, create it once and the command errors if it's unreachable: - ```bash - createdb -h 127.0.0.1 -U mendix "$(basename app.mpr .mpr | tr '[:upper:]' '[:lower:]')" - ``` - - Defaults: `127.0.0.1:5432`, user `mendix`, db derived from the project name. - Override with `--db-host/--db-name/--db-user/--db-password`. If the database is - unreachable the command stops with an actionable message (it does not provision it). + ```bash + createdb -h 127.0.0.1 -U mendix "$(basename app.mpr .mpr | tr '[:upper:]' '[:lower:]')" + ``` ## The intended loop @@ -76,6 +77,7 @@ mxcli exec add-page.mdl -p app.mpr |------|---------|---------| | `--local` | — | Required; run without Docker | | `--watch` | off | Rebuild + hot-apply on each change | +| `--ensure-db` | off | Provision local Postgres + app database if missing | | `--screenshot` | off | Playwright PNG after boot + each change | | `--screenshot-path` / `--screenshot-url` | `.mxcli/run-local.png` / app root | Screenshot output / page (URL or `/path`) | | `--screenshot-user` / `--screenshot-password` | — | Log in once, reuse session (pages behind login) | diff --git a/CLAUDE.md b/CLAUDE.md index dcaf6cf2c..0c8ab939b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -523,7 +523,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) -- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` +- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing (fresh-session bootstrap). `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - OQL query execution against running runtime (`mxcli oql`) - Business event services (SHOW/DESCRIBE/CREATE/DROP) - Project settings (SHOW/DESCRIBE/ALTER) diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index ecfe987d5..ddd9558f3 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -50,6 +50,7 @@ Examples: } watch, _ := cmd.Flags().GetBool("watch") + ensureDB, _ := cmd.Flags().GetBool("ensure-db") appPort, _ := cmd.Flags().GetInt("app-port") adminPort, _ := cmd.Flags().GetInt("admin-port") servePort, _ := cmd.Flags().GetInt("serve-port") @@ -69,6 +70,7 @@ Examples: AdminPort: adminPort, ServePort: servePort, Watch: watch, + EnsureDB: ensureDB, Screenshot: screenshot, ScreenshotPath: screenshotPath, ScreenshotURLs: screenshotURLs, @@ -94,6 +96,7 @@ Examples: func init() { runCmd.Flags().Bool("local", false, "Run locally without Docker (warm serve + standalone runtime)") runCmd.Flags().Bool("watch", false, "Rebuild and hot-apply on every project change") + runCmd.Flags().Bool("ensure-db", false, "Provision the local Postgres + app database if missing (fresh-session bootstrap)") runCmd.Flags().Int("app-port", 0, "HTTP port for the app (default 8080)") runCmd.Flags().Int("admin-port", 0, "M2EE admin API port (default 8090)") runCmd.Flags().Int("serve-port", 0, "mxbuild --serve port (default 6543)") diff --git a/cmd/mxcli/docker/ensuredb.go b/cmd/mxcli/docker/ensuredb.go new file mode 100644 index 000000000..6869bfa34 --- /dev/null +++ b/cmd/mxcli/docker/ensuredb.go @@ -0,0 +1,210 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "io" + "os" + "os/exec" + "regexp" + "strings" + "time" +) + +// ensuredb.go provisions the local PostgreSQL a standalone runtime needs, so a +// fresh session comes up testable without a manual createdb (slice 2 of the +// warm-loop proposal). It is best-effort and devcontainer-shaped: it starts the +// local Postgres service if the port is down, then ensures the app role and +// database exist via a superuser (`sudo -u postgres psql`). For a non-local DB +// host it does nothing but verify reachability — provisioning a remote database +// is not mxcli's business. + +// pgIdent is a conservative PostgreSQL identifier (unquoted): a safe database or +// role name. We refuse anything else rather than quote/escape it into DDL. +var pgIdent = regexp.MustCompile(`^[a-z_][a-z0-9_]*$`) + +// splitHostPort splits "host:port" into host and port, defaulting the port to +// 5432 when absent. +func splitHostPort(hostPort string) (host, port string) { + if i := strings.LastIndexByte(hostPort, ':'); i >= 0 { + return hostPort[:i], hostPort[i+1:] + } + return hostPort, "5432" +} + +// isLocalHost reports whether host refers to the local machine (so it is safe to +// start a service / use a local superuser). +func isLocalHost(host string) bool { + switch host { + case "127.0.0.1", "localhost", "::1", "": + return true + } + return false +} + +// quoteSQLString wraps s in single quotes for use as a SQL string literal, +// doubling any embedded single quotes (used only for the role password, which +// cannot be a bind parameter in CREATE ROLE). +func quoteSQLString(s string) string { + return "'" + strings.ReplaceAll(s, "'", "''") + "'" +} + +// EnsureDatabase makes db reachable and the app role + database present. It is a +// no-op when the app can already connect. For a local host whose port is down it +// starts Postgres, then creates the role and database if missing. +func EnsureDatabase(db DBConfig, w io.Writer) error { + if strings.ToLower(db.Type) != "postgresql" { + return fmt.Errorf("--ensure-db only supports PostgreSQL (DatabaseType=%q)", db.Type) + } + if !pgIdent.MatchString(db.Name) { + return fmt.Errorf("unsafe database name %q (expected %s)", db.Name, pgIdent) + } + if !pgIdent.MatchString(db.User) { + return fmt.Errorf("unsafe database user %q (expected %s)", db.User, pgIdent) + } + host, port := splitHostPort(db.Host) + + // Already usable? Then we're done. + if canConnectDB(db) { + return nil + } + + // Port down: start the local Postgres service (best-effort) or bail for remote. + if err := pingTCP(db.Host, 2*time.Second); err != nil { + if !isLocalHost(host) { + return fmt.Errorf("database not reachable at %s and host is not local; "+ + "start it and create the %q database (user %q)", db.Host, db.Name, db.User) + } + fmt.Fprintln(w, " Starting local PostgreSQL...") + if err := startLocalPostgres(); err != nil { + return fmt.Errorf("starting local PostgreSQL: %w", err) + } + if err := waitPGReady(host, port, 20*time.Second); err != nil { + return err + } + } + + // Ensure the role and database exist (needs a local superuser). + if err := ensureRole(db, w); err != nil { + return err + } + if err := ensureDatabase(db, w); err != nil { + return err + } + + if !canConnectDB(db) { + return fmt.Errorf("provisioned PostgreSQL but still cannot connect to %q as %q at %s", + db.Name, db.User, db.Host) + } + fmt.Fprintf(w, " Database ready: %s (user %q) at %s\n", db.Name, db.User, db.Host) + return nil +} + +// canConnectDB reports whether the app can connect to its database as its user. +func canConnectDB(db DBConfig) bool { + host, port := splitHostPort(db.Host) + cmd := exec.Command("psql", "-h", host, "-p", port, "-U", db.User, "-d", db.Name, + "-tAc", "select 1") + cmd.Env = append(os.Environ(), "PGPASSWORD="+db.Password, "PGCONNECT_TIMEOUT=3") + return cmd.Run() == nil +} + +// startLocalPostgres starts the local PostgreSQL service, trying the common +// service managers in turn. Success is confirmed later by waitPGReady. +func startLocalPostgres() error { + attempts := [][]string{ + {"service", "postgresql", "start"}, + {"pg_ctlcluster", "--", "start"}, // placeholder; real cluster args vary + } + var lastErr error + for _, a := range attempts { + if _, err := exec.LookPath(a[0]); err != nil { + lastErr = err + continue + } + cmd := exec.Command(a[0], a[1:]...) + if err := cmd.Run(); err == nil { + return nil + } else { + lastErr = err + } + // `service postgresql start` is the reliable path in the devcontainer; if + // it ran (even non-zero) Postgres may still be coming up — let waitPGReady + // decide rather than failing here. + return nil + } + if lastErr != nil { + return lastErr + } + return fmt.Errorf("no known service manager found to start PostgreSQL") +} + +// waitPGReady polls pg_isready until the server accepts connections or timeout. +func waitPGReady(host, port string, timeout time.Duration) error { + if _, err := exec.LookPath("pg_isready"); err != nil { + // Fall back to a raw TCP check. + return waitTCP(host+":"+port, timeout) + } + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + cmd := exec.Command("pg_isready", "-h", host, "-p", port) + if cmd.Run() == nil { + return nil + } + time.Sleep(500 * time.Millisecond) + } + return fmt.Errorf("PostgreSQL did not become ready at %s:%s within %s", host, port, timeout) +} + +// waitTCP polls a host:port until it accepts a connection or timeout. +func waitTCP(hostPort string, timeout time.Duration) error { + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + if pingTCP(hostPort, time.Second) == nil { + return nil + } + time.Sleep(500 * time.Millisecond) + } + return fmt.Errorf("%s did not accept connections within %s", hostPort, timeout) +} + +// superuserPSQL runs a psql command as the postgres superuser (sudo -u postgres). +func superuserPSQL(args ...string) *exec.Cmd { + full := append([]string{"-u", "postgres", "psql", "-v", "ON_ERROR_STOP=1"}, args...) + return exec.Command("sudo", full...) +} + +// ensureRole creates the app login role if it does not already exist. +func ensureRole(db DBConfig, w io.Writer) error { + check := superuserPSQL("-tAc", fmt.Sprintf("select 1 from pg_roles where rolname='%s'", db.User)) + out, _ := check.Output() + if strings.TrimSpace(string(out)) == "1" { + return nil + } + fmt.Fprintf(w, " Creating role %q...\n", db.User) + ddl := fmt.Sprintf("CREATE ROLE %s WITH LOGIN PASSWORD %s CREATEDB", db.User, quoteSQLString(db.Password)) + cmd := superuserPSQL("-c", ddl) + if out, err := cmd.CombinedOutput(); err != nil { + return fmt.Errorf("creating role %q: %w\n%s\n"+ + " (need a local postgres superuser via 'sudo -u postgres'; create the role manually if unavailable)", + db.User, err, strings.TrimSpace(string(out))) + } + return nil +} + +// ensureDatabase creates the app database owned by the app role if it is absent. +func ensureDatabase(db DBConfig, w io.Writer) error { + check := superuserPSQL("-tAc", fmt.Sprintf("select 1 from pg_database where datname='%s'", db.Name)) + out, _ := check.Output() + if strings.TrimSpace(string(out)) == "1" { + return nil + } + fmt.Fprintf(w, " Creating database %q owned by %q...\n", db.Name, db.User) + ddl := fmt.Sprintf("CREATE DATABASE %s OWNER %s", db.Name, db.User) + cmd := superuserPSQL("-c", ddl) + if out, err := cmd.CombinedOutput(); err != nil { + return fmt.Errorf("creating database %q: %w\n%s", db.Name, err, strings.TrimSpace(string(out))) + } + return nil +} diff --git a/cmd/mxcli/docker/ensuredb_test.go b/cmd/mxcli/docker/ensuredb_test.go new file mode 100644 index 000000000..5cd0ab470 --- /dev/null +++ b/cmd/mxcli/docker/ensuredb_test.go @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "io" + "testing" +) + +func TestSplitHostPort(t *testing.T) { + cases := []struct{ in, host, port string }{ + {"127.0.0.1:5432", "127.0.0.1", "5432"}, + {"db.example.com:6000", "db.example.com", "6000"}, + {"localhost", "localhost", "5432"}, + } + for _, c := range cases { + h, p := splitHostPort(c.in) + if h != c.host || p != c.port { + t.Errorf("splitHostPort(%q) = (%q,%q), want (%q,%q)", c.in, h, p, c.host, c.port) + } + } +} + +func TestIsLocalHost(t *testing.T) { + for _, h := range []string{"127.0.0.1", "localhost", "::1", ""} { + if !isLocalHost(h) { + t.Errorf("isLocalHost(%q) = false, want true", h) + } + } + for _, h := range []string{"db.example.com", "10.0.0.5", "postgres"} { + if isLocalHost(h) { + t.Errorf("isLocalHost(%q) = true, want false", h) + } + } +} + +func TestQuoteSQLString(t *testing.T) { + if got := quoteSQLString("mendix"); got != "'mendix'" { + t.Errorf("quoteSQLString = %q", got) + } + if got := quoteSQLString("it's"); got != "'it''s'" { + t.Errorf("quoteSQLString(apostrophe) = %q, want doubled", got) + } +} + +func TestEnsureDatabase_Validation(t *testing.T) { + // Non-Postgres type is rejected. + if err := EnsureDatabase(DBConfig{Type: "HSQLDB", Name: "a", User: "u"}, io.Discard); err == nil { + t.Error("expected error for non-PostgreSQL type") + } + // Unsafe identifiers are rejected (before any exec). + if err := EnsureDatabase(DBConfig{Type: "PostgreSQL", Name: "bad-name;DROP", User: "u", Host: "127.0.0.1:5432"}, io.Discard); err == nil { + t.Error("expected error for unsafe database name") + } + if err := EnsureDatabase(DBConfig{Type: "PostgreSQL", Name: "ok", User: "bad user", Host: "127.0.0.1:5432"}, io.Discard); err == nil { + t.Error("expected error for unsafe database user") + } +} diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index fbf721716..acd7e5d96 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -42,6 +42,9 @@ type LocalRunOptions struct { DB DBConfig // Watch keeps running, rebuilding+applying on every project change. Watch bool + // EnsureDB provisions the local Postgres + app database if missing (otherwise + // the DB must already exist). Intended for a fresh session / SessionStart boot. + EnsureDB bool // PollInterval is how often Watch checks for changes (default 1s). PollInterval time.Duration // Screenshot, when set, captures a PNG of the app after boot and after each @@ -259,10 +262,17 @@ func RunLocal(opts LocalRunOptions) error { return fmt.Errorf("setting up runtime: %w", err) } - // 3. Check the database is reachable (we don't provision it here). - if err := pingTCP(opts.DB.Host, 3*time.Second); err != nil { + // 3. Ensure the database is available. With --ensure-db, provision it (start + // local Postgres + create the role/db if missing); otherwise just check + // reachability and point the user at --ensure-db. + if opts.EnsureDB { + fmt.Fprintln(w, "Ensuring database...") + if err := EnsureDatabase(opts.DB, w); err != nil { + return fmt.Errorf("ensuring database: %w", err) + } + } else if err := pingTCP(opts.DB.Host, 3*time.Second); err != nil { return fmt.Errorf("database not reachable at %s: %w\n"+ - " Start Postgres and create the '%s' database (user %q), or pass --db-* flags.", + " Pass --ensure-db to provision it, or start Postgres and create the '%s' database (user %q).", opts.DB.Host, err, opts.DB.Name, opts.DB.User) } diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index 942464368..0502db9b7 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -44,15 +44,18 @@ so structural changes need a restart; behavioural changes do not. - **Mendix 11.x** project. The runtime is launched under **JDK 21**; version-aware JDK selection for Mendix 9/10 is a follow-up. -- A reachable **PostgreSQL**, with the database already created. The devcontainer - provides one. Defaults: `127.0.0.1:5432`, user `mendix`, database derived from the - project file name (`App1112.mpr` → `app1112`). If the DB is unreachable, `run - --local` stops with an actionable message rather than booting. - -```bash -# devcontainer Postgres, one-time DB creation -createdb -h 127.0.0.1 -U mendix app1112 -``` +- A **PostgreSQL** database. Defaults: `127.0.0.1:5432`, user `mendix`, database + derived from the project file name (`App1112.mpr` → `app1112`). Two ways to have it: + - **`--ensure-db`** (recommended for a fresh session) provisions it: starts the + local Postgres service if the port is down, and creates the app role + database + if missing (via a local `sudo -u postgres` superuser). For a non-local `--db-host` + it only verifies reachability — mxcli won't provision a remote database. + - Otherwise create it once yourself; without `--ensure-db`, `run --local` stops with + an actionable message if the DB is unreachable: + + ```bash + createdb -h 127.0.0.1 -U mendix app1112 + ``` ## Flags @@ -60,6 +63,7 @@ createdb -h 127.0.0.1 -U mendix app1112 |------|---------|---------| | `--local` | — | Required; run without Docker | | `--watch` | off | Rebuild + hot-apply on every project change | +| `--ensure-db` | off | Provision local Postgres + the app database if missing (fresh-session bootstrap) | | `--app-port` | 8080 | App HTTP port | | `--admin-port` | 8090 | M2EE admin API port | | `--serve-port` | 6543 | `mxbuild --serve` port | diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 1054c42cf..8d113129b 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -216,6 +216,14 @@ distinct PNGs in one change cycle. ## Proposed CLI +> **Shipped name (slice 1):** the warm loop shipped as **`mxcli run --local`**, not +> `mxcli dev`. The separate `dev reload`/`dev exec`/`dev status`/`dev stop` +> subcommands envisioned below were folded into a single long-lived +> `mxcli run --local --watch` (it watches the model source and hot-applies each +> change itself), so no `reload`/`exec` subcommands were needed. The `mxcli dev …` +> names in the rest of this proposal refer to that command (and the not-yet-built +> `dev up`/`dev serve` bootstrap + preview pieces of slices 2–3). + ### Scenario A: `mxcli dev` — Docker-free warm run loop A long-lived local dev supervisor. Boots (or reuses) the standalone runtime + @@ -519,8 +527,8 @@ builds on the previous. | # | Slice | Delivers | Depends on | Size / risk | |---|-------|----------|------------|-------------| -| 1 | **Warm local loop** — `mxcli dev` (serve daemon + M2EE admin client + `restartRequired` × `get_ddl_commands` branching) | Docker-free ~1 s edit→test loop, locally | nothing new | medium / low — highest bang-for-buck | -| 2 | **Provisioning** — `mxcli dev up` bootstrap + `mxcli init` SessionStart hook + prompt template | a fresh Claude Code Web session comes up testable; iPad-native start | slice 1 | small / low | +| 1 | **Warm local loop** — shipped as `mxcli run --local [--watch]` (serve daemon + M2EE admin client + `restartRequired` branching; + client bundling & Playwright screenshots) | Docker-free ~1 s edit→test loop, locally | nothing new | ✅ **shipped** | +| 2 | **Provisioning** — database provisioning (`run --local --ensure-db`, ✅ done) + `mxcli init` SessionStart hook + prompt template (remaining) | a fresh Claude Code Web session comes up testable; iPad-native start | slice 1 | small / low | | 3 | **Single-app external preview** — `mxcli dev serve` + chisel client → one static relay + `ApplicationRootUrl` wiring | a shareable live preview URL (the iPad two-container flow) | slice 1 | medium / medium — the `app.github.dev` WebSocket hop is unverified (a VPS relay avoids it) | | 4 | **Tunnel hub** — `mxcli tunnel-hub` + `mxcli dev --hub` + registration API + admin overview | many dev containers behind one ingress; fleet overview + per-container change lists | slice 3 | large / higher — a product in its own right, with a multi-tenant auth surface | From 1d3863ab76a0e2601bc2a6b9dd9792f5f4654a83 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 21:20:57 +0000 Subject: [PATCH 22/26] feat(run-local,init): SessionStart hook + bootstrap prompt (slice 2 complete) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Finishes slice 2 (provisioning): a fresh Claude Code Web session self-bootstraps, and an empty repo can be seeded from a prompt — no GitHub template needed. run --local --setup: a non-blocking bring-up. It does the idempotent prerequisites (detect version, cache MxBuild+runtime, ensure the database with --ensure-db) and RETURNS without booting serve/runtime — the long-lived loop would block a SessionStart hook. The agent runs the full `run --local` on demand. mxcli init: emits a Claude Code SessionStart hook into .claude/settings.json (ensureSessionStartHook / addSessionStartHook) running `test -x ./mxcli && ./mxcli run --local --setup --ensure-db -p || true`. The merge is generic over the JSON, so existing settings/hooks are preserved and re-running init is idempotent (marker match); invalid existing JSON is left untouched rather than clobbered. Verified E2E: the hook merged cleanly into the settings.json init already writes (env + permissions retained). Bootstrap prompt: docs-site/src/tools/bootstrap-prompt.md — the copy-paste seed for an empty repo (web/iPad), using the shipped commands, with the two robustness rules (commit the config; mxcli delivery is an environment concern). This is the primary iPad path, chosen over a GitHub template repo (mobile shows only a small template subset, and a template repo needs per-version upkeep). Docs (site page + SUMMARY, skill, CLAUDE.md, proposal) updated; slices 1 and 2 marked shipped. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/run-local.md | 1 + CLAUDE.md | 2 +- cmd/mxcli/cmd_run.go | 3 + cmd/mxcli/docker/runlocal.go | 13 +++ cmd/mxcli/init.go | 10 ++ cmd/mxcli/init_hook.go | 98 ++++++++++++++++ cmd/mxcli/init_hook_test.go | 106 ++++++++++++++++++ docs-site/src/SUMMARY.md | 1 + docs-site/src/tools/bootstrap-prompt.md | 57 ++++++++++ docs-site/src/tools/run-local.md | 12 ++ .../PROPOSAL_mxcli_dev_warm_loop.md | 30 +++-- 11 files changed, 321 insertions(+), 12 deletions(-) create mode 100644 cmd/mxcli/init_hook.go create mode 100644 cmd/mxcli/init_hook_test.go create mode 100644 docs-site/src/tools/bootstrap-prompt.md diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index 275e66177..dd90761f5 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -78,6 +78,7 @@ mxcli exec add-page.mdl -p app.mpr | `--local` | — | Required; run without Docker | | `--watch` | off | Rebuild + hot-apply on each change | | `--ensure-db` | off | Provision local Postgres + app database if missing | +| `--setup` | off | Cache MxBuild+runtime + ensure DB, then exit (SessionStart bring-up) | | `--screenshot` | off | Playwright PNG after boot + each change | | `--screenshot-path` / `--screenshot-url` | `.mxcli/run-local.png` / app root | Screenshot output / page (URL or `/path`) | | `--screenshot-user` / `--screenshot-password` | — | Log in once, reuse session (pages behind login) | diff --git a/CLAUDE.md b/CLAUDE.md index 0c8ab939b..d6f0eb6f9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -523,7 +523,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) -- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing (fresh-session bootstrap). `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` +- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing; `--setup` does the non-blocking prerequisites (cache mxbuild+runtime, ensure DB) and exits — `mxcli init` wires it into a Claude Code SessionStart hook so a fresh/reaped web session self-bootstraps, and `docs-site/src/tools/bootstrap-prompt.md` is the empty-repo seed prompt. `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - OQL query execution against running runtime (`mxcli oql`) - Business event services (SHOW/DESCRIBE/CREATE/DROP) - Project settings (SHOW/DESCRIBE/ALTER) diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index ddd9558f3..66ecafb15 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -51,6 +51,7 @@ Examples: watch, _ := cmd.Flags().GetBool("watch") ensureDB, _ := cmd.Flags().GetBool("ensure-db") + setupOnly, _ := cmd.Flags().GetBool("setup") appPort, _ := cmd.Flags().GetInt("app-port") adminPort, _ := cmd.Flags().GetInt("admin-port") servePort, _ := cmd.Flags().GetInt("serve-port") @@ -71,6 +72,7 @@ Examples: ServePort: servePort, Watch: watch, EnsureDB: ensureDB, + SetupOnly: setupOnly, Screenshot: screenshot, ScreenshotPath: screenshotPath, ScreenshotURLs: screenshotURLs, @@ -97,6 +99,7 @@ func init() { runCmd.Flags().Bool("local", false, "Run locally without Docker (warm serve + standalone runtime)") runCmd.Flags().Bool("watch", false, "Rebuild and hot-apply on every project change") runCmd.Flags().Bool("ensure-db", false, "Provision the local Postgres + app database if missing (fresh-session bootstrap)") + runCmd.Flags().Bool("setup", false, "Prepare prerequisites (cache MxBuild+runtime, ensure DB) and exit without booting — for a SessionStart hook") runCmd.Flags().Int("app-port", 0, "HTTP port for the app (default 8080)") runCmd.Flags().Int("admin-port", 0, "M2EE admin API port (default 8090)") runCmd.Flags().Int("serve-port", 0, "mxbuild --serve port (default 6543)") diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index acd7e5d96..d60e3c803 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -45,6 +45,11 @@ type LocalRunOptions struct { // EnsureDB provisions the local Postgres + app database if missing (otherwise // the DB must already exist). Intended for a fresh session / SessionStart boot. EnsureDB bool + // SetupOnly does the idempotent prerequisites (cache mxbuild+runtime, ensure + // the database) and returns without booting serve/runtime — the non-blocking + // bring-up a SessionStart hook runs each session. The agent then runs the full + // (blocking) loop on demand. + SetupOnly bool // PollInterval is how often Watch checks for changes (default 1s). PollInterval time.Duration // Screenshot, when set, captures a PNG of the app after boot and after each @@ -276,6 +281,14 @@ func RunLocal(opts LocalRunOptions) error { opts.DB.Host, err, opts.DB.Name, opts.DB.User) } + // Setup-only: prerequisites are ready (mxbuild+runtime cached, database up). + // Stop here without booting — this is the non-blocking SessionStart bring-up. + if opts.SetupOnly { + fmt.Fprintf(w, "Setup complete: MxBuild %s + runtime cached, database %q ready.\n", version, opts.DB.Name) + fmt.Fprintln(w, "Run 'mxcli run --local -p ' to boot the warm dev loop.") + return nil + } + // 4. Start the warm build server. fmt.Fprintln(w, "Starting mxbuild --serve...") serve, err := StartServe(ServeOptions{ diff --git a/cmd/mxcli/init.go b/cmd/mxcli/init.go index b3afdb93e..df8c83281 100644 --- a/cmd/mxcli/init.go +++ b/cmd/mxcli/init.go @@ -505,6 +505,16 @@ Container Runtime: } } + // Emit a Claude Code SessionStart hook so a fresh/reaped web session + // self-bootstraps (cache MxBuild+runtime, provision the DB) before use. + if slices.Contains(tools, "claude") && claudeDir != "" { + if changed, err := ensureSessionStartHook(claudeDir, mprFile); err != nil { + fmt.Fprintf(os.Stderr, " Warning: SessionStart hook: %v\n", err) + } else if changed { + fmt.Println("\nAdded SessionStart hook to .claude/settings.json (self-bootstrap on session start)") + } + } + // Create .gitignore if it doesn't exist gitignorePath := filepath.Join(absDir, ".gitignore") if _, err := os.Stat(gitignorePath); os.IsNotExist(err) { diff --git a/cmd/mxcli/init_hook.go b/cmd/mxcli/init_hook.go new file mode 100644 index 000000000..aa4304422 --- /dev/null +++ b/cmd/mxcli/init_hook.go @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" +) + +// init_hook.go emits a Claude Code SessionStart hook into .claude/settings.json so +// a fresh (or reaped-and-resumed) session self-bootstraps: it caches MxBuild + +// runtime and provisions the local database, leaving the session ready to +// `mxcli run --local`. Background processes (Postgres, the JVM) are reaped on +// idle, so this idempotent bring-up must run on every session start. +// +// The hook is setup-only (non-blocking): it must return, so it prepares +// prerequisites and exits rather than booting the long-lived warm loop. + +// sessionStartHookMarker identifies our hook command so re-running init is +// idempotent and we never clobber a user's own SessionStart hooks. +const sessionStartHookMarker = "run --local --setup" + +// sessionStartHookCommand is the shell command the hook runs. It is guarded so a +// missing ./mxcli (or a setup hiccup) never blocks the session from starting. +func sessionStartHookCommand(mprFile string) string { + return fmt.Sprintf("test -x ./mxcli && ./mxcli run --local --setup --ensure-db -p %s || true", mprFile) +} + +// ensureSessionStartHook adds (idempotently) the mxcli bring-up to +// .claude/settings.json, preserving any existing settings/hooks. It returns +// whether a change was written. If settings.json exists but is not valid JSON it +// is left untouched (changed=false) with an error, so we never destroy content. +func ensureSessionStartHook(claudeDir, mprFile string) (changed bool, err error) { + path := filepath.Join(claudeDir, "settings.json") + + settings := map[string]any{} + if data, readErr := os.ReadFile(path); readErr == nil { + if json.Unmarshal(data, &settings) != nil { + return false, fmt.Errorf("%s exists but is not valid JSON; leaving it untouched — add a SessionStart hook manually", path) + } + } + + if updated := addSessionStartHook(settings, sessionStartHookCommand(mprFile)); !updated { + return false, nil // already present + } + + out, err := json.MarshalIndent(settings, "", " ") + if err != nil { + return false, err + } + out = append(out, '\n') + if err := os.WriteFile(path, out, 0o644); err != nil { + return false, err + } + return true, nil +} + +// addSessionStartHook inserts a SessionStart command hook into a parsed settings +// map, preserving existing keys and hooks. It returns false if a SessionStart +// hook whose command contains sessionStartHookMarker already exists (idempotent). +// Exported-for-test via the package; operates on the generic JSON shape so it +// never drops unknown settings. +func addSessionStartHook(settings map[string]any, command string) bool { + hooks, _ := settings["hooks"].(map[string]any) + if hooks == nil { + hooks = map[string]any{} + } + sessionStart, _ := hooks["SessionStart"].([]any) + + for _, entry := range sessionStart { + em, ok := entry.(map[string]any) + if !ok { + continue + } + inner, _ := em["hooks"].([]any) + for _, h := range inner { + hm, ok := h.(map[string]any) + if !ok { + continue + } + if c, _ := hm["command"].(string); strings.Contains(c, sessionStartHookMarker) { + return false // already configured + } + } + } + + sessionStart = append(sessionStart, map[string]any{ + "hooks": []any{ + map[string]any{"type": "command", "command": command}, + }, + }) + hooks["SessionStart"] = sessionStart + settings["hooks"] = hooks + return true +} diff --git a/cmd/mxcli/init_hook_test.go b/cmd/mxcli/init_hook_test.go new file mode 100644 index 000000000..305e20d4d --- /dev/null +++ b/cmd/mxcli/init_hook_test.go @@ -0,0 +1,106 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestAddSessionStartHook_Empty(t *testing.T) { + s := map[string]any{} + if !addSessionStartHook(s, "cmd A") { + t.Fatal("expected a change on empty settings") + } + hooks := s["hooks"].(map[string]any) + ss := hooks["SessionStart"].([]any) + if len(ss) != 1 { + t.Fatalf("SessionStart len = %d, want 1", len(ss)) + } +} + +func TestAddSessionStartHook_Idempotent(t *testing.T) { + s := map[string]any{} + addSessionStartHook(s, "x run --local --setup y") + // A second add with the marker present must be a no-op. + if addSessionStartHook(s, "run --local --setup --ensure-db -p App.mpr") { + t.Error("expected no change when the marker is already present") + } + ss := s["hooks"].(map[string]any)["SessionStart"].([]any) + if len(ss) != 1 { + t.Errorf("SessionStart len = %d, want 1 (no duplicate)", len(ss)) + } +} + +func TestAddSessionStartHook_PreservesExisting(t *testing.T) { + // Existing unrelated settings + a different SessionStart hook must survive. + s := map[string]any{ + "model": "opus", + "hooks": map[string]any{ + "SessionStart": []any{ + map[string]any{"hooks": []any{map[string]any{"type": "command", "command": "echo hi"}}}, + }, + "PostToolUse": []any{map[string]any{"hooks": []any{}}}, + }, + } + if !addSessionStartHook(s, "run --local --setup -p App.mpr") { + t.Fatal("expected a change") + } + if s["model"] != "opus" { + t.Error("unrelated top-level setting was dropped") + } + hooks := s["hooks"].(map[string]any) + if _, ok := hooks["PostToolUse"]; !ok { + t.Error("unrelated hook group was dropped") + } + ss := hooks["SessionStart"].([]any) + if len(ss) != 2 { + t.Errorf("SessionStart len = %d, want 2 (existing + new)", len(ss)) + } +} + +func TestEnsureSessionStartHook_WritesFile(t *testing.T) { + dir := t.TempDir() + changed, err := ensureSessionStartHook(dir, "App.mpr") + if err != nil || !changed { + t.Fatalf("ensureSessionStartHook = (%v,%v), want (true,nil)", changed, err) + } + data, err := os.ReadFile(filepath.Join(dir, "settings.json")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(data), "run --local --setup --ensure-db -p App.mpr") { + t.Errorf("settings.json missing the hook command:\n%s", data) + } + // Valid JSON round-trips. + var check map[string]any + if err := json.Unmarshal(data, &check); err != nil { + t.Errorf("settings.json is not valid JSON: %v", err) + } + // Second call is idempotent (no change). + changed, err = ensureSessionStartHook(dir, "App.mpr") + if err != nil || changed { + t.Errorf("second call = (%v,%v), want (false,nil)", changed, err) + } +} + +func TestEnsureSessionStartHook_InvalidJSONUntouched(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "settings.json") + _ = os.WriteFile(path, []byte("{ not json"), 0o644) + changed, err := ensureSessionStartHook(dir, "App.mpr") + if err == nil { + t.Error("expected an error for invalid existing settings.json") + } + if changed { + t.Error("must not report a change when leaving invalid JSON untouched") + } + // The original content is preserved. + data, _ := os.ReadFile(path) + if string(data) != "{ not json" { + t.Errorf("invalid settings.json was modified: %q", data) + } +} diff --git a/docs-site/src/SUMMARY.md b/docs-site/src/SUMMARY.md index 4e5b3b62d..7537125cf 100644 --- a/docs-site/src/SUMMARY.md +++ b/docs-site/src/SUMMARY.md @@ -152,6 +152,7 @@ - [Credential Management](tools/credentials.md) - [Database Connector Generation](tools/connector-generation.md) - [Local Dev Loop](tools/run-local.md) + - [Bootstrap Prompt](tools/bootstrap-prompt.md) - [Docker Integration](tools/docker.md) - [mxcli docker build](tools/docker-build.md) - [mxcli docker check](tools/docker-check.md) diff --git a/docs-site/src/tools/bootstrap-prompt.md b/docs-site/src/tools/bootstrap-prompt.md new file mode 100644 index 000000000..b31044905 --- /dev/null +++ b/docs-site/src/tools/bootstrap-prompt.md @@ -0,0 +1,57 @@ +# Bootstrap prompt (empty repo → running Mendix app) + +The **primary** way to start a Mendix + mxcli project from the web or an iPad — no +local CLI, no GitHub template to pick from a (short) mobile list. Open an **empty +repo** in Claude Code Web and paste the prompt below; the agent provisions +everything and commits the result so future sessions self-bootstrap. + +Why a prompt instead of a GitHub template repo: the mobile "New repository" template +dropdown shows only a small subset of templates, and a template repo needs per-Mendix- +version upkeep. A prompt starts from a *truly empty* repo, runs *current* mxcli, and +can seed the model from a design prototype in the same session — nothing to maintain. + +## The prompt + +```text +This is an empty repo. Provision it as a Mendix app developed with mxcli: + +1. Ensure `mxcli` is available. It should be pre-installed by the environment; if + not, install it (`go install github.com/mendixlabs/mxcli/cmd/mxcli@latest`, or + `mxcli setup mxcli`), and make a `./mxcli` available at the repo root. +2. Create the app at the repo root: `mxcli new App --version 11.6.3` + (or `mxcli init` if an .mpr already exists). +3. Ensure the Claude tooling is set up: `mxcli init --tool claude`. This adds a + SessionStart hook to `.claude/settings.json` that self-bootstraps future sessions. +4. Bring prerequisites up: `./mxcli run --local --setup --ensure-db -p App.mpr` + (caches MxBuild + runtime, starts Postgres, creates the app database). +5. COMMIT everything now — `App.mpr`, `.devcontainer/`, and `.claude/` (including the + SessionStart hook) — so that after idle reaping the next session bootstraps from + files, not from re-running this prompt. +6. Boot and verify: `./mxcli run --local -p App.mpr` in the background, then confirm + the app answers HTTP 200 at http://localhost:8080/ and report. + +(Optional) Seed the domain model, pages, and microflows from this prototype: . +``` + +## Two rules that make this robust + +- **Committing the config (step 5) is mandatory.** The prompt is a *one-time seed*. + Its output — `.mpr` + `.devcontainer/` + `.claude/` with the SessionStart hook — must + be committed so the steady state is file-driven and deterministic. After that, every + new session runs the hook (`run --local --setup --ensure-db`) automatically; you + never re-paste the prompt. +- **mxcli delivery is an environment concern, not the prompt's.** Step 1 is the fragile + part in a gated web session (a GitHub release `curl` may be blocked). The robust fix + is for the Claude Code Web **environment image / setup script to pre-install mxcli** + (and pre-cache MxBuild + runtime); `go install` via `proxy.golang.org` is the fallback + and needs mxcli published as a public Go module. + +## After bootstrap — the inner loop + +```bash +./mxcli run --local -p App.mpr --watch --screenshot # warm dev loop + screenshots +./mxcli exec change.mdl -p App.mpr # edit the model; the loop hot-applies +``` + +See [mxcli run --local](run-local.md) for the warm loop, `--watch`, `--ensure-db`, and +the screenshot flags. diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index 0502db9b7..437b72f92 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -64,6 +64,7 @@ so structural changes need a restart; behavioural changes do not. | `--local` | — | Required; run without Docker | | `--watch` | off | Rebuild + hot-apply on every project change | | `--ensure-db` | off | Provision local Postgres + the app database if missing (fresh-session bootstrap) | +| `--setup` | off | Prepare prerequisites (cache MxBuild+runtime, ensure DB) and exit without booting — for a SessionStart hook | | `--app-port` | 8080 | App HTTP port | | `--admin-port` | 8090 | M2EE admin API port | | `--serve-port` | 6543 | `mxbuild --serve` port | @@ -135,5 +136,16 @@ mxcli run --local -p app.mpr --watch --screenshot \ --screenshot-url /p/customer_overview ``` +## Fresh sessions (Claude Code Web) + +Background processes (Postgres, the JVM) are reaped on idle, so a resumed web session +needs to bring prerequisites back up. `mxcli init` emits a **SessionStart hook** into +`.claude/settings.json` that runs `./mxcli run --local --setup --ensure-db -p ` +on every session start — the non-blocking `--setup` mode caches MxBuild+runtime and +provisions the database, then exits, leaving the session ready to `run --local`. + +To start from an **empty repo** on the web or an iPad, use the +[bootstrap prompt](bootstrap-prompt.md) instead of a GitHub template. + See also: [PROPOSAL_mxcli_dev_warm_loop](../../../docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md), [mxcli docker run](docker-run.md), [Playwright Testing](playwright.md). diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index 8d113129b..dfa6e4bf9 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -360,18 +360,26 @@ Two rules make this robust: Codespaces and local dev containers, but Claude Code Web needs one more piece: a **SessionStart hook** (or devcontainer `postStart`) that idempotently brings the runtime up on **every** session, because background processes (Postgres, the JVM) are **reaped -on idle** — observed repeatedly during this investigation. Proposed: `mxcli init` also -emits an `mxcli dev up` bootstrap, wired to SessionStart, that idempotently: +on idle** — observed repeatedly during this investigation. + +**Shipped:** `mxcli init` now emits a Claude Code **SessionStart hook** into +`.claude/settings.json` (merged idempotently, preserving existing settings) that runs +`./mxcli run --local --setup --ensure-db -p `. That `--setup` mode is the +non-blocking bring-up — it does steps 1–3 below and **returns** (a SessionStart hook +must not block), leaving the session ready to `mxcli run --local` on demand: 1. **Ensure mxcli** — prefer a prebuilt binary via `mxcli setup mxcli` (fast) over a - from-source build (needs antlr4 + Go, ~70 s). -2. **Ensure MxBuild + runtime cached** — `mxcli setup mxbuild -p app.mpr` and - `mxcli setup mxruntime -p app.mpr` (both already exist). One-time ~700 MB / ~30–40 s; - **bake into the devcontainer image** so the first session is instant. -3. **Start Postgres + create the app DB** — re-run every session (survives reaping). -4. **Boot the runtime + `mxbuild --serve`** — the standalone-boot recipe - (BasePath / RuntimePath / MicroflowConstants / data-dirs → `start` → DDL if needed), - leaving a warm serve daemon. + from-source build (needs antlr4 + Go, ~70 s). *(mxcli delivery is the environment's + job; the hook guards on `test -x ./mxcli`.)* +2. **Ensure MxBuild + runtime cached** — `run --local --setup` runs DownloadMxBuild + + the runtime resolve. One-time ~700 MB / ~30–40 s; **bake into the devcontainer + image** so the first session is instant. +3. **Start Postgres + create the app DB** — `--ensure-db`, re-run every session + (survives reaping). + +Step 4 — **booting the runtime + `mxbuild --serve`** — is deliberately *not* in the +hook (it is long-lived and would block session start). The agent runs the full +`mxcli run --local` when it wants to test; the heavy prerequisites are already warm. End state: the session comes up **ready to prompt → build → test**. Then: @@ -528,7 +536,7 @@ builds on the previous. | # | Slice | Delivers | Depends on | Size / risk | |---|-------|----------|------------|-------------| | 1 | **Warm local loop** — shipped as `mxcli run --local [--watch]` (serve daemon + M2EE admin client + `restartRequired` branching; + client bundling & Playwright screenshots) | Docker-free ~1 s edit→test loop, locally | nothing new | ✅ **shipped** | -| 2 | **Provisioning** — database provisioning (`run --local --ensure-db`, ✅ done) + `mxcli init` SessionStart hook + prompt template (remaining) | a fresh Claude Code Web session comes up testable; iPad-native start | slice 1 | small / low | +| 2 | **Provisioning** — `run --local --ensure-db` (DB) + `run --local --setup` (non-blocking bring-up) + `mxcli init` SessionStart hook + bootstrap prompt template | a fresh Claude Code Web session comes up testable; iPad-native start | slice 1 | ✅ **shipped** | | 3 | **Single-app external preview** — `mxcli dev serve` + chisel client → one static relay + `ApplicationRootUrl` wiring | a shareable live preview URL (the iPad two-container flow) | slice 1 | medium / medium — the `app.github.dev` WebSocket hop is unverified (a VPS relay avoids it) | | 4 | **Tunnel hub** — `mxcli tunnel-hub` + `mxcli dev --hub` + registration API + admin overview | many dev containers behind one ingress; fleet overview + per-container change lists | slice 3 | large / higher — a product in its own right, with a multi-tenant auth surface | From 7e36ef41d11fa9f53de9b9e32e460f130a8f95f7 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 21:35:37 +0000 Subject: [PATCH 23/26] docs: promote the bootstrap prompt to the README + clarify install version Adds a prominent Quick Start subsection ("From the web or an iPad") pointing at the bootstrap prompt and the warm local dev loop (run --local --watch --screenshot), so the empty-repo web/iPad path is discoverable from the README. Documents which mxcli version each install path yields, in bootstrap-prompt.md: - `go install ...@latest` -> latest tagged release (not nightly; pin with @vX.Y.Z) - `mxcli setup mxcli` -> matches the running binary's version (nightly->nightly, vX.Y.Z->that release); override with --tag - environment image pre-install -> whatever the image bakes (recommended for pinning a version fleet-wide) Recommends pinning for a reproducible bootstrap. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- README.md | 14 ++++++++++++++ docs-site/src/tools/bootstrap-prompt.md | 16 ++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/README.md b/README.md index 2c7ea0f72..21de20df0 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,20 @@ mxcli new MyApp --version 11.8.0 This downloads MxBuild, creates a blank Mendix project, sets up AI tooling and a Dev Container, and installs the correct mxcli binary. Open the resulting folder in VS Code and reopen in the Dev Container — you're ready to go. +### From the web or an iPad (empty repo, no local install) + +No machine with a CLI? Open an **empty repo** in [Claude Code on the web](https://claude.ai/code) (works on an iPad) and paste the **bootstrap prompt** — the agent provisions the whole project (creates the app, wires the Dev Container + AI tooling, provisions the database) and commits it so future sessions self-bootstrap. This is the recommended web/iPad path; you don't need to pick a GitHub template. + +> See **[Bootstrap Prompt](https://mendixlabs.github.io/mxcli/tools/bootstrap-prompt.html)** for the exact copy-paste text. In short: it runs `mxcli new` → `mxcli init` → commits the config → `mxcli run --local --setup --ensure-db` so the app comes up testable. + +Then iterate with the **warm local dev loop** — a Docker-free ~1-second edit→test cycle: + +```bash +mxcli run --local -p app.mpr --watch --screenshot # hot-reload + auto screenshots +``` + +`mxcli run --local` keeps `mxbuild --serve` and a standalone runtime hot: a page/microflow edit is hot-applied in seconds (a hot `reload_model`, or a restart + DDL for entity changes), and `--screenshot` captures each page with Playwright. See **[Local Dev Loop](https://mendixlabs.github.io/mxcli/tools/run-local.html)**. + ### Existing project For an existing Mendix project, use `mxcli init` to add AI tooling and a Dev Container: diff --git a/docs-site/src/tools/bootstrap-prompt.md b/docs-site/src/tools/bootstrap-prompt.md index b31044905..736ce7fc5 100644 --- a/docs-site/src/tools/bootstrap-prompt.md +++ b/docs-site/src/tools/bootstrap-prompt.md @@ -33,6 +33,22 @@ This is an empty repo. Provision it as a Mendix app developed with mxcli: (Optional) Seed the domain model, pages, and microflows from this prototype: . ``` +## Which mxcli version gets installed + +- **`go install github.com/mendixlabs/mxcli/cmd/mxcli@latest`** installs the **latest + tagged release** (highest semver tag) — *not* the nightly. Pin a specific one with + `@vX.Y.Z`. +- **`mxcli setup mxcli`** downloads the release asset that **matches the mxcli already + running it** — a nightly build pulls the moving `nightly` release, a `vX.Y.Z` build + pulls that exact release. Override with `--tag nightly` or `--tag vX.Y.Z`. (This is + mainly for getting the *Linux* binary that matches your host, e.g. into a Dev + Container.) +- **Environment pre-install** (the robust path) installs whatever the Claude Code Web + image bakes in — the recommended way to pin a known-good version fleet-wide. + +For a reproducible bootstrap, pin the version (`@vX.Y.Z` or `--tag vX.Y.Z`) rather than +tracking `@latest`; nightly is opt-in only. + ## Two rules that make this robust - **Committing the config (step 5) is mandatory.** The prompt is a *one-time seed*. From 938e3b50ebf06683a7cabf79e438b0903e57c13a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 21:46:08 +0000 Subject: [PATCH 24/26] =?UTF-8?q?docs:=20correct=20install=20path=20?= =?UTF-8?q?=E2=80=94=20go=20install=20@latest=20doesn't=20work=20(uncommit?= =?UTF-8?q?ted=20parser)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Delivery-pipeline check findings: - github.com/mendixlabs/mxcli IS a public Go module (tags v0.1.0–v0.16.0; @latest -> v0.16.0 per proxy.golang.org). - release.yml (on v* tags) runs `make release` and uploads bin/mxcli-* via softprops/action-gh-release; make release builds mxcli-{linux,darwin,windows}- {amd64,arm64}(.exe), which exactly match what `mxcli setup mxcli` downloads. nightly.yml maintains a rolling `nightly` release. So the prebuilt-binary path is correct and consistent with setup mxcli's naming. - BUT `go install …/cmd/mxcli@latest` does NOT work: the generated ANTLR parser (mdl/grammar/parser/) is a hard import yet is gitignored and not committed, so the tagged module source is missing the package and the build fails. Fix bootstrap-prompt.md to stop recommending `go install @latest`: use the prebuilt release binary (mxcli setup mxcli / direct mxcli-- download) or the environment pre-install; from-source builds need `make grammar`. Enabling go install (commit the generated parser, or generate during module build) is flagged as a maintainer decision. (Could not verify the release assets are attached to the tags from this session — GitHub is scoped to ako/mxcli, so api.github.com/github.com return proxy 403 for mendixlabs/mxcli — but the release CI is present and correct.) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- docs-site/src/tools/bootstrap-prompt.md | 31 +++++++++++++++---------- 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/docs-site/src/tools/bootstrap-prompt.md b/docs-site/src/tools/bootstrap-prompt.md index 736ce7fc5..3a74fee01 100644 --- a/docs-site/src/tools/bootstrap-prompt.md +++ b/docs-site/src/tools/bootstrap-prompt.md @@ -16,8 +16,10 @@ can seed the model from a design prototype in the same session — nothing to ma This is an empty repo. Provision it as a Mendix app developed with mxcli: 1. Ensure `mxcli` is available. It should be pre-installed by the environment; if - not, install it (`go install github.com/mendixlabs/mxcli/cmd/mxcli@latest`, or - `mxcli setup mxcli`), and make a `./mxcli` available at the repo root. + not, download a prebuilt binary from the GitHub releases for your OS/arch (e.g. + `mxcli-linux-amd64`) and put it at `./mxcli`. (`go install …@latest` does **not** + work yet — the ANTLR parser is generated at build time and isn't committed; a + from-source build needs `make grammar`. Use the prebuilt binary.) 2. Create the app at the repo root: `mxcli new App --version 11.6.3` (or `mxcli init` if an .mpr already exists). 3. Ensure the Claude tooling is set up: `mxcli init --tool claude`. This adds a @@ -35,19 +37,24 @@ This is an empty repo. Provision it as a Mendix app developed with mxcli: ## Which mxcli version gets installed -- **`go install github.com/mendixlabs/mxcli/cmd/mxcli@latest`** installs the **latest - tagged release** (highest semver tag) — *not* the nightly. Pin a specific one with - `@vX.Y.Z`. -- **`mxcli setup mxcli`** downloads the release asset that **matches the mxcli already - running it** — a nightly build pulls the moving `nightly` release, a `vX.Y.Z` build - pulls that exact release. Override with `--tag nightly` or `--tag vX.Y.Z`. (This is - mainly for getting the *Linux* binary that matches your host, e.g. into a Dev - Container.) +- **Prebuilt release binary** (`mxcli setup mxcli`, or a direct download of + `mxcli--` from GitHub releases) is the working install path. Releases are + published by CI on every `vX.Y.Z` tag (latest is v0.16.0), plus a rolling `nightly` + pre-release. `mxcli setup mxcli` downloads the asset that **matches the mxcli already + running it** (a nightly build → the `nightly` release; a `vX.Y.Z` build → that exact + release); override with `--tag nightly` / `--tag vX.Y.Z`. Because `setup mxcli` needs + an mxcli to run it, it's mainly for replicating a version onto another OS/arch (e.g. + the Linux binary in a Dev Container), not the first install. - **Environment pre-install** (the robust path) installs whatever the Claude Code Web image bakes in — the recommended way to pin a known-good version fleet-wide. +- **`go install …@latest` does not work yet.** The module *is* public (tags v0.1.0– + v0.16.0), but the generated ANTLR parser (`mdl/grammar/parser/`) is gitignored and + not committed, so a `go install` from the tagged source fails on the missing package. + Building from source works only via `make build`/`make release` (which run + `make grammar` first). Enabling `go install` would require committing the generated + parser (or generating it during module build) — a maintainer decision. -For a reproducible bootstrap, pin the version (`@vX.Y.Z` or `--tag vX.Y.Z`) rather than -tracking `@latest`; nightly is opt-in only. +For a reproducible bootstrap, pin the version (`--tag vX.Y.Z`); nightly is opt-in only. ## Two rules that make this robust From a9bbbd9dac5f786e476ebd11f14ec1c657be75cf Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 22:05:20 +0000 Subject: [PATCH 25/26] docs(bootstrap): recommend the nightly build for install Nightly is already downloadable (`mxcli setup mxcli --tag nightly`, or the `.../releases/download/nightly/mxcli--` asset). Recommend it for the bootstrap while mxcli is fast-moving alpha: the warm-loop surface referenced by the prompt (run --local, --watch, --ensure-db, --setup, screenshots) lands in nightly before it reaches a tagged release, so the documented flow needs it. Pinning a vX.Y.Z release remains the reproducibility/stability option, and the go-install caveat stays. Docs-only. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- docs-site/src/tools/bootstrap-prompt.md | 45 ++++++++++++++++--------- 1 file changed, 30 insertions(+), 15 deletions(-) diff --git a/docs-site/src/tools/bootstrap-prompt.md b/docs-site/src/tools/bootstrap-prompt.md index 3a74fee01..6c0331741 100644 --- a/docs-site/src/tools/bootstrap-prompt.md +++ b/docs-site/src/tools/bootstrap-prompt.md @@ -16,10 +16,20 @@ can seed the model from a design prototype in the same session — nothing to ma This is an empty repo. Provision it as a Mendix app developed with mxcli: 1. Ensure `mxcli` is available. It should be pre-installed by the environment; if - not, download a prebuilt binary from the GitHub releases for your OS/arch (e.g. - `mxcli-linux-amd64`) and put it at `./mxcli`. (`go install …@latest` does **not** - work yet — the ANTLR parser is generated at build time and isn't committed; a - from-source build needs `make grammar`. Use the prebuilt binary.) + not, download a prebuilt binary for your OS/arch and put it at `./mxcli`, e.g.: + + ```bash + curl -fsSL -o ./mxcli \ + https://github.com/mendixlabs/mxcli/releases/download/nightly/mxcli-linux-amd64 + chmod +x ./mxcli + ``` + + Use the **`nightly`** build: this is fast-moving alpha software and the warm-loop + commands used below (`run --local`, `--setup`, `--ensure-db`) land in `nightly` + before they appear in a tagged release. For a reproducible setup, pin a specific + release instead (`.../releases/download/vX.Y.Z/mxcli--`). Note: `go + install …@latest` does **not** work — the generated ANTLR parser isn't committed, + so use the prebuilt binary (a from-source build needs `make grammar`). 2. Create the app at the repo root: `mxcli new App --version 11.6.3` (or `mxcli init` if an .mpr already exists). 3. Ensure the Claude tooling is set up: `mxcli init --tool claude`. This adds a @@ -37,16 +47,23 @@ This is an empty repo. Provision it as a Mendix app developed with mxcli: ## Which mxcli version gets installed -- **Prebuilt release binary** (`mxcli setup mxcli`, or a direct download of - `mxcli--` from GitHub releases) is the working install path. Releases are - published by CI on every `vX.Y.Z` tag (latest is v0.16.0), plus a rolling `nightly` - pre-release. `mxcli setup mxcli` downloads the asset that **matches the mxcli already - running it** (a nightly build → the `nightly` release; a `vX.Y.Z` build → that exact - release); override with `--tag nightly` / `--tag vX.Y.Z`. Because `setup mxcli` needs - an mxcli to run it, it's mainly for replicating a version onto another OS/arch (e.g. - the Linux binary in a Dev Container), not the first install. +Prebuilt binaries are the working install path. CI publishes them on every `vX.Y.Z` +tag (latest is v0.16.0) **and** as a rolling `nightly` pre-release, with assets named +`mxcli--`. + +- **`nightly` — recommended while mxcli is fast-moving alpha.** New features (the whole + warm-loop surface: `run --local`, `--watch`, `--ensure-db`, `--setup`, screenshots) + land in `nightly` before they reach a tagged release, so the bootstrap flow above + needs it. Download `.../releases/download/nightly/mxcli--`, or once mxcli is + present, `mxcli setup mxcli --tag nightly`. +- **`vX.Y.Z` — pin for reproducibility / stability.** The CI marks nightly a + pre-release ("use tagged releases for production"). Download + `.../releases/download/vX.Y.Z/mxcli--` or `mxcli setup mxcli --tag vX.Y.Z`. + With no `--tag`, `mxcli setup mxcli` matches the mxcli already running it (nightly → + `nightly`, `vX.Y.Z` → that release) — mainly useful for replicating a version onto + another OS/arch (e.g. the Linux binary in a Dev Container), not the first install. - **Environment pre-install** (the robust path) installs whatever the Claude Code Web - image bakes in — the recommended way to pin a known-good version fleet-wide. + image bakes in — the way to pin a known-good version fleet-wide. - **`go install …@latest` does not work yet.** The module *is* public (tags v0.1.0– v0.16.0), but the generated ANTLR parser (`mdl/grammar/parser/`) is gitignored and not committed, so a `go install` from the tagged source fails on the missing package. @@ -54,8 +71,6 @@ This is an empty repo. Provision it as a Mendix app developed with mxcli: `make grammar` first). Enabling `go install` would require committing the generated parser (or generating it during module build) — a maintainer decision. -For a reproducible bootstrap, pin the version (`--tag vX.Y.Z`); nightly is opt-in only. - ## Two rules that make this robust - **Committing the config (step 5) is mandatory.** The prompt is a *one-time seed*. From 82e1fc86bcfb2e455d650d5a605cab0846294a5c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 18 Jul 2026 22:10:50 +0000 Subject: [PATCH 26/26] =?UTF-8?q?docs:=20sweep=20install=20guidance=20?= =?UTF-8?q?=E2=80=94=20correct=20asset=20format,=20go=20install,=20recomme?= =?UTF-8?q?nd=20nightly?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swept all user-facing docs for install accuracy. Two real inaccuracies fixed beyond the go-install issue: - Release assets are RAW binaries named mxcli-- (not mxcli__. tar.gz archives). quickstart.md and installation.md told users to download and `tar xzf` a .tar.gz that doesn't exist — replaced with a direct curl of the raw binary + chmod +x. - `go install …@latest` was listed as "build from source"/an install option in quickstart.md and installation.md; it fails (uncommitted generated parser). Replaced with `git clone && make build`, plus a caveat note. Also: recommend the rolling `nightly` build across README/quickstart/installation while mxcli is fast-moving alpha (features land there before a tagged release), with vX.Y.Z pinning for reproducibility. Flagged the go-install reality in the warm-loop proposal's release row (it claimed a public Go module suffices). Files: README.md, docs-site/src/tutorial/{quickstart,installation}.md, docs-site/src/tools/bootstrap-prompt.md (earlier), PROPOSAL_mxcli_dev_warm_loop.md. Docs-only. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- README.md | 13 ++++++++++- docs-site/src/tutorial/installation.md | 22 ++++++++++++------- docs-site/src/tutorial/quickstart.md | 14 +++++++----- .../PROPOSAL_mxcli_dev_warm_loop.md | 2 +- 4 files changed, 35 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 21de20df0..338dcc365 100644 --- a/README.md +++ b/README.md @@ -180,7 +180,16 @@ mxcli add-tool cursor ## Installation -Download the latest release for your platform from the [releases page](https://github.com/mendixlabs/mxcli/releases), or build from source: +Download a pre-built binary from the [releases page](https://github.com/mendixlabs/mxcli/releases) — the assets are raw binaries named `mxcli--` (nothing to extract). While mxcli is fast-moving alpha, the rolling `nightly` build is recommended (new features land there first); pin a `vX.Y.Z` release for reproducibility: + +```bash +# Linux/macOS — nightly +curl -fsSL -o mxcli \ + https://github.com/mendixlabs/mxcli/releases/download/nightly/mxcli-linux-amd64 +chmod +x mxcli && sudo mv mxcli /usr/local/bin/ +``` + +Or build from source (Go + Make — `make build` runs the ANTLR parser generation that `go install` can't): ```bash git clone https://github.com/mendixlabs/mxcli.git @@ -189,6 +198,8 @@ make build # binary is at ./bin/mxcli ``` +> `go install …@latest` is not supported: the generated ANTLR parser isn't committed, so a module-source build fails. Use a pre-built binary or `make build`. + ## Core Features ### Explore Project Structure diff --git a/docs-site/src/tutorial/installation.md b/docs-site/src/tutorial/installation.md index c6652b452..25657bfdf 100644 --- a/docs-site/src/tutorial/installation.md +++ b/docs-site/src/tutorial/installation.md @@ -26,19 +26,25 @@ When you're ready to work on your own Mendix project, use one of the installatio ## Binary download -Pre-built binaries are available for Linux, macOS, and Windows on both amd64 and arm64 architectures. +Pre-built binaries are published for Linux, macOS, and Windows on both amd64 and arm64. Release assets are **raw binaries** named `mxcli--` (Windows: `.exe`) — there is nothing to extract. -1. Go to the [GitHub Releases page](https://github.com/mendixlabs/mxcli/releases). -2. Download the archive for your platform (e.g., `mxcli_linux_amd64.tar.gz` or `mxcli_darwin_arm64.tar.gz`). -3. Extract the binary and move it somewhere on your `PATH`: +While mxcli is fast-moving alpha, use the rolling **`nightly`** build (new features land there before a tagged release); pin a `vX.Y.Z` release for reproducibility. ```bash -# Example for Linux/macOS -tar xzf mxcli_linux_amd64.tar.gz -sudo mv mxcli /usr/local/bin/ +# Linux/macOS — nightly (recommended for now) +curl -fsSL -o mxcli \ + https://github.com/mendixlabs/mxcli/releases/download/nightly/mxcli-linux-amd64 +chmod +x mxcli && sudo mv mxcli /usr/local/bin/ + +# ...or pin a specific release +curl -fsSL -o mxcli \ + https://github.com/mendixlabs/mxcli/releases/download/v0.16.0/mxcli-darwin-arm64 +chmod +x mxcli && sudo mv mxcli /usr/local/bin/ ``` -On Windows, extract the `.zip` and add the folder containing `mxcli.exe` to your system PATH. +Assets: `mxcli-linux-amd64`, `mxcli-linux-arm64`, `mxcli-darwin-amd64`, `mxcli-darwin-arm64`, `mxcli-windows-amd64.exe`, `mxcli-windows-arm64.exe`. On Windows, download the `.exe` and add its folder to your PATH. You can browse them on the [GitHub Releases page](https://github.com/mendixlabs/mxcli/releases). + +> `go install github.com/mendixlabs/mxcli/cmd/mxcli@latest` does **not** work: the ANTLR parser is generated at build time and isn't committed, so a `go install` from the module source fails. Use a pre-built binary, or **Build from source** below (which runs `make grammar`). ## Build from source diff --git a/docs-site/src/tutorial/quickstart.md b/docs-site/src/tutorial/quickstart.md index 4b56df0d7..b650c19d5 100644 --- a/docs-site/src/tutorial/quickstart.md +++ b/docs-site/src/tutorial/quickstart.md @@ -12,18 +12,20 @@ Open the [mxcli Playground](https://github.com/mendixlabs/mxcli-playground) in a **Option B: Binary download** -Download from the [GitHub Releases page](https://github.com/mendixlabs/mxcli/releases) and extract: +Release assets are raw binaries named `mxcli--` (nothing to extract). Use the rolling `nightly` build while mxcli is alpha: ```bash # macOS / Linux -tar xzf mxcli_.tar.gz -sudo mv mxcli /usr/local/bin/ +curl -fsSL -o mxcli \ + https://github.com/mendixlabs/mxcli/releases/download/nightly/mxcli-linux-amd64 +chmod +x mxcli && sudo mv mxcli /usr/local/bin/ ``` -**Option C: Build from source** +**Option C: Build from source** (Go + Make) ```bash -go install github.com/mendixlabs/mxcli/cmd/mxcli@latest +git clone https://github.com/mendixlabs/mxcli.git && cd mxcli && make build +# binary at ./bin/mxcli (go install @latest doesn't work — see Installation) ``` Verify: `mxcli --version` should print the version number. @@ -132,6 +134,6 @@ No errors? You're done. Open in Studio Pro and everything is there. mxcli setup mxbuild -p your-app.mpr ``` -**"CGO not available"** -- mxcli uses pure Go SQLite. No C compiler needed. If you see CGO errors, ensure you're using the official binary or `go install`. +**"CGO not available"** -- mxcli uses pure Go SQLite. No C compiler needed. If you see CGO errors, ensure you're using the official pre-built binary or a `make build` from source. **Project won't open in Studio Pro after changes** -- Close Studio Pro before running mxcli write commands, then reopen. See [F4 sync support](../appendixes/version-compatibility.md) for details. diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index dfa6e4bf9..acbe24d20 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -501,7 +501,7 @@ downloads/caches mxbuild + runtime and speaks the M2EE admin API. | `cmd/mxcli/init.go` | Emit a **SessionStart hook** (Claude Code Web) and/or devcontainer `postStart` that runs `mxcli dev up`; keep the existing devcontainer + `.claude/` scaffolding. | | `cmd/mxcli/new.go` | Add `--emit-template`: write a GitHub-template-ready repo (config + starter `.mpr`), for CI-per-version publishing of `mendix-mxcli-starter`. | | `docs-site/.../bootstrap-prompt.md` | Ship the canonical **prompt template** (the web/iPad seed prompt) as documented, copy-pasteable text. | -| release / go.mod | Ensure mxcli is deliverable into a gated web session — **public Go module** (`go install`) and/or an **environment setup-script** that pre-installs it; do not rely on a GitHub release `curl` (may be gate-blocked). | +| release / go.mod | Ensure mxcli is deliverable into a gated web session. **Status:** the module is public (tags to v0.16.0) but **`go install` does not build** — the generated ANTLR parser (`mdl/grammar/parser/`) is gitignored/uncommitted, so the tagged source is missing the package. Working paths today: prebuilt release binaries (`mxcli--`, incl. a rolling `nightly`) via `mxcli setup mxcli` / direct download, or an **environment setup-script** pre-install. Enabling `go install` needs the parser committed (or generated during module build) — open decision. | | `cmd/mxcli/tunnelhub/*_test.go` | Tests: slot allocation/isolation, authfile + vhost reload, register/heartbeat/deregister lifecycle, admin-page rendering. | | `cmd/mxcli/dev_test.go` | Tests for the serve client and the `restartRequired` branch logic (mock the serve/admin HTTP endpoints). |