036 — Marketplace install gaps surfaced during 2026-05-12 install

A real marketplace install (kv-mike-test1 / backend.nicedune-6314ad94.uksouth.azurecontainerapps.io, operator amorgan@altien.com) hit a chain of failures that, taken together, mean a Bicep-default marketplace deploy cannot reach a working main-app sign-in without manual az intervention. This issue collects every gap surfaced, with file:line references and root-cause evidence. Each item should be doable as a discrete PR; sequencing notes below.

Status legend

• CONFIRMED — reproduced with concrete evidence in this install
• OBSERVED — saw the symptom but didn't run the reproducer
• THEORETICAL — implied by reading the code, not yet observed

Critical fixes (block marketplace install end-to-end)

1. Main app reads Entra config from process.env; install configurator writes it to KV — they don't meet [CONFIRMED]

backend\src\routes\auth.ts:117-122, 251-257 reads:

• process.env.ENTRA_TENANT_ID
• process.env.ENTRA_CLIENT_ID (or ENTRA_FRONTEND_CLIENT_ID)
• process.env.ENTRA_BACKEND_CLIENT_ID
• process.env.ENTRA_CLIENT_SECRET
• process.env.ENTRA_BACKEND_SCOPE / ENTRA_AUTH_SCOPES

backend\src\routes\install.ts:1115-1116, 1202-1204 reads the same logical values from KV via getConfig():

• KV secret entra-tenant-id
• KV secret entra-client-id
• KV secret entra-client-secret

Outcome on a fresh marketplace install: /install works fine (KV is populated by create-entra-apps.ps1), but /api/auth/login returns 500 {detail: "Missing Entra OpenID configuration"} because the env vars are unset. Operator must run az containerapp secret set + az containerapp update --set-env-vars for five values to bridge the gap.

Fix: change main-app auth.ts to read from KV via the same getConfig() helper /install uses. Eliminates the env-var contract entirely and removes the need to redeploy the Container App when an Entra value changes.

File touchpoints: backend\src\routes\auth.ts (entraClientId / entraScopes / token-exchange and authorize-URL paths), backend\src\lib\config.ts (the getConfig helper).

2. Bicep doesn't wire 4 of 5 ENTRA_* env vars even in authProvider=entra mode [CONFIRMED]

infra\modules\containerapp-backend.bicep:74-78:

var modeSpecificEnv = authProvider == 'entra' ? [
  { name: 'ENTRA_BACKEND_SCOPE', value: entraBackendScope }
] : [ ... ]

Only ENTRA_BACKEND_SCOPE is wired, and it comes from a Bicep parameter (main.bicep:95, default ''). The other four env vars main-app auth.ts needs (ENTRA_TENANT_ID, ENTRA_CLIENT_ID, ENTRA_BACKEND_CLIENT_ID, ENTRA_CLIENT_SECRET) are never set by Bicep at all.

Fix path A (preferred): ride on Fix #1 — when auth.ts reads from KV, Bicep doesn't need to wire any of these env vars.

Fix path B (if Fix #1 isn't done first): wire all five from KV secret references using the same secretRef pattern already in use for anthropic-api-key / openai-api-key. Caveat: requires the KV secrets to exist at Bicep deploy time, which is the chicken-and-egg with create-entra-apps.ps1 running post-deploy. Marketplace deploy would need a placeholder + a Container App revision restart after create-entra-apps.ps1. Operationally awkward — strongly prefer path A.

3. Bicep default for authProvider is 'supabase' [CONFIRMED]

infra\main.bicep:91-92:

@description('Auth provider applied to backend + PostgREST: supabase | local | entra')
param authProvider string = 'supabase'

For marketplace SaaS where the only sensible mode is Entra (Supabase requires env vars marketplace never sets), this default lands the deployment in a guaranteed-broken state — main app crashes with Supabase client requested but NEXT_PUBLIC_SUPABASE_URL / NEXT_PUBLIC_SUPABASE_PUBLISHABLE_DEFAULT_KEY are not set.

Fix: either flip default to 'entra', or make the marketplace createUiDefinition.json require an explicit pick. Bicep should also fail-fast if authProvider='supabase' and the Supabase config isn't supplied — silently deploying broken is worse than refusing.

4. entra-backend-scope KV row has no paste-form fix path [CONFIRMED]

backend\src\lib\install\manifest.ts:383-397 marks the row as fixedBy: { type: "auto" }, which renders only a description label (backend\src\routes\install.ts:192-194) — no paste form, no download button. Description says "create-entra-apps.ps1 sets this", but the script is only offered for download from the entra-tenant-id row's alsoAsScript block (manifest.ts:318-322), per the deliberate decision at manifest.ts:288-296.

Operators who took the paste-each-row path (the rows for tenant ID / backend app / frontend app all have in-app forms) get every other Entra row green while this row remains stuck without an actionable affordance from that row.

Fix options:

• Best: drop the secret entirely — compute scope as api://<entra-backend-client-id>/access_as_user at read-time. There is no scenario where it can legitimately diverge from the backend client ID.
• Otherwise: give the row its own in-app paste form (value is deterministic, format is api://<guid>/access_as_user).
• Minimal: surface the create-entra-apps.ps1 download on this row too. Less clean but unblocks the paste-path operator.

5. AUTH_PROVIDER not in the install manifest [CONFIRMED]

The install configurator exhaustively checks Entra app regs, redirect URIs, scopes, secrets, group GUIDs, and groupMembershipClaims — but never inspects process.env.AUTH_PROVIDER. So a marketplace install can show every row green while the main app is hardwired to Supabase mode and crashes immediately.

Fix: add a manifest entry under Foundations:

• id: 'auth-provider-mode'
• check: read process.env.AUTH_PROVIDER; fail if not entra when any Entra KV secret is set.
• fixedBy: until install backend can update its own Container App revision (requires escalated RBAC), use type: 'auto' with a copy-pasteable az containerapp update --set-env-vars AUTH_PROVIDER=entra line. Eliminate entirely once Fix #1 lands (because then auth mode is derived from KV state, not env).

High-leverage UX gaps (operators consistently trip on these)

6. Admin-group check has no fallback for Entra groups-claim overage [CONFIRMED, root cause of install lockout]

backend\src\lib\install\installAuth.ts:192-204 reads claims.groups directly. When Entra emits overage signaling (hasgroups: true on implicit flow, _claim_names/_claim_sources on auth-code flow), claims.groups is undefined and isInAdminGroup returns false — even when the user is legitimately in the configured admin group at directory level.

Reproducer captured in this session: amorgan@altien.com is in 57 transitive group memberships (well under the documented 200-group JWT limit), yet implicit-flow id_token returned hasgroups: "true" instead of an inline groups array. The documented 200 is a ceiling, not a guarantee — Entra budgets by total token size, and Mike's broad Graph scopes + 5 optional claims push the budget below the user's actual group count.

Fix: when claims.groups is empty/missing AND (claims.hasgroups === "true" OR claims._claim_names exists), call Microsoft Graph /me/memberOf using the access_token already held at install.ts:1273-1281. Walk the result, match against entra-admin-group-ids.

This same fallback is also needed in backend\src\routes\auth.ts for the main-app sign-in path if it uses groups for downstream authorization.

7. The 403 "Not in admin group" page is unhelpful [CONFIRMED]

backend\src\routes\install.ts:1257-1263 says "Either add yourself to the configured admin group, or update via bootstrap-authed /install" — without surfacing:

• What admin group GUID is configured (Graph-resolve to display name)
• What groups the user's id_token actually carried (so the operator can tell "wrong group" from "no groups claim at all")
• Where to find the bootstrap token if they lost the Bicep deploy output
• How to detect / handle groups overage (related to Fix #6)

Fix: enrich the 403 page. Use the access_token (install.ts:1273-1281) to Graph-resolve admin group names, and dump the claims that arrived. After Fix #6 lands, most of this can be removed — the user just gets in.

8. No marketplace-installer identity captured [CONFIRMED]

Marketplace SaaS handshake exposes the purchaser's oid + email. Bicep currently doesn't accept either as a parameter, so once an operator gets locked out (e.g., Fix #6 not yet shipped + their admin group isn't picked up), there's no durable identity hook to recover.

Fix:

• Bicep: accept param initialAdminObjectId string = '' + param initialAdminEmail string = ''. Write to KV as install-initial-admin-oid / install-initial-admin-email at deploy time.
• Marketplace deploy pipeline: pass these from the SaaS subscription metadata.
• Install auth: isInAdminGroup (or sibling isInstaller) honors the initial-admin OID as a permanent escape hatch — independent of any group claim.

9. Bootstrap token retires too eagerly [OBSERVED]

install.ts:1284-1290 retires on the first successful Entra admin sign-in. If the operator hasn't yet finished configuration when that first sign-in happens (e.g., signs in too early), the only escape hatch disappears.

Fix: retire only when the manifest has zero fail rows in required: true items, or behind an explicit "I'm done — retire bootstrap" button. Less surface area for the operator to footgun themselves.

10. Slice-9 redirect-URI Graph check returns "info: Graph denied" instead of fail [OBSERVED]

backend\src\lib\install\checks\redirectUris.ts falls back to info when the backend UAMI lacks app-reg ownership. So operators who took the paste-each-row path (never ran create-entra-apps.ps1 -ResourceGroup ... which grants UAMI ownership at scripts\install\create-entra-apps.ps1:275-304) never get a passing or failing redirect-URI verification — it just stays informational.

Fix path A: when Graph access isn't available, surface the expected list of URIs ("the following should be registered: ...") so the operator can visually verify. Don't silently degrade.

Fix path B (better): split the UAMI-ownership grant into its own action, keyed off entra-client-id rather than bundled with create-entra-apps.ps1. Then any operator (paste-path or script-path) can fix it independently.

Robustness / sequencing

11. Operator scripts drift silently — no version stamping [CONFIRMED]

This session reproduced exactly this regression: operator's local register-redirect-uris.ps1 predated the fix at scripts\install\register-redirect-uris.ps1:39-46 that adds $WebRedirectInstall. The local script ran "successfully" but never added the install URI, causing AADSTS50011 on operator sign-in.

Fix:

• Put a # version: N banner at the top of each script.
• Have /install/scripts/<name> serve a Last-Modified header and a discoverable version manifest.
• Surface "script as of <date>, version <N>" next to each Download button on the manifest page.
• Operators see drift at a glance.

12. Stale comment in manifest [TRIVIAL, CONFIRMED]

backend\src\lib\install\manifest.ts:294 says "ALL THREE secrets in one pass" — the script now writes five. Cosmetic but a signal that the comment-vs-code drift isn't being caught.

Fix: update the comment when fixing anything else in the file.

13. Install configurator should manifest-check its own auth dependencies [OBSERVED]

Today the manifest checks Entra secrets but doesn't check whether the frontend app reg actually has groupMembershipClaims=SecurityGroup (or ApplicationGroup, post-workaround) and the groups optional claim. Either the install backend's OIDC flow needs these to work, or the workaround for #6 reduces dependence on them — but as long as the install flow relies on them, the manifest should verify them.

Fix: add manifest items that Graph-introspect the frontend app reg's optionalClaims.idToken and groupMembershipClaims. Mark as fail if groups is missing or groupMembershipClaims is unset. These would have caught the user's lockout before the first sign-in attempt.

Architectural / longer-horizon

14. Provision Entra app regs from /install directly via admin-consent OIDC

scripts\install\create-entra-apps.ps1 is ~300 lines of operator-facing PowerShell, requires the operator to download + Unblock-File + run with the right parameters, and bakes in script-drift risk (Fix #11) by definition. The same logic could run server-side in the install backend:

1. /install offers "Sign in with Microsoft to provision app registrations" alongside the bootstrap-token form.
2. OIDC flow requests Application.ReadWrite.OwnedBy scope (Entra shows admin-consent prompt — typical marketplace buyer is Global Admin / Cloud Application Administrator).
3. Backend uses the resulting access_token to call Graph: create backend app, create frontend app, stamp scope, stamp groups claim, register redirect URIs, mint client secret, grant UAMI ownership of frontend app.
4. Writes the five KV secrets in-process.
5. Redirects to the configured /install checklist with every Entra row already green.

Eliminates: create-entra-apps.ps1, register-redirect-uris.ps1, the bootstrap-token path as the primary onboarding step, the UAMI-ownership-missing problem (Fix #10), and the script-drift problem (Fix #11).

Keep the scripts as documented break-glass for restricted tenants (operator's tenant has "Users can register applications: No" enabled — only an admin can perform the grant, and even Application.ReadWrite.OwnedBy won't work). Bootstrap token survives as secondary escape hatch.

Cost: ~200-300 lines of TypeScript in backend\src\routes\install.ts using @microsoft/microsoft-graph-client. Replaces an even larger volume of operator-facing PowerShell.

15. Reconsider whether AUTH_PROVIDER should be an env var at all

Once Fix #1 lands, the backend auth mode is functionally determined by which KV secrets are populated. process.env.AUTH_PROVIDER becomes redundant with the more authoritative KV state. Deriving the mode at runtime eliminates the "set the secrets but forgot the env var" failure mode entirely (Fix #5).

Not blocking; do once Fixes #1 and #5 are in.

New feature — custom-domain wizard in /install

19. Build a custom-domain wizard for /install [DESIGN]

The current install configurator validates Entra apps, KV secrets, group IDs, etc., but custom-domain setup (point a friendly hostname like miketest1.altien.com at the backend) is entirely tribal knowledge. Today an operator has to:

1. Pick a hostname and add a DNS record at their DNS provider (Cloudflare, Route53, GoDaddy, …)
2. Read Azure's customDomainVerificationId and add a second DNS record for domain-ownership proof
3. Run az containerapp hostname add to register the hostname on the Container App
4. Decide on cert strategy (managed cert / BYO) — and if managed-cert + Cloudflare proxied, learn that HTTP-01 won't work and pick DNS validation
5. Read another verification token from Azure and add a third DNS record for cert provisioning
6. Run az containerapp managed-cert create and az containerapp hostname bind
7. Add https://<hostname>/api/auth/openid-callback/microsoft and https://<hostname>/login to the frontend app reg's redirect URIs
8. Add https://<hostname> to backend CORS allowedOrigins (currently hardcoded from FRONTEND_URL env var)
9. Update Container App's FRONTEND_URL to the new hostname (revision restart)
10. Sign out everywhere, sign back in, verify

Ten manual steps across four UIs (DNS provider, Azure portal/CLI, Entra portal, Container App config). Error-prone, no progress indication, partial states leave broken installs.

Wizard UX shape

A new section on /install titled "Custom domain (optional)", with a single-row entry point that expands into a multi-step in-page wizard:

┌─ Custom domain ─────────────────────────────────────────────────┐
│  ○ Not configured — backend reachable at default URL only       │
│                                              [ Set up domain ]  │
└─────────────────────────────────────────────────────────────────┘

When clicked, replaces the row with a stepper UI:

┌─ Custom domain setup ───────────────────────────────────────────┐
│                                                                 │
│  [✓] 1. Choose hostname                                         │
│  [✓] 2. Add DNS records at your provider                        │
│  [▶] 3. Verify and register on Container App                    │
│  [ ] 4. Provision TLS certificate                               │
│  [ ] 5. Update auth (Entra redirect URIs + FRONTEND_URL)        │
│  [ ] 6. Test end-to-end                                         │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Each step renders inline, with: state (queued/in-progress/done/failed), what's happening, what the operator needs to do, and exact records to paste into their DNS provider where applicable.

Per-step content

Step 1 — Choose hostname

• Single input: Hostname (e.g., miketest1.altien.com)
• Validation: valid DNS hostname; not already bound to this Container App; not the backend's default FQDN
• Hidden field: useCloudflare checkbox — when checked, the next step's copy explains the Cloudflare-proxied caveat (DNS validation required for cert)

Step 2 — DNS records at your provider (this is the "explain Cloudflare/etc." panel the user asked for)

Inline panel with two records to add. Each record is a copy-button + tabular display + provider-specific notes:

Record 1 — Points your hostname at the backend
─────────────────────────────────────────────
Type:    CNAME
Name:    miketest1                    (provider strips zone automatically)
Target:  backend.nicedune-XYZ...      [Copy]
TTL:     Auto / 300s
Proxy:   ⚠ Cloudflare-specific — see note below

Record 2 — Proves you own this domain (Azure verification)
──────────────────────────────────────────────────────────
Type:    TXT
Name:    asuid.miketest1
Content: E633926AA75A097E59EEE685BE17D763A2446D9907CC1B899F515610AEC067BC  [Copy]
TTL:     Auto / 300s
Proxy:   N/A (TXT records cannot be proxied)

Cloudflare-specific note (rendered when "Cloudflare" is selected from a provider dropdown):

Set Record 1's Proxy status to Proxied (orange cloud) if you want Cloudflare's CDN/WAF features. Container Apps' managed-cert provisioning uses DNS validation (Record 3 in the next step), so Cloudflare-proxied mode is fully supported — you don't need to flip the proxy off.

Provider dropdown: Cloudflare / Route 53 / GoDaddy / Other. Each provides 1-2 lines of provider-specific guidance (e.g., Cloudflare's "use @ for root domain" warning, Route 53's hosted-zone considerations).

Operator confirms records are saved → "Continue" button.

Step 3 — Verify and register on Container App

• Background: az containerapp hostname add equivalent (the install backend uses its own UAMI to call ARM directly)
• Status display: "Polling DNS to confirm records have propagated…" with retry every 10s for up to 5min
• On success: green check, hostname appears in the Container App's customDomains list
• On failure: shows the specific DNS lookup that failed + remediation hint

Step 4 — Provision TLS certificate

• Sub-choice: Managed (free, auto-renewing) (default) / Bring your own
• For managed: install backend initiates Microsoft.App/managedEnvironments/certificates provisioning via Graph/ARM. Container Apps generates a CNAME validation record; the wizard surfaces it as Record 3:

  Record 3 — TLS cert validation (DNS challenge)
  ──────────────────────────────────────────────
  Type:    CNAME
  Name:    _acme-challenge.miketest1
  Target:  <provided-by-Azure>.azurecontainerapps.io     [Copy]
  Proxy:   ⚠ DNS-only (grey cloud) for validation; can be flipped later
  

  Operator pastes into DNS provider → wizard polls for the record → cert provisions (~2-5 min) → green check.
• For BYO: file upload widget for PFX + password, validates expiry/SAN matches hostname, uploads to env.

Step 5 — Update auth (Entra redirect URIs + FRONTEND_URL)

Two automated sub-actions, shown in a checklist with current state and a "Apply" button:

• [ ] Add https://<hostname>/api/auth/openid-callback/microsoft to frontend app reg's web redirect URIs (requires Graph access; install backend already has it via the UAMI)
• [ ] Add https://<hostname>/login to frontend app reg's SPA redirect URIs
• [ ] Update Container App env var FRONTEND_URL to https://<hostname> (causes one revision restart)

After "Apply": each sub-action shows pending → done, with detail (e.g., "added URI to app reg 267c319b-...").

Step 6 — Test end-to-end

• "Open https://<hostname>/install in a new tab" button — verifies the wizard's own host can be reached
• Shows latency to confirm the cert is valid and Cloudflare is proxying
• Final green-check + collapse back into the original /install checklist row, now showing Custom domain: miketest1.altien.com ✓

Cloudflare proxy vs. cert validation — critical caveat surfaced 2026-05-12

Reproduced this session. With Cloudflare's orange-cloud (proxied) mode active for the hostname CNAME, Container Apps cannot complete managed-cert validation — either method (CNAME or TXT) fails because Container Apps' pre-flight check requires the hostname's CNAME chain to be externally resolvable, and Cloudflare proxy hides it (the world sees Cloudflare's edge A records, not the *.azurecontainerapps.io target).

Concrete failure modes observed:

• --validation-method CNAME fails fast with (FailedCnameValidation) Not found CNAME of hostname 'miketest1.altien.com' directly pointing to a default hostname.
• --validation-method TXT stays in Pending for ~15 min then transitions to Failed with error: "Operation timed out." — the validation order expires before Azure can confirm the chain.

Resolution: flip Cloudflare from Proxied → DNS-only (grey cloud) for the hostname CNAME, wait ~30s for DNS to propagate, then az containerapp env certificate create --validation-method CNAME. Cert provisions in ~3-4 min. After it's Succeeded and bound, flip back to Proxied.

This is what the wizard MUST automate or document prominently. The renewal cycle (~3 months) hits the same wall: managed certs use the same validation method at renewal. Two solid resolutions for the wizard:

1. Detect Cloudflare from DNS lookup before cert provisioning (miketest1.altien.com resolves to Cloudflare ranges 104.x / 172.x / 2606:4700::/32) and prompt operator to flip to DNS-only for the next 5 minutes. Auto-poll cert state, tell operator when they can flip back. Set a calendar/email reminder for next renewal cycle.

2. Recommend BYO Cloudflare Origin Certificate as the default for Cloudflare-proxied installs. Cloudflare's Origin Certs are valid for 15 years and never need renewal. Avoids the proxy dance permanently. Trade-off: certificate-management UI complexity in the wizard; the operator has to generate the cert in Cloudflare and paste it.

Implementation notes

• New backend module backend\src\lib\install\customDomain\* for the orchestration logic. Pure server-side; the wizard is just HTML form posts + polling endpoints.
• Each step is restartable. State stored in KV (a single secret install-custom-domain-state holding JSON: { hostname, currentStep, txtRecordVerified, hostnameAdded, certId, certProvisioned, redirectUrisUpdated, frontendUrlUpdated }). Operator can leave the page mid-wizard and resume; the manifest row reflects partial progress.
• Failure handling: each Azure operation logs the underlying ARM/Graph error verbatim so operators (and support) can see exactly what failed.
• Idempotent: re-running any step is safe.
• Telemetry: count which steps fail in the wild; informs future copy improvements.

Why this matters

• The current "operator runs az containerapp hostname add + managed-cert create + hostname bind + remembers to update redirect URIs" sequence has ten distinct points of failure, each with their own opaque error. Operators report what they see; we burn support cycles diagnosing.
• The same DNS-verification-then-Azure-action pattern applies to email-domain setup, BYO storage account, etc. — this wizard is the prototype for a broader "guided setup" idiom in /install.
• For marketplace SaaS specifically, a custom domain is a near-universal requirement and currently the highest-friction post-install task.

Dependencies / sequencing

• Depends on Fix #17 (FRONTEND_URL ↔ redirect-URI cross-validation) in principle — the wizard subsumes that fix because the post-bind step explicitly updates both.
• Depends on the install backend's existing Graph access (install.ts:1273-1281 already caches an access token after admin sign-in) — required for the auto-update of redirect URIs in step 5.
• ARM access for the Microsoft.App/managedEnvironments/certificates and Microsoft.App/containerApps/*/hostname calls requires the install backend's UAMI to have Contributor (or a narrower custom role) on the Container App + the environment. Today the UAMI has KV permissions only. Adding ARM permissions widens its trust boundary — discuss.

Database initialization gaps (surfaced 2026-05-13)

20. db-migrate Container App job is never auto-triggered on greenfield deploy [CONFIRMED]

infra\modules\containerapp-job-migrate.bicep deploys a manual-triggered job. The Bicep parameter authProvider is wired in (with the comment "AUTH_PROVIDER=entra; in supabase/local mode they're harmless") but Bicep itself never calls az containerapp job start, so the job sits with zero executions after a marketplace install. The app's frontend hits 500 immediately because PostgREST tries to query roles/tables that the migration was supposed to create.

Concrete symptom on this install: PostgREST returned Failed to read user profile: role "web_anon" does not exist on every authenticated endpoint, blocking the entire main app.

Fix options:

1. Bicep deploymentScripts that runs az containerapp job start db-migrate post-deploy — fires once at greenfield, idempotent for migrations.
2. Install manifest item that detects un-migrated state (e.g., probe Postgres for the web_anon role) and surfaces a "Run migrations" button. Cleaner UX, also catches "fresh migrations need to run after upgrade" cases.
3. Container App init container that runs migrations on first start of the backend. Self-contained but mixes concerns.

Option 2 is the wizard-aligned answer.

21. db-migrate job's AUTH_PROVIDER defaults to supabase and has no DATABASE_URL wired [CONFIRMED]

Even when manually triggered, the job fails immediately with DATABASE_URL is required when AUTH_PROVIDER is not 'entra' because:

• Job env has AUTH_PROVIDER=supabase (inherits Bicep's bad default, same as Fix #14)
• Job env has no DATABASE_URL env var (Bicep doesn't wire one)
• Job has zero secrets configured (verified via az containerapp job show --query properties.configuration.secrets)

Compare with the backend Container App, which has FIVE secrets KV-mapped (anthropic-api-key, openai-api-key, supabase-secret-key, openai-base-url, appinsights-connection-string). The migration job got none of that.

Fix: Bicep should wire a database-url secret on the migration job (KV ref to pgrst-db-uri or a dedicated db-admin-url secret) AND set DATABASE_URL=secretref:database-url on the env. With Fix #14 also done (authProvider default flipped or made required), the script's AUTH_PROVIDER check would route correctly. Even better, Fix #16's "main app reads from KV" architecture extended to the migration script eliminates the env-var dependency entirely.

22. Migration container's DATABASE_URL needs ?sslmode=require but the canonical KV value (pgrst-db-uri) lacks it [CONFIRMED]

Postgres Flexible Server requires TLS for connections from outside the server's own subnet (no encryption error from pg_hba.conf). The KV secret pgrst-db-uri is postgres://mikeadmin:...@pg-mike-test1.postgres.database.azure.com:5432/postgres — no ?sslmode=require. node-postgres defaults to non-SSL when the parameter is missing, so the migration fails with code 28000 ("invalid_authorization_specification").

PostgREST itself apparently negotiates TLS automatically (didn't fail with this), but the migration script does not.

Fix: append ?sslmode=require to pgrst-db-uri in Bicep/KV-seeding. Belt-and-braces: have the migration runner default-add sslmode=require to DATABASE_URL if not present.

23. Postgres Flexible Server provisioned with Entra auth disabled [CONFIRMED]

Bicep parameter authProvider='entra' would suggest the install supports Entra-based Postgres auth, and there's a param pgMigrationMiUsername string for the migration MI's Postgres role name (main.bicep:100). But on this install, az postgres flexible-server show returned:

"authConfig": {
  "activeDirectoryAuth": "Disabled",
  "passwordAuth": "Enabled",
  "tenantId": null
}

So Bicep is parameterising for an entra-auth path that the marketplace install never enables. Either:

• The marketplace install defaulted to supabase/password mode end-to-end (consistent with #14, #21)
• Or Entra auth on Postgres requires post-deploy steps (enable AD auth, add admin, create role for UAMI) that Bicep doesn't handle and the install configurator doesn't surface

Fix: pick a lane. Either (a) Bicep fully provisions Entra auth on Postgres + adds the UAMI as a Postgres principal (this needs an az postgres flexible-server execute step OR a deploymentScript), or (b) accept that password auth is the supported mode and remove the entra-auth migration scaffolding to reduce surface area. Mixed-mode "params exist but aren't honored end-to-end" is the worst of both.

Partial resolution (2026-05-18): lane (b) chosen for the migrate job specifically — see 037-migrate-job-auth-decoupling.md. runMigrations.ts now prefers DATABASE_URL (KV-sourced admin URL) and falls back to MI-token auth only when PG_HOST + PG_MI_USERNAME are set. Surfaced via failed install in rg-mike-test2 on image 1.0.3.

24. PostgREST is configured with admin credentials in pgrst-db-uri [SECURITY CONCERN, CONFIRMED]

The KV secret pgrst-db-uri resolves to postgres://mikeadmin:<password>@... — i.e., the Postgres superuser. PostgREST design assumes a dedicated authenticator role with the ability to SET ROLE to web_anon / user roles, but no inherent privileges of its own. Giving PostgREST the admin role means any successful auth bypass in PostgREST = total compromise of the database.

This is a 2026-05-13 finding but applies broadly across Mike installs that share this scaffolding.

Fix: migration should create an authenticator role with the minimal LOGIN + SET ROLE permissions, and pgrst-db-uri should connect as that role. The migration job needs admin creds (legitimately); PostgREST does not.

32. Install configurator has no step-by-step narrative — checklist only [OPERATOR UX]

/install shows a flat list of status badges grouped by section. There's no "start here", no progress indicator, no explanation of what each section does in plain English, no "you're done" final state. Operators have to derive the right order from requires fields and status badges. For an engineer this is fine. For the marketplace buyer it's hostile.

Fix paths:

• Progress banner at top: "Step N of M complete · Estimated 5 minutes remaining"
• Section intros: 1-2 sentences in plain English explaining what each section configures
• Inline narrative cues between sections ("When the rows above are green, move on to ...")
• Final "Install complete — go to your app" state with a CTA button to the main app

Minimum viable: banner + section intros + done state. Doesn't need to be a full wizard. See also gap #19 (custom-domain wizard) which is the bigger "guided flow" idea for a specific sub-task; #32 is the always-on framing.

31. Install configurator copy is engineer-jargon, not buyer-language [OPERATOR UX]

Every row's label, helpText, and section header is written for someone who already knows what an Entra tenant / app registration / OAuth scope is. Examples:

• Section header Entra ID — meaningless to non-IT buyers; should be "Microsoft sign-in"
• Row label Backend app registration — assumes knowledge of Entra concepts; should be something like "Microsoft sign-in identity for Mike's backend"
• helpText for tenant id: "GUID of the Entra tenant. Find it in Azure portal → Microsoft Entra ID → Overview." — assumes the operator knows what a "GUID" is, what "Entra tenant" means, and how to navigate the portal

Fix: copywriting pass on every manifest entry. Plain English in labels and helpText; technical strings preserved where they're needed for az commands or KV secret names (kept inside <code> tags). Section headers renamed to plain English: Foundations → Core setup, Entra ID → Microsoft sign-in, Tenant policy → Access rules, Lifecycle → Cleanup, Optional → Optional.

Doesn't change behavior; doesn't touch tooling; just text. But it's the highest-leverage change for non-engineer buyers.

30. Set values are invisible until you click Edit [OPERATOR UX]

checkKvSecret deliberately returns { status: "pass" } with no detail when a value is set (manifest.ts:29-31: "Pass with no detail — the previous 'length=N' output was operator-noise (no concrete value, no actionable signal)."). The intent (drop length=42 noise) was right; the swing to "literally nothing" was too far. A green row tells you the field is set but not WHAT it's set to — operators can't verify at a glance that the tenant ID is the right one, that the admin group is the intended GUID, etc.

Fix: each manifest entry specifies how to display the value when set:

• GUIDs (tenant id, group ids, client ids) → full GUID
• URLs / FQDNs (backend-public-url, FRONTEND_URL) → full URL
• Display-name-annotated values (<guid> # Admins) → display name + truncated GUID
• Secrets (client-secret, auth-state-secret) → redacted (••••)
• Onboarding mode / auth provider → full literal (auto, entra)

Implementation: add a displayValue? formatter or redacted? flag to checkKvSecret's options; default behavior continues "show full value" for safe types, opt into redaction for secrets.

29. Marketplace install doesn't auto-launch /install for the buyer [OPERATOR UX]

Bicep emits installUrl as an output (main.bicep:264). The Azure portal shows it in the deployment's Outputs tab, but the marketplace SaaS subscription page doesn't prominently surface it. Non-technical buyers who completed an install have no obvious next step — they have to find the deployment, find the outputs tab, copy the URL.

Fix:

• In this repo: nothing to fix beyond making the output description more prominent — "OPEN THIS URL TO COMPLETE SETUP" rather than the current low-key comment. Bicep can't pop a browser.
• In the marketplace listing config (outside this repo): the publisher's createUiDefinition.json or post-deploy artefact should surface the install URL as the primary CTA. This is a publisher pipeline task.

Add an output description that makes the URL self-explanatory for any tooling that consumes it.

28. PostgREST authorization is delegated entirely to the backend middleware chain — no enforcement at the data layer [SECURITY, STRUCTURAL]

The entra-mode design (issue 014) made an intentional trade: PostgREST runs every query as a fixed role (PGRST_DB_ANON_ROLE, now service_role which has BYPASSRLS per migrations/0005_postgres_roles.sql:28), with no per-request JWT. Security is provided by:

1. Network isolation — PostgREST is only reachable from inside the VNET (the Container App Environment's internal subnet)
2. Backend middleware — every route that touches PostgREST is expected to go through tenantAccess (backend/src/middleware/tenantAccess.ts) or equivalent, which gates on tenant membership + group whitelist

This means PostgREST itself has no way to refuse an unauthorized request. If a backend route reaches PostgREST without first going through tenantAccess, PostgREST will happily execute the read/write — there is no second line of defence in the database.

A legitimate exception is /api/auth/callback and similar pre-session handlers (e.g., the user-profile upsert that runs during sign-in, before a session cookie exists, so tenantAccess literally cannot evaluate res.locals.principal because it isn't populated yet). This session's 2026-05-13 reproducer hit exactly this path: the user-profile upsert succeeded as anonymous → service_role → full table access, before any group check fired.

The risk: any backend route that calls into the PostgREST client and isn't mounted behind tenantAccess is an authorization bypass. A future PR that adds a new route — say /api/foo/bar — and mounts it without the middleware would let any unauthenticated request (or any request from a non-whitelisted tenant or group) read or write the database. PostgREST has no way to detect or refuse it.

This is fundamentally a "the developer must remember to apply the middleware" trust model, which is fragile. The mitigation has to be in tooling, not memory.

Required defences

Multi-layered, smallest to largest:

1. Pre-merge review checklist item — docs/dev/checklists/pre-merge-review.md (or add to existing): "Any new backend route that imports createServerSupabase or otherwise calls into PostgREST must be mounted behind tenantAccess middleware, OR explicitly justified as a pre-session exception (with the justification in the PR description)."

2. Static analysis / lint rule — an ESLint rule (custom, or eslint-plugin-import-zones) that flags any file in backend/src/routes/** that imports ../lib/supabase and isn't on an allowlist of known-protected mount points. Allowlist tracked in eslint.config.mjs with comment-justifications.

3. Runtime invariant test — backend/src/__tests__/auth-middleware.invariant.test.ts. At test setup, walk the Express router tree on the assembled app, and for each handler, inspect the middleware stack via router.stack[i].handle.toString() or Express's internal _router introspection. Fail if any non-allowlisted route has a handler that closes over createServerSupabase without tenantAccess in its middleware chain.

4. Production-time observability — add a metric mike.postgrest.requests with dimensions {route, principal_present, tenant_id_present}. Alert if principal_present=false on a route that isn't on the auth-callback allowlist. Catches the "regression slipped past review + lint + test" case.

File touchpoints

• backend/src/middleware/tenantAccess.ts — current middleware, no changes needed, just becomes mandatory upstream of PostgREST consumers
• backend/src/lib/supabase.ts — single import point for the PostgREST client; make it the lint marker
• backend/src/routes/* — audit existing routes for the invariant; expect 1-2 legitimate exceptions (auth callback, install configurator routes that have their own session model)
• New: backend/src/lib/dev/auth-middleware-invariant.ts — the runtime invariant checker, plus its test file
• docs/dev/checklists/pre-merge-review.md — the checklist item

Priority

This is not blocking marketplace install — the current routes all go through the right middleware as far as we know. It's preventing the next regression, which is far harder to catch after the fact. Recommend doing this BEFORE adding any significant new backend feature, especially anything tenant-facing.

27. Systematic env-var/KV mismatch sweep — Bicep wires 0/4 high-impact bridges that the backend code needs [CONFIRMED 2026-05-13]

Audit done this session. Enumerated all process.env.<X> references in backend/src/, intersected with KV secrets named <x> (snake-cased), checked which env vars the backend Container App actually has set. Result: four env vars the backend code reads, with corresponding KV secrets that the install configurator populates, but no Bicep wiring:

This is in addition to the per-symptom gaps already documented (#13–14 for AUTH_PROVIDER, #17 for ENTRA_*, #25 for PGRST_DB_ANON_ROLE, #26 for TENANT_ONBOARDING_MODE). The total: at least 10 distinct env vars that Bicep doesn't wire correctly for entra mode.

This is the single highest-leverage fix in the inventory. Don't paper over each gap individually:

• The correct architectural fix is #16: change backend code so the runtime values come from KV (via getConfig()) at request time, not from process.env at process start. The install configurator already writes to KV; the install configurator validates KV; the install configurator's UX promises that setting the value works. Right now half of those promises are broken because the value never reaches the running code.
• The wrong fix is to wire each missing env var in Bicep one-by-one as we discover them. That's what we've been doing reactively all session. It works locally but reproduces the same architecture in the marketplace: brittle env-var sprawl, secrets that have to exist before Bicep runs but get populated after Bicep runs (chicken-and-egg with create-entra-apps.ps1), no single source of truth.

Workarounds applied to THIS install (2026-05-13, all via az containerapp update --set-env-vars / secret set):

• AUTH_STATE_SECRET=secretref:auth-state-secret (added Container App secret with KV ref)
• BACKEND_PUBLIC_URL=https://backend.nicedune-6314ad94...
• ENTRA_ADMIN_GROUP_IDS=60c62ed1-...
• ENTRA_MEMBER_GROUP_IDS=7e061842-...

These workarounds are disposable. The actual fix lives in the codebase.

26. TENANT_ONBOARDING_MODE env var never wired; middleware reads env not KV [CONFIRMED]

backend\src\middleware\tenantAccess.ts:48 reads process.env.TENANT_ONBOARDING_MODE ?? "manual". KV has a tenant-onboarding-mode secret which the install configurator surfaces and validates (manifest.ts:447-470), but the env var is never wired by Bicep. So even when an operator sets KV to auto (the install configurator's stated purpose), the middleware sees undefined, falls back to manual, and rejects every cross-tenant sign-in with TENANT_UNKNOWN.

This is the same architectural split called out in Fix #16. The middleware (like auth.ts) should read from KV via getConfig(). Either fix:

• A (preferred, same as #16): middleware reads tenant-onboarding-mode from KV. No env var needed; install configurator's value is authoritative.
• B: Bicep wires TENANT_ONBOARDING_MODE env var from the tenant-onboarding-mode KV secret (secretRef). Bridges the gap but doesn't fix the architectural issue.

Workaround applied in this install: az containerapp update --set-env-vars TENANT_ONBOARDING_MODE=auto on the backend.

25. PGRST_DB_ANON_ROLE=web_anon in entra mode causes every write to fail [CONFIRMED, KNOWN FOR ≥1 WEEK]

This was diagnosed and fixed on 2026-05-05 in session e40de1d0 (local docker-compose stack) and again on 2026-05-06 in the same session (deployed PostgREST). The prior-session note: "This is another 023 manifest item I should add: postgrest-anon-role-correct — required when AUTH_PROVIDER=entra." It never landed in the marketplace Bicep.

Mechanism (from issue 014's "entra-mode trust model"):

1. In entra mode, the backend deliberately strips auth headers before talking to PostgREST (security is provided by network isolation — PostgREST is only reachable from inside the VNET).
2. PostgREST therefore receives every request as anonymous.
3. The anonymous role is whatever PGRST_DB_ANON_ROLE env var names.
4. Bicep sets it to web_anon (the supabase-mode default), which is SELECT-only per migrations/0005_postgres_roles.sql:49.
5. Any INSERT/UPDATE/DELETE → "permission denied for table user_profiles" (or any other table). Reads work; writes don't.

Fix already applied locally on 2026-05-13 in this install: az containerapp update --name postgrest --set-env-vars PGRST_DB_ANON_ROLE=service_role. service_role is defined as nologin bypassrls (migrations/0005_postgres_roles.sql:28) — full table access, safe because PostgREST is VNET-only in entra mode.

Permanent fix (the manifest item that was promised but never delivered):

• containerapp-postgrest.bicep should set PGRST_DB_ANON_ROLE conditionally on authProvider:
  • entra → service_role
  • supabase / local → web_anon (current default, correct for those modes because JWTs DO travel and PostgREST routes by claim)
• AND add a manifest item to /install: postgrest-anon-role-correct — check reads the env var on the running PostgREST Container App, fails if it doesn't match the authProvider mode.

Critical assessment: this is exactly the failure mode the manifest is supposed to catch. The bug was diagnosed, the fix was identified, the manifest item was named — and a week later, a new install hits it again. The doc-to-Bicep handoff is the actual bottleneck for closing these.

Unresolved in 2026-05-12 session — needs follow-up

16. After Entra sign-in at /install, operator lands on paste form instead of checklist [UNRESOLVED]

After full fresh sign-in (MFA challenged & passed), user ended up at /install with the bootstrap-token paste form — not the 403 page and not the checklist. Code review of install.ts:1166-1291 shows no silent redirect path — every error branch renders a titled error page. Only the success path issues a 303 to /install after setting the install-session cookie at install.ts:1266 via issueInstallSession(res, "entra") (installAuth.ts:54-82, path=/install, sameSite=lax, secure when NODE_ENV=production).

Possible causes (not yet narrowed):

• Cookie set but browser drops it on the 303 hop (security policy edge case)
• Cookie set but middleware doesn't read it (less likely — bootstrap flow uses the same cookie reader)
• req.protocol returning http behind Container Apps TLS ingress, causing some downstream issue — but redirect_uri works, so probably not
• Cookie set but path/scheme combination rejected — investigate browser DevTools Network tab + Application → Cookies for the actual Set-Cookie header on the callback response

Next step: instrument with DevTools Network tab during a real sign-in; check whether mike-install-session cookie appears in the Set-Cookie header on the /install/auth/microsoft/callback response AND whether it's stored in the cookie jar afterwards. If set on response but not stored, browser is rejecting it (likely SameSite/Secure mismatch). If stored but not sent on /install GET, middleware isn't reading it. If absent from response entirely, callback hit some error path my code review missed.

17. FRONTEND_URL ↔ frontend-app-reg redirect URIs not cross-validated [OBSERVED 2026-05-12]

FRONTEND_URL is a Container App env var set from Bicep param frontendUrl at install time (main.bicep:76, threaded through containerapp-backend.bicep:83). It's used by the frontend bundle to construct OIDC returnUrls. But:

• The install manifest doesn't check whether the host of FRONTEND_URL appears in the frontend app reg's web.redirectUris.
• The marketplace deploy typically takes a frontendUrl value the operator intends to use post-deploy (custom domain pointed via Cloudflare / A record / CNAME). That DNS wiring happens out-of-band from the Azure deploy.
• Until DNS is in place AND the app reg has the matching redirect URI, the main-app sign-in flow appears to half-work: Entra successfully authenticates (its redirect_uri is the backend FQDN, which IS registered), but the final hop back to returnUrl lands on an unresolvable domain.

In the 2026-05-12 install, FRONTEND_URL=https://miketest1.altien.com while the app reg has mike.altien.com (leftover from a different install). Operator intent is to wire miketest1.altien.com via Cloudflare and add it to redirect URIs once that's done.

Fix: add a manifest item that compares URL(process.env.FRONTEND_URL).host against the redirect URIs read via Graph (the same Graph call slice-9 already makes). Report:

• pass — FRONTEND_URL host appears at least once in web.redirectUris AND spa.redirectUris
• info with diagnostic — appears in some but not all expected paths
• fail — host doesn't appear at all; surface the exact https://<host>/api/auth/openid-callback/microsoft + https://<host>/login URIs to register

Less critical than the architecture fixes above, but it would have caught the leftover mike.altien.com URIs in this install and saved the operator from a confusing "sign-in completes then breaks at the end" experience.

18. Install start route uses prompt=select_account, not prompt=login [THEORETICAL]

install.ts:1162 sets prompt=select_account, which asks Entra to show the account picker but does NOT force a fresh token mint. If Entra returns a cached id_token (from before optional-claim or app-reg-assignment changes landed), isInAdminGroup evaluates on stale claims. Suspected during this session but not confirmed.

Fix: change to prompt=login on the install flow (forces re-authentication AND fresh claims). Slightly worse UX for operators (they re-enter password) but during install they're authenticating once and never again until token expiry — acceptable tradeoff.

Suggested sequencing for a future PR cycle

1. Quick wins (independent, low-risk): Fix #3 (Bicep default), Fix #4 (scope row), Fix #12 (stale comment), Fix #17 (prompt=login).
2. Architecture fix: Fix #1 (auth.ts reads from KV). Unlocks Fix #2 (Bicep stops needing to wire env vars) and Fix #5 (auth-provider manifest item) and Fix #15.
3. Robustness: Fix #6 (Graph fallback for groups overage) — the actual root cause of this install's lockout. Fix #7 (better 403 page).
4. Sequencing-sensitive: Fix #8 (marketplace-installer identity in Bicep). Fix #13 (manifest checks app-reg claim config).
5. Operator-script ergonomics: Fix #10 (split UAMI grant), Fix #11 (version stamping).
6. Investigation: Fix #16 (install session-cookie mystery).
7. Architectural: Fix #14 (in-app Entra provisioning) — biggest payoff, biggest scope.

Each item is small enough to PR independently. The architecture fix (#1) is the highest leverage of the lot because it removes a whole class of "I set the value in KV but the main app doesn't see it" problems and lets Bicep stop trying to wire something it fundamentally can't (the KV secrets don't exist at Bicep deploy time — only after create-entra-apps.ps1 or its successor runs).

Evidence captured in this session

• Operator UPN: amorgan@altien.com
• Operator OID: dda38dd7-1e15-43bd-9bca-f46de548bcaa
• Tenant: 7994793f-3dc8-4db8-ae7d-68ba9898985d (Altien Ltd.)
• KV: kv-mike-test1
• Backend Container App: backend in rg-mike-test1
• Frontend app reg appId: 267c319b-8135-46cf-82e8-25a9570745b0
• Backend app reg appId: 5e7f4433-4673-4a0d-8946-c119586eb5b1
• Admin group: 60c62ed1-932e-417e-a473-f9644448562c (Systems Administrators)
• Member group: 7e061842-7e52-4cf0-ae8c-746a477be676 (Altiens)
• amorgan transitive memberOf count: 57 (well under documented 200-group overage threshold; still triggered hasgroups: true — concrete reproducer for Fix #6)