Web release flow
How Today web freezes a release branch, validates lane staging, promotes a production candidate to today.ai, and backmerges fixes to dev.
dev is the trunk branch. Product work merges there through the normal PR and
merge-queue path.
Production releases do not use main. A release is frozen on a temporary
release/web-* branch, promoted from a Vercel production candidate, and
archived by a protected web-prod-* tag. Release-only fixes return to dev
through a controlled backmerge PR.
On Vercel, Web and Embed are one exact-SHA deployment unit. Every Web Preview, lane, staging, candidate, and production build deploys Embed from the same checkout. Web builds once, then the workflow deploys Embed followed by the prebuilt Web artifact as the composed candidate. Embed executes its namespaced BFF directly through shared package logic; Web is not its runtime upstream. This order is required because Vercel resolves the available child fallback when the default-app deployment is created; a child deployed afterward is not retroactively attached. Only the composed Web URL may receive candidate or stable aliases.
Embed reports the shared SHA from /api/embed/health. The paired action checks
that endpoint both on the child deployment and through the composed Web URL, so
a failed deployment or broken Microfrontends association fails the owning Web
workflow before any stable Web domain moves.
/embed/build-info.json is the public, shallow identity endpoint for the Embed
artifact selected by Microfrontends routing. It reports the standard build
metadata plus application: today-embed without contacting Web. The protected
/api/embed/health endpoint remains the release gate because it also proves the
composed ingress selects the expected Embed artifact.
The today-web-embed Microfrontends group uses today-web as the default app
and SAME_ENV fallback. Both projects are connected to the same GitHub repo,
with Vercel roots apps/web and apps/embed; Vercel needs that metadata to
associate Preview deployments by commit and environment. Removing either Git
connection causes composed verification to fail closed. Embed still declares
git.deploymentEnabled: false; the connection supplies association metadata,
while the paired GitHub workflows remain the only deployment authority.
The paired action synchronizes TODAY_DEPLOYMENT_VERIFICATION_SECRET into the
Embed Vercel project's target environment as a sensitive variable, using the
Protection Bypass for Automation secret over stdin. The protected health route
uses it to authenticate deployment verification. The action fails closed if the
sensitive variable is not present after the project environment is pulled;
Vercel intentionally does not make sensitive values available for plaintext
comparison. VERCEL_PROJECT_ID_EMBED is required in every paired workflow.
Lane staging also passes one explicit Auth/API/lane profile from the Web
workflow into the paired action. The action requires the complete profile,
overrides the Embed build environment with the same APP_ENV, traffic lane,
App URL, API URL, and public/internal OIDC authority, and repeats those values
as deployment runtime variables. Protected health verification compares the
effective Embed profile with the Web profile before either candidate or stable
staging aliases move. Vercel project variables remain the baseline for manual
inspection, but they are not the deployment workflow's source of truth.
The current Web bundle uses /api/embed/auth/session. A transitional
/api/auth/embed-session alias lives only in Embed for already-open Web bundles;
native clients do not call either URL directly. Main Web owns neither handler.
The normal cadence is Monday cut and Thursday promotion. The release branch is
the week's first-parent append-only release line: keep ordinary fixes linear,
never rebase or force-push
it after cut, and allow it to lag the continuously moving dev trunk.
Naming
Use two names consistently:
- lane staging:
https://staging.today.ai, built from arelease/web-*branch withNEXT_PUBLIC_TRAFFIC_LANE=staging. Browser requests and BFF upstream calls sendX-Traffic-Lane: staging. - production candidate: a production-mode Vercel deployment built from the
same release branch without
NEXT_PUBLIC_TRAFFIC_LANE. It has a preview URL and is promoted tohttps://today.aionly after approval.
Both artifacts keep an immutable random preview URL and a workflow-context URL.
Lane staging uses *.staging.preview.today.ai; production candidates use
*.prod.preview.today.ai. The workflow-context URL is the review and GitHub
environment link. Stable traffic domains move only after the random URL passes
smoke: staging.today.ai for lane staging and today.ai through
vercel promote for production.
The lane staging artifact cannot be promoted to production because the browser bundle already contains the staging lane value. The production candidate is a separate rebuild from the release SHA.
The manual Deploy Web * Candidate workflows may build an arbitrary repository
ref for inspection, but they never move stable domains and do not count as
release evidence. Only this release flow may assign staging.today.ai or
promote today.ai.
Normal Release
- On Monday, run
Cut Web Release Branchfromdev. This createsrelease/web-YYYYMMDDat the chosendevSHA and freezes the week's release scope. Selectweekly; the workflow rejects any source other than the currentdevtip and refuses to cut while staging still points to an unpromoted release SHA. - The release branch push starts CI and
Deploy Web Lane Staging. The workflow builds against Vercel's customstagingtarget withAPP_ENV=production, production API/auth settings, andNEXT_PUBLIC_TRAFFIC_LANE=staging; it assigns unique and contextual aliases under*.staging.preview.today.ai, smokes the unique alias, then aliases the deployment tohttps://staging.today.ai. Before aliases move, the workflow deploys Embed from the same release SHA, deploys the composed Web candidate, and verifies/api/embed/healthreports both that SHA and the matching Auth/API/lane profile through both ingresses. Neither staging deployment runsvercel promote; promotion is reserved for the separate production workflow. Confirm theCI Successaggregate job, then manually dispatchE2E Fullfrom the release ref againsthttps://staging.today.ai; the release deploy does not trigger Full E2E automatically. - During validation, add release-specific fixes only through linear PRs to the
release branch. Unrelated work continues to merge to
dev. Do not rebase, force-push, or mergedevinto the release branch. - On Thursday, after the prerelease checklist and human go/no-go approval, run
Promote Web Productionfromdev, passing the frozen release branch assource_ref. This keeps deployment automation trusted while the artifacts still come from the exact release SHA. - The workflow builds Web and Embed production candidates from the release SHA
without a traffic lane, assigns Web unique and workflow aliases under
*.prod.preview.today.ai, and smokes the unique alias. - When required reviewers are configured, the
web-productionenvironment approval pauses the workflow. Validate the candidate preview URL, then approve. - The workflow checks that the release branch still points at the same SHA,
promotes the paired Embed candidate first and then the Web candidate,
verifies
https://today.ai, creates a protectedweb-prod-*tag, and opens the required backmerge/backport PRs. - After production metadata, the tag, and all reconciliation refs match the promoted SHA, the workflow deletes the promoted source branch. A failed reconciliation setup or archive retains the branch and fails visibly.
For an explicitly approved direct-to-production release, manually dispatch
Promote Web Production with skip_staging enabled. This bypasses lane-staging
and Full E2E evidence only; exact-SHA CI Success, the production candidate
build and smoke, human production approval, production verification, tagging,
reconciliation, and archive checks still run. Record the skipped gate as a
release caveat.
After promotion, staging and production remain sourced from the same release SHA until the next release is cut. They are still separate builds: lane staging contains the staging traffic lane and production does not. The staging slot is logically idle during this interval; the next Monday cut moves it to the next release while production remains on the last promoted tag and SHA.
Hotfix Production Candidate
Hotfixes use Promote Web Hotfix, not the weekly production workflow. The
standard hotfix path intentionally skips lane staging and builds a production
candidate directly. After its immutable *.prod.preview.today.ai alias passes
smoke, the workflow moves hotfix.prod.preview.today.ai for human review and
pauses at the production approval gate.
Dispatch the trusted workflow from dev and pass the strict hotfix branch in
hotfix_ref. The workflow freezes that remote branch tip before building, so a
hotfix based on an older production tag does not need to contain the latest
deployment automation.
An optional isolated staging candidate is available when a fix specifically
needs the staging traffic lane. It receives only random and workflow-context
*.staging.preview.today.ai aliases. It never moves staging.today.ai, and the
staging artifact is never promoted to production.
Backmerge And Backport
Release fixes must land back on dev. The production workflow creates a
mechanical PR after promotion when dev does not already contain the promoted
release SHA. It uses a backmerge/web-* branch while the immutable
web-prod-* tag remains the durable release evidence.
Reconcile that PR against current dev semantically. If dev deleted or
replaced code that the release line modified, preserve the deletion while
resolving the production-tag merge and add focused tests. Do not merge dev
into the release branch, restore obsolete code, or treat a non-trivial
reimplementation as conflict cleanup.
After approval and green merge-result CI, run Merge Web Release Reconciliation from dev with the PR number and exact production tag. The
trusted workflow uses the existing Today Platform CI App to verify the PR,
immutable tag, source-side topology, current base/head SHAs, and CI-tested merge
tree. It then compare-and-set fast-forwards the target to a two-parent merge
commit whose first parent is the previous target tip and whose second parent is
the production tag. Normal PRs remain on rebase-only rules and the dev merge
queue.
gh workflow run merge-web-release-reconciliation.yml --ref dev \
-f pr_number=<approved-pr> \
-f production_tag=<web-prod-tag>The workflow materializes a clean merge automatically. If Git reports a conflict, prepare that exact two-parent merge on the existing reconciliation branch, resolve the result against the current target architecture, preserve the workflow-provided reconciliation trailers, push with lease, wait for refreshed PR CI, and rerun the workflow. Do not add commits after the prepared merge node.
Hotfix backports use the same workflow with an unpromoted strict weekly release as the target. This permits only production-tagged reconciliation merge nodes; the release first-parent line and all ordinary release fixes remain linear.
Archive Lifecycle
Keep a protected web-prod-* tag for every successful promotion. The tag is the
permanent Git archive and immutable production event; delete the promoted
release/web-* source branch only after production verification and creation
of every required reconciliation ref.
Keep web-abandoned-* tags as well. They are immutable evidence that an
unpromoted line was explicitly replaced; they prevent the retained abandoned
branch from blocking every future weekly cut.
Keep web-hotfix-open-* and web-hotfix-closed-* lifecycle tags. The cut
workflow creates the open marker atomically with the branch, and successful
promotion creates a closed marker for the exact promoted tip. These markers
make the single hotfix slot auditable even before its first fix commit.
Backmerge and backport branches are work branches only. Delete them after their PRs merge or close.
At any moment only one release branch is active. Historical branches created before automatic archival may still exist, so identify the active release by the latest successful lane-staging deployment, deployed build metadata, and absence of a production tag at that exact SHA, not by branch existence alone.
If production needs a hotfix, cut release/web-YYYYMMDD-hotfix-* from the exact
current production tag SHA. Only one unpromoted hotfix line may exist. A
correction before promotion advances the same line; a later hotfix starts from
the newly current production tag SHA. Never build it from staging or current
dev.
After promotion, reconcile the final hotfix behavior to dev through a
backmerge/web-* PR and to every active unpromoted weekly release through a
backport/web-* PR. A weekly release must not promote while it is missing a
production hotfix that applies to it.
Automated Guardrails
The release workflows share .github/scripts/web-release-state.mjs and enforce
these rules before mutating release state:
Cut Web Release Branchis globally serialized. A weekly cut must use the workflow-resolved currentdevSHA and a real calendar date inrelease/web-YYYYMMDD[.N]. An existing unpromoted strict weekly branch reserves the lifecycle before staging finishes. Same-day replacements use.2,.3, and so on. An emergency hotfix must userelease/web-YYYYMMDD-hotfix-*and start from the current productionweb-prod-*tag SHA.Deploy Web Lane Staginghas one global concurrency group because every run owns the samestaging.today.aialias. Same-branch updates must fast-forward. A different unpromoted release can replace it only withallow_staging_preemption=trueplus the exact current branch and full SHA.- Strict hotfix branches are excluded from the lane-staging push trigger and rejected by the staging ownership guard. Optional hotfix staging validation goes through the isolated candidate workflow only.
- A release Full E2E dispatch is accepted only when its URL is exactly
https://staging.today.ai. Promote Web Productionalways requiresCI Success. By default it also requires the latest successful staging run, stagingbuild-info.json, and a newer manually dispatched Full E2E run to match the weekly branch and SHA; an explicitly selectedskip_stagingbypass records that evidence exception.Promote Web Hotfixalways skips stable staging evidence but retains production candidate smoke, approval, SHA revalidation, production verification, tagging, and reconciliation PRs.- Every workflow that creates a Vercel Web artifact calls the shared paired
Embed deploy action. That action verifies the checkout SHA before building
Embed, redeploys the existing prebuilt Web artifact after Embed exists, and
verifies the deployed
/api/embed/healthSHA both directly and through the final Web deployment's composed ingress. Preview affected-package detection treats either Web or Embed as a change to the combined unit.
The repository's Protected Branches ruleset supplies the default branch-level
contract for release/*: no deletion, no force-push, linear history, PRs, and
required CI Success. Normal actors cannot bypass it. Trusted release workflows
use the Today Platform CI App only after verifying the protected production tag
and reconciliation topology: the promotion workflow archives the promoted
source ref, while the reconciliation workflow lands the one allowed class of
two-parent merge commit. Protected tag rules prevent release lifecycle tags
from being moved or deleted.
An intentionally abandoned weekly release is also explicit state, not an age
heuristic. Re-run Cut Web Release Branch with
allow_release_replacement=true and expected_abandoned_releases, with one
branch@fullSHA blocker per line. The guard requires the complete set to match
exactly, rejects stale or partial state, and records each abandonment with an
immutable web-abandoned-* tag. Retain both the abandoned branches and marker
tags. Existing markers are excluded from future blocker detection and authorize
the audited replacement branch to claim staging. A deleted recorded source is
also treated as an orphaned assignment, but only the explicit replacement cut
can clear the weekly release guard.
The staging concurrency group retains one running and one pending run. Rapid same-release pushes therefore converge on the newest SHA. Hotfixes do not enter this concurrency group or change the stable staging alias.
Merge queue
How GitHub's merge queue interacts with `gh pr merge` on this repo. The `UNSTABLE` vs `CLEAN` vs `BLOCKED` states, what cancels auto-merge, and the speculative-branch double-run.
CI affected scope
How `scripts/affected.ts` computes which packages a PR actually touches and how that scopes typecheck / test / lint / visual-regression on CI.