Orbi

Guide · how Orbi's CI gates work · engine source verified 2026-09-12

What you put in CI is what Orbi guarantees

Orbi's two gates — the review gate and the release gate — read one thing: the conclusion of each GitHub check run. They do not know pytest from Playwright: success, neutral or skipped passes, anything else blocks. So the suite you put in CI is exactly the promise Orbi enforces on every delivery — and with no check runs at all, both gates pass, which is the first boundary below. Here is the mechanism, two copyable workflows, and the four boundaries, each traced to the engine's code.

Is your Issue ai-ready? The 12 factors

The four boundaries first How a release runs, end to end →

THE MECHANISM

Two gates, one input: check-run conclusions

The review gate holds a clean review against the PR head's check runs before anything merges; the release gate holds the release against the release commit's check runs before Orbi freezes the SHA and cuts the tag. Both ask GitHub's Checks API for the commit's check runs and decide on the conclusion field — quoted here verbatim from the delivery engine, src/orbi/runner.py (check_review_ci) and src/orbi/release.py (check_release_gates):

src/orbi/runner.py — check_review_ci (the review gate)
failed = [
    check for check in check_runs
    if check.get("conclusion") not in ("success", "neutral", "skipped")
]
if failed:
    check = failed[0]
    reference = check.get("html_url") or check.get("details_url") or "no run URL"
    raise RuntimeError(
        f"review gate: CI check '{check.get('name')}' failed on PR head "
        f"{commit} ({reference})"
    )
if not check_runs:
    evidence = f"CI on review head {commit}: no check runs (nothing to gate)"
src/orbi/release.py — check_release_gates (the release gate)
for check in check_runs:
    name = check.get("name")
    status = check.get("status")
    conclusion = check.get("conclusion")
    if status != "completed" or conclusion not in (
        "success", "neutral", "skipped",
    ):
        raise RuntimeError(
            f"release gate: CI check '{name}' is {status}/{conclusion} "
            f"on the release commit {release_commit}"
        )

Both quotes verbatim, verified 2026-09-12. Pending checks are never judged early: each gate polls until every check run completes, and a wait that exceeds the budget fails the gate — a queued suite blocks, it does not pass. The conclusion values are GitHub's own enum (Checks API): success, failure, neutral, cancelled, skipped, timed_out, action_required, stale; the gates' pass set is the subset success, neutral, skipped.

The gate never sees a test name, a framework, or a coverage number. It sees one bit per check run — pass or not — which is why the contents of your CI, not Orbi, define the guarantee.

WHAT THAT MAKES ORBI GUARANTEE

Your tests decide what “guaranteed” means

The gate enforces whatever verification lives in CI — no more, and importantly no less. The layers stack: each one widens what a green delivery proves.

What lives in CIWhat Orbi then guarantees on every delivery
Unit tests onlyThe code logic those tests cover passes. A delivery that breaks them cannot merge — and that is the whole guarantee.
+ integration testsModules also agree at the boundaries the integration tests exercise.
+ business-flow e2e (Playwright)The business loop closes. The flow a user actually runs — open the app, do the thing, see the result — is exercised and green on every single delivery.

Put pytest unit tests in CI and Orbi guarantees code logic. Put Playwright business flows in CI and Orbi guarantees the loop closes. That is not a metaphor — it is a literal reading of the code above.

COPY, PASTE, DONE

Two complete workflows, ready to commit

Full files, not fragments — either one is a complete .github/workflows/*.yml. Each triggers on pull_request (so the review gate sees the PR head) and on push to the delivery branch (so the release gate sees the release commit). push.branches must include the branch Orbi delivers to; the Playwright file assumes your playwright.config starts the app itself (webServer) — the Playwright docs own that part.

.github/workflows/ci.yml — pytest
name: CI

on:
  pull_request:
  push:
    branches: [main]

jobs:
  tests:
    name: pytest
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest
.github/workflows/e2e.yml — Playwright business flows
name: E2E

on:
  pull_request:
  push:
    branches: [main]

jobs:
  e2e:
    name: playwright
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test

From the next delivery on, Orbi checks these runs on every PR head and every release commit — and refuses to merge or tag past a red one. What running them costs per delivery is measured, not guessed: the cost page puts the mean delivery at about $0.06–0.12 at DeepSeek list prices.

READ THE BOUNDARIES FIRST

Four things this page does not claim

  • BOUNDARY 1 · FAIL-OPEN
    No check runs, nothing to gate — both gates pass

    A repository without CI runs deliveries end to end. The engine's own evidence line says it plainly: no check runs (nothing to gate). Nothing blocks — and nothing is verified, because the only automated acceptance gate is the one that isn't installed. One refinement, same code: on a just-pushed release commit in a repository whose frozen base already showed checks, an empty list means “not registered yet”, and the gate waits for the first check instead of passing. Fail-open is for repositories without CI — not for CI that hasn't spoken yet.

  • BOUNDARY 2 · SKIPPED ≠ RAN
    Neutral and skipped conclusions count as passing

    The pass set is literally success, neutral, skipped. An e2e job skipped by an if: condition, or filtered out by a paths: filter, concludes skipped — and does not block the merge. A job that didn't run is a job that didn't fail. Point the triggers at the right paths and keep conditions off the jobs a merge depends on.

  • BOUNDARY 3 · THE AGENT WRITES THE TESTS
    Coverage proves the tests touch the code — not that the code is right

    The same agent that writes the feature writes the tests. Playwright can prove the button renders and responds; it cannot prove the button belongs on that screen, or that the error message is kind. Small UX drift is exactly what these gates do not catch — judgment about feel stays with a human reviewer, and a green check run is not a UX sign-off.

  • BOUNDARY 4 · REPORTING, KNOWN LIMIT
    The progress comment's tests line is pytest-specific

    The tests passed / tests failed line in a delivery's progress comment is parsed from the repository's test log, and the parser is tuned to pytest's summary line (1 failed, 155 passed in 4.43s). A Playwright run's summary is not parsed into counts, so a non-Python repository may show no tests line at all. That is a reporting limitation, not a gate change: the gates decide on check runs and never read the test log.

PROVENANCE

Sources and verification dates

Every engine claim on this page was checked against the delivery engine's source on 2026-09-12 — quoted from the code, not paraphrased from documentation.

ONE COMMIT AWAY

Install the gate you want Orbi to enforce

Pick the layer you want guaranteed — logic, integration, or the business loop — and put its suite in CI. From the next delivery on, Orbi checks it on every PR head and every release commit, and refuses to merge or tag past a red run.

Set up Orbi Or let Cloud run the line →