rubric ⟶ ai code review
Rubric
liveChecks whether a pull request's implementation matches its stated intent — not just code quality.
Tech stack: Claude API, Zod, GitHub Actions, CLI
summary
Rubric reviews a pull request against its own description: does the diff actually do what the PR says it does? It extracts the explicit and implicit promises from the PR's intent, then hunts for changes that no promise accounts for.
The engine is a single Claude call with a Zod schema as its required output format — one definition serves as both the prompt contract and the runtime validator. It counts tokens with Claude's real tokenizer instead of a chars-over-four heuristic, and it keeps mechanical facts (like whether input was truncated) owned by code and never trusted from the model. The model judges; the code states facts.
It ships two front doors that share one engine — a GitHub Action for CI and a read-only CLI for any PR — and it is dogfooded on its own pull requests.
features
Intent vs implementation
Checks whether a PR's diff matches its stated intent, not just whether the code is clean.
Schema-validated output
A single Claude call with a Zod schema as the required output — one definition is both prompt spec and runtime validation.
Real token budgeting
Filters noise and truncates to a token budget using Claude's real tokenizer, not a chars-over-four heuristic.
Code-owned facts
Mechanical truths like the truncated flag are set by code and never trusted from the model.
Two front doors
A GitHub Action for CI and a read-only CLI for local review of any PR, sharing one engine.
Dogfooded
Runs on its own PRs — and caught one labeled docs-only that had quietly shipped code.
architecture
A monorepo with a shared core engine and two front doors (GitHub Action + CLI). The engine builds a prompt from the PR's intent and diff, filters and ranks files, budgets tokens with Claude's real countTokens, and makes one Claude call constrained by a Zod ReviewSchema. Code sets mechanical flags; the model returns claims, unstated changes, and a verdict.
changelog
- Feature: Static demo replaying the dogfooded reviews, with an editorial dev-tool redesign.
- Feature: Two front doors: a GitHub Action that comments on PRs and a read-only CLI (rubric scan) for any PR.
- Feature: Initial release: an intent-vs-implementation review engine — a single Claude call with a Zod schema as the required output, real token budgeting, and code-owned mechanical facts.