AI · Developer Tooling
gitvaluation
Point it at any public GitHub repo; it pulls recent merged PRs, runs the diffs through an LLM with a structured-output prompt, and returns a per-contributor impact breakdown — a demo of GitHub API integration, prompt engineering, and JSON-mode structured output.
The idea
Line counts and commit totals say almost nothing about the quality of a contribution — deleting 400 lines of dead code can be worth more than adding 400. gitvaluation is a small experiment in using an LLM to read the actual diffs and reason about impact: confidence (does a change fix the root cause or just a symptom?), complexity, and net code value.
How it works
- A server action parses the repo URL and pulls recent merged PRs — with their diffs, additions, deletions, and changed-file counts — from the GitHub REST API.
- Commits are grouped by author and each diff is truncated to keep the prompt bounded before it's sent to the model.
- OpenAI runs in JSON mode against a fixed schema, scoring each contributor on confidence, complexity, and net code value, and assigning an archetype.
- The scores are aggregated per contributor and rendered as a ranked dashboard.
Decisions worth calling out
JSON-mode structured output
LLMs are unreliable at free-form JSON. Running the model in JSON mode against an explicit schema means the response parses deterministically — no brittle regex clean-up, and the UI can trust the shape of what comes back.
Bounded prompts via diff truncation
Real PRs can be enormous. Truncating each diff to a fixed budget keeps token cost and latency predictable while preserving the signal that actually drives the score.
A keyless demo via mock mode
Without API keys the app serves a realistic mock analysis, so the live demo works for anyone without spending tokens or requiring setup — a small decision that makes the project genuinely try-able.
What it demonstrates
The interesting part isn’t the scores themselves — it’s the plumbing: pulling structured data from a third-party API, shaping it into a bounded prompt, and getting reliable, typed output back from a model that a UI can render without guesswork. That’s the same pattern most production LLM features come down to.
Want to talk about building AI features like this?
Get in touch