Building an LLM-Powered Code Review Sidebar

I review a lot of pull requests. Doing a thorough code review is impossible without context: who wrote it, what problem they’re solving, why they chose that approach. Without that context, I’m just staring at diffs, guessing.

This is especially true if you aren’t as familiar with the specific project as the PR author.

That’s where Sherpa comes in.

The Problem

You open a PR to review. Thirty files changed. Two thousand four hundred lines of diff. Your brain does the math: this will take an hour, minimum.

You have a few options:

1. Skim it. Trust the diff looks reasonable. Approve. Move on. Hope nothing bad ships.
2. Read it all. Understand every change. Ask clarifying questions. Actually review the code.
3. Outsource it. Rely on LLMs like Copilot / Claude / etc to do all the heavy lifting for you.

All options suck. Option one is risky. Option two is time-consuming. Option three? What’s the point?

What if there was a better option? What if someone explained the PR to you as you read it?

That’s the idea behind Sherpa. It doesn’t post review comments. It doesn’t approve PRs. It doesn’t tell you what to think about the code. Instead, it gives you context. The kind of context you’d get if the author sat next to you and walked you through their changes. You’re still the one deciding if the approach is right, if the tests are sufficient, if the error handling is sane. Sherpa just makes sure you’re not doing that blind.

What Sherpa Does

Sherpa is a Chrome extension that guides you through pull requests with live, LLM-powered explanations.

When you open a GitHub PR’s “Files changed” tab, Sherpa renders a sidebar with three layers of explanation:

1. PR Summary

One LLM call using the title, description, commit messages, and file list. You get the “why” before you even start reading code.

2. File-Level Explanations

Click or scroll to a file, and Sherpa explains what that file does in context of the full diff. Not just “this line changed” but “this line changed because the auth service needed a fallback.”

3. Hunk-Level Detail

Drill into individual code changes for line-by-line explanation. When a refactor isn’t obvious, Sherpa tells you what changed and why.

Sherpa also recognizes generated files: package-lock.json, go.sum, yarn.lock, minified bundles, protobuf outputs, and other artifacts you’d never review manually. It collapses them automatically and marks them with a “gen” badge. No more scrolling past two thousand lines of lock file to find the three lines that matter.

The key is that you only pay for what you explore. Most PRs you read three files and close the tab. Sherpa costs you three file explanations, not thirty.

How It Works

The project is split into two workspaces:

• browser/: The extension itself. WXT (Manifest V3, Vite-based) for the framework, Preact + Tailwind for the side panel UI, and the Vercel AI SDK with provider adapters for LLM abstraction. Tests run on Vitest.
• worker/: A Cloudflare Worker that handles GitHub OAuth and caches explanations with Durable Objects.

All LLM calls happen from your browser. There’s no intermediary server that sees your code. Your API key stays local and encrypted at rest.

The flow:

Explanations stream in real time as they generate, so you’re reading before the LLM finishes. They’re cached locally and remotely. Revisit a PR? Instant. The Cloudflare Worker caches across devices. Start a review on your laptop, pick it up on your desktop, and the explanations are already there.

The sidebar also syncs its scroll position with GitHub’s diff view. Click a file in the diff, the sidebar jumps to that file’s explanation. It sounds like a small thing until you’re bouncing between fifteen files.

Under the Hood

A few things worth noting:

GitHub DOM fragility. The content script reads GitHub’s HTML. Their class names aren’t a stable API. I isolated all selectors in src/providers/github/selectors.ts so updates are a single-file change when GitHub inevitably changes their markup.

Provider abstraction. The CodeReviewProvider interface + DOMAdapter pattern means adding GitLab (or Bitbucket) is a new adapter, not a rewrite. Today it’s GitHub only, but the architecture supports more.

No server-side processing. Some PR tools send your code to their servers. Sherpa calls the LLM from your browser. Only the summaries are stored by a server you control. Your code never leaves your machine except to Anthropic/OpenAI/Google directly.

Token budget management. A massive PR could blow through your API budget fast if you sent the entire diff in one shot. The lazy-loading architecture is the main defense. File and hunk explanations only fire when you ask for them, so a thirty-file PR where you only care about five files costs you five calls, not thirty. Generated files get collapsed automatically, so you’re not burning tokens on lock file diffs.

Where It Goes From Here

This is v0. It works for GitHub. It works for Anthropic, OpenAI, and Google.

What I’d like to add:

• GitLab support. The abstraction is there. Just need to write the DOM adapter.
• More providers. Local providers are next. Including Ollama and LMStudio support.
• Team features. Shared caches, organization-wide summaries.

But first, I want to see if anyone else finds this useful.

If you’re interested in more developer tooling with AI, I also wrote about running AI agents in devcontainers.

Try It

Sherpa is open source and MIT licensed: github.com/markphelps/sherpa. Chrome Web Store listing is coming soon, or you can build from source.

If you try it, I’d love to hear how it goes. Open an issue, send me a message, or tell me I’m wrong about something. That’s how this stuff gets better.

Thanks for reading.