Code Review Feedback Loop Experiment

Thesis

Code review automation is useful only when it reduces reviewer load without hiding risk or generating confident noise. This project explores whether repository context, recent changes, and prior review notes can make automated suggestions more grounded.

It does not replace human review. The goal is to test a feedback loop: what should be checked mechanically, what should be surfaced as a question, and what should stay with reviewers.

What It Explores

The current direction combines conventional checks with constrained LLM prompts:

• Repository indexing to retrieve nearby examples and conventions
• Vector search to compare related code paths
• Rule candidates derived from repeated review comments
• GitHub Actions integration to test pull-request workflow fit
• Reviewer feedback capture to measure accepted, rejected, and ignored suggestions

Current Evidence

The prototype can index a repository, retrieve related snippets, and attach experimental suggestions to a review workflow. The unresolved work is quality control: deciding when the system has enough context to comment and when silence is the better outcome.

The next useful measurements are:

• False positive rate by suggestion category
• Suggestions accepted without edits
• Comments that duplicate existing static analysis
• Reviewers' perceived interruption cost

Open Questions

The hard part is not generating comments. It is deciding which comments deserve to exist. This experiment is useful only if it can make that decision with evidence from reviewer behavior and repository-specific examples.

Near-term work stays narrow:

1. Tighten retrieval around changed files
2. Separate deterministic rules from model-generated observations
3. Add review feedback labels for accepted and rejected suggestions
4. Compare output against existing linters before adding new categories