The day we almost shipped an API key to production
It was a Friday evening. A developer had pushed an urgent fix for a client. The code worked, the tests passed, the PR looked clean. Except there was a hardcoded Stripe API key in a config file. In plain text. In the repo.
Our automated reviewer caught it 45 seconds after the push. Forty-five seconds. By the time the developer opened Slack to ask for a review, the bot had already commented on the exact line with an unambiguous message: "Secret detected. Merge blocked."
That's the day I stopped treating automated code review as a nice-to-have.
Why we automated our reviews
Before, our PR reviews took an average of 4 hours. Not because the code was complex, because the reviewers were busy. A developer opens a PR at 10am, the reviewer looks at it at 2pm, leaves three comments about formatting, one about a variable name, and one real architecture comment. The developer fixes, re-pushes, and waits another 2 hours for approval.
Four hours for what should have taken 20 minutes. Multiply that by 8 to 12 PRs per day depending on the project. The math was simple: we were losing 30 to 50 hours per week to review friction.
The problem wasn't reviewer competence. The problem was that we were asking them to do work a machine does better and faster: checking formatting, scanning for forbidden patterns, counting test coverage.
Global rules: the CLAUDE.md and hooks
The first thing we set up was a global rules file. At JADEV, it's the CLAUDE.md. Every repo has one. It contains the project conventions: code style, allowed patterns, forbidden patterns, coverage thresholds, naming rules.
But a rules file is useless if nobody reads it. So we made it executable. Every rule maps to a CI check that runs automatically on every PR.
Specifically, our checks verify:
- Security: no secrets, no API keys, no passwords in code. We use secret scanners that block the merge immediately. No warnings, no "fix it later". Blocked.
- Naming conventions: files, variables, functions follow the patterns defined in the CLAUDE.md. A React component called `data_handler` instead of `DataHandler`? Rejected.
- Test coverage: every PR must maintain or improve the coverage threshold. For us, that's 80% minimum on modified code. Non-negotiable.
- Forbidden patterns: `console.log` in production, `any` in TypeScript without justification, circular imports, files over 400 lines. All detected and commented automatically.
- PR size: beyond 500 modified lines, the PR gets flagged. Not blocked, but flagged. Large PRs hide bugs. That's a fact.
These pre-merge hooks run in under 90 seconds. The developer gets feedback before they've even left their terminal.
AI as the first reviewer
CI checks catch mechanical violations. But there's a layer above that: contextual review. That's where AI reviewers come in.
We use Claude and CodeRabbit on our repos. As soon as a PR is opened, the AI reviewer analyzes the full diff and leaves comments in under 2 minutes. It catches potential bugs, uncovered edge cases, inconsistencies with the rest of the codebase.
What's interesting is the complementarity. The AI is excellent at spotting an unhandled `null` in a function that receives data from an API. It's excellent at saying "this function does 3 things, you should split it". It's excellent at pointing out a missing test for the error case.
It's bad at judging whether a module's architecture is right. It's bad at knowing if a refactor will cause a performance problem in 6 months. It's bad at understanding the business context that explains why that "weird" code is actually correct.
That's why we keep human reviewers.
The concrete workflow
Here's how it works, from push to approval:
- The developer opens a PR.
- Within 30 seconds, CI checks run: linting, security, coverage, conventions.
- Within 2 minutes, the AI reviewer analyzes the diff and leaves inline comments.
- The human reviewer arrives to an already-annotated PR. Mechanical issues are resolved or flagged. They can focus on what matters: business logic, architecture, design choices.
- The human reviewer approves or requests changes. On the points that actually matter.
Result: average review time went from 4 hours to 1h20. Human reviewer comments went from 60% syntax / 40% substance to 10% syntax / 90% substance. The quality of review discussions improved because humans stopped wasting energy on details that machines handle better.
Trust but verify
I often hear two extreme positions. There are those who want to automate everything, "if the AI says it's good, we merge". And those who trust nothing, "a bot doesn't understand my code".
Both are wrong.
100% automated review is a fantasy. You need humans for architecture decisions, for business context, for choices that will impact technical debt two years from now. No AI knows that today.
But 80% automated review for syntax, patterns, and security? That's not innovation. That's discipline. It's like using a spell checker before sending an important email. Nobody says "I trust my spelling, no need to proofread". Well, almost nobody.
Our approach is "trust but verify". We trust the automated system for the 80% that's mechanical. And we verify with humans for the 20% that requires judgment.
What we learned
Three things we didn't expect:
First, developers write better code when they know a bot will review it. It's not fear, it's a feedback loop. When you know every forgotten `console.log` will come back as a comment in 30 seconds, you stop leaving them around. In six months, the number of violations detected per PR dropped by 70%.
Second, human reviewers are happier. Nobody enjoys spending 20 minutes counting spaces and checking imports. When you take that work away, they can do what they do best: think.
Third, confidence in shipped code went up. We sleep better. Because we know every line that reaches production has been scanned for secrets, checked for coverage, and reviewed by a human who had time to think instead of counting semicolons.
How to get started
If you have nothing in place, start with three things:
- A secret scanner in your CI. This is non-negotiable. A secret in production costs more than six months of tooling.
- A linter with strict rules and a blocking check. Not warnings. Errors.
- An AI reviewer, Claude, CodeRabbit, doesn't matter, that runs on every PR and leaves comments before the human reviewer does.
It takes a day to set up. It saves you hundreds of hours per quarter. And it keeps you from shipping API keys on a Friday evening.