From Gatekeepers to Guardrails

This post was written with ❤️ by a human. No LLM was used in it's creation, thinking or writing. As such there almost will be issues you take with it. That's good!

Code review was designed for a world where writing code was slow and reading it was cheap. That world is gone. Human attention is the scarcest resource on an engineering team, and we spend most of it on pull requests that don't really need it.

The Toll Booth

Let's picture code review as a toll booth on a busy highway. In the before times a booth agent would sit and collect the toll before the car is allowed to drive on it, if there was a bus then the toll was higher, it paid a slightly higher cost. The booth agent gets into a monotonous roll, next - that will be 75 cents, thanks, next 75 cents - and so on. With technology advances we've gained toll passes, you can drive right through the toll booths that are equipped with sensors. Forget to load money into your account? Don't worry the toll operators will send a bill.

We can think of Code Review in this same way, your teammates and you are looking at code, most of us more code than ever before in our careers. I find myself getting into that monotonous roll, does this break our system in the ways I am concerned about? No? approved and next please. Does it break? Sorry you forgot the toll increased this month, pay the new toll please with this change - and now you can go.

But now the toll booth is backing up, causing traffic jams. Teams that are finding their LLM agents producing more code than ever before may try to split them into smaller Stacked PRs but that exhausts the capacity for the toll booth to process them. You have the same amount of code, but the throughput required on PRs that are smaller are significantly higher. If you take a 12k line feature and break it into smaller PRs that can be upwards of 20-40 PRs depending on how it's split.

Perhaps your team has asked each other to try to keep lines in PRs between 300-500 lines and limit the number of files changed at once. These are objectively good engineering principles to uphold even in an era of AI agents, but now 12k lines of code your agent can churn out in a few days of wall time, that needs to be split up.

You need to convey the context required for each of the 20-40 PRs, and then you need to keep track of changes if any are requested or required across all of them. I'd rather do anything else than try to keep 20 stacked branches up to date when the 3rd one needs a major change.

As a Senior Engineer I've viewed my job in Code Reviews to be a semi-gatekeeper from bad code or architecture getting in the codebase. The codebase is my home, I need to try and protect it from starting to decay from a lack of maintenance.

In code reviews by humans we're catching style differences, finding bugs, sharing knowledge of the new patterns introduced by others and dissemination of information, spotting the risky changes.

You know what we could do? We can bring that up out of reviews into the machine enforcing them, and then raising when something needs judgement (that's largely around the risky changes).

We can turn the style changes into linter rules, turn recurring review comments into the lint rules too. Add evals for behavior and we can make the CI the thing that says no, but it would be even better if we can run it locally so our agents can run them as checks before it even gets to CI.

Build the guardrails that prevent the toll booth from backing up and you produce higher quality. That doesn't mean every car always pays that automated toll, but toll booths always get their money, you can get your quality upheld in other ways, after the fact, turn the things that slip through into Root Cause Analysis' and you can find new linter rules and ways to catch it moving forward.

Border Crossings

You as your engineering team needs to pick the border crossings. These are what paths you'd define as critical. For me, the ones that come to mind right away include Billing, Authentication/Authorization, Anything that touches security or could potentially leak data. Are you collecting Personally Identifiable Information, or risk leaking it if you store it? Tag a human in. Every app will likely vary here, if you're at a FinTech startup, whew you might have more border crossings here than not. If you're at a healthcare software startup, well I hope you have good tests to ensure pacemakers don't stop on a glitch.

You (likely) don't need to sit a guard (engineers) at every crossing. Only the ones with railroads or the ones that cross into school zones.

What do we lose?

With PR reviews always done by humans you share information through osmosis, I see a new pattern or system someone does in a code review I'm doing for them, I might do it that way next time. We'll need to find new places for this in the new way. Pairing, design docs or asynchronous walkthroughs can all supplement and replace this kind of behavior. Record a dope Loom or Screenrecording of the work you saw and guided your agent to do and implement, share what worked and even better share what didn't work.

Guardrails need maintenance

Your review process gets moved upstream, you start reviewing things about the linters. What's working, what's no longer needed because Opus 5.5 is not as bad as Opus 4.1 was, etc. Evals would be super helpful in this process I bet.

Find a rule that keeps ruining your life in the automated reviews? Fix the rule if it's not just you.

What else is needed?

End to end testing, architecture checks are both things that would ease the burden and lower the risk in adopting auto approving LLM workflows.

You want end to end testing to be able to run against potential changes, but also to run against production to try to quickly identify regressions that slip through (some will in this new world, it's likely a small toll to pay for the volume of cars you'll see move every day).

Humans get to use judgement, machines shouldn't

Your agent cannot think for you and be a replacement for your thinking. I'm sorry but it's just not feasible at this point. You should consume information and make an informed judgement about a situation or thing. Be it your own PR or another person's.

I am so tired of reading my own agent's walls of text each day, putting a PR up, then reviewing coworker's agent's walls of text. We don't need to, we can write for humans, and then even better humans can then read that. We can restore a little trust that we're all doing our best in this new world. We can disseminate what we found useful or not so useful when we include Claude or ChatGPT to collect potential paths forward.

Use your human judgement to spend time scoping down the context you share with your coworkers.

If we shift to mostly-automated reviews, we move quicker.

Spend your engineering tokens on guardrails and then you can have them move quicker, but you also start to enable the possibility of non-engineers to ask their own agents to help make a change to your system.

A wired designer using their agents can help give your UI a glow-up, and not need an engineer in the room reviewing the code.

Your product managers could make a change to expand a text limit on a text area from 100 to 5000, which otherwise would have had to be a fully written ticket for an engineer.

The list goes on, perhaps you have a support agent who needs to fix a bug for a specific customer, if it required engineering escalation they'd have to wait for who knows what priorities to be shifted for that fix. If you get auto-approving PRs small scoped fixes can get through.

The old ways do not always have to be the only way

The way in which we have done things historically does not have to be the way we always do things. We're in an era of unprecedented access to generate code that works! We can level us all up and reduce the pain of the backed up toll booth by guiding your agents and organization into the proper paths.

Thank you all for reading,

Andrea