Why We Built a QA Agent for Our Background Agent

If a QA agent for your projects sounds interesting, we'd love if you tried out Ranger Feature Review here.

‍Our background agent writes the code like any other coding agent, but can now use Ranger to close the loop by testing itself and showcasing its work back to us for review.

The results of this have been pretty transformational for us internally. We just don’t manually test features anymore or even open up the preview branch. We queue off multiple background agents at once, come back when it's fully verified its work with Ranger, leave feedback, and put up a PR. We’ve isolated ourselves to just provide our input and then get out of the way.

What We Saw

When background agents have a way to verify their work, they can work further on their own. Karpathy, Ramp Inspect, Gas Town, and others have shown how closed loops create AGI-esque leverage.

When we set up our background agent, we knew prioritizing the best verification loop was key for best performance. The more effectively our agent could verify its work, the longer the agent could productively run and stay on track. The canonical solution was to give our agent a browser MCP, which worked fine but had shortcomings. Here’s what we saw:

Reviewing agent work sucks

We pretty quickly got annoyed by scrolling through pages of agent flibbertigibbetings just to find a couple screenshots of what the agent did. And leaving granular UI feedback for the agent was a struggle. Even if we might not read code anymore, we needed a way to iterate on and review features before they were shipped.

One coding agent can’t do everything

One agent doing everything (writing code, testing it, etc.) is slow and context-inefficient. Rather, we wanted our agent to be able to spawn multiple QA teammates so that they could verify quickly without degrading the main agent’s context.

Complex features need structured testing flows

Just like agents need to plan for a new feature, complex features need a testing plan – data setup, edge cases, etc. One happy path is not enough when your coding agent is ripping ten screens in one-shot.

What We Built

We set out to build a QA teammate that our background agent could a) delegate work to b) verify complex scenarios with c) use to showcase its work back to us. Here are a few takeaways from what we did.

Feature Reviews need a new type of UI

Feature reviews are not code reviews. Github PRs were not built as a way to provide design feedback to agents, so we had to build our own paradigm. Originally, we dumped everything into a markdown file, which worked, but it was hard to a) share b) render any media and c) give any feedback on. So we made our own feature review UI and embedded that UI into our background agent’s UI. Now, we can natively view all the screenshots/videos that Ranger took during a session, leave comments on the screenshot, and then immediately resume the agent.

Sub agents and plugins are context efficient, faster, and reliable

Our first attempt was browser MCP with some skills for how specifically we wanted to verify features. But it was still slow and inefficient and we found skills to be flaky and regularly forgotten mid-usage by the coding agent. So we built a CLI accompanied with a plugin instead that allowed the main agent to create and spawn sub-agents so it could verify features in the background. We use hooks in the plugin to continuously remind the agent to keep going until Ranger has fully verified the feature (thanks Ralph!). This resulted in a higher one-shot completion rate.

Multi-player matters – everyone can ship now

Just like a design review or bug bash, the more eyes the better. Instead of having to record a video to add to your PRs, we can instantly attach the work Ranger did on a PR for our teammates, and they can leave comments directly in the Feature Review UI. Now, we just send Feature Review links to each other when we need feedback from others.

Most exciting is how this enables any Ranger teammate to ship high quality changes. We integrated our background agent into our product feedback and troubleshooting slack channels. When tagged, the agent reads the task and gets to work. When it thinks it’s done, it verifies with Ranger, does as many iterations as necessary, and then posts a link to the Ranger Feature Review in slack. Now, anyone can push a new production-ready feature end to end without having to open the code or terminal.

Closing Thoughts

We at Ranger find ourselves living in some of the craziest times for software engineers, and prioritizing a purpose-built QA agent for our background agent has really helped us harness all the power of these tools to ship many high quality features in parallel. 

Starting today, you can integrate Ranger into your background agent with these docs. If you’d like to integrate Ranger into your background agents’ UI, we’d love to hear from you at eng@ranger.net.