The Dream Quality System: How AI Can Finally Solve Software Testing

I spent 5 years as an engineer at Google working on a variety of mobile products, from the Android operating system to Google Maps. We had the world's smartest engineers and excellent quality practices, from a complete testing suite (unit, integration, & smoke) to an army of manual QA testers.

Yet we still shipped critical bugs to users. Check out any comment about the "Pixel phone alarm bug" or "Google Maps app crashes". Even after all of the quality checks, why do we still ship bugs? Well, there are a couple of challenges.

The Dream Quality System

If I had an infinite budget and headcount, here's how I would design the best quality system. First of all, I would ensure that there are engineers who are not writing any features and only writing tests. Secondly, for these engineers, I would only provide the API surfaces of the software, not the implementation to prevent their coding agents from leveraging implementation during the tests. This should significantly improve the quality of automated tests. Finally, I would create a Github bot that doesn't allow an engineer to submit changes unless they updated test cases based on their PR.

Now that we have a guarantee that the code is well tested and that the test cases are always up-to-date, we can focus on the manual QA process. I would double my QA team immediately. Then I would ensure that testing requests are distributed across the team such that each person does not test the same feature twice. This guarantees a couple of things. First, that fresh eyes are testing features as opposed to the same person, who may miss minor details. Second, that the entire QA team has a strong understanding of the app.

Unfortunately, it is impossible to double my QA team overnight because we need to source new hires and then spend a few weeks onboarding them. If only we could click a button and increase QA headcount.

Can AI Replace QA

This is an interesting question, and one that's been top-of-mind for many folks in the industry for years. Pre-LLMs it was impossible for sure. Even now, it appears to be a long-shot. A strong AI product with the potential to replace QA would have a couple of attributes:

1. Take my existing test cases without extra work. I don't want to import my test cases into a new system or re-write them for an AI. I want the AI to understand the same way my manual QA testers do.

2. Interact with my app just like a human. This is obvious: if the AI uses code or some hack to control the app rather than using the device like a human, then it will not catch bugs your users will see.

3. Run on an array of real devices to guarantee that the AI sees what users would see.

Can AI Do Better than Manual QA

This is especially exciting. If the AI product mentioned above is possible, then it can be extended to truly disrupt the current QA processes and help teams ship bug-free software at speeds never seen before.

Here's what I would imagine it has:

1. Run in CI / CD pipelines

2. Run in parallel. If it's automated, shouldn't the test suite take 10 minutes instead of 10 hours?

This Isn't Theory Anymore

That system now exists. Leading mobile teams are already using it to replace much of their manual QA while shipping with higher confidence.

The solution is QualGent AI, a system that understands your app like a QA tester, runs on real devices, and plugs directly into your workflow. Companies like Knowt are already using it to cut regression cycles from a week to an hour.