Can an AI Agent Do QA? Our Overnight Audit Pipeline
We run AI agents through client sites overnight: what they catch that manual QA misses, what they still get wrong, and how human review keeps the bar up.
Yes - an AI agent can do useful QA when it runs against written charters, captures evidence, and hands findings to humans for severity and taste. It cannot own release authority or invent product judgement. We run overnight agent walkthroughs on client and studio sites: they expand coverage past what a tired manual pass reaches, they also produce false positives, and the bar only holds when a person triages. This piece is how that pipeline works, what it catches, where it fails, and how it shows up inside Support & Growth.
What the overnight pipeline actually does
Can AI do QA testing?
AI can perform structured exploratory and checklist QA - walking flows, checking states, flagging accessibility and content issues - but production sign-off stays human.
What can an AI agent test on a website?
An agent can test primary journeys, form validation, broken links and assets, responsive breakpoints, basic accessibility smells, console errors, and charter-defined edge paths - especially overnight when humans are offline.
Our pipeline in plain language:
- Charters. Written missions (“complete enquiry on mobile”, “add to basket as guest”, “check footer legal links on key templates”) - not vibes.
- Walkthroughs. Agents navigate the live or staging site, following the charter and branching into likely failure states.
- Evidence capture. Screenshots, short clips, URLs, console notes - a finding without evidence is a rumour.
- Structured report. Issues grouped by journey and template, ready for triage.
- Human review next morning. Severity, duplicates, taste, and “ship / defer / ignore”.
That is the same senior-directed agent pattern as what is agentic AI and the build-time discipline in our agent build log: agents execute; humans decide. We describe capability here, not internal tooling names.
What agents catch that manual passes miss
Does AI QA replace human testers?
No. AI QA widens coverage and stamina; humans still own prioritisation, domain judgement and whether a release ships.
In our experience overnight runs surface classes of issues manual afternoon passes under-sample:
- Template dark corners. Legal pages, empty states, logged-out vs logged-in variants, and “rare” locales nobody clicked after lunch.
- Regression drift. A component change that breaks a secondary template the feature owner never opened.
- Form edge paths. Wrong keyboards, validation order, double-submit, and error copy that only appears on one field combination.
- Accessibility smells. Missing labels, contrast failures, focus traps - cheap to catch mechanically, easy to skip when you are hunting a visual bug.
- Console and network noise. Failures that do not always paint a red UI but poison analytics or payments.
Agents catch breadth humans skip when time is short; humans catch meaning agents invent. Named issue counts wait for client-approved disclosure.
When those findings feed a productised review, they land in a UX audit fix list rather than a Slack dump.
What agents still get wrong
False confidence is the failure mode.
False positives. Agents flag “broken” states that are intentional (auth walls, geo blocks, feature flags). Triage time is real cost.
Context blindness. A visually noisy page may convert fine; a clean page may hide a trust problem only a practitioner spots. Taste is not a checkbox.
Flaky environments. Staging data resets mid-run; third-party widgets timeout; the agent retries and writes a novel. Harness discipline matters more than model fashion.
Encoded bugs in tests. Agents sometimes assert the broken behaviour as correct - we learned that the hard way in build review; humans must check test intent, not only pass/fail.
Security theatre. Automated clicking is not a penetration test. Do not market overnight QA as a security audit.
Mastery is knowing when to distrust a tidy report.
The human layer: severity, triage and sign-off
Every morning (or after each run) a human:
- Deduplicates and merges template-level causes
- Scores severity against the money path (enquiry, checkout, booking, activation)
- Separates ship-this-week patches from redesign-class issues
- Assigns owners and release windows
- Signs off whether the run is “clean enough” for the agreed bar
Without that layer you have an expensive screenshot generator. With it, overnight coverage becomes a force multiplier for small teams - especially when paired with the judgement model in UX audit.
Release authority never moves to the agent. If a finding is contested, a senior decides. That is non-negotiable on client work and on our own properties.
What this means for support clients
How much does automated QA cost?
Standalone automated QA tooling licences often sit from tens to a few hundred pounds a month; agency-run exploratory QA and audit passes are usually scoped as project fees or retained hours rather than a raw tool price - in the UK, serious retained care that includes monitoring and human developer time commonly lands in the hundreds to around a thousand pounds a month depending on depth.
At Code23, overnight agent QA is not a separate gimmick SKU for most clients. It sits inside how we operate Support & Growth:
| Tier | Monthly | QA-relevant reality |
|---|---|---|
| Standard | £350 | Monitoring, security posture, investigation route; agent assists triage where useful |
| Enhanced | £500 | More maintenance hours and health checks - room for scheduled charter runs |
| Gold | £1,000 | Higher change volume and tighter ops - broader overnight coverage and faster human follow-up |
Extras outside the tier remain transparent fixed change bands. Build projects still get Harden before Launch; the overnight pipeline is how we keep catching drift after go-live instead of waiting for a customer tweet. Dual hubs: AI development for agentic systems work, and Support & Growth for retained care.
Est 2005, 350+ projects - the lesson is boring: coverage compounds when someone owns triage. Agents give you the overnight shift; they do not give you accountability.
Overnight agent coverage without a human triage owner is noise. Retained care that includes both lives under Support & Growth.
