Why judgement matters in AI-assisted software development

By Andrew Ward, Managing Director of Scorchsoft

AI can help us produce work that would once have required far more time, money or specialist training. I think that is extraordinary. It also leaves us with a question that a polished result can make surprisingly easy to overlook: how do we know whether it is any good?

I encountered that question when I used AI to explore one of the hardest problems in physics. Anna Bromley encountered a different version of it when she built an app around a problem she understood personally. Between those two examples sits something increasingly important in software development: judgement.

Judgement helps us recognise a worthwhile problem, choose an appropriate solution, and establish what evidence we need before trusting the result. AI changes how we do that work. It also makes the consequences of skipping it easier to miss.

I could produce a physics paper. I could not personally validate it.

In 2024, I used AI to explore whether quantum entanglement could help connect general relativity and quantum mechanics. I wrote about it in How I Unified General Relativity with Quantum Mechanics. The title was deliberately tongue-in-cheek. Watching a substantial amount of PBS Space Time had, regrettably, failed to qualify me to solve quantum gravity.

I later revisited the experiment in an article about dark matter, dark energy and the information structure of spacetime, narrowing the question and exploring how a proposed explanation might be tested.

AI helped me develop arguments and mathematical material that I could never have produced unaided. It also helped me interrogate the ideas. Yet I do not have the physics and mathematics expertise to independently establish whether the resulting work is sound, novel or consistent with the relevant evidence.

That does not make the experiment worthless. It made difficult ideas more accessible and gave my curiosity somewhere useful to go. But a speculative, AI-assisted paper remains speculative until suitably qualified scrutiny and evidence justify stronger claims.

My ability to generate the work had moved ahead of my ability to evaluate it.

The same gap can appear in software. An application can look professional, run successfully and pass a demonstration while containing problems that its creator does not know to investigate.

What judgement contributes to a software project

When people discuss AI development, the conversation often centres on how much code a tool can generate. Code matters, but clients usually want something further down the chain: fewer administrative mistakes, a better customer experience, a new service, or a process their team can actually use.

Reaching that outcome involves several kinds of judgement:

• Choosing the problem: which difficulty deserves attention, and whose experience tells us it matters?
• Choosing the approach: what should we build, what should we simplify, and where would an existing tool be sufficient?
• Checking the result: what would demonstrate that the solution works, and what would reveal that it does not?
• Taking responsibility: who monitors it, handles exceptions and learns from what happens after launch?

AI can contribute to each of these activities. Experienced people can supply context, challenge assumptions and arrange checks that go beyond the system's own account of its work. Their contribution should be visible in the decisions and evidence, rather than assumed from their job titles.

For a new product, app idea validation helps establish whether the problem deserves a build. Choosing the approach is a separate decision: our article on when to use ChatGPT, builders or code explores how to match the tool to the work.

Sometimes an established offering fits the requirement. For businesses seeking AI-assisted customer conversations across phone, web and mobile, our AI Enablement Package brings those channels together. The judgement is whether that package fits the need, or whether a different solution would serve it better.

Judgement runs through the project. The questions in this infographic are a practical Scorchsoft interpretation, rather than a measured research model.

Research shows why confidence needs checking

What changed between 2024 and 2026?

Older studies cannot tell us how capable today's models are. A paper published in 2026 may still report an experiment run with 2023 tools. To understand progress, we need to check both dates and compare like with like.

METR's January 2026 Time Horizon 1.1 assessment provides one comparison. In its updated task suite and evaluation setup, GPT-4o, a 2024 model, had an estimated 50%-success horizon of about five minutes; late-2025 Claude Opus 4.5 reached about 320 minutes. Those are estimates of how long a human expert would take on tasks the agent has a 50% predicted chance of completing. They are not AI running times or a measure of dependable production delivery. The tasks are self-contained and well-specified, and the estimates have wide uncertainty. Even with those limits, this is substantial progress in the kinds of work an agent can attempt.

Real-world productivity is harder to compare. METR's early-2025 developer experiment found tasks took 19% longer with AI. Its February 2026 update reported an estimated 18% reduction in task time for ten returning developers in a later study begun in August 2025. However, that estimate's confidence interval ranged from a 38% reduction to a 9% increase. Developers and tasks were increasingly selected out because people did not want to work without AI; concurrent agents also complicated timing. The authors regard the newer data as an unreliable estimate of the current effect. This is a promising signal, rather than a cleanly established reversal.

A May 2026 survey of 349 technical workers asked a different question: how much more valuable was their work? Using the same wording, respondents retrospectively estimated a median 1.3-times multiplier for March 2025 and two-times for March 2026. These are perceptions from a convenience sample, not measured business returns. Across its current-value questions, the median estimates were 1.4–2 times; the separate speed estimate was three times.

That distinction is useful. Making a task cheaper does not automatically make it worth doing. My conclusion is that as AI handles more implementation, judgement about priorities and evidence becomes a larger part of the value a development partner should contribute. Our AI Opportunity Planning service addresses that decision by weighing value, feasibility, risk and data readiness before recommending a build-or-buy route.

What earlier experiments still help us understand

The positive case for AI is substantial. Three randomised field experiments covering 4,867 developers estimated a 26.08% increase in completed tasks with an AI coding assistant. Less experienced developers had greater productivity gains. The measure was task throughput, rather than every dimension of software quality.

A useful study with 758 Boston Consulting Group consultants examined what researchers call the jagged technological frontier: AI can help with some tasks and hinder others, even within a familiar workflow.

Participants using GPT-4 took, on average, 25.1% less time to complete the first 17 suitable tasks. On a different task selected to fall beyond the model's capabilities, AI users were, on average, 19 percentage points less likely to reach the correct answer than the control group.

The experiment used 2023 technology, and examined one deliberately difficult task outside the frontier. Those figures are not forecasts for today's tools. What makes the study useful is its demonstration that the same AI tool can improve performance on one task and reduce accuracy on another.

These are different measures from different tasks, not a single productivity score. Source: Dell'Acqua and colleagues, Navigating the Jagged Technological Frontier.

For a business commissioning software, that makes verification a practical concern. You need to know which parts of the work have been checked, what those checks establish, and which assumptions remain untested.

A human reviewer needs something useful to contribute

A 2024 meta-analysis in Nature Human Behaviour examined 106 experiments and 370 effect sizes. Human–AI combinations generally improved on humans working alone, but performed worse than whichever was better individually: the human or the AI.

When humans already outperformed AI on a task, the combination tended to outperform both. The evidence covers studies from 2020 to mid-2023, so it does not evaluate today's coding agents. It does challenge the assumption that adding human approval automatically improves an AI system.

My interpretation is that review needs relevant skill, time and useful evidence. A person who cannot recognise an incorrect output may approve it quite sincerely. A knowledgeable reviewer can still miss it if the process gives them little opportunity to investigate.

The thinking shifts towards checking and integration

Microsoft Research and collaborators surveyed 319 knowledge workers, collecting 936 examples of AI use. Participants described critical thinking shifting towards verifying information, integrating responses and overseeing the task. Higher confidence in AI was associated with less critical thinking; higher confidence in their own ability was associated with more.

This was self-reported, observational research. It does not show that AI causes people to become less intelligent. It does describe a change in where people put their mental effort.

For software teams, that suggests an opportunity: spend some of the time saved in production on understanding the requirements, investigating risks and assessing the result.

Healthcount: recognising a problem worth solving

Checking whether an answer is sound is one part of judgement. Recognising which problem deserves an answer is another.

Anna Bromley oversees and builds Healthcount's GLP-1 tracker, which focuses on maintenance and coming off GLP-1 medication, such as Mounjaro and Wegovy. Its public product description centres on recording food, prescribed doses and appetite through a short diary entry.

Anna's account of why she built it describes her difficult relationship with food, the burden of constantly doing calculations, and the support she felt people needed around maintenance. Alongside that understanding, she brought experience working with software projects.

That combination matters. She could recognise a meaningful problem and translate her understanding into choices about how an app should behave.

At Scorchsoft, we have supported Anna in a coaching and trusted-adviser role. Her founder account describes me as her sounding board and credits my guidance on AI coding tools: setting clear rules, separating jobs and checking generated work before release. Anna oversees and builds Healthcount herself. Her understanding drives the product; our contribution is guidance about the development process.

Consider some of Healthcount's published design decisions:

• A short entry in ordinary language reduces the work involved in recording information.
• Weekly trends receive more attention than daily fluctuations.
• The interface avoids streaks and red/green food scoring.
• Clinical decisions remain with the appropriate professionals; the platform describes itself as a tracking tool, rather than a prescribing service.

Those choices show judgement about the experience the product should create. Adding streaks might look like an obvious way to encourage engagement. Someone who understands how tracking can become burdensome may instead ask whether that feature serves the user.

There is a pleasing detail here: professional judgement helped shape a product that avoids moral judgement about the people using it.

Healthcount's public homepage, captured on 12 October 2026. This is a founder and product-design example, rather than evidence of clinical effectiveness.

The underlying maintenance challenge has independent research behind it. In the STEP 1 trial extension, participants previously receiving semaglutide regained, on average, around two-thirds of their prior weight loss during the year after medication and structured lifestyle support stopped. That finding establishes the relevance of the challenge. It does not establish that a particular app can prevent regain.

The lesson from Anna's example is the value of understanding a problem well enough to make purposeful product decisions. AI may help implement decisions like these, but faster implementation alone cannot tell us whether we chose well.

What verifying software actually means

Imagine a customer portal generated with AI. You can register, sign in and view an invoice. The screens look good. The happy-path demonstration works.

Now change the question. Can one customer access another customer's invoice by altering a request? Does a revoked user retain access through an old session? What happens when an external payment service times out after taking payment?

These are illustrative engineering questions, rather than findings from the studies above. They show how verification depends on knowing which situations to investigate.

An experienced team turns intended behaviour into checks that can fail for useful reasons. For example, “customers can see their invoices” needs a corresponding rule: “customers cannot see anyone else's invoices”. A good demonstration of the first rule says little about the second.

We should also test the intended business benefit. Can customers find the invoice they need without calling support? Observing that task with real users tells us something a functioning invoice screen cannot establish on its own.

Independent evidence matters here. If the same AI generates an implementation and then generates tests from its interpretation of that implementation, both may share the same mistaken assumption. AI-generated tests can be valuable, but they need comparison with independently established requirements and expected results.

Useful checks may include:

• permissions and access boundaries;
• known examples with independently confirmed results;
• integration failures, duplicate requests and interrupted workflows;
• realistic user testing and accessibility checks;
• monitoring, recovery and support arrangements after launch.

The depth should fit the consequences. A disposable prototype and a service holding sensitive customer records need different levels of scrutiny. Choosing that level is itself an engineering and business decision. If you already have a codebase and need to decide whether it is a sound foundation, a code audit and viability review can help assess the implementation and priorities for further work.

Research on human expertise in algorithmic prediction provides another useful perspective. It examines how experts can distinguish cases that look identical to an algorithm by contributing additional information. That work concerns prediction, including medical classification; it does not measure software agencies.

The parallel I draw is contextual: a project team may know that staff regularly bypass a workflow, that a customer exception occurs every Friday, or that an apparently minor integration constraint will determine the whole design. A system cannot reliably act on context nobody has given it.

We also need to develop the people who will check future work

There is a longer-term question for development teams. How do people acquire the understanding they need to supervise AI if they delegate unfamiliar work before learning from it?

In a 2026 randomised study of 52 developers, researchers examined learning an unfamiliar Python library with and without AI assistance. AI use reduced measured understanding, including code reading and debugging, without a statistically significant overall speed improvement.

The researchers also observed more cognitively engaged patterns of use, including conceptual questions and requests for explanations, which preserved stronger learning outcomes. Those usage patterns were not separately randomised. The study concerns a particular learning task, rather than every use of AI or permanent skill loss.

For me, the practical question is what a developer understands after the task. Can they explain the important decisions? Can they debug the result? Do they know what would make the approach unsuitable?

AI can help someone learn by explaining, comparing approaches and making unfamiliar concepts accessible. Teams need to create opportunities for that learning, alongside the pressure to complete work quickly.

Kahneman and Klein's research on intuitive expertise adds a useful caution: dependable intuition needs an environment with sufficiently predictable cues and opportunities to learn from them. Confidence alone does not establish accuracy.

That is a reason to value feedback, review and exposure to real outcomes. An agency should be able to explain how experience improves its decisions, rather than asking you to take “we have been doing this for years” as proof.

What you should expect from an AI-enabled development agency

The results differ across tools, people and settings. The useful question for your business is what changes in your own work. Our Managed AI & Automation Improvement Programme starts with operational friction, agrees an intervention and measures the result against a baseline. That makes the benefit something to investigate, rather than infer from a model benchmark.

As someone who runs a development agency, I think this raises our responsibility to make the benefit visible. Faster production should give clients a better route to the outcome they need. It should also create room for the work that makes a solution dependable.

For a small, low-risk experiment, you may reasonably build it yourself. As the commercial stakes, integrations or consequences grow, an experienced team can help you define the problem, evaluate options and establish proportionate evidence before release.

Our AI software development service connects implementation with the requirements, integrations and checks a working product needs. The value should show up in a useful, dependable outcome, rather than the volume of generated code.

Before trusting a result, ask:

1. What problem are we solving, and what tells us it matters?
2. What would count as a successful outcome?
3. Who can evaluate the result at the level this project requires?
4. What evidence would show that our assumptions are wrong?
5. Who takes responsibility when real use exposes something we missed?

I enjoyed using AI to explore physics. Anna used her understanding to make purposeful choices for Healthcount. In a commercial software project, I want that same curiosity and willingness to build, accompanied by the expertise and evidence needed to decide what we can safely rely on.

If you are building with AI and want help establishing whether the result is ready for real use, talk to Scorchsoft.