A new question, now that AI can run the calculation for you
Generative AI has advanced to the point where describing a problem in plain language is often enough to run specialized calculations, the kind that used to require dedicated engineering software. At Mathematical Physics Labo, we're building a pipeline that takes a natural-language request, assembles the analysis conditions, runs the calculation, and verifies the result.
That last step raises a question that won't go away: how do you know an AI-produced result is actually right?
The tempting but flawed answer: have AI check AI
The intuitive fix is to have a second AI (or the same one) check the first AI's work. This has a fundamental problem.
The checking AI is prone to the same kind of mistake as the AI it's checking. A failure where the formula itself is correct, but the number written down as the final answer doesn't match what that formula actually produces, is obvious to a human eye but can look perfectly plausible to another AI, so it tends to slip past AI-based checking. If the judge is also an AI, you've just built a loop: AI output, guaranteed by AI judgment.
A design built on mechanical verification
To break that loop, the analysis pipeline we're building never lets AI make the call. Judgment is left entirely to ordinary code, that only does what it's told, running in a fixed sequence. We built in two layers of checks.
- Input-stage checks: beyond confirming units are consistent and no disallowed function calls appear, the pipeline does something specific. The AI outputs a predicted value and the formula behind it, together, in the same response. The program independently evaluates that formula and checks whether the result matches the predicted value. Since both the formula and the value come from the same AI output, this is really a check for self-consistency, not correctness against the world. What it occasionally catches: the formula is right, but the number in the answer field is different from what that formula actually evaluates to, as if the arithmetic was done correctly but the wrong digit got written down. It's a common failure mode for AI
- Result-stage verification: once the input passes, we run the real simulation, the software that actually computes from physical law. Its output is compared against the value the AI had predicted before that simulation ever ran. This is a check of whether the AI's prediction matches reality, a different kind of check from the input stage. The AI commits to a prediction before seeing the answer, which matters: it's not the same as showing it the answer and asking "does this look right to you?"
When a check fails, what goes back to the AI isn't the correct answer, it's an error message describing what's wrong. The AI never learns the actual result; it has to fix its own work blind.
The same AI failure patterns, across unrelated fields of calculation
We applied this same verification approach to several distinct kinds of engineering calculation. What's notable: even when the underlying physics had nothing in common, the AI made the same kinds of mistakes.
- Wrapping the answer in an extra layer instead of returning it in the requested shape directly, like sealing a completed form inside another envelope before handing it over. The content can be entirely correct, but it no longer matches what the receiving system expects, and can't be read automatically
- Writing down a final number that doesn't match what the (correct) formula actually produces
The fact that the same failures showed up regardless of the calculation domain has a useful flip side: mechanical verification works regardless of the domain, too. If the AI's weaknesses are universal, the countermeasure is universal as well.
The harder problem that remains
Everything above assumes "correct" can be clearly defined. For simple geometries and simple physical conditions, you can check against an exact theoretical solution. As the problem gets more complex, though, an exact solution often stops existing at all.
We ran into this directly: conditions where an exact solution was available at first, and then weren't once the problem got more complex. In those cases we had to fall back on empirical correlations, approximations grounded in experimental data rather than theory. The very definition of "correct" gets fuzzier as complexity increases.
There's no single fix for this. Instead, we stack several independent mechanical checks, each imperfect on its own, to build up confidence without ever asking the AI to be the judge.
- Convergence checks: does the result settle toward a stable value as resolution changes? This doesn't tell you the result is correct, but it does tell you, mechanically, whether it's numerically stable
- Cross-checking against an independent method: solve the same problem with a different approximation or technique and see if the results agree. Without an exact solution to compare to, two independent paths arriving at the same answer is itself evidence
- Building complexity on a verified foundation: rather than trying to verify a complex case outright, start from a simple, verifiable one and add complexity incrementally. That way, only the newest addition is in question at any point, which keeps the problem tractable
- Being honest about confidence: a comparison against an exact solution (verification) and a comparison against an empirical correlation (validation) carry different weights of confidence. Naming that difference explicitly in the report, rather than blurring it, is part of what makes the verification honest
And when even all of that is exhausted and something still can't be settled mechanically, the answer isn't to hand the decision to AI, it's to escalate to a human expert. The principle isn't "when mechanical checks run out, let AI decide," it's "when mechanical checks run out, a person decides." Keeping that consistent is the core of the design.
A question that will only matter more as AI improves
As AI-generated calculations and decisions work their way into more of everyday operations, the question of how to verify that output isn't going away. Letting AI make the call can look efficient, but it may just be a deeper reliance on AI judgment dressed up as verification.
At Mathematical Physics Labo, we're testing this approach to verification in the field, starting from engineering and physics, domains where the standard for "correct" is relatively well defined.
How to start a conversation
"Is it safe to hand this analysis to AI?" "I don't know how to verify what the AI gave me." Inquiries at that stage are very welcome. Mathematical Physics Labo can support you with AI used actively throughout, paired with a built-in mechanism for verifying its output mechanically.