{"id":5624,"date":"2026-09-22T06:08:47","date_gmt":"2026-09-22T06:08:47","guid":{"rendered":"https:\/\/falcoxai.com\/main\/ai-chatbot-accuracy-financial-queries-saturn-research\/"},"modified":"2026-09-22T06:08:47","modified_gmt":"2026-09-22T06:08:47","slug":"ai-chatbot-accuracy-financial-queries-saturn-research","status":"publish","type":"post","link":"https:\/\/falcoxai.com\/main\/ai-chatbot-accuracy-financial-queries-saturn-research\/","title":{"rendered":"AI Chatbot Accuracy: 57% Wrong on Financial Queries"},"content":{"rendered":"

Technology firm Saturn put more than 10,000 money-related questions to 18 AI models, including ChatGPT, Claude, Copilot, Grok and Gemini. The answers were wrong 57 per cent of the time. On harder questions requiring more than one calculation, the average error rate hit 88 per cent, and some models got 99 per cent of them wrong. Calculation errors, omitted tax changes, invented rules. If your team is using a general-purpose chatbot to answer anything with numbers and regulations attached, those are your odds.<\/p>\n

That finding says less about AI capability than about deployment discipline. A consumer chatbot with no grounding data, no validation layer and no audit trail was never built for rules-based work. Below, we break down why these failures happen, and the specific controls that separate a reliable deployment from a liability.<\/p>\n

10,000 Questions, 18 Models, and a 57% Error Rate<\/h2>\n

Look at how Saturn ran the test, reported by the FT on 19 September 2026. More than 100 money-related questions, each put through free and paid tiers, repeated up to five times. That repetition matters. The same question asked twice did not reliably produce the same answer, which is the opposite of what any auditable process requires. Free versions performed worse than paid ones, so the tier your team quietly signed up for is part of the risk.<\/p>\n

These were rules-based questions with published, checkable answers. Now apply that to your own work. Cost-per-unit calculations, scrap rate math, tolerance conversions, CAPA write-ups citing clause numbers, supplier audit summaries. If a model invents a tax rule, it will invent a standard clause just as confidently.<\/p>\n

\"Bar<\/figure>\n

Where the Models Actually Broke Down<\/h2>\n

Three distinct failures showed up in the Saturn results: calculation errors, omitted upcoming tax changes, and hallucinated rules. Lumping them together as “AI is unreliable” is lazy. Each one has a different cause and a different fix, and only one of them is really about the model itself.<\/p>\n

Multi-step maths: why the error rate jumped from 57% to 88%<\/h3>\n

A language model predicts tokens. It does not compute. On a single-step question it can often pattern-match its way to a plausible number, which is why the overall failure rate sat at 57 per cent rather than something worse.<\/p>\n

Chain two or three operations together and the plausibility trick collapses. Errors compound across steps, and nothing in the architecture flags the drift. That is why the jump to 88 per cent on multi-calculation questions is predictable rather than surprising. The fix is not a better prompt. It is routing arithmetic to a deterministic tool, a calculator, a spreadsheet engine, a rules engine, and using the model only to interpret the question and present the result.<\/p>\n

Stale and invented rules: models that missed upcoming tax changes<\/h3>\n

Missing an upcoming tax change is a retrieval problem. The model’s training data has a cut-off, and without a live connection to an authoritative source, it answers confidently from whatever it last saw. Your ISO revision, your customer’s updated spec, your new regulatory threshold: same exposure.<\/p>\n

Hallucinated rules are worse because there is no source at all. The model invents a plausible-sounding regulation and states it flatly.<\/p>\n

Notably, free-to-use models gave more inaccurate responses than paid versions. That gap is the distance between casual use and deployment-grade tooling, and most teams are sitting on the wrong side of it.<\/p>\n

Why Financial Queries and Manufacturing Queries Fail the Same Way<\/h2>\n

Tax rules are structured, versioned, jurisdiction-specific, and they change on dates that are published years in advance. That description fits your world almost word for word. Tolerance bands, ISO clause requirements, calibration intervals, customer-specific PPAP thresholds, supplier scorecard weightings: all structured, all versioned, all revised on a schedule.<\/p>\n

So when someone in your quality team types “does this batch meet spec?” into a general-purpose chatbot, they are running the same pattern that produced Saturn’s results. The model retrieves a half-remembered rule from training data of unknown vintage, then performs arithmetic it was never engineered to perform reliably. It does not know which revision of the spec is current. It does not know your customer waived clause 8.5.1 last quarter.<\/p>\n

The queries in your plant that are secretly multi-step calculations<\/h3>\n

Most people underestimate how many steps their question contains. “Is this supplier still approved?” means: pull the current scorecard formula, weight OTD against PPM against corrective action closure rate, compare to the threshold in the current supplier manual, then check whether any of those inputs are stale. Four steps, minimum. Saturn’s models failed 88 per cent of multi-calculation questions on average.<\/p>\n

Same story with “when is this gauge next due?”, which needs the calibration interval, the last certified date, the usage-based adjustment rule, and the holiday calendar. Or “can we ship this lot?”, which spans spec limits, deviation status, and customer notification requirements.<\/p>\n

Ask people to list the questions they put to a chatbot in a week. Then count the calculation steps hiding in each one. Anything above one step belongs in a governed system with a deterministic calculation layer, not in a chat window.<\/p>\n

\"Side-by-side<\/figure>\n

What Separates a Governed AI Deployment From a Chatbot Guess<\/h2>\n

The controls that would have caught most of Saturn’s failures are not exotic. They are architectural decisions made before anyone types a prompt. A governed deployment answers three questions on demand: where did this fact come from, what computed this number, and which version of the rule applied on which date.<\/p>\n

Separate retrieval, reasoning and calculation into different components<\/h3>\n

Stop asking one model to do three jobs. Retrieval pulls the clause, spec or threshold from a single controlled source you own, not from model memory. Calculation goes to a deterministic tool: a function, a query, a spreadsheet engine. The model’s only job is to read the question, call the right tool, and write the answer in plain language.<\/p>\n

Two more controls sit on top. Every rule set carries an effective date, so the system knows which version applies to a batch produced in March. And the system must be allowed to refuse. If retrieval returns nothing relevant or coverage is thin, the correct output is “no confident answer, escalate,” not a fluent paragraph.<\/p>\n

Build your own benchmark: 100 real questions with verified answers<\/h3>\n

Saturn used more than 100 money-related questions, repeated up to five times per model. Copy the method. Pull 100 real questions your team actually asks, have a subject matter expert write the verified answer for each, and lock that set away as a held-out test.<\/p>\n

Run it before go-live, then again on every model update, prompt change or document reload. Repetition matters: ask each question five times and check whether the answers match each other, not just the key. Track pass rate over time. A vendor demo is not AI accuracy testing; your own benchmark is.<\/p>\n

The ROI Case for Narrow AI Over General Chatbots<\/h2>\n

Price the risk before you price the savings. The formula is simple: error rate multiplied by cost per error, set against hours saved multiplied by loaded hourly rate. A tool that is wrong most of the time only pays off when a wrong answer is cheap to catch and cheap to fix.<\/p>\n

In regulated quality work, it almost never is. A misquoted tolerance that reaches a customer costs a containment action, a corrective action report, and a slot on your next audit finding list. No amount of saved typing covers that. Saturn’s researchers framed the financial version bluntly: relying on AI answers to tax questions “could lead to serious financial losses.”<\/p>\n

Tasks where 57% accuracy is still useful, and tasks where it is not<\/h3>\n

It is useful when a human reviews every output anyway and the model’s job is to reduce blank-page time. First-draft deviation write-ups. Extracting fields from a supplier certificate into a structured form a reviewer confirms. Summarising a 40-page audit report into candidate findings someone then verifies against the source. In all three, the error surfaces inside the workflow, before it reaches a customer or a regulator.<\/p>\n

It is not useful anywhere the output becomes the decision. Spec conformance calls, calibration due dates, clause applicability, anything feeding a certificate of conformance. Those need deterministic calculation and a controlled source of record, with the model restricted to finding and phrasing, never deciding.<\/p>\n

Narrow deployments also give you something a general chatbot cannot: a measurable accuracy figure on your own task. Run a hundred known cases, count the misses, and you have a number you can put in a business case and defend in an audit.<\/p>\n

\"Bar<\/figure>\n
\n

Ready to find AI opportunities in your business?<\/strong>
\nBook a Free AI Opportunity Audit<\/a>. It is a 30-minute call where we map the highest-value automations in your operation.<\/p>\n<\/div>\n

What Accuracy Testing Becomes Standard Practice in 2026<\/h2>\n

Saturn’s work marks a shift from AI enthusiasm to AI evidence. Vendors have spent three years selling capability demos. Buyers are about to start asking for measured error rates on their own tasks, in their own jurisdiction, against their own verified answers.<\/p>\n

Expect internal AI governance to start looking like calibration management rather than IT procurement. You already run gauge R&R on measurement devices. An AI tool that answers rules-based questions is a measurement device, and repeatability is part of the spec. Saturn repeated each question up to five times for exactly that reason.<\/p>\n

The three questions to ask any AI vendor about measured accuracy<\/h3>\n

Ask these before the pricing conversation, not after:<\/p>\n