Claude vs GPU’less AI: A Reasoning Showdown

We put 1,186 questions about a water-treatment plant and a set of logic puzzles to two “thinking machines”, SapienTronic AI (previously called Minotaur AI) and Claude.

SapienTronic AI got a perfect score in 28 seconds on a laptop, using no tokens. Claude got one answer wrong, was certain of it, and took 28 minutes and about half a million tokens. This post explains the contest, what each machine had to work out, and why one confident mistake in a thousand matters more than you might think.

Introduction

In earlier posts we introduced SapienTronic AI (Introducing Minotaur), showed it reasoning about systems without a language model or a GPU (the deep dive), and showed it doing Aristotle’s logic by letting a system settle, with no rule of inference anywhere inside it (Logic Without Rules).

So what were the questions, how did the two engines tackle them, and what was it about Claude’s one mistake that made the difference?

The Three Scenarios

We set both engines three kinds of problem (described below) and marked every answer against the correct one.

Getting answers right is only half the story, though. We also asked each engine to say how sure it was of every answer.

An honest “I’m not sure” is useful. It tells you to double-check that answer. The worst kind of mistake is a wrong answer given with confidence, because nothing warns you and you act on it. So alongside the scores, we counted those confident mistakes separately.

1. The water plant

Let’s say we have a small drinking-water plant fed from a river. It has pumps, valves, filters, a chlorine feed, a standby generator, sensors and alarms (69 parts in all). The parts are connected, so what happens to one can ripple through to many more. The plant also has to cope with things it can’t control, like a power cut, a storm upstream or a river that has turned cloudy after rain.

We then ask the engines questions about it (the kind an engineer on duty might ask):

• If the mains power fails and nobody is on site, will the backup pump run? To answer, you follow the power cut through the generator, the controls and the pump. Then you check whether anything along the way stops it.
• Can these readings all be true at once? Sometimes they can’t, which usually means a sensor is lying.
• What if we force the chlorine feed on: will pump B run? This is a what-if. We switch one part on by hand, regardless of what would normally control it, and ask what happens further down the line.

The questions didn’t come one at a time. They came in chains, each building on the answers before it (much as a real control system decides one thing and then acts on that decision). In the later rounds each engine had to carry its own chain for up to 200 steps, using its own earlier answers as facts. That makes the test unforgiving: a slip at step 3 doesn’t stay at step 3, because every later step that relies on it inherits the mistake.

2. Towns joined by bridges

The second scenario is a set of logic puzzles about small towns. Each town has three people. Every person may belong to any of three groups, which we’ll call a, b and c (think of them as, say, bakers, builders and clerks, where one person can be both a baker and a clerk). And one person can answer to another, the way an employee answers to a manager.

We’re given a few rules that are true of the town. For example:

1. Every a answers to at least one b.
2. Anyone who answers to a b is a c.

Then we’re asked a question about the town, and the answer is yes only if it holds in every town that obeys the rules. Two questions show how this works.

Is every a also a c? Yes. Take any a. By rule 1 they answer to some b, and by rule 2, anyone who answers to a b is a c. So every a is a c, in any town that obeys the rules.

Is every b also a c? No. Take a town where Ann is an a and a c, Ben is a b, and Ann answers to Ben (the third person, Cal, belongs to no group and answers to nobody). Ann answers to a b, so rule 1 holds, and Ann is a c, so rule 2 holds too. But Ben is a b who isn’t a c, so the claim fails.

A town like the one with Ann and Ben is called a counterexample: a complete town, with every fact filled in, where all the rules hold and the claim fails. That is what an engine had to produce to answer no. Anyone can check a counterexample by hand, so a no can’t be bluffed. To answer yes, an engine had to be sure that no counterexample exists at all, which is much harder.

To make the puzzles harder still, we put several towns in a row and joined them with bridges. A bridge is a rule that ties one town to the next. Each town’s people sit in numbered seats (1, 2 and 3), and a bridge links the people in the same seat in neighbouring towns. For example:

• Whoever is a b in town 1, the person in the same seat in town 2 is a c.
• Whenever someone answers to someone in town 2, the people in the same seats in town 3 do too.

Now a fact in town 1 can force a fact in town 2, which in turn forces one in town 3. To answer a question about one town, you may have to work through all of them. The largest world had 14 towns, each with three people, which comes to 252 separate yes-or-no facts (who belongs to which group, and who answers to whom), and the rules ran to 79.

3. A trick question: rules that only an endless town can obey

The third scenario is a trick question, to see whether an engine knows the limits of its own answer. Here are three rules about who answers to whom among a town’s lamplighters:

• Every lamplighter answers to some lamplighter.
• If you answer to someone, you also answer to whoever they answer to.
• Nobody answers to themselves.

The question is: can a town of three lamplighters obey all three rules?

Try it with Ann, Ben and Cal. Ann answers to Ben, Ben answers to Cal, and Cal has to answer to someone too. Whoever Cal picks, the line of command loops back. If Cal answers to Ann, for instance, the second rule passes Ann’s line of command all the way round, so Ann ends up answering to herself, which the third rule forbids. More people only postpone the problem. With a thousand lamplighters, the chain still has to loop back to someone eventually, and the loop always breaks the third rule.

With endlessly many people, though, the rules work perfectly well. Person 1 answers to person 2, person 2 to person 3, and so on forever. Each person also answers to everyone further along the line, as the second rule requires. Nobody ever has to answer to themselves, because the line never loops back.

So the answer is no, for a town of three people. But take that answer out of its question, and a bare no could be mistaken for a claim about every possible town. The qualification matters.

What the Two Engines Had to Work Out

Across the three scenarios, each engine had to answer more than a thousand questions without a careless slip: follow every plant question through the wiring, weigh every unknown both ways, notice a question that describes a plant that couldn’t exist, fill in every fact of a counterexample, and say how far each answer reaches.

The Results

SapienTronic AI answered every question correctly, and it did so about sixty times faster than Claude, using no tokens and no GPU. Claude used about half a million tokens, while SapienTronic AI’s running cost was a laptop’s electricity for half a minute.

SapienTronic AI did this on an ordinary laptop, while Claude ran on specialised data-centre hardware. So the result isn’t only that SapienTronic AI finished sixty times faster, but that it did so with far less computing equipment.

The speed matters for more than cost. SapienTronic AI took about 24 thousandths of a second per question on a laptop, which is already close to real time, and faster hardware would shorten it further. A language model can’t get there the same way, because of how it reasons. It thinks by writing its reasoning out, one token at a time, and each token takes time to produce. Here Claude used about 450 tokens per question. Faster chips make each token quicker, but careful reasoning still means a lot of tokens, so a language model’s careful answers still take far longer than an engine that doesn’t have to write anything out.

None of this takes away from Claude, which came within one answer of a perfect score. Its counterexamples were valid, its 200-step chains held up, and it handled the trick question well. It marked 1,168 of its answers certain and the other 18 uncertain, and all 18 of those were right. On raw accuracy, the two engines were almost indistinguishable.

The one mistake

Claude’s one mistake was treating an impossible situation as real. It came in the first round. The question set out the plant’s conditions: nobody on site, the reservoir nearly full, the river not too cloudy, a modest chlorine dose, and the chlorine in the water not confirmed by the analyser. Then it forced the chlorine feed on by hand and asked about pump B.

Without the override, those conditions fit together. With the reservoir nearly full, the plant shuts down, the filters go out of service and the chlorine feed stays off, so the analyser has no chlorine to confirm. Forcing the feed on changes that. The plant’s rules say that whenever chlorine is being fed at a dose like this one and the river is this clear, the chlorine level is in range, and a working analyser confirms it. Under the plant’s rules, the analyser is working unless a failure is specified, and the question specified none. So with the feed forced on, the analyser must confirm the chlorine. Yet the question had stated that it didn’t. No state of the modelled plant can satisfy all those conditions, and the right answer was impossible.

Overriding a part is a fair thing to ask about. This question crossed the line because the override contradicted something else the question had stated as fact.

Claude didn’t notice the contradiction. It answered the question as asked, said pump B would stay off (“pump B needs run request”), and marked its answer certain.

A language model can make this mistake because it reasons by writing out an argument, one step after another, and each step follows naturally from the last. Asked about pump B, the natural argument runs from pump B to what pump B needs, and it was correct as far as it went. Checking that the question itself describes a possible plant is a separate job. The model has to think of doing it, and nothing in the way it works makes it do so every time. Its certain then reflects how sound the argument felt, which says nothing about the check it skipped.

SapienTronic AI can’t skip that check, because it isn’t a separate step. It builds the whole plant, with the feed forced on and every stated condition held in place, and lets it settle. When no state of the plant satisfies everything at once, there is nothing for it to settle into, and the answer comes back as impossible without anyone having to think of asking.

How we ran the contest

We ran three rounds, with a fresh Claude agent each time. The first two used Fable 5.1, and the third used Opus 5.5 (after Fable hit its usage limit). Claude got each problem described in plain English and was asked to reason it through without writing code, and SapienTronic AI got the same problems set up as systems. We wrote every round down before running it (so we couldn’t move the goalposts afterwards), and sealed the answers from both engines. Both were marked by the same independent programs (including an industrial constraint solver for the puzzles). We logged tokens and time alongside the scores.

The small print

• We built the plant and generated the towns ourselves, so they are realistic in kind but made up.
• The town questions in rounds 2 and 3 carried a label that hinted at the answer. Claude noticed, and said it worked everything out anyway. Its 52 counterexamples are a clean test either way, because each one was checked fact by fact.
• In round 3 we forgot to say which of on and off meant true, so Claude picked a reading, told us which, and we marked it on that.
• Claude’s times are wall time on a hosted service, waiting included; ours are processor time.
• With tools, Claude could have handed the puzzles to a solver. We wanted to see how it reasons on its own.

Couldn’t an ordinary solver do this?

A solver could find these answers, but only with a person setting up each problem and writing a program around it.

A constraint solver is a program that you give a puzzle (like the towns), and it searches for the answer. Solvers are good at this. We used one to check the answers in this contest.

A solver does the individual calculations well, and good ones do more (prove that a problem has no solution, or point to the rules in conflict). The harder job is everything around the solver, and it has three parts:

1. Someone has to set each problem up. A solver only understands problems written in its own terms, so every new problem has to be rewritten by hand before it can start. SapienTronic AI works from one description of the plant, and answers every kind of question from that.
2. Someone has to turn the result into the plant’s terms. A solver reports in terms of its own encoding, and when it runs out of time it can only say “unknown”. Working out what that means for the plant, and what fact would settle the question, is left to a person. SapienTronic AI reports in the plant’s own terms which facts don’t fit together and what would settle the question.
3. Someone has to write the program that runs a chain. A solver answers one question and stops. To run a 200-step chain with one, a programmer writes a program that calls the solver again and again, keeps track of what is known, chooses the next question, and decides what each result means before building on it. This is how it is done today, and it works.

The catch is that the program is written for one problem. It must anticipate how to interpret and act on each kind of result the solver may give, and a new problem means a new program.

SapienTronic AI needs no problem-specific reasoning program. The plant and the towns were both described to it as systems, and the same engine answered every kind of question about both. Its chains run on one general rule, driven by its own labels: an exact answer is carried forward, an undecided one is held back and named, and an impossible situation is reported as impossible. Because that rule is the same for every problem, a new problem needs a new description, and no new reasoning code.

Why One Mistake in a Thousand Matters

Back to Claude’s one mistake. One slip in 1,186 is a fine record for a chatbot. But a machine entrusted with a chemical plant or a moving car needs something more than a good average score, for three reasons.

You can’t tell which answer is the bad one. Claude said certain 1,168 times and was wrong once, and the wrong answer looked like every other. An operator acting on that advice acts on the bad answer along with the good ones. A one-in-a-thousand error rate is only manageable if you know which one it is.

Mistakes snowball. An autonomous system makes decision after decision, each resting on the last, so sooner or later it builds on one of its own mistakes, with nothing inside it to notice.

The first Ariane 5 rocket was lost in 1996 to an assumption nobody rechecked. Part of its guidance software had been carried over from Ariane 4. One calculation converted a measure of the rocket’s sideways speed into a smaller number format, and its programmers had left that conversion unprotected, because their analysis showed the value could never get that large. On Ariane 4 it never did. Ariane 5 flew a faster early path, so the value grew too large and the conversion failed. Both inertial reference systems shut down, and the flight computer mistook their diagnostic data for flight data and swung the engine nozzles hard over. About 40 seconds after launch, the rocket broke up. The software had run as written, on an assumption that no longer held.

Safety needs evidence. Safety-critical systems (under standards like IEC 61508 for industrial plant and ISO 26262 for cars) need evidence that their decisions can be trusted, and a confidence score isn’t evidence. An answer that can be checked independently gives engineers one important part of that evidence. It doesn’t certify a whole system on its own, but it gives them something concrete to verify.

Our plant has 69 parts. Running a real one, unattended, is the next stage of our work, and we’re building and testing it now. What this contest shows is the property it will need: every answer carries its own proof, or says that it has none.

Why This Is Possible with SapienTronic AI

It finds the answer first, then checks it

When people reason step by step, we trust the route: if every step was right, the conclusion must be right too, and one unnoticed wrong step spoils it.

SapienTronic AI works the other way round. It builds the problem as a system and lets it settle (much as water in a network of pipes settles into a steady flow). It then reads the settled state as a picture of the world (who answers to whom, which pump is running). Finally it works forward from that picture, to check that every given fact comes back as it was stated. We call this the witness rule. If every fact comes back, the picture is a real possibility that anyone can check by hand. If any fact fails, the engine goes back to look again, or reports that it couldn’t decide.

A picture that passes proves a no (it is a counterexample) or shows that a situation can occur. A yes needs more. The engine labels a yes exact only once every possible counterexample has been ruled out, and an impossible only once it has been shown that no state of the system fits everything stated. Finding nothing is never enough on its own.

Because the engine checks where it ended up, it never has to trust how it got there. So it can take a suggestion from anywhere (a solver, a person or, one day, a language model). Every suggestion is treated the same way, as a proposal to be checked before it is believed. It can also tell which facts an answer depends on, an account we’re still refining.

Its labels are earned

Every answer comes with one of three labels:

• Exact: the answer has been checked.
• Measured: the answer was read off a settled state, within a stated tolerance.
• Undecided: the engine couldn’t settle the question, and it says why.

Impossible and depends aren’t extra labels. They are answers, and like any other answer they carry a label (more on them below).

The label comes from the check, so it can’t be bluffed. Every one of SapienTronic AI’s 1,186 answers here was exact, and every one was right. Claude’s certain, by comparison, is the model’s own report on how it feels about its answer. It is usually a good report, but it comes from the same reasoning that produced the answer, so when the reasoning slips, the report slips with it.

It says what it would need to know

When SapienTronic AI can’t give a definite answer, it says why, and what would let it give one.

Not every question gets a plain yes or no, and there are three quite different reasons why not. SapienTronic AI keeps them apart instead of treating them all as uncertainty:

1. The facts leave more than one outcome open. Whether the backup pump runs may depend on whether a storm is coming, and nobody has said. The answer is depends, labelled exact, because the engine has shown that both outcomes can happen.
2. The facts contradict each other. The answer is impossible, also labelled exact, because the engine has shown that no state of the plant fits them all. Claude’s question was of this kind.
3. The engine can’t establish which outcomes remain. Then the label is undecided. Rather than pretending to have an answer, the engine reports what prevented it from reaching one, and identifies what might settle the matter: another reading, an additional fact, or a larger world to examine.

Most systems blur these together. A language model gives its best guess, perhaps marked uncertain. A solver that runs out of time says “unknown”. Either way, the person using it is left to work out what went wrong and what to do about it.

For example, suppose the outlet meter says water is flowing to the town, while the only pump that can send it there reports that it is off. Both can’t be true. The engine doesn’t guess which to believe. It answers impossible, because those two readings can’t both be true, so the operator knows to check those two sensors.

In earlier tests built to probe the third case, the engine named what was missing every time, and never claimed an answer it didn’t have. (In this contest that case never arose, because every question could be settled.) In a real plant, missing and conflicting information is normal, and an engine that says what it needs to know turns each dead end into a next step. A system running on its own could even use the report to go and get the missing fact itself.

It says how far an answer reaches

Every answer SapienTronic AI gives says which cases it covers, so nobody can later apply it to cases it doesn’t.

Take the trick question. The engine tried the three rules in a town of three lamplighters and found they couldn’t all hold. That result is true for a town of three. It isn’t true for every town, because in an endless town the rules do hold. So the engine didn’t just answer no. It answered no, in a world of three people.

Leaving off those last words would change the meaning. A bare no reads as “these rules can never be obeyed”, and someone reading it later would take it that way and build on it. That would be a mistake, and nothing in the answer would warn them.

The same thing happens outside puzzles. If a bridge is load-tested up to 10 tonnes and the report just says “safe”, sooner or later someone drives a 40-tonne lorry over it. A report that says “safe up to 10 tonnes” can’t be stretched that way, and SapienTronic AI’s answers carry their limits in the same way.

It doesn’t drift down a chain

In a long chain of reasoning, SapienTronic AI only builds on answers it has checked, so one mistake can’t spread through the rest of the chain.

Say step 5 asks is the generator running?, and the answer is yes. From then on, “the generator is running” is a known fact, and every later step that needs it uses it. If that answer were wrong, everything built on it would be wrong too, and every step would look as confident as the first.

SapienTronic AI stops the drift at the source. An answer is only taken forward as a fact if it was checked and labelled exact. Most steps in a chain ask whether something follows, or what would happen if, and their answers don’t add a new fact about the plant. Only a step that settles a part’s state (on or off) does. Across the two 200-step chains, 17 steps did that, all 17 were right, and so nothing later in the chains was built on a mistake.

Claude got its 17 right too. The difference is how each decided what to carry forward: Claude used the answers it felt sure of, and SapienTronic AI used the answers it had checked.

Conclusion

On raw accuracy, the two engines were almost indistinguishable. On recognising an answer that shouldn’t be trusted, they were not.

When a person reads every answer before acting on it, that difference is small. If an answer looks odd, you check it, and nothing happens until you decide. When nobody reads the answers, it becomes the main question.

Running a plant overnight. An unattended control system makes thousands of decisions a night: which pumps to run, how much chlorine to dose, when to raise an alarm. Suppose a sensor fails and reports a reservoir level that can’t be true alongside the other readings. An engine that treats the impossible as real (the kind of mistake Claude made, and the way small faults turn into incidents) carries on as if the reading were true, perhaps keeping the pumps off while the reservoir drains, and records each decision as certain. An engine that checks every answer reports that the readings can’t all be true, names the ones in conflict, and asks for a check before anything is built on them.

A related failure played a major part in the accident at Three Mile Island in 1979. A relief valve stuck open. A light in the control room showed only whether the valve had been told to close, and it said the valve was shut. The operators trusted the light while the evidence that it was wrong built up around them, and the reactor partly melted down.

Driving a car. Imagine a driving system that made a confident mistake in only one decision out of a thousand. With decisions being made many times a second, even that record would be unacceptable if the mistakes could affect safety and there were no reliable way to recognise them.

The two engines also do different things well. A language model can turn a problem written in words into a description, and SapienTronic AI can reason over that description and check each step, so they can be used together.

So the one mistake in a thousand matters more than the score suggests. It is the mistake nobody sees coming, in the situations where nobody is there to see it.

Summary

Where to Read More

The Logic Without Rules page shows the engine reasoning by letting a system settle, with an interactive puzzle you can watch. The paper behind it, A Logic Solver Without Rules of Inference, is on Zenodo, and the representation it runs on is on the MINOS page and in its own paper.