Same score, same mind? We opened the box to find out

Three students hand in the same exam and all score 90. Do they think the same way? You cannot tell from the grade. Maybe they used the same method. Maybe they used three completely different methods that happen to land on the same number. The score is the outside. To know what is going on you have to look at their working.

This post is about doing exactly that to three tiny neural networks, getting the answer badly wrong on the first try, catching it, and what we found once we fixed it.

The setup: a tie we did not understand

In an earlier note we found that when the task is learning (work out which button pays off, from reward alone, and re-work it when we secretly swap the buttons), three different kinds of network all do about equally well. A network that remembers with its activity, one that rewires itself with a single rule, and one that rewires with a rule per connection: same score, near enough.

We had a tidy explanation for the tie. This kind of learning works by averaging reward over many tries, and averaging is a smoother, so we guessed it simply blurs away whatever makes the three networks different. Neat. Satisfying. And completely untested, because it is a claim about the inside of the networks, and all we had measured was the outside (the score).

So we opened the box.

The mind-reader

The trick: take a trained network, let it play, and at every step peek at its internal state (the activity it is holding, or the wiring it has rewired) and ask one question. From this alone, can I tell which button is currently the good one? We train a small decoder to try. Score it: 1.0 means the network’s insides perfectly carry that knowledge, 0.5 is a coin flip (they carry nothing), and below 0.5 should be impossible, because it would mean the insides are reliably, informatively wrong.

The part where we were wrong

The first time we ran this, it produced a bombshell. One network scored below a coin flip at reading its own mind. Taken at face value, it said a network was solving the task while carrying reliably-wrong gibberish inside: a genuinely exciting result, and one that would have forced us to walk back an earlier finding.

It was wrong, and it was our fault. Before we signed anything, a reviewer whose entire job is to attack our work read the code and found the flaw. Without meaning to, we had built the test so the mind-reader practised on one kind of day and was graded on the mirror-image kind. A reader that had learned the real pattern would then score perfectly inside-out: not random, but reversed. Our shocking “below chance” number was not an empty network. It was our own answer key printed upside-down.

We fixed the split and ran it again. The bombshell evaporated:

We kept the broken run on the record, right next to the corrected one, and retracted nothing, because the finding we had been about to walk back was never actually wrong.

What the fixed test showed

With the answer key the right way up, the honest picture appears: all three networks read the good button beautifully, and almost identically (0.91 to 0.97). Their insides are not different-things-blurred-into-agreement. They are the same skill, arrived at three separate ways.

Which quietly kills our own earlier guess. The networks do not tie because a smoother blurs away their differences. They tie because there are no differences to blur: each independently learns the same good habit of tracking which button pays.

One more trap: reading, or just a stopwatch?

A sceptic could still object. We always swapped the buttons at the same moment. What if a network is not reading the world at all, just running an internal stopwatch (“switch at the halfway mark”) that happens to line up? Same score, no actual thinking.

Cheap to check: move the swap. Make it early (trial 20) or late (trial 40) instead of the trained halfway point (30), and see whether performance holds.

Where it lands

The machines converge because they each learn the same real skill, not because a filter hides how they differ. That is a smaller, sturdier sentence than the one we started with, and we reached it by looking inside instead of trusting the score.

The part we are proudest of is not the result. It is that the mistake in the first version was caught by a check built to catch exactly that, before it hardened into a published claim. Publishing our failures is the visible half of honest science. Building the checks that catch them early is the other half, and this is the first time one paid for itself in a single afternoon.

Postscript: how deep is that tracking?

Same skill, three ways — but how deep does the skill run? When a network tracks which button pays, is it patiently weighing all its evidence since the start (a rich running tally of a whole lifetime), or just reacting to the last handful of tries (a short memory)? We looked. And once again the first look fooled us, in a way worth telling.

The first measurement said deep: the network seemed to carry the good-button knowledge spread thinly across many past trials, as if integrating a long history. Our reviewer caught that this was the measuring tool, not the network. The mind-reader was too simple to represent one particular combination — “this reward and that action, together” — so it smeared what was really a short recent-memory signal across many trials, making a shallow thing look deep. We gave the reader the missing combination, and the picture flipped.

So the learning is a short reactive memory, not a deep integration of a whole lifetime — and the same shallow window for all three network kinds. A smaller, truer sentence again, reached the same way: a check built to distrust our own tools caught the first version before it hardened.

Read the rigorous version

Every number here is a signed, dated entry in the open corpus, including the upside-down run we kept. Follow the sources under this post, or start with the Programme 3 synthesis.