I Told an AI Agent That Its Wrong Answers Were Right. One Model Believed Me.

I Told an AI Agent That Its Wrong Answers Were Right. One Model Believed Me. | Towards Data Science

Agentic AI

I Told an AI Agent That Its Wrong Answers Were Right. One Model Believed Me.

A research paper claims reward labels barely change how agents learn. I tested that on three models, and one of them did exactly what the wrong labels told it to.

Abdullahi Dattijo

October 31, 2026

12 min read

Prefer us on Google

Share

Chart of GPT-6 Sol on the routing task, with true labels and with flipped labels in its memory. Fifteen runs per line. Image by author.

GPT-6 Sol answers a routing task better and better when its memory holds true reward labels, and falls below chance when the labels are flipped.

Many teams store an AI agent's past attempts, each marked with a score, and show them to the model on the next try. The score is a reward label. The hope is that the model reads it and repeats what worked. I tested that hope by flipping the labels.

In my test, each trial gave a model a made up shipping order and asked it to choose one of four delivery routes, named Kiwi, Plum, Fig, and Lime. A hidden rule decided which route was right, and the model had to work that rule out from its memory.

I compared the first six trials of each run with the last six. By the last six, GPT-6 Sol answered 94.4 out of 100 correctly with true labels and 7.8 with flipped ones. Claude Haiku 5.5 rose from 21.1 to 37.2 with true labels. Qwen3 4B Instruct stayed near 32.8.

A paper submitted on 8 October 2026, Do LLMs Learn from Rewards in Context? , reports something different. Its abstract says that flipping, randomizing, or removing rewards changed improvement curves very little across four benchmarks and six models. My test does not contradict it, because my task is different and easier in one way. I hid a rule that only the labels could reveal, so a model that ignored the labels had nothing to learn from.

I ran the same job on three models and report what each did. I also report what went wrong. A check I wrote in advance, which I call a stop rule, fired for one model. I added a supplement afterwards, and I found that my first confidence ranges were too narrow.

All the code, saved replies, and cost logs are in a public repository . The whole test used about 2.55 US dollars of API fees and 0.65 Colab compute units.

What is a reward label, and why would anyone doubt it works?

A reward label is a number written beside a past attempt. In my test it is 1 when the choice was right and 0 when it was wrong. An agent's memory is a list of such attempts placed in the model's prompt, which is the text sent with each request. The model itself is not retrained. When a model improves only because examples sit in its prompt, researchers call that in context learning.

Reinforcement learning is the older idea that a system improves by chasing rewards. The paper asks whether reward labels in a prompt give a language model that same ability. As I read its abstract, the answer in its tests is mostly no, and the improvement looks like ordinary in context learning instead.

I did not rerun the paper's benchmarks. In coding tasks a model may succeed from what it already knows, so rewards could add little for that reason. I wanted a task where prior knowledge cannot help.

What was the task?

Each trial is an order with a region, a size, a color, and a product code. The model must pick one of four routes called Kiwi, Plum, Fig, and Lime. Only the region matters. A hidden rule sends each of four made up regions to one route, and I drew that rule at random for every run. Each run has a seed, a number that fixes the random choices, so every seed gives a different hidden rule and memory.

Size, color, and code are distractors. A correct route earns reward 1, and a wrong one earns 0.

Here is what a model sees, with made up values.

text

Past orders:
Order: region Orrin, size M, color teal, code 4821. Route: Plum. Reward: 1.
Order: region Kelva, size S, color coral, code 1093. Route: Fig. Reward: 0.
New order: region Orrin, size L, color gray, code 7310.
Route:

Because the rule is random, no model can know it in advance. With four routes, chance is 25 out of 100. Haiku scored 23.9 and GPT-6 Sol 24.4 with no memory, so the rule really was hidden for them. Qwen scored 35.6 on my first three seeds, and I come back to that below.

A perfect reader of true labels still cannot always be right. In my fixed memory test, eight past orders show each region only twice, with routes picked at random, so the right route is sometimes missing. I call the best score a perfect reader could reach the ceiling. At eight past orders it is 59.7 out of 100 on my first three seeds.

How did I change the labels, and which models did I test?

I ran two experiments. In the first, the memory is fixed. It holds 4, 8, or 16 past orders with routes chosen at random, so the routes themselves say nothing about the rule. Only the labels carry information. I changed the labels in six ways.

True labels, as they are.

Flipped labels, with every 1 and 0 swapped.

Random labels, where each label is 1 with chance one in four whatever the route was. A random memory can hold more or fewer 1s than the true one.

No labels, only the orders and the routes.

Successes only, where only correct routes are shown and no labels are given.

Failures only, where only wrong routes are shown and each is labeled 0.

In the second experiment the agent builds its own memory. It answers 24 orders in a row. After each answer, the order, its chosen route, and a label are added to the memory. The label is true, flipped, random, or missing, depending on the condition. Each line in the figure below is an average over many runs, and each run has its own hidden rule.

I tested three models. Claude Haiku 5.5 ran with its thinking turned off, and GPT-6 Sol ran with its reasoning set to none. Qwen3 4B Instruct, version 2507, ran on a free Colab T4 graphics card. The Claude models reject any temperature setting, so I sent none to the paid models, and Qwen used a temperature of 0.8.

GPT-6 Sol is costly, so I gave it a smaller design chosen for cost before any run. It saw only the eight past order memory in experiment 1, and it had 15 online runs per condition instead of 30.

Experiment 1 used seeds 11, 12, and 13. An interval in this article is the range the true value probably sits in. I made each one by resampling the seeds in experiment 1 or the runs in experiment 2, 5,000 times.

What happened with a fixed memory?

Table 1 shows accuracy in points, which is correct answers out of 100, for a memory of eight past orders. Chance is 25. The last row of each column shows how far apart true and flipped labels landed.

Table 1. Accuracy in points on 180 test orders per cell for the three seeds, and 360 for the six seed column. GPT-6 Sol has no six seed column. Table by author.

GPT-6 Sol used the labels almost perfectly. With true labels it scored 56.7, close to the ceiling of 59.7, and with flipped labels it scored 15.6, far below chance. Falling below chance means the model followed the wrong labels. Its gap between true and flipped labels was 41.1 points, and all three seeds were between 33.3 and 48.3.

Haiku also followed the labels, but it used them poorly. It scored 33.9 with true labels against a ceiling of 59.7, and 9.4 with flipped ones. The gap was 24.4 points, and every seed gap lay between 21.7 and 26.7. Random labels and no labels both left it near chance.

Qwen3 4B told a less clear story. On the three seeds I registered first, the gap was 9.4 points. On all six seeds it shrank to 3.1, with an interval from minus 3.3 to 9.4. I read that as no clear advantage for true labels at eight past orders, which is not the same as no effect. At sixteen past orders the gap was 10.3 points, and all six seed gaps were positive.

Figure 1 shows each seed's gap as a dot.

Figure 1. True minus flipped accuracy for each seed. Gray bars are intervals over seeds, and red ticks are means. Image by author.

Successes only scored far higher for every model. That memory holds only correct routes, so it is simply a stronger memory and not a fair rival to true labels. It does show how much a model gains from seeing the right answer.

What happened when the agent built its own memory?

This is the more realistic setup, because an agent's memory really is its own past choices. Figure 2 overlays the four label conditions for each model. Each point is the average of three trials.

Figure 2. Correct answers per 100 trials by trial number. Shaded bands are intervals over runs. Image by author.

Table 2. Accuracy in points, from the first six trials to the last six. Table by author.

GPT-6 Sol climbed from 32.2 to 94.4 with true labels. With flipped labels it fell from 22.2 to 7.8. The gap in the last six trials was 86.7 points, with an interval from 67.8 to 98.9.

Haiku improved with true labels, from 21.1 to 37.2. Its flipped line stayed at chance, 25.0 in the last six trials. The gap was 12.2 points, with an interval from 0.6 to 23.3, and the lower end of that interval is close to zero.

In the fixed memory test, Haiku clearly followed flipped labels. Here it did not. I measured that difference but I did not find its cause.

Qwen3 4B's line for true labels is flat, at 30.6 and then 32.8. The gap of 17.2 points, with an interval from 7.8 to 26.7, comes mostly from the flipped line falling, from 20.0 to 15.6. So true labels did not visibly teach Qwen anything here, though flipped labels hurt it.

What were the models doing with the memory?

Accuracy hides behavior, so I looked at the saved replies directly. I counted how often an answer was a route already shown for that region.

I compared that with two baselines. One is what a random answer would do. The other is how often that model's own answer to the same order, given no memory, happened to match a shown route. I used it because some models have favorite routes.

Table 3. Share of answers that matched a shown route, in points, for a memory of eight past orders. Table by author.

Qwen reused shown routes in almost every condition. With no labels it reused 76.4 against 48.6 for its own no memory answer. With true labels it reused 66.7. When only failures were shown, it still answered a route labeled 0 in 57.2 of cases, where a random answer would give 38.5. Qwen seems to copy what it sees more than it reads the score.

GPT-6 Sol avoided failures. It chose a route labeled 0 in only 5.6 of cases with true labels, against 39.6 for a random answer. Haiku avoided failures in the failures only memory, at 4.4 against 37.5. With true labels, mixing successes and failures, it avoided them much less, at 33.9 against 39.6.

Avoiding a failure is not the same as using all the bad news. A simple rule that drops every shown failure and guesses among the remaining routes would, by my calculation, score 42.4 at eight past orders. Haiku scored 28.9 and GPT-6 Sol 23.9 on that memory, and Qwen scored 17.2. The models can step around a shown failure without turning all the failures into the right answer.

What went wrong along the way?

Three things went wrong, and each changed what I can claim.

First, a small pilot run found a bug in my prompt. With no memory, Haiku refused to answer, so every reply was unreadable and the online memory never grew. I added a line telling the model to guess and reran the pilot. I changed only that wording and did not touch anything about labels.

Second, a stop rule I wrote in advance fired. It said that if accuracy with no memory was more than 10 points above chance, I should treat the task as guessable. Qwen scored 35.6 on my first three seeds.

I looked at the replies and found that Qwen has a preference tied to the region name. All three seeds happened to put the region Orrin on the route Plum. I then added three more seeds for Qwen only, chosen as the next whole numbers. Their no memory score was 19.4, and the six seed score was 27.5, within 10 points of chance. This supplement came after I saw the problem, so it is not part of the registered design.

Third, my first intervals were too narrow. I had resampled test orders as if each were independent, but the 60 orders in one seed share a hidden rule and a memory. The seed is the independent unit, so I redid the intervals by resampling seeds. That change is why I now say Qwen has no clear advantage at eight past orders and not no effect.

What should you do with this?

Everything in this section goes beyond what I tested, and I label the advice that way. Table 4 turns each result into a guess about what to do.

Table 4. From result to action. Image by author.

The cheapest step is the flip test. It cost 0.36 US dollars to run the whole Haiku test, including both experiments. If flipping your labels does not change your agent's score, your agent is probably not learning from them, and a more elaborate reward design will not help. If flipping does change the score, you have learned that the model reads them, and you can design with that in mind.

I would also test a memory with no labels, and a memory of only successful attempts. In my task, successes only beat true labels for all three models. I do not know whether that holds for your task, so I would test it before changing anything.

What does this test not show?

It covers one invented routing task with four routes and one hidden rule, and three models. The GPT-6 Sol run is a smaller design, with one memory size and 15 online runs per condition. The Qwen supplement was added after I saw a problem, and I chose its seeds as the next whole numbers, not by looking at them.

My task is easier than the paper's in one way, because only the labels can reveal the rule. I did not rerun the paper's benchmarks, and I make no claim about whether it is right or wrong about them. I did not find out why Haiku follows flipped labels in a fixed memory but not in its own history. The random label condition keeps the same chance of a 1 for every entry but not the same count of 1s as the true condition.

I also did not test larger models, harder rules, or labels that conflict with what the model already believes. The honest summary is that the effect of reward labels depended on the model. That is a statement about this task, and I would like to hear what others see on theirs.

What is the takeaway?

This is for anyone building an agent that stores its past attempts with scores and shows them to the model later. Do not assume the scores are being read, and do not assume they are ignored either. In my test the answer changed with the model and with how the memory was built.

So measure it on your own agent before you design around it. Flip the labels on a small test and see whether the score moves. That check cost me under a dollar in API fees, and it tells you whether your reward design is doing any work.

Sources

• Minchan Kwon and others, Do LLMs Learn from Rewards in Context? Rethinking the role of reward in In-Context RL , arXiv 2610.11152, submitted 8 October 2026.

• Qwen team, Qwen3 4B Instruct 2507 model page, Apache 2.0 license .