Berk Bayri

If AI helps you solve it, you have learned it

A common misconception about AI, work and organizations, tested against the evidence.

The myth
If AI helps you complete a problem correctly and faster, it has improved your learning.
The reality
Assisted performance and durable learning are different outcomes. AI can make you look better while it is present without improving what you can do when it is gone.

Explanation and evidence

The argument

AI can improve your performance on a task without improving your ability to perform the task.

A randomized field experiment with roughly 1,000 high-school students makes the distinction unusually visible. Students using unrestricted GPT-4 performed substantially better while practicing mathematics, but later scored worse than the no-AI control group on a closed-book exam after the tool was removed. A guardrailed tutor that pushed students through the reasoning process avoided that collapse.

A 2026 large-scale observational analysis of millions of math-learning interactions found a similar pattern worth taking seriously: students completed AI-susceptible problems faster, while later proctored retention on those problem types declined. That study is a preprint and uses a quasi-experimental design, so it should not be treated as universal proof. It strengthens the reason to separate completion from learning.

AI can help you produce the evidence of knowing without producing the knowledge itself.

Why people believe it

The feedback arrives immediately. You solve more problems, finish the reading faster and get unstuck.

That feels like learning because traditional study often couples progress and understanding. AI can uncouple them: the task moves forward even when some of the cognitive work moved to the model.

What the evidence says

The effect depends heavily on how AI is used. A system that gives hints, asks questions and preserves the learner's effort can support learning differently from one that supplies finished reasoning and answers.

So the relevant distinction is not "AI versus no AI." It is AI as tutor versus AI as substitute.

The better question

After using AI, ask:

Could I explain this from scratch, solve a similar problem without the tool, or recognize when the AI's answer is wrong?

If not, the session may have improved completion more than learning.

Sources

Generative AI without guardrails can harm learning: Evidence from high school mathematics

PNAS · 2025-06-24

Faster Completion, Less Learning: Generative AI Reduced Study Time on Math Problems and the Knowledge They Build

arXiv · 2026-05-27