When does an AI answer become knowledge?

By Luuk Joseph Soons · CounterProof Research · 30 October 2026

An AI model gives you an answer. It is clear, detailed and plausible. You ask another model to check it. The second agrees. What do you now know that you did not know before?

Perhaps quite a lot. Perhaps only that two models can produce the same answer.

The difference matters when the answer becomes a line of code, a scientific claim or a decision that someone else must live with. It also matters over time. What will the next researcher, colleague or AI agent inherit: a useful discovery, an unresolved conjecture, or a mistake that has acquired authority through repetition?

This is the question behind our research: under what conditions does AI help us learn more about the world, and under what conditions does it merely help us produce more claims about it?

A small example with real consequences

Imagine a developer building a tool that exports records in batches. Each batch begins after the timestamp of the last record in the previous batch. An AI assistant writes the code and explains why no records will be missed. A second reviewer accepts the explanation. The existing tests pass.

Then someone asks a small question: can two records have the same timestamp?

They can. A new test puts such records on opposite sides of a batch boundary. One disappears from the export. The code moves past its timestamp before fetching it.

This is an invented example, but it shows several different kinds of progress. The first answer supplied an implementation and a claim. The second supplied agreement. The test exposed a condition under which the claim was false. A repair and a successful rerun could then establish that the revised code handles this particular case.

That would still leave other questions open. Does the export behave correctly while records are changing? Does it include what the user intended to export? A passing test establishes less than the sentence “the software is correct”. Its value depends on knowing what it actually checked.

The useful contribution might come from a person, a model or a tool that neither can dispense with. What matters here is whether the work gives us better grounds for a conclusion, and whether those grounds remain available for examination.

The origin of an answer does not settle its value

It would be a mistake to dismiss a sound argument because a model generated it. It would be the same mistake to accept an unsound one because a professor wrote it.

Consider a recent mathematical example. In Remarks on the Disproof of the Unit Distance Conjecture, nine mathematicians present what they describe as a human-verified reconstruction of an OpenAI-generated counterexample to a conjecture of Erdős. Their version simplifies and generalises the argument, and identifies earlier mathematics on which it depends. The reader is offered a proof to examine, beyond the model's assurance that it has solved the problem. Alon and colleagues, abstract and §1.

The distinction is important even for those of us who cannot personally check that proof. We depend on expertise, published arguments and the possibility of competent challenge. Naming a human checker does not make an argument infallible. It does make sense to ask what was checked and where the reasoning can be inspected.

Nor does the phrase “next-token prediction” decide whether a contribution is valuable. Describing how an answer was generated does not tell us whether its argument holds. A system may find a solution through a process very different from ours and still give us something we can establish and use.

There is a separate philosophical question about whether the model itself knows or understands. We need not settle it before asking whether its contribution helps people acquire knowledge. Equally, useful contributions do not settle that philosophical question in the model's favour.

This essay offers no final definition of knowledge. Philosophers disagree about its analysis. But we can distinguish a true claim, good grounds for accepting it, and understanding why it holds. A lucky correct answer does not by itself amount to knowledge. Conversely, reasonable grounds can support a belief later shown to be false. Confidence and certainty are not the same thing. Stanford Encyclopedia of Philosophy, “The Analysis of Knowledge”.

When agreement adds less than it appears to

Return to the export tool. Suppose both reviewers worked from the same design note, which assumed that timestamps were unique. Their agreement would not give us two separate checks of that assumption. Both could carry it forward untouched.

Different model names do not settle this problem. Models may receive the same material or share influences from training. Similar answers can also arise because the answer is straightforwardly correct. We must distinguish these possibilities before treating resemblance as evidence of dependence, or agreement as independent corroboration.

Agentic Stemmatics investigates such questions about transmission, shared errors and model relationships. In the paper's terminology, a single answer is a draw from a model lineage, rather than the lineage itself. When we infer relationships from outputs, we need repeated observations and tests against alternative explanations. Documentary records may provide more direct evidence. Resemblance in one answer is not enough to reconstruct a family tree.

This is one part of a larger inquiry. When an executed test can settle the relevant code question, use it. When it cannot, we still need to weigh arguments, sources and uncertainty. We do not have to reconstruct every model's ancestry before making any justified use of AI.

An accurate sentence can belong to a misleading report

Now imagine the release summary says: “Both AI reviewers approved the implementation, and all existing tests passed.” Every word could be true. Yet the account would mislead if it omitted the newly discovered missing record.

The problem lies in what the report leads its reader to conclude. A manager may reasonably understand it as a recommendation to release. The omitted qualification changes that decision.

This is the concern of Agentic Representation: how selection, emphasis and connections between statements shape an account. Checking sentences individually is necessary work, but it may leave the most important problem untouched.

The same distinction applies when a professor uses AI. Assistance with finding literature, comparing arguments or drafting prose need not diminish the work. The question is whether the resulting account fairly presents the evidence and whether the professor can answer for its claims. A bibliography that looks complete may omit the strongest objection. A precise description of an association may be arranged to suggest a cause.

Human authorship has never guaranteed these virtues. A human signature on an AI-assisted report does not guarantee them either. Responsibility requires identifiable people with the ability, time and authority to arrange competent checks, challenge the account and correct it. This can involve a division of expertise; it need not require one person to repeat every specialist check.

What the next session must inherit

Suppose the export bug is fixed. A month later, a new agent reads the original design note and rebuilds the same mistake. The previous team learned something, but the arrangement failed to carry that lesson into later work.

Saving a transcript is only a beginning. The obsolete assumption must be identified. The relevant test must be available. The next participant must encounter the correction and apply it appropriately. A later change in requirements may then require another revision.

Agentic Institutions studies these conditions of continuity and correction. A library, retrieval system, working rules and human oversight can help. Their presence does not demonstrate learning. We need to look at subsequent work: did the correction change what happened, and was that change useful?

Our essays on musical traditions approach this through another experience. A tradition survives through people who preserve, interpret, teach and revise it. Preservation alone cannot tell us whether what survives deserves its authority. A scientific institution must make room for a successor to discover that the inherited account was wrong.

That freedom includes criticism of the institution's founder, its preferred models and its own research methods. Otherwise, an archive can become a mechanism for repeating authority more efficiently.

More output is not yet more discovery

These questions extend beyond checking individual answers. AI may change which problems researchers choose, which explanations they consider and which results receive attention. We could become better at answering familiar questions while becoming less willing to ask unfamiliar ones.

AI could widen inquiry too: by helping us explore unfamiliar literature, proposing alternatives we had overlooked, or turning a vague conjecture into something we can test. Those contributions would matter even before a new discovery was established. We should investigate when assistance expands our capacity to inquire, as well as when it makes familiar answers easier to repeat.

Lisa Messeri and M. J. Crockett examine these risks in their work on AI and illusions of understanding in science. They warn that tools promising productivity may encourage researchers to overestimate their understanding and narrow the range of approaches they consider. This is a reason to investigate how AI changes inquiry, not evidence that every use of AI has those effects. Messeri and Crockett, Nature, 2024.

There is also a concrete question about what future models learn from. Research on model collapse shows that certain forms of repeated training on generated data can lose information. Other experiments found that retaining original real data while accumulating generated material avoided collapse in the settings tested. These findings concern particular training regimes; they establish neither inevitable decline nor a universal cure. Shumailov and colleagues; Gerstgrasser and colleagues.

Using an AI-generated document in a later conversation is not itself the same operation as retraining a model. Both raise questions about inheritance, but evidence about one cannot simply be transferred to the other.

We should therefore ask what keeps inquiry in contact with something beyond its own previous assertions: new observations, experiments, checkable arguments and serious objections. A larger collection of fluent text is not an adequate substitute. Nor should we assume that people necessarily become less capable through AI use. Whether assistance develops or displaces their ability to investigate is a question we can study.

What this asks of us in practice

For a consequential AI-assisted claim, a reader should be able to ask:

• What exactly is being claimed? About which version, circumstances or population?
• What supports it? An observation, an argument, a test, a source, or agreement that may share the same assumptions?
• What would change the conclusion? Which objection has been considered, and what remains unknown?
• What will survive a correction? Who will update the account, the working rule and the next decision?

The amount of checking should fit the decision. An exploratory suggestion and a release decision do not require identical treatment. Extra reviewers, longer records and more elaborate procedures must justify their cost. If a simpler method gives equally useful grounds, we should prefer it.

Much of this belongs to established scientific and engineering practice. “Agentic Epistemics” is our name for a research programme within that wider tradition, not a claim that AI has made earlier epistemology obsolete. The work is to understand what changes when generation, criticism and memory are increasingly performed with models, and to test which arrangements actually help.

CounterProof begins with software review. A client should receive a clear account of what was examined, what the evidence supports and what remains unresolved. Our broader ambition is to build an institution that can improve its work across people, sessions and models. That remains an ambition to be examined, not a benefit established by naming the institution.

The three programmes address different parts of that task. Tracing a source does not establish its truth. A faithful account does not ensure that anyone acts on it. An institution can preserve both with great care and still preserve an error. Each programme must establish its own contribution.

The philosophical questions remain: what should count as knowledge when discovery is distributed across people and machines? How should authority be shared? What would progress look like if no individual could survey all the work on which a result depends? These deserve a fuller paper.

For now, we can begin with the next answer on the screen. What does it give us reason to believe? What can we do to find out? And if we discover that it is wrong, will that discovery reach the person or agent who comes after us?

Source and scope note: This is an explanatory essay, not a report of a new experiment or a demonstrated improvement from CounterProof's methods. The software example is fictional. The mathematical example follows the cited authors' account; we have not independently verified their proof. The research discussions draw on the cited publications and retain their stated limits.

Prepared with Codex (OpenAI). This essay has not undergone scholarly peer review.

CounterProof Research is part of a commercial adversarial-review practice of Clavestra Capital Limited (Malta, C 113987). The author has a financial interest in methods that could benefit that practice.

Read alongside Different writers, shared traditions and Earned, not inherited. The research papers develop the programmes and their limits; the introductions offer shorter explanations.

Cite as: Soons, L. J. (2026). When does an AI answer become knowledge? CounterProof Research, 30 October 2026.