# Collation and the microscope: how agentic stemmatics and interpretability could check each other *CounterProof Research note, 16 October 2026, version 1.1 (version 1.0 is the deposited text; the post scriptum on Siglum was added the same evening and read cold by one Codex seat before it landed). Drafted by Luuk Joseph Soons with a Claude model (Anthropic) and revised by the author. Reviewed cold, before publication, in six rounds by two outside seats, Codex (OpenAI) and Kimi (Moonshot AI); the review records, the brief and packet each seat received, and the seat outputs are published as the supplement to this note's Zenodo record (Competing interest, below). This is a proposal, not a result.* ## Summary Interpretability asks why a model does what it does. Agentic stemmatics asks which models share errors or ancestry, and how much their agreement is worth as evidence. We argue the two could check or sharpen each other: internal features offer a second measurement of the descent signals our method relies on, and the philological reasoning about shared errors, completed with the probability under descent, the term that likelihood methods in phylogenetics carry, suggests how to weigh similarities between models as evidence of relationship rather than of convergence or other common causes. We drafted an experiment and put it through six rounds of review by models from two families other than the one used in drafting. The reviews did not converge on a design. The list of conditions in this note is our rewrite after the fourth round, read cold in a fifth and, after revision, in a sixth, revised on what each round found, and not read cold since by a model from outside the drafting family; it is our current statement of what a design that makes these claims has to meet, and it may change at pre-registration. We publish the list and move the design to a pre-registration. This is a proposal, not a result. **A grown system can be produced reliably long before anyone understands it.** Brewers made sound beer for centuries without knowing what yeast was. They kept and reused it from batch to batch, but the Bavarian decree of 1516 listed only water, barley and hops; the organism doing the work went unnamed (Niebrzydowski 2016). Quality was governed through what could be observed: the inputs, the practice, and the judgement of people who could tell a sound batch from a spoiled one. When microbiology reached the brewery, one of its early applied results came from inside one: Emil Christian Hansen's pure-culture brewing yeast at the Carlsberg Laboratory in 1883 (Carlsberg Group, n.d.; Hansen 1886, p. 167). Medicine offers a sharper version of the same pattern. In 1854 John Snow investigated cholera deaths around a water pump in London's Broad Street; the map he drew of them, published in 1855, supported the waterborne theory he had set out in 1849 (Snow 1849, 1855). The bacterium had been described the same year by Filippo Pacini (Pacini 1854), whose work was largely overlooked in his lifetime. Robert Koch's commission worked in Egypt in 1883 and then in Calcutta, where a report dated 7 January 1884 announced the bacillus isolated in pure culture (Koch 1884); Koch's claim that it caused cholera was disputed for years afterwards, notably by Max von Pettenkofer. Snow's object was transmission across a population, not the organism. He set the theory out in 1849 and had the evidence by 1855, some thirty years before the organism was isolated, and longer before it was generally accepted as the cause. Large language models are in a similar position. They are produced by training, not written line by line, so their builders know the recipe better than the result. Two research traditions approach that gap from opposite sides. **Interpretability** opens the model and tries to read its internal computation: its physiology. **Agentic stemmatics**, the subject of our main paper, stays outside and compares the outputs of many models, borrowing the philological method of collation, and the stemmatic inference built on it, for reconstructing the relationships among surviving copies of a text: something closer to its epidemiology. Our paper describes the two as complementary and proposes a division of labour (paper §10). This note asks whether they can do more than divide the work: whether each can test or sharpen the other at a known weak point. ## Two questions, two kinds of access Interpretability asks *why* a model produced an output. Mechanistic interpretability, the branch this note concerns, typically works on weights and activations: it identifies features the model represents, traces how they combine, and intervenes on them to test the account. Agentic stemmatics asks *which models share errors or ancestry, and how much their agreement is worth as evidence*. It treats each model output as a witness. Witnesses that share an improbable error may share a source; agreement among dependent witnesses counts for less than agreement among independent ones. Whether two witnesses are independent is itself something to measure, not assume. Turning these relationships into a judgement about which outputs are correct needs further assumptions, about ground truth and calibration, which our paper treats separately. | | Collation (agentic stemmatics) | Mechanistic interpretability | |---|---|---| | Access needed | Outputs only; works on closed models and across laboratories | Weights or internal activations; open models, or a laboratory's own | | Typical cost profile | Many queries, but each comparison is cheap and can run continuously | Expensive per investigation today, though automation is lowering the cost | | Strengths | Correlated error across many models; candidate descent relationships; where witnesses disagree | Internal representations and their causal role; in controlled settings, a contribution to recovering a planted objective | | Limits | What every witness omits; behaviour that leaves no trace in any output | Restricted to models with internal access; for this note's question, no settled way to weigh a cross-model similarity as evidence of ancestry | On the last clause of the interpretability column's Strengths cell, the evidence is still narrow. In a blind audit of one model with one deliberately planted hidden objective, three of four teams recovered it, two of them using sparse-autoencoder interpretability among other tools; the fourth, with API access only, did not (Marks et al. 2025). This does not show that concealed objectives can be detected reliably in arbitrary models. Access has an institutional consequence. A regulator, auditor or insurer can reanalyse a collation if the model outputs, the reviewers' outputs and the method were recorded at the time, with dates and model versions; hosted models change behind the same name, so outputs that were not recorded may not be regenerable later. Without the laboratory's cooperation, nobody outside it can in practice perform the activation-level analysis this note concerns on a closed model, so they can only take the laboratory's word for it, or fall back on its outputs, queried or recorded. Open weights remove that barrier, and independent analysis then depends on the ordinary conditions of reproducibility: the model revision, the analysis code, the extracted features and enough compute. For assurance regimes that need evidence a third party can re-examine, dated and versioned output records are therefore valuable in their own right, and they can only be made at the time. They also give later work a behavioural record against which an interpretability account of a historical model can be checked for consistency, though not a record of what the model contained internally, since different internal computations can produce the same outputs. ## 1. Internal features as a second measurement of descent Our paper names its own weakest link. Planted marks, such as watermarks, give a controlled test of inheritance: a student model trained on a marked teacher's outputs can carry a detectable mark (Sander et al. 2024; Gu et al. 2024). Fine-tuning on ordinary text can erase a learned mark (Gu et al. 2024), and an adversary can weaken an inherited mark by paraphrasing the training data, and remove it where the watermarking rules are known or recovered (Pan et al. 2025). But the method's larger ambition depends on **naturally occurring** habits: phrasings, error patterns and preferences that no one planted. Output similarity among public models has been shown to track known family membership (PhyloLM; Yax et al. 2025), without separating descent from the common causes set out in §2. Our paper has not established, and we have not found work that establishes, whether particular naturally occurring characters can be weighed as evidence of descent against those alternatives, on a population whose descent is known and whose common causes are recorded (paper §13; condition 2 below). Interpretability offers a second kind of measurement. Internal features can be compared across models, with crosscoders (Lindsey et al. 2024), whose experiments across models are preliminary, or with sparse autoencoders whose features are matched across models that share a tokeniser, tested on pairs within a family (Lan et al. 2025). Known relationships can then be reconstructed twice, once from output habits and once from shared internal features, and each reconstruction scored against the known relationships (paper §7.1). Two measurements on the same models are not independent by construction. Shared training data, objectives, architecture and fine-tuning can produce similar internal features and similar outputs without the relationship we are trying to detect, and matching features on the same prompts that elicit the output habits adds further dependence. Each arm is therefore judged by how well it recovers the known relationships, with controls built to separate descent from those common causes, rather than by how well the two arms agree with each other. Whether the two check each other, in the sense condition 8 below defines, is a further and separate question: whether one arm's errors are not the other's, and whether a combination rule fixed in advance beats both single arms on held-out lineages. Two separately scored reconstructions answer neither question. A failure to find an internal counterpart for an output habit has many possible causes: incomplete feature recovery, distributed representations, the wrong layers, poor matching across models, or an inherited behaviour implemented differently in a descendant. Such a failure is reported as "not detected by this instrument". For characters of a kind on which the instrument's sensitivity has not been measured, that is all it says; it is not by itself decisive against inheritance. ## 2. Weighing similarities between models Interpretability has a long-standing hypothesis, usually called **universality**: networks trained separately often learn similar features (Olah et al. 2020). The evidence is mixed rather than settled. Some features recur across separately trained networks and others do not (Li et al. 2016). Induction heads appear across a series of Anthropic models of many sizes, and a comment invited by the authors reported potential induction heads in public models (Olsson et al. 2022). Similarity between two models, neither of which descends from the other, has more than one cause, and they need separating. Some of it is convergence: separately trained networks arrive at the same feature because the task rewards it, one kind of what biologists call homoplasy, and what the philologist calls polygenesis, the easy mistake two unrelated copyists make alike. Some of it is common descent without direct descent: two descendants of one parent share what they inherited from it without either descending from the other, as two manuscripts copied from the same exemplar share its errors; such pairs are related, and for a target of direct descent they are the hard negatives of condition 2. And some of it comes from common causes short of ancestry: shared training data, shared architecture or tokeniser, or shared tooling at inference. For architecture, tokeniser and tooling the nearest textual analogue is two scriptoria that share conventions. For shared training data the nearest analogue is contamination from a source outside the line of descent that both witnesses drew on, of which the scribe's own memory of the text is the limiting case. A model's corpus is a source of that kind, far larger, and contamination is a case the manuscript method handles poorly, which is why a shared corpus is recorded as an attribute of its own (condition 2 below). None of these makes a shared feature useless as evidence. They mean that its evidential weight has to be measured, not assumed. The measure is a comparison: how probable is the shared feature if the two models are related by descent, against how probable it is under an alternative? The ratio of those two probabilities is the feature's weight against that alternative. A feature shared with probability 0.8 between related models and 0.4 between unrelated ones supports descent modestly. A feature that related models share only occasionally, but unrelated models almost never, can support it strongly when it does appear. A feature that is rare under independent origin but rarer still under descent counts against descent, however rare it is. A feature that related and unrelated models share alike has a ratio near one and carries no weight. Which features discriminate depends on the comparison: against unrelated models a rare inherited character can tell strongly, even one every descendant retains, while within an ancestry only characters that vary across it separate one branch from another; among them, the stemmatist's conjunctive errors and the cladist's shared innovations are the ones whose shared state also groups a branch. Support here means the likelihood ratio; turning it into a probability that two models are related needs the base rate of descent in the population, which a ranking endpoint does not require and a decision does. Our paper frames its rule in terms of improbability under independent origin (paper §4); that is one of the two terms, and a feature's weight depends on both. Because both probabilities concern the feature *as observed*, instrument error enters each of them. With several alternatives there are several weights, and how they combine into one score for a pair of models is a decision the test design has to state. Each alternative needs its own estimate. Seed replicas, models trained from the same recipe with different random seeds (Sellam et al. 2022; van der Wal et al. 2025), estimate how often features recur without weight descent under a shared recipe, data and tokeniser; they do not separate task-driven convergence from the effect of shared data. At the level of single neurons it has been measured on GPT-2 seed replicas: 1 to 5 percent of neurons had best-match counterparts in the other seeds whose activation correlation, averaged over those seeds, exceeded a random-rotation baseline by more than 0.5, a threshold the authors say is not a principled one (Gurnee et al. 2024), a recurrence rate at neuron level rather than feature level, so the instrument question remains. Shared data, shared architecture and shared tooling need controls built for the purpose, or must be held constant and declared as such. The instrument varies too: sparse autoencoders trained on the same model with different seeds can recover substantially different feature sets (Paulo and Belrose 2025), so instrument variance has to be measured, or held fixed and declared, alongside model variance. **Where this sits in existing work.** Lineage has been reconstructed from model outputs (PhyloLM; Yax et al. 2025), from weights (Horwitz et al. 2025) and from weights and outputs together (Gallagher et al. 2026). It has also been tested from internal representations: REEF compares centred kernel alignment between a suspect's and a victim's activations on the same inputs, and reports high values for fine-tuned, pruned and merged derivatives of one base model and low values for unrelated models (Zhang et al. 2025); it reports no probability under descent or under the alternatives, and its control for a shared corpus is a single independently trained model. Cross-model similarity of sparse-autoencoder features has been tested against a random-pairing null (Lan et al. 2025), and output-level agreement on model errors has been adjusted for the agreement expected between independent models given their accuracies (CAPA; Goel et al. 2025). None of these comparisons has a descent-side term, and CAPA does not use internal features. The comparison itself is old: likelihood phylogenetics compares trees by the probability of the data under each (Felsenstein 1981), and computational stemmatics has applied phylogenetic methods to manuscript traditions (Barbrook et al. 1998), the methods compared on artificial traditions by Roos and Heikkilä (2009) including parsimony, distance, network, compression and clustering methods. What we have not found is prior work that scores shared internal features of models, as an instrument observes them, by the ratio of their probability under descent to their probability under convergence and other common causes. Our earlier literature searches each missed at least one close work, so this should be held loosely; we would welcome references. ## 3. Collating automated interpretability Interpretability is scaling through automation. In one influential protocol, a model writes an explanation of what a neuron responds to, a simulator model, in the original work the same model by default, predicts the neuron's activations from that explanation, and the prediction is scored against the measured activations (Bills et al. 2023). Because scores in this protocol are anchored to measured activations, they are not mere agreement between models. Correlated error can still enter: explainers and simulators from related model families may share blind spots about what a description covers, and pipelines that let a model judge an explanation's plausibility are more exposed than pipelines anchored to held-out activations. These are risks to test, not established failures. The stemmatic toolkit suggests how: - Use a crossed design: every explainer paired with every simulator, across model families, with at least two models per family so that same-family pairs are not only self-pairs, and with capability measured separately. This separates the explainer's effect, the simulator's effect and their interaction. Then look at *errors*, not scores: on held-out inputs chosen to expose what an explanation omits, do related explainer and simulator pairs fail on the same inputs more often than unrelated pairs of similar capability? - Test whether fluency predicts error. Copyists tended toward easier readings, and the philologists' preference for the harder reading was a response to that; whether fluent model explanations are more often wrong is a hypothesis to check against accuracy, not a rule to assume. - Plant features with keys registered in advance and test whether the automated interpreter finds them without being told. This adapts the planted-error design our paper proposes for review panels (paper §7.2) to interpretability. None of this disputes the progress of automated interpretability. It would give its results a check that does not depend on the instrument's own family. ## An open direction: written reasoning and internal evidence Studies of reasoning faithfulness show that a model's written chain of thought can misrepresent the reasons for its answer (Turpin et al. 2023; Lanham et al. 2023). It is tempting to treat the written reasoning and the internal evidence as two witnesses to the same computation and collate them. We do not yet have a method for this. Internal states are not semantic steps that line up with sentences; an interpretability account of them is a partial reconstruction with its own errors; and written reasoning can itself take part in the computation rather than merely report it. Any disagreement could lie in either account or in how they were aligned. We note the direction and make no claim for it. ## Conditions a valid test has to meet We drafted an experiment to test the empirical claims of §1 and the estimable terms of §2 and put it through six rounds of review, each by two models from families other than the one used in drafting (see Competing interest), each given the draft and a review brief and no other document from us, in a fresh session. The reviews did not converge on a design. In every round after the first, each objection that both reviewers raised and rated blocking fell on text written to answer the previous round: six such objections across rounds two to five and none in the sixth, by our own classification of records that are not yet published and that no one outside the drafting lineage has audited. The most fully specified version scored two reconstructions separately rather than testing whether the methods check each other. The list below is our rewrite after the fourth round, built from the objections raised in review, whether by both reviewers or by one, and from our own re-reading, including screening passes by a model of the drafting family before and after the fifth round and before and alongside the sixth; the fifth round read it cold, rated two of its conditions blocking, one of them by both reviewers, and the list was revised on that round; the sixth round read the revised list cold and rated three objections to it blocking, touching conditions 3, 4, 7 and 8, none by both reviewers, and the list was revised on that round; it has not been read cold since that revision by a model from outside the drafting family. It is our current statement of what a design that makes these claims has to meet, and it may change at pre-registration. We publish it here and move the design to a pre-registration that will be published, with its own review, before any model is run. The list concerns the lineage experiment; the tests sketched in §3 have conditions of their own. The review records, with model identifiers, versions and dates for the drafting, reviewing and screening models, and the packet each reviewer received, are published with this note, as the supplement to its Zenodo record. 1. **The relationship is defined and the channel is matched.** Weight descent (a model trained from another's weights) and distillation (a model trained on another's outputs) transmit by different mechanisms and are scored separately; each related pair is scored as a positive under one channel, assigned in advance, and the design says whether a pair related under one channel enters the other channel's evaluation as a negative, labelled by channel; a pair related under both channels is assigned to one and enters neither as a negative. If cross-channel negatives are not used, the design tests detection within each channel and not discrimination between them. Model merging is out of scope unless the design carries it through its ground truth, recorded attributes and controls. The target is named: immediate parent and child, any ancestor and descendant, or relatedness. The similarity scores this design uses are symmetric, so it credits no direction to the method; directional signals, such as the weight statistics of Horwitz et al. (2025), are out of scope here, deferred rather than unavailable. The manuscript method's own source of direction, the separative error, presupposes a character a descendant could not plausibly lose; the class on which this has been tested, learned watermarks, can be lost (Gu et al. 2024), and whether any natural character meets that is untested. Where direction is reported it comes from known training order and is labelled as such. 2. **Every scored pair has known ground truth and recorded attributes.** For each pair of models the scored relationship is known, and the common causes the design claims to separate are recorded as attributes of the pair: shared recipe, pretraining data, fine-tuning data, architecture, tokeniser, tooling at inference. For most public models these attributes are undocumented; a population built for the purpose, or an open-data seed-replica suite that documents them within the one recipe it varies, is the way to know them. Which attributes the population can distinguish, and which it holds constant, is declared; declaring is not controlling, so a separation is claimed only where the population has identifying variation for it, matched or crossed positive and negative pairs with independent lineages in each cell, and nowhere else. Pairs that share an ancestor without either descending from the other inherit characters too; for a descent target (parent and child, or ancestor and descendant) they are the hard negatives and stay in the evaluation, and for a relatedness target they are positives. For the immediate parent-and-child target, ancestor-descendant pairs at greater distance are hard negatives as well and stay in the evaluation. 3. **The observation model is complete.** For each character, or for each class of character where the pre-registration so states, both probabilities are estimated from pairs of known relationship (condition 2), for the character as observed by the instrument: the rate at which pairs related by descent share it, and the rate at which pairs under each alternative the design distinguishes share it; no label of which characters were transmitted is needed or assumed. Not sharing a character carries information and enters the score. The rule for combining alternatives into one score for a pair, the handling of zero counts and of characters unseen in calibration, and the dependence model among characters are stated; grouping correlated characters reduces double counting and is not independence. 4. **Controls exercise the channel under test, and their scope is stated.** A positive control for weight descent is a marker implanted in a parent whose child is trained from the parent's weights; a student trained on a teacher's outputs controls the distillation channel. Markers are chosen, and their retention in the child established, by a route independent of the pipeline judged on them. Non-retention is decided only by that route, and its verdict is recorded before the pipeline scores any marker; a lineage whose marker that route finds non-retained is excluded before scoring and the exclusion reported; no lineage is excluded or replaced after scoring, whatever the pipeline reports. A marker counts as detected when the instrument stage of the pipeline, before any label-fitted scoring, reports a shared character matching a key registered in advance in a form that does not depend on the instrument's own features, under a criterion fixed in advance; the label-fitted scorer is not judged on markers. For the internal arm, behavioural retention does not establish that the internal feature was retained, and the design names its route to internal retention, with its own false-positive rate on marker-free models and declared as a weaker standard than the output arm's, or states that retention for that arm is behavioural only, in which case a miss is reported as either instrument insensitivity or reimplementation, the control curve describes detection given behavioural retention and is not a bound on sensitivity to internally retained features, and a failed bound in that arm is reported as uncontrolled and attributed to neither cause. Marker-free models go through the same detector; the marker-free unit is the same seed replica with the marker omitted and any implantation step applied without it, so that the pair differs only in the marker. A detection curve on implanted markers describes the instrument on those markers. The sensitivity hypothesis is a pair of bounds fixed in advance, each stated at the unit it is tested on: the proportion of marked lineages in which every marker at or above a stated strength is detected is at or above a stated rate; and the proportion of marker-free units on which any registered key is reported is at or below a stated rate, with the number of registered keys stated. Each bound is tested as a one-sided test at a stated level whose null hypothesis is that the bound is not met, so an instrument with too few marked lineages to reject it does not pass; the unit of each test is the seed replica, its marked lineage for the detection bound and its marker-free pair for the false-detection bound, with the number of lineages of each kind, and of marked lineages at each strength, stated. Misses below the stated strength are points on the curve, not failures. The marker test is a capability check on the instrument stage that every character-based method in an arm shares; sensitivity to markers is not shown to transfer to natural characters in either direction, so passing it licenses no claim about natural characters, and failing it bears on how that arm's non-detections are read (condition 7). A negative on a natural character is reported as "not detected; sensitivity to true sharing for characters of this kind unmeasured" unless the design has measured it. 5. **Fitting that uses the labels stays inside the fold.** A lineage here is a seed replica together with everything trained from its weights. For the distillation channel the held-out unit is the teacher's lineage together with every model trained on the outputs of any of its members, membership decided by a criterion fixed in advance; a pair whose members fall in different folds is not scored. A student initialised from existing weights belongs to its base's weight lineage as well as to its teacher's distillation unit; units that share a model are joined and held out together, so no model is in the fitting fold under one channel and in an evaluated pair under the other. Because joining is transitive, a design shows the fold structure after joining and the number of units left in each fold, and constrains cross-lineage initialisation (students trained from scratch, or bases and teachers drawn from disjoint blocks of lineages) so that something remains to hold out. Character selection, weights, thresholds and any grouping fitted on known relationships are fitted on lineages held out from evaluation, and every evaluated pair has both members held out; a prompt split alone leaves the lineage in sample. Holding out lineages tests transfer to new weight ancestry within the same recipe; transfer across recipes needs recipes held out, which the design states if it claims it. The marker-control lineages of condition 4, and their marker-free pairs, sit in a fold of their own, outside both the fitting and the evaluation lineages for natural characters; the pipeline's fitted parameters are frozen before any marker is scored, and the markers do not enter the fitting of natural characters. The instrument instance judged on markers is the same training run, parameters frozen, that scores the evaluation lineages; where its training is unsupervised, the marker-control lineages and the evaluation lineages enter its training set on the same terms, both in or both out, and the choice is declared (training on the activations is how an unsupervised instrument finds any feature, marker or natural, so the clause fixes the regime, not the exposure); where instances differ, instrument-instance variance enters both the marker curve and the intervals of condition 6. Unsupervised fitting on the models under test, such as training a crosscoder on their activations, is permitted when it uses no labels and is fixed in advance, and the design says whether its claim is about lineages not seen in fitting. 6. **Uncertainty is computed at the unit that varies.** Model pairs share models, prompts and the feature pipeline, and are not independent observations. Intervals come from separately seeded lineages, and over instrument seeds where instrument variance is not held fixed, and are labelled conditional on whatever is shared across them; power is computed from the design, by simulation where the dependence has no closed form, not from the number of pairs. 7. **Endpoints, comparators and the claims they test are fixed before any result is seen, and the claims are kept apart.** The design fixes in advance the primary metric; the margin that counts as improvement; numeric definitions of a ceiling effect and of a ratio near one; and the handling of multiple comparisons across arms, channels, targets, characters and the control tests of condition 4. The probability comparison of §2 is a rule of evidence, and this experiment does not test it. The experiment tests two empirical claims, each with a failure condition written down in advance, on held-out lineages, at the unit condition 6 requires: that natural characters in this population, as observed by the stated instrument, carry descent signal that at least one stated method extracts, the failure of which is a bound on those methods and not on the characters; and that an estimator built on the probability comparison of §2 extracts that signal. Comparators of both kinds, unsupervised and label-fitted, are stated in advance, and the role of each is named: what a gain, a tie and a loss against each one means for each claim is written down before any result is seen. A gain over any comparator is reported as the estimator's and is not attributed to the probability-ratio structure, which a discriminative model fitted on the same characters can learn as well. A ranking metric, if chosen, is preserved by any increasing transform of the score, so it validates no claim about a character's weight on the probability scale; any such claim needs a separate calibration assessment, fixed in advance. If a control bound of condition 4 is not passed, or the arm is reported as uncontrolled under condition 4, the arm's results on natural characters are reported together with the failed control; its non-detections are then not read as evidence against inheritance, since the instrument's sensitivity is unshown, while a held-out discrimination result stands on its own evidence against the known relationships, since sensitivity to markers is not shown to transfer to natural characters in either direction. The comparators, their forms and the dispositions belong to the pre-registration; this note states that they are fixed in advance, that each claim can fail, and what a gain may not be read as. 8. **"Check each other" has a definition and two endpoints.** The arms check each other in two senses, each with an endpoint written down in advance on held-out lineages, and the design states which combinations of the two outcomes license which conclusion. First, that one arm's errors are not the other's, with the error defined in advance for each pair, at an operating point fitted on the fitting lineages or as a rank error defined per pair, measured as each arm's error rate given the other's error against its error rate overall, in both directions, with uncertainty at the unit condition 6 requires, and with the null stated and a route to estimate it named: since the arms share models, prompts and pipeline, their errors cannot be assumed independent, so the endpoint is stated against a null for their co-occurrence fixed in advance, with its assumptions, and with what a fail means. Second, that a combination rule whose form is fixed in advance and whose parameters, if any, are fitted on the fitting lineages only, beats both single arms on the primary metric, with paired uncertainty at the unit condition 6 requires and multiplicity handled, so that what the combination loses counts against it as what it gains counts for it. Both endpoints are read only for arms that have passed the first claim of condition 7, on the method whose scores the arm contributes to them, named in advance. That is a restriction of scope to arms with standalone signal, stated as such and not as a necessary condition, since two arms each at chance can determine a target jointly; it is imposed because an arm at chance would otherwise pass the first endpoint by having no signal to share. If either arm fails that first claim, neither endpoint is read and the two-arm claim of §1 is untested; each arm's own results are still reported under condition 7. An arm whose control bound of condition 4 was not passed enters the endpoints with that reported, and how its non-detections count there is fixed in advance. Without endpoints of this kind, the experiment measures two reconstructions and says nothing about that claim. Conclusions are limited to the population tested; if the population is built for the purpose, its small models may not behave like production models. No design we have meets these eight conditions. The pre-registration will have to, in whatever form they then take, and until it does the claims in this note stand as argument, not as tested results. ## Limits - **Access.** The internal measurements work only where weights or activations are available. For closed models, collation remains available where internal analysis is not. - **No design yet.** The conditions above come from review of our own drafts, not from a design that meets them. Meeting them takes a pre-registration with its own review and a training budget for models with known descent. - **Open items the pre-registration must settle.** Among them: the route to internal retention for the internal arm, or the decision that its retention is behavioural only; whether a pair related under one channel enters the other channel's evaluation; the null for error co-occurrence and the route to estimate it; which method's scores each arm contributes to the endpoints of condition 8, and how an arm whose control bound was not passed enters them; the membership criterion for a distillation unit; which comparators, in what form, carry which dispositions, representation similarity of the REEF kind among them for the internal arm; whether the probabilities of condition 3 are estimated per character or per class of character, since characters inherited within one lineage need not recur in another; and the effect sizes and dependence model the power simulation of condition 6 draws on. - **The budget is in lineages, not pairs, and the floor is high.** Rejecting "detection rate below 0.8" at the five percent level with no misses needs fourteen marked lineages at that strength per channel, and twenty-nine for a bound of 0.9; the false-detection bound sets its own count, which exceeds fourteen for any bound below about 0.19 and reaches twenty-nine at 0.1; a distillation student trained from scratch is a further pretraining run; the fitting and evaluation lineages come on top, their number set by the power simulation of condition 6. That is a floor from the control fold alone, and a floor for a test that tolerates no miss at an uncorrected level: an instrument whose true rate is 0.95 passes it with probability about 0.49, and under the multiplicity rule of condition 7 the no-miss count rises, to twenty-three at a Bonferroni level over the eight control tests alone, and higher over the full family condition 7 names. The pre-registration sets the count by power, about thirty lineages at the uncorrected level for 0.8 power at a true rate of 0.95 against the 0.8 bound with two misses allowed, and may train the control fold's lineages before the fitting and evaluation lineages. Existing seed-replica suites (MultiBERTs; PolyPythias) may serve as a base population, PolyPythias for an output arm that collates on generated text. - **A first design separates less than §2 distinguishes.** A population of one recipe, one architecture and one tokeniser separates descent from same-recipe non-descent and from sibling descent; the other alternatives of §2 need populations that vary them. In such a population the weight-descent channel may sit near ceiling for a representation-similarity comparator of the REEF kind: REEF separates derivatives of a base from unrelated bases; whether it separates them from seed replicas of the base is untested; if it does, the character estimator has little room to improve on it in that channel, and the distillation channel, where no weights are shared, carries the informative comparison. We state this as a hypothesis to be measured, not a result. - **The internal arm may be uninterpretable.** Given the seed variance of sparse autoencoders, the internal arm failing its own sensitivity bound is a possible outcome; the design names a route for establishing internal retention (for example a probe fitted on the parent and applied unchanged to the child, declared as that arm's retention route with its own false-positive rate on marker-free units) or accepts that the arm's non-detections are then uninterpretable, and says in advance how that arm enters the endpoints of condition 8. - **Instrument reliability.** Interpretability tools have their own variance and failure modes. An internal measurement is a second measurement, not an oracle. - **Universality may cut deep.** If most internal features prove convergent, internal measurements may carry little descent signal. That would itself be worth reporting. - **No measurement yet.** This note proposes a combination and sets out the conditions a test has to meet. It reports no result. ## What this note does not claim It does not claim that interpretability is solved or unsolved, that agentic stemmatics can detect deception, or that internal similarity establishes descent. It claims that the two methods have different reach, that each has an open problem the other may help with, and that a test of the combination has to meet the conditions above, as we currently state them, before it is worth running. ## Post scriptum: Siglum (added 16 October 2026) Siglum, the project named in Competing interest, is being developed to measure whether two reviewers on a panel make the same mistakes more often than chance, with planted errors and known ground truth as its intended instrument (paper §7.2); it has measured nothing yet. It bears on this note in three ways, each prospective. A panel that discounts agreement between seats it takes to be related relies today on vendor labels and on output habits, both proxies for lineage and neither a measure of dependence; the second measurement of descent proposed in §1 could, if validated, give such a panel, where weights are open, an estimate of relatedness whose scoring inputs are neither. Whether reconstructed descent predicts the error co-occurrence a panel measures between its seats is a separate hypothesis, not a criterion of the reconstruction's validity: shared data or tooling could predict shared errors without descent, and accurate ancestry need not predict errors on a given task. The pre-registration may state it as an endpoint of its own, beside those of condition 8, which concern the two measurement arms and not reviewer seats; testing it on an existing panel would train no model, though the reconstruction it tests needs the population the Limits describe. And the planted-error design that §3 adapts to automated interpretability is the design Siglum is intended to implement, a statement of the project's intent, not something this note establishes. The interest declared in Competing interest applies to this paragraph as to the rest of the note. ## Competing interest CounterProof Research offers review and assessment services that use the methods described in our paper, and is developing Siglum, a project to measure reviewer independence. We therefore have a commercial interest in these methods being shown to work, and the review process described in this note is itself an instance of them. This note was drafted with a Claude model (Anthropic) and revised by the author; the developer of the drafting model is also the source of several of the interpretability works cited here. The six review rounds were run against our own drafts by models from two other families, OpenAI and Moonshot, each given the draft and a review brief and no other document from us, in a fresh session, though what a model's own harness adds to its context is outside the packet and was not recorded; the fifth-round Moonshot review is a re-run, on another model of the same family with a bounded output, after two attempts that were cut off at the output cap before reaching a verdict; the findings those attempts did produce are in the records and were dispositioned; the sixth-round Moonshot review ran on that model with the same bound, in one attempt. A model of the drafting family also checked references and facts during the rounds and screened the rewrite before the fifth round, the revisions after it before the sixth, and the sixth-round text alongside that round; those findings were applied but are not counted as independent review. Independence by family membership is a proxy, and §2 of this note lists the common causes that proxy does not exclude; our paper says independence between reviewers has to be measured (paper §7.2), and we have not measured it here. The review records, with model identifiers, versions and dates for the drafting, reviewing and screening models, and the packet each reviewer received, are published with this note, as the supplement to its Zenodo record; the pre-registration will be published so that it can fail in public. ## Sources - L. J. Soons, "Agentic Stemmatics", preprint. The text cited here is v1.38, the site copy at https://research.counterproof.io/agentic-stemmatics.html, with section numbers as in that copy, deposited on Zenodo on 16 October 2026 as version DOI 10.5281/zenodo.23424032 (concept DOI 10.5281/zenodo.22030516). Sections cited: §4 (similarity without descent: the improbability rule; engineered markers), §7.1 (validation against known relationships), §7.2 (independence measurement; planted errors; Siglum), §10 (related work and the division of labour), §13 (status of natural markers). - T. Sander, P. Fernandez, A. Durmus, M. Douze, T. Furon, "Watermarking Makes Language Models Radioactive", NeurIPS 2024 (spotlight). arXiv:2402.14904. - C. Gu, X. L. Li, P. Liang, T. Hashimoto, "On the Learnability of Watermarks for Language Models", ICLR 2024. arXiv:2312.04469. - L. Pan, A. Liu, S. Huang, Y. Lu, X. Hu, L. Wen, I. King, P. S. Yu, "Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?", ACL 2025 (long papers), pp. 13228-13251. arXiv:2502.11598. - C. Olah et al., "Zoom In: An Introduction to Circuits", Distill, 2020. https://distill.pub/2020/circuits/zoom-in/ - Y. Li, J. Yosinski, J. Clune, H. Lipson, J. Hopcroft, "Convergent Learning: Do Different Neural Networks Learn the Same Representations?", ICLR 2016. arXiv:1511.07543. - C. Olsson et al., "In-context Learning and Induction Heads", 2022. arXiv:2209.11895. - W. Gurnee, T. Horsley, Z. C. Guo, T. R. Kheirkhah, Q. Sun, W. Hathaway, N. Nanda, D. Bertsimas, "Universal Neurons in GPT2 Language Models", Transactions on Machine Learning Research, 2024. arXiv:2401.12181. - J. Lindsey et al., "Sparse Crosscoders for Cross-Layer Features and Model Diffing", Transformer Circuits Thread, 2024. https://transformer-circuits.pub/2024/crosscoders/index.html - M. Lan, P. Torr, A. Meek, A. Khakzar, D. Krueger, F. Barez, "Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders", 2025 (arXiv:2410.06981; first version 2024, titled "Sparse Autoencoders Reveal Universal Feature Spaces Across Large Language Models"). - E. Horwitz, A. Shul, Y. Hoshen, "Unsupervised Model Tree Heritage Recovery", ICLR 2025 (arXiv:2405.18432; first version 2024, titled "On the Origin of Llamas: Model Tree Heritage Recovery"). - S. K. Gallagher, S. Rallapalli, T. Brooks, C. Loughin, M. Sezgin, R. Yurko, "Analysis and Explainability of LLMs Via Evolutionary Methods", 2026. arXiv:2605.02930. - J. Zhang, D. Liu, C. Qian, L. Zhang, Y. Liu, Y. Qiao, J. Shao, "REEF: Representation Encoding Fingerprints for Large Language Models", ICLR 2025 (oral). arXiv:2410.14273. - S. Goel, J. Strüber, I. A. Auzina, K. K. Chandra, P. Kumaraguru, D. Kiela, A. Prabhu, M. Bethge, J. Geiping, "Great Models Think Alike and this Undermines AI Oversight", Proceedings of Machine Learning Research 267 (2025), pp. 19621-19678. arXiv:2502.04313. - J. Felsenstein, "Evolutionary trees from DNA sequences: a maximum likelihood approach", Journal of Molecular Evolution 17 (1981), pp. 368-376. DOI 10.1007/BF01734359. - A. C. Barbrook, C. J. Howe, N. Blake, P. Robinson, "The phylogeny of The Canterbury Tales", Nature 394 (1998), p. 839. DOI 10.1038/29667. - T. Roos, T. Heikkilä, "Evaluating methods for computer-assisted stemmatology using artificial benchmark data sets", Literary and Linguistic Computing 24 (2009), pp. 417-433. DOI 10.1093/llc/fqp002. - G. Paulo, N. Belrose, "Sparse Autoencoders Trained on the Same Data Learn Different Features", 2025. arXiv:2501.16615. - T. Sellam et al., "The MultiBERTs: BERT Reproductions for Robustness Analysis", ICLR 2022. arXiv:2106.16163. - O. van der Wal et al., "PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs", ICLR 2025. arXiv:2503.09543. - S. Bills et al., "Language Models Can Explain Neurons in Language Models", OpenAI, 2023. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html - M. Turpin, J. Michael, E. Perez, S. Bowman, "Language Models Don't Always Say What They Think", NeurIPS 2023. arXiv:2305.04388. - T. Lanham et al., "Measuring Faithfulness in Chain-of-Thought Reasoning", 2023. arXiv:2307.13702. - S. Marks et al., "Auditing Language Models for Hidden Objectives", 2025. arXiv:2503.10965. - N. Yax, P.-Y. Oudeyer, S. Palminteri, "PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting Their Performances in Benchmarks", ICLR 2025. arXiv:2404.04671. - J. Snow, *On the Mode of Communication of Cholera*, London: John Churchill, 1849 (a 31-page pamphlet); 2nd edition, much enlarged, London: John Churchill, 1855. https://www.gutenberg.org/ebooks/72894 - F. Pacini, "Osservazioni microscopiche e deduzioni patologiche sul cholera asiatico", Gazzetta Medica Italiana, Federativa Toscana, 2nd series, 4 (1854), nos. 50 and 51, pp. 397-401 and 405-412. - R. Koch, reports I, II and IV to VII of the German Cholera Commission (the third had not been published when the fourth appeared, by the editors' footnote), Deutsche Medicinische Wochenschrift 9 (1883), pp. 615 (Alexandria, 17 September 1883) and 743 (Suez, 10 November 1883), and 10 (1884), pp. 63 and 111-112 (Calcutta, 16 December 1883 and 7 January 1884), 191 and 221 (2 February and 4 March 1884); the report of 7 January 1884 announces the isolation in pure culture. Collected in *Bericht über die Thätigkeit der zur Erforschung der Cholera im Jahre 1883 nach Egypten und Indien entsandten Kommission*, ed. G. Gaffky, Arbeiten aus dem Kaiserlichen Gesundheitsamte 3, Berlin: Julius Springer, 1887. Also R. Koch, "Ueber die Cholerabakterien", Deutsche Medicinische Wochenschrift 10 (1884), no. 45, pp. 725-728; and the editorial on the differences between Koch and Pettenkofer, Deutsche Medicinische Wochenschrift 10 (1884), p. 464. - E. C. Hansen, "Undersøgelser over Alkoholgjærsvampenes Fysiologi og Morfologi. V. Methoder til Fremstilling af Renkulturer af Saccharomyceter og lignende Mikroorganismer", Meddelelser fra Carlsberg Laboratoriet 2, Hefte 4 (1886), pp. 152-167, which dates the practical pure-yeast trials to 1883 (p. 167). The 1883 paper in the same series is Part II, "Om Askosporedannelsen hos Slægten Saccharomyces", Meddelelser fra Carlsberg Laboratoriet 2, Hefte 2 (1883), pp. 29-86. French résumés in Résumé du compte-rendu des travaux du Laboratoire de Carlsberg, vol. 2. https://archive.org/details/meddelelser-fra-carlsberg-laboratoriet - Carlsberg Group, "Purifying yeast", n.d.: https://www.carlsberggroup.com/pursuit-of-better/scientific-discoveries/purifying-yeast/ - P. Niebrzydowski, "Keeping Beer Pure: 1516 Reinheitsgebot", Origins (Ohio State University), April 2016: https://origins.osu.edu/milestones/april-2016-keeping-beer-pure-1516-reinheitsgebot --- *[CounterProof](https://counterproof.io/) is a small practice of Clavestra Capital Limited (Malta, C 113987) that reviews AI-generated code and AI-assisted review processes for organisations where a wrong answer is expensive. Our interests in the methods this note discusses are set out in the Competing interest section above.* *Cite as: Soons, L. J. (2026). Collation and the microscope: how agentic stemmatics and interpretability could check each other. CounterProof Research, 16 October 2026, version 1.1; version 1.0 deposited as doi:10.5281/zenodo.23424154. Companion to [Agentic Stemmatics](https://research.counterproof.io/agentic-stemmatics.html), doi:10.5281/zenodo.22030516.*