Training a small language model on compositional opaque arithmetic did not reliably improve its ability to compose a new ordered operation—even when every primitive fact in the target domain was available.
Question
Correct answers to multistep problems leave an important ambiguity: did the model learn reusable structure, or did it benefit from exposure, labels, frequency, or a shortcut? We tested whether structural lessons in one domain transfer to a semantically disjoint target domain.
Design
We generated 16 independent semantic worlds. Each contains a fresh four-value operation that is deliberately noncommutative and nonassociative. The target test has 80 frozen nested expressions per world. The source is five-value opaque modular arithmetic with randomized digit and operator names.
Every critical curriculum contained 139 training examples: 16 target primitive facts, 75 source primitive facts, and 48 matched lessons. The structural arm received nested opaque-arithmetic compositions. Its control received an exact atomic placebo: a visible depth-one fact with the same answer, length, vocabulary, character multiset, label distribution, example count, exposure, batch schedule, and optimizer budget—but no valid composed operator application.
Primary result
All 16 paired worlds passed the prespecified full-training and primitive-mastery gates; none were excluded. Source structure won in 7 worlds, the placebo won in 8, and 1 tied. The one-sided exact paired-world sign test was 22,819 / 32,768 = 0.6964.
Interpretation
The preregistered conclusion is narrow: for this causal decoder, opaque modular source, ordered finite target population, exact atomic placebo, and fixed training budget, we found no reliable positive cross-domain transfer. This is a negative result, not an impossibility theorem about all language models or all forms of reasoning transfer.
What was held fixed
Each fit used 2,048 optimizer updates. The primary model was a small scratch-trained, character-tokenized, decoder-only Transformer. Eligibility was fail-closed: both compared models had to master every training example and every target primitive before their held-out target predictions counted.