• A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation AuthorsRuss Webb, Jason Ramapuram** View publication Copy Bibtex What research can be pursued with small models trained to complete true programs? • Typically, researchers study program synthesis via large language models (LLMs) which introduce issues such as knowing what is in or out of distribution, understanding fine-tuning effects, understanding the effects of tokenization, and higher demand on compute and storage to carry out experiments. • We present a system called Cadmus which includes an integer virtual machine (VM), a dataset composed of true programs of diverse tasks, and an autoregressive transformer model that is trained for under $200 of compute cost. • The system can be used to study program completion, out-of-distribution representations, inductive reasoning, and instruction following in a setting where researchers have effective and affordable fine-grained control of the training distribution and the ability to inspect and instrument models. • Smaller models working on complex reasoning tasks enable instrumentation and investigations that may be prohibitively expensive on larger models. • To demonstrate that these tasks are complex enough to be of interest, we show that these Cadmus models outperform GPT-5 (by achieving 100% accuracy while GPT-5 has 95% accuracy) even on a simple task of completing correct, integer arithmetic programs in our domain-specific language (DSL) while providing transparency into the datasetâ s relationship to the problem.

Article Summaries:

  • A research team has released Cadmus, a low‑cost system for autoregressive program synthesis that uses an integer virtual machine, a curated dataset of true programs, and a transformer model trained for under $200 of compute. Cadmus enables controlled experimentation on program completion, out‑of‑distribution reasoning, inductive inference, and instruction following, offering researchers fine‑grained control over training data and model inspection. In benchmark tests, Cadmus achieved 100 % accuracy on a simple domain‑specific language for integer arithmetic, outperforming GPT‑5’s 95 % accuracy, and highlighted how large LLMs can introduce unknown priors that obscure task‑specific analysis.

Sources: