• Computer Science > Computation and Language [Submitted on 23 Jan 2026] Title:Gated Tree Cross-attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs View PDF HTML (experimental)Abstract:Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for downstream reasoning. • However, directly injecting explicit syntactic structure into an existing checkpoint can interfere with its pretrained competence. • We introduce a checkpoint-compatible gated tree cross-attention (GTCA) branch that reads precomputed constituency chunk memory while leaving backbone architecture unchanged. • Our design uses a token update mask and staged training to control the scope and timing of structural updates. • Across benchmarks and Transformer backbones, GTCA strengthens syntactic robustness beyond continued-training baselines without compromising Multiple-Choice QA performance or commonsense reasoning, providing a practical checkpoint-compatible route to more syntax-robust decoder-only LLMs. • References & Citations export BibTeX citation Loading…
Article Summaries:
- Summary
A new method, Gated Tree Cross‑Attention (GTCA), enables the injection of explicit syntactic structure into decoder‑only large language models without altering their pretrained weights. GTCA adds a lightweight branch that reads precomputed constituency chunk memory, applies a token‑update mask, and uses staged training to control when structural updates occur. Experiments across multiple Transformer backbones show that GTCA improves robustness to minor grammatical perturbations while preserving performance on multiple‑choice question answering and commonsense reasoning tasks. The approach offers a practical, checkpoint‑compatible route to more syntax‑robust decoder‑only LLMs.
Sources: