• Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates AuthorsNikhil Prakashâ  , Donghao Ren, Dominik Moritz, Yannick Assogba View publication Copy Bibtex Prior studies investigating the internal workings of LLMs have uncovered sparse subnetworks, often referred to as circuits, that are responsible for performing specific tasks • Additionally, it has been shown that model performance improvement through fine-tuning often results from the strengthening of existing circuits in the model • Taken together, these findings suggest the possibility of intervening directly on such circuits to make precise, task-targeted updates • Motivated by these findings, we propose a novel method called Constructive Circuit Amplification which identifies pivotal tokens from model reasoning traces as well as model components responsible for the desired task, and updates only those components • Applied to mathematical reasoning, it improves accuracy by up to +11 • 4% across multiple models while modifying as little as 1

Article Summaries:

  • Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates AuthorsNikhil Prakashâ , Donghao Ren, Dominik Moritz, Yannick Assogba Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates AuthorsNikhil Prakashâ , Donghao Ren, Dominik Moritz, Yannick Assogba Prior studies investigating the internal workings of LLMs have uncovered sparse subnetworks, often referred to as circuits, that are responsible for performing specific tasks. Additionally, it has been shown that model performance improvement through fine-

Sources: