Reinforcement Learning for Mathematical Reasoning in Small-Scale Language Models with Structured Policy Optimization
Identifikátory výsledku
Kód výsledku v IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26220%2F26%3A0200028" target="_blank" >RIV/00216305:26220/26:0200028 - isvavai.cz</a>
Výsledek na webu
<a href="https://ieeexplore.ieee.org/document/11268643" target="_blank" >https://ieeexplore.ieee.org/document/11268643</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/ICUMT67815.2025.11268643" target="_blank" >10.1109/ICUMT67815.2025.11268643</a>
Alternativní jazyky
Jazyk výsledku
angličtina
Název v původním jazyce
Reinforcement Learning for Mathematical Reasoning in Small-Scale Language Models with Structured Policy Optimization
Popis výsledku v původním jazyce
Advancing mathematical reasoning in small-scale language models remains a challenge due to their limited capacity and the high computational demands of standard reinforcement learning methods like Proximal Policy Optimization (PPO). To address this, a resource-efficient training pipeline based on Group Relative Policy Optimization (GRPO) is proposed, enabling fine-tuning of compact models under strict memory constraints. The proposed method introduces structured prompting to explicitly separate reasoning steps from final answers and applies a dual reward system to jointly optimize for format adherence and mathematical correctness. The training incorporates low-overhead techniques such as 8-bit optimization, mixed-precision training, gradient checkpointing, and accelerated decoding for efficient rollout and policy updates. Experimental results show a 50.95% accuracy on a benchmark reasoning dataset—GSM8K, outperforming several larger models—while training entirely on a single GPU. These findings demonstrate that small-scale models, when trained with structured reinforcement learning, can achieve competitive performance in mathematical reasoning tasks. The approach offers a practical pathway for deploying interpretable, reasoning-capable models in low-resource environments.
Název v anglickém jazyce
Reinforcement Learning for Mathematical Reasoning in Small-Scale Language Models with Structured Policy Optimization
Popis výsledku anglicky
Advancing mathematical reasoning in small-scale language models remains a challenge due to their limited capacity and the high computational demands of standard reinforcement learning methods like Proximal Policy Optimization (PPO). To address this, a resource-efficient training pipeline based on Group Relative Policy Optimization (GRPO) is proposed, enabling fine-tuning of compact models under strict memory constraints. The proposed method introduces structured prompting to explicitly separate reasoning steps from final answers and applies a dual reward system to jointly optimize for format adherence and mathematical correctness. The training incorporates low-overhead techniques such as 8-bit optimization, mixed-precision training, gradient checkpointing, and accelerated decoding for efficient rollout and policy updates. Experimental results show a 50.95% accuracy on a benchmark reasoning dataset—GSM8K, outperforming several larger models—while training entirely on a single GPU. These findings demonstrate that small-scale models, when trained with structured reinforcement learning, can achieve competitive performance in mathematical reasoning tasks. The approach offers a practical pathway for deploying interpretable, reasoning-capable models in low-resource environments.
Klasifikace
Druh
D - Stať ve sborníku
CEP obor
—
OECD FORD obor
20203 - Telecommunications
Návaznosti výsledku
Projekt
<a href="/cs/project/FW07010015" target="_blank" >FW07010015: Monitorování kvality vnitřního prostředí budov pomocí senzorů pachu a umělé inteligence </a><br>
Návaznosti
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Ostatní
Rok uplatnění
2025
Kód důvěrnosti údajů
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Údaje specifické pro druh výsledku
Název statě ve sborníku
2025 17th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT)
ISBN
979-8-3315-7675-2
ISSN
—
e-ISSN
2157-023X
Počet stran výsledku
6
Strana od-do
272-277
Název nakladatele
IEEE
Místo vydání
Florence, Italy
Místo konání akce
Florence, Italy
Datum konání akce
3. 11. 2025
Typ akce podle státní příslušnosti
WRD - Celosvětová akce
Kód UT WoS článku
—