Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AQG6WDX7Q" target="_blank" >RIV/00216208:11320/25:QG6WDX7Q - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/record/display.uri?eid=2-s2.0-85204443275&origin=resultslist&sort=plf-f&src=s&sid=e2b9c7bf82ada12b524d66c7a293503a&sot=b&sdt=b&s=TITLE-ABS-KEY%28Dependency+Transformer+Grammars%3A+Integrating+Dependency+Structures+into+Transformer+Language+Models%29&sl=114&sessionSearchId=e2b9c7bf82ada12b524d66c7a293503a&relpos=0" target="_blank" >https://www.scopus.com/record/display.uri?eid=2-s2.0-85204443275&origin=resultslist&sort=plf-f&src=s&sid=e2b9c7bf82ada12b524d66c7a293503a&sot=b&sdt=b&s=TITLE-ABS-KEY%28Dependency+Transformer+Grammars%3A+Integrating+Dependency+Structures+into+Transformer+Language+Models%29&sl=114&sessionSearchId=e2b9c7bf82ada12b524d66c7a293503a&relpos=0</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2024.acl-long.84" target="_blank" >10.18653/v1/2024.acl-long.84</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models

Popis výsledku v původním jazyce

Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transformers, we introduce Dependency Transformer Grammars (DTGs), a new class of Transformer language model with explicit dependency-based inductive bias. DTGs simulate dependency transition systems with constrained attention patterns by modifying attention masks, incorporate the stack information through relative positional encoding, and augment dependency arc representation with a combination of token embeddings and operation embeddings. When trained on a dataset of sentences annotated with dependency trees, DTGs achieve better generalization while maintaining comparable perplexity with Transformer language model baselines. DTGs also outperform recent constituency-based models, showing that dependency can better guide Transformer language models. Our code is released at https://github.com/zhaoyd1/Dep_Transformer_Grammars.

Název v anglickém jazyce

Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models

Popis výsledku anglicky

Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transformers, we introduce Dependency Transformer Grammars (DTGs), a new class of Transformer language model with explicit dependency-based inductive bias. DTGs simulate dependency transition systems with constrained attention patterns by modifying attention masks, incorporate the stack information through relative positional encoding, and augment dependency arc representation with a combination of token embeddings and operation embeddings. When trained on a dataset of sentences annotated with dependency trees, DTGs achieve better generalization while maintaining comparable perplexity with Transformer language model baselines. DTGs also outperform recent constituency-based models, showing that dependency can better guide Transformer language models. Our code is released at https://github.com/zhaoyd1/Dep_Transformer_Grammars.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics

ISBN

979-8-89176-094-3

ISSN

—

e-ISSN

—

Počet stran výsledku

14

Strana od-do

1543-1556

Název nakladatele

ACL

Místo vydání

—

Místo konání akce

Bangkok, Thailand

Datum konání akce

1. 1. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—