Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AQG6WDX7Q" target="_blank" >RIV/00216208:11320/25:QG6WDX7Q - isvavai.cz</a>
Result on the web
<a href="https://www.scopus.com/record/display.uri?eid=2-s2.0-85204443275&origin=resultslist&sort=plf-f&src=s&sid=e2b9c7bf82ada12b524d66c7a293503a&sot=b&sdt=b&s=TITLE-ABS-KEY%28Dependency+Transformer+Grammars%3A+Integrating+Dependency+Structures+into+Transformer+Language+Models%29&sl=114&sessionSearchId=e2b9c7bf82ada12b524d66c7a293503a&relpos=0" target="_blank" >https://www.scopus.com/record/display.uri?eid=2-s2.0-85204443275&origin=resultslist&sort=plf-f&src=s&sid=e2b9c7bf82ada12b524d66c7a293503a&sot=b&sdt=b&s=TITLE-ABS-KEY%28Dependency+Transformer+Grammars%3A+Integrating+Dependency+Structures+into+Transformer+Language+Models%29&sl=114&sessionSearchId=e2b9c7bf82ada12b524d66c7a293503a&relpos=0</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.18653/v1/2024.acl-long.84" target="_blank" >10.18653/v1/2024.acl-long.84</a>
Alternative languages
Result language
angličtina
Original language name
Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models
Original language description
Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transformers, we introduce Dependency Transformer Grammars (DTGs), a new class of Transformer language model with explicit dependency-based inductive bias. DTGs simulate dependency transition systems with constrained attention patterns by modifying attention masks, incorporate the stack information through relative positional encoding, and augment dependency arc representation with a combination of token embeddings and operation embeddings. When trained on a dataset of sentences annotated with dependency trees, DTGs achieve better generalization while maintaining comparable perplexity with Transformer language model baselines. DTGs also outperform recent constituency-based models, showing that dependency can better guide Transformer language models. Our code is released at https://github.com/zhaoyd1/Dep_Transformer_Grammars.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
—
Others
Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics
ISBN
979-8-89176-094-3
ISSN
—
e-ISSN
—
Number of pages
14
Pages from-to
1543-1556
Publisher name
ACL
Place of publication
—
Event location
Bangkok, Thailand
Event date
Jan 1, 2025
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—