The MorPhEMe Machine: An Addressable Neural Memory for Learning Knowledge-Regularized Deep Contextualized Chinese Embedding

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3APVVK8D38" target="_blank" >RIV/00216208:11320/25:PVVK8D38 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85187261096&doi=10.1109%2fTASLP.2024.3364610&partnerID=40&md5=80e95f1bb7f89a2d456950b15b6c8fb1" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85187261096&doi=10.1109%2fTASLP.2024.3364610&partnerID=40&md5=80e95f1bb7f89a2d456950b15b6c8fb1</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/TASLP.2024.3364610" target="_blank" >10.1109/TASLP.2024.3364610</a>

Jazyk výsledku

angličtina

Název v původním jazyce

The MorPhEMe Machine: An Addressable Neural Memory for Learning Knowledge-Regularized Deep Contextualized Chinese Embedding

Popis výsledku v původním jazyce

Deep contextualized embeddings, as learned by large pre-training models, have proven highly effective in various downstream natural language processing tasks. However, the embedding space in these large models lacks explicit regularization, leading to underfitting and substantial costs during large-scale training on huge corpora. In this paper, we present a novel approach to learning deep contextualized embeddings, introducing linguistic knowledge regularization. Specifically, our proposed model, MorPhEMe (Morphology and Phonology Embedding Memory), features an external addressable memory with two additional addressable memories for storing morphology and phonology knowledge. MorPhEMe can be seamlessly stacked into a deep architecture. Notably different from existing pre-training models, MorPhEMe boasts two distinctive features: i) compositional encoding and decompositional decoding facilitated by a dynamic addressing mechanism; and ii) explicit memory embedding regularization through cross-layer memory sharing. Theoretical analysis suggests that the inclusion of morphology and phonology enables MorPhEMe to reduce the modeling complexity of natural language sequences. We evaluate MorPhEMe across a diverse set of Chinese natural language processing tasks, including language modeling, word similarity computation, word analogy reasoning, relation extraction, and machine reading comprehension. Experimental results demonstrate that MorPhEMe, in contrast to state-of-the-art models, achieves remarkable improvements with fewer parameters and rapid convergence. © 2014 IEEE.

Název v anglickém jazyce

The MorPhEMe Machine: An Addressable Neural Memory for Learning Knowledge-Regularized Deep Contextualized Chinese Embedding

Popis výsledku anglicky

Deep contextualized embeddings, as learned by large pre-training models, have proven highly effective in various downstream natural language processing tasks. However, the embedding space in these large models lacks explicit regularization, leading to underfitting and substantial costs during large-scale training on huge corpora. In this paper, we present a novel approach to learning deep contextualized embeddings, introducing linguistic knowledge regularization. Specifically, our proposed model, MorPhEMe (Morphology and Phonology Embedding Memory), features an external addressable memory with two additional addressable memories for storing morphology and phonology knowledge. MorPhEMe can be seamlessly stacked into a deep architecture. Notably different from existing pre-training models, MorPhEMe boasts two distinctive features: i) compositional encoding and decompositional decoding facilitated by a dynamic addressing mechanism; and ii) explicit memory embedding regularization through cross-layer memory sharing. Theoretical analysis suggests that the inclusion of morphology and phonology enables MorPhEMe to reduce the modeling complexity of natural language sequences. We evaluate MorPhEMe across a diverse set of Chinese natural language processing tasks, including language modeling, word similarity computation, word analogy reasoning, relation extraction, and machine reading comprehension. Experimental results demonstrate that MorPhEMe, in contrast to state-of-the-art models, achieves remarkable improvements with fewer parameters and rapid convergence. © 2014 IEEE.

Druh

J_SC - Článek v periodiku v databázi SCOPUS

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

IEEE/ACM Transactions on Audio Speech and Language Processing

ISSN

2329-9290

e-ISSN

—

Svazek periodika

32

Číslo periodika v rámci svazku

2024

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

14

Strana od-do

1673-1686

Kód UT WoS článku

—

EID výsledku v databázi Scopus

2-s2.0-85187261096