The MorPhEMe Machine: An Addressable Neural Memory for Learning Knowledge-Regularized Deep Contextualized Chinese Embedding

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3APVVK8D38" target="_blank" >RIV/00216208:11320/25:PVVK8D38 - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85187261096&doi=10.1109%2fTASLP.2024.3364610&partnerID=40&md5=80e95f1bb7f89a2d456950b15b6c8fb1" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85187261096&doi=10.1109%2fTASLP.2024.3364610&partnerID=40&md5=80e95f1bb7f89a2d456950b15b6c8fb1</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/TASLP.2024.3364610" target="_blank" >10.1109/TASLP.2024.3364610</a>

Result language

angličtina

Original language name

The MorPhEMe Machine: An Addressable Neural Memory for Learning Knowledge-Regularized Deep Contextualized Chinese Embedding

Original language description

Deep contextualized embeddings, as learned by large pre-training models, have proven highly effective in various downstream natural language processing tasks. However, the embedding space in these large models lacks explicit regularization, leading to underfitting and substantial costs during large-scale training on huge corpora. In this paper, we present a novel approach to learning deep contextualized embeddings, introducing linguistic knowledge regularization. Specifically, our proposed model, MorPhEMe (Morphology and Phonology Embedding Memory), features an external addressable memory with two additional addressable memories for storing morphology and phonology knowledge. MorPhEMe can be seamlessly stacked into a deep architecture. Notably different from existing pre-training models, MorPhEMe boasts two distinctive features: i) compositional encoding and decompositional decoding facilitated by a dynamic addressing mechanism; and ii) explicit memory embedding regularization through cross-layer memory sharing. Theoretical analysis suggests that the inclusion of morphology and phonology enables MorPhEMe to reduce the modeling complexity of natural language sequences. We evaluate MorPhEMe across a diverse set of Chinese natural language processing tasks, including language modeling, word similarity computation, word analogy reasoning, relation extraction, and machine reading comprehension. Experimental results demonstrate that MorPhEMe, in contrast to state-of-the-art models, achieves remarkable improvements with fewer parameters and rapid convergence. © 2014 IEEE.

Czech name

—

Czech description

—

Type

J_SC - Article in a specialist periodical, which is included in the SCOPUS database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

IEEE/ACM Transactions on Audio Speech and Language Processing

ISSN

2329-9290

e-ISSN

—

Volume of the periodical

32

Issue of the periodical within the volume

2024

Country of publishing house

US - UNITED STATES

Number of pages

14

Pages from-to

1673-1686

UT code for WoS article

—

EID of the result in the Scopus database

2-s2.0-85187261096