SynCSE: syntax graph-based contrastive learning of sentence embeddings

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AF8REJIEU" target="_blank" >RIV/00216208:11320/26:F8REJIEU - isvavai.cz</a>

Result on the web

<a href="http://dx.doi.org/10.1016/j.eswa.2025.128047" target="_blank" >http://dx.doi.org/10.1016/j.eswa.2025.128047</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.eswa.2025.128047" target="_blank" >10.1016/j.eswa.2025.128047</a>

Result language

angličtina

Original language name

SynCSE: syntax graph-based contrastive learning of sentence embeddings

Original language description

Pre-trained language models (PrLMs) trained via contrastive learning methods achieved state-of-the-art performance on various natural language processing (NLP) tasks. Most PrLMs for sentence embedding focuses on context similarity as an objective function of contrastive learning. However, we found that these PrLMs, including recently released large language models (LLMs) like LLaMA, underperform when analyzing syntax information on probing tasks. This limitation becomes particularly noticeable in applications that depend on nuanced sentence understanding, such as the Retrieval Augmented Generation (RAG) framework in LLMs. This paper introduces a new sentence embedding model named SynCSE: Syntax Graph-based Contrastive Learning of Sentence Embeddings. Our approach enables meaningful sentence embeddings of language models through learning the syntactic features. To accomplish this, we train a PrLM with graph neural networks (GNNs) receiving a directed syntax graph. We then detach additional GNN layers from PrLM for inference; which does not require a syntax graph. The proposed model gains improvement on baselines in sentence textual similarity (STS) tasks, transfer tasks, and especially probing tasks. Additionally, we observe that our model has improved alignment and competitive uniformity compared to the baseline. © 2025 Elsevier Ltd

Czech name

—

Czech description

—

Type

J_SC - Article in a specialist periodical, which is included in the SCOPUS database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Expert Systems with Applications

ISSN

0957-4174

e-ISSN

—

Volume of the periodical

287

Issue of the periodical within the volume

2025

Country of publishing house

US - UNITED STATES

Number of pages

16

Pages from-to

128047

UT code for WoS article

—

EID of the result in the Scopus database

2-s2.0-105005497338