Implementation Notes for the Soft Cosine Measure

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14330%2F18%3A00101853" target="_blank" >RIV/00216224:14330/18:00101853 - isvavai.cz</a>

Result on the web

<a href="https://arxiv.org/abs/1808.09407" target="_blank" >https://arxiv.org/abs/1808.09407</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3269206.3269317" target="_blank" >10.1145/3269206.3269317</a>

Result language

angličtina

Original language name

Implementation Notes for the Soft Cosine Measure

Original language description

The standard bag-of-words vector space model (VSM) is efficient, and ubiquitous in information retrieval, but it underestimates the similarity of documents with the same meaning, but different terminology. To overcome this limitation, Sidorov et al. proposed the Soft Cosine Measure (SCM) that incorporates term similarity relations. Charlet and Damnati showed that the SCM is highly effective in question answering (QA) systems. However, the orthonormalization algorithm proposed by Sidorov et al. has an impractical time complexity of O(n^4), where n is the size of the vocabulary. In this paper, we prove a tighter lower worst-case time complexity bound of O(n^3). We also present an algorithm for computing the similarity between documents and we show that its worst-case time complexity is O(1) given realistic conditions. Lastly, we describe implementation in general-purpose vector databases such as Annoy, and Faiss and in the inverted indices of text search engines such as Apache Lucene, and ElasticSearch. Our results enable the deployment of the SCM in real-world information retrieval systems.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/TD03000295" target="_blank" >TD03000295: Intelligent software for semantic text search</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach<br>I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2018

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18)

ISBN

9781450360142

ISSN

—

e-ISSN

—

Number of pages

4

Pages from-to

1639-1642

Publisher name

Association for Computing Machinery

Place of publication

Torino, Italy

Event location

Torino, Italy

Event date

Oct 22, 2018

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

000455712300190