An LP-based hyperparameter optimization model for language modeling

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61989100%3A27510%2F18%3A10239458" target="_blank" >RIV/61989100:27510/18:10239458 - isvavai.cz</a>

Výsledek na webu

<a href="https://link.springer.com/article/10.1007/s11227-018-2236-6" target="_blank" >https://link.springer.com/article/10.1007/s11227-018-2236-6</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/s11227-018-2236-6" target="_blank" >10.1007/s11227-018-2236-6</a>

Jazyk výsledku

angličtina

Název v původním jazyce

An LP-based hyperparameter optimization model for language modeling

Popis výsledku v původním jazyce

In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models&apos; hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most popular cost functions. In this study, we propose a fractional nonlinear programming model that finds the optimal perplexity value. The special structure of the model allows us to approximate it by a linear programming model that can be solved using the well-known simplex algorithm. To the best of our knowledge, this is the first attempt to use optimization techniques to find perplexity values in the language modeling literature. We apply our model to find hyperparameters of a language model and compare it to the grid search algorithm. Furthermore, we illustrate that it results in lower perplexity values. We perform this experiment on a real-world dataset from SwiftKey to validate our proposed approach. (C) 2018 Springer Science+Business Media, LLC, part of Springer Nature

Název v anglickém jazyce

An LP-based hyperparameter optimization model for language modeling

Popis výsledku anglicky

In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models&apos; hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most popular cost functions. In this study, we propose a fractional nonlinear programming model that finds the optimal perplexity value. The special structure of the model allows us to approximate it by a linear programming model that can be solved using the well-known simplex algorithm. To the best of our knowledge, this is the first attempt to use optimization techniques to find perplexity values in the language modeling literature. We apply our model to find hyperparameters of a language model and compare it to the grid search algorithm. Furthermore, we illustrate that it results in lower perplexity values. We perform this experiment on a real-world dataset from SwiftKey to validate our proposed approach. (C) 2018 Springer Science+Business Media, LLC, part of Springer Nature

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10102 - Applied mathematics

Projekt

Výsledek vznikl pri realizaci vícero projektů. Více informací v záložce Projekty.

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2018

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Journal of Supercomputing

ISSN

0920-8542

e-ISSN

—

Svazek periodika

74

Číslo periodika v rámci svazku

5

Stát vydavatele periodika

NL - Nizozemsko

Počet stran výsledku

10

Strana od-do

2151-2160

Kód UT WoS článku

000430412400016

EID výsledku v databázi Scopus

2-s2.0-85040232951