Adapting Value Iteration with Adversarial Risk Analysis for Secure Reinforcement Learning

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985556%3A_____%2F25%3A00638655" target="_blank" >RIV/67985556:_____/25:00638655 - isvavai.cz</a>

Výsledek na webu

—

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Adapting Value Iteration with Adversarial Risk Analysis for Secure Reinforcement Learning

Popis výsledku v původním jazyce

The increasing automation of processes via machine learning (ML) algorithms necessitates robust systems, particularly in reinforcement learning (RL) where agents learn through interaction. However, RL systems are highly vulnerable to adversarial attacks that manipulate rewards or observations, compromising their integrity. Traditional adversarial machine learning (AML) often focuses on supervised learning or relies on common knowledge assumption, which is unsuitable for scenarios where adversaries actively conceal information. Moreover, standard single-agent RL methods often falter in dynamic setting with adapting opponents due to the emergent non-stationarity. This work employs framework for enhancing the security of RL agents by leveraging Adversarial Risk Analysis (ARA), by modifying the standard Markov Decision Process (MDP) framework to explicitly account for the presence of an adversary affecting state transition and reward dynamics. The modification incorporates the decision maker's (DM’s) subjective beliefs about potential adversarial actions. This enables the DM to use Bayesian principles for predicting adversarial actions in dynamic interactions. The framework adapts the value iteration process by forecasting and averaging over the predicted actions based on DM’s beliefs, thereby aiming to enhance policy robustness.To model the DM's uncertainty regarding the adversary's strategy, this approach focuses on methods for belief elicitation and updating. Specifically, the DM maintains and updates a probability distribution over the adversary's likely actions based on past observations. For instance, Bayesian methods with Dirichlet priors can be used to estimate action frequencies in discrete settings, allowing the DM to continuously refine its understanding and adapt its strategy even when the adversary's true policy is unknown and potentially evolving. The primary contribution of this work is a methodology demonstrating how ARA facilitates secure RL by enabling the explicit modelling, prediction, and adaptation to interfering adversarial actions within the augmented MDP formulation. The efficacy and robustness of this approach is tested through a Coin Game experiment. This strategic environment serves as a testbed for adversarial interaction. By operating under the modified MDP, which explicitly models and adapts to the opponent's behaviour, agent aims to manage adversarial interactions more effectively than standard dynamic programming baselines that typically do not incorporate such explicit opponent modelling. This highlights the crucial benefits of incorporating accurate adversarial prediction models and underscores the framework's potential for robustness even in the presence of partially misspecified opponent models. This work offers a principled and practical pathway towards developing more secure and resilient RL agents in adversarial settings.

Název v anglickém jazyce

Adapting Value Iteration with Adversarial Risk Analysis for Secure Reinforcement Learning

Popis výsledku anglicky

The increasing automation of processes via machine learning (ML) algorithms necessitates robust systems, particularly in reinforcement learning (RL) where agents learn through interaction. However, RL systems are highly vulnerable to adversarial attacks that manipulate rewards or observations, compromising their integrity. Traditional adversarial machine learning (AML) often focuses on supervised learning or relies on common knowledge assumption, which is unsuitable for scenarios where adversaries actively conceal information. Moreover, standard single-agent RL methods often falter in dynamic setting with adapting opponents due to the emergent non-stationarity. This work employs framework for enhancing the security of RL agents by leveraging Adversarial Risk Analysis (ARA), by modifying the standard Markov Decision Process (MDP) framework to explicitly account for the presence of an adversary affecting state transition and reward dynamics. The modification incorporates the decision maker's (DM’s) subjective beliefs about potential adversarial actions. This enables the DM to use Bayesian principles for predicting adversarial actions in dynamic interactions. The framework adapts the value iteration process by forecasting and averaging over the predicted actions based on DM’s beliefs, thereby aiming to enhance policy robustness.To model the DM's uncertainty regarding the adversary's strategy, this approach focuses on methods for belief elicitation and updating. Specifically, the DM maintains and updates a probability distribution over the adversary's likely actions based on past observations. For instance, Bayesian methods with Dirichlet priors can be used to estimate action frequencies in discrete settings, allowing the DM to continuously refine its understanding and adapt its strategy even when the adversary's true policy is unknown and potentially evolving. The primary contribution of this work is a methodology demonstrating how ARA facilitates secure RL by enabling the explicit modelling, prediction, and adaptation to interfering adversarial actions within the augmented MDP formulation. The efficacy and robustness of this approach is tested through a Coin Game experiment. This strategic environment serves as a testbed for adversarial interaction. By operating under the modified MDP, which explicitly models and adapts to the opponent's behaviour, agent aims to manage adversarial interactions more effectively than standard dynamic programming baselines that typically do not incorporate such explicit opponent modelling. This highlights the crucial benefits of incorporating accurate adversarial prediction models and underscores the framework's potential for robustness even in the presence of partially misspecified opponent models. This work offers a principled and practical pathway towards developing more secure and resilient RL agents in adversarial settings.

Druh

O - Ostatní výsledky

CEP obor

—

OECD FORD obor

10101 - Pure mathematics

Projekt

—

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů