Adaro-RL

ADARO-RL

A unified framework of Adversarial Attacks for Robust Reinforcement Learning.

📚 Table of Contents

1. 📂 Content of the Repository
2. 💾 Installation
3. 📦 Content of the Library
4. 🛡️ Adversarial Attacks
   • Available Methods
     • Random Attacks
     • Fast Gradients Method Based Attacks
     • Fast Gradients Sign Method Based Attacks
   • Parameters of the Attacks
5. 🛠️ Usage
   • Usage of Attacks Only
   • Usage with the Scripts
     • Command Lines
     • Python

💾 Installation

Use python 3.10

git clone git@github.com:IRT-SystemX/adaro-rl.git
pip install -e adaro-rl

📦 Content of the Library

adaro_rl/
├── attacks/
├── pipelines/
├── wrappers/
├── zoo/
└── __init__.py

• src/adaro_rl/attacks/ contains the implementation of many adversarial attacks in a unified framework.
• src/adaro_rl/pipelines/ contains the implementation of scripts to train, test, attack, and adversarially train agents.
• src/adaro_rl/wrappers/ contains the implementations of wrappers of gymnasium environments to apply attacks from different perspectives (the agent or the adversary) and with different supports for perturbation (observations or environment).
• src/adaro_rl/zoo/ contains the implementation of tools and functionality to run the scripts.

🛡️ Adversarial Attacks

Available Methods

Random Attacks

• RandomUniformAttack or RUA: generates perturbation with random uniform sampling.
• RandomNormalAttack or RNA: generates perturbation with random normal sampling.
• RandomSignedAttack or RSA: generates perturbation with random sampling between -1 and 1.

Fast Gradients Method Based Attacks

• FGM_D for FastGradientMethod_DiscreteAction: generates perturbation with fast gradient method to attack discrete action policies.
• FGM_C for FastGradientMethod_ContinuousAction: generates perturbation with fast gradient method to attack continuous action policies.
• FGM_V for FastGradientMethod_V_Critic: generates perturbation with fast gradient method to attack value critics.
• FGM_QC for FastGradientMethod_Q_Critic: generates perturbation with fast gradient method to attack the q critic of a q actor-critic agent.
• FGM_QAC for FastGradientMethod_Q_ActorCritic: generates perturbation with fast gradient method to attack both models of a q actor-critic agent.

Fast Gradients Method Based Attacks

• FGSM_D for the sign version of FGM_D.
• FGSM_C for the sign version of FGM_C.
• FGSM_V for the sign version of FGM_V.
• FGSM_QC for the sign version of FGM_QC.
• FGSM_QAC for the sign version of FGM_QAC.

Parameters of the Attacks

The common parameters for all attacks are the following:

• eps defines the amount of perturbation applied with an attack.
• norm defines the norm of the delta. Options: 0, 1, 2, ... inf.
• max_eps defines the maximum amount allowed on each dimension of the perturbation. It can be a number, the same for each dimension. It can also be an array, defining a specific amount of perturbation for each dimension. By default, max_eps is set as delta.
• is_proportional_mask is an array of booleans that defines for each dimension, if the perturbation has to be applied proportionally to the value of the support on which the perturbation is applied. By default, is_prop_perturb_mask is None.

Other parameters for gradient attacks are the following:

• target defines the target followed by the attack. Options: 'untargeted', 'targeted', 'min', 'max' or 'target_fct'. Default is 'untargeted'.

🛠️ Usage

Usage of Attacks Only

The first way to use the ADARO-RL library is to directly apply the attack within your infrastructure with your training and testing scripts.

from adaro_rl.attacks import make_attack
attack = make_attack(attack_name, **attack_kwargs)
perturbation = attack.generate_perturbation(observation)
adv_obs = observation + perturbation

• Here, use one of the attack_name listed in adaro_rl.attacks.attack_names.
• Add the attack_kwargs needed for the attack_name specified.
• Use an observation of your environment|dataset.

Usage with the Scripts

The second way to use the ADARO-RL library is to use the application available in adaro_rl in command line. This enables to train, test, attack, and adversarial train agents, it is needed to choose a configuration of an existing use case provided in the adaro_rl.zoo, add your configurations in the zoo, or provide a new zoo module to use.

Command Lines

Here are some basic command line examples :

adaro_rl train --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo'
adaro_rl test --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo'
adaro_rl online_attack --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo' --attack-name 'FGM_D' --eps 0.01 --norm "2"

The different pipelines that can be executed by the application are the following:

• train
• test
• online_attack
• adversarial_train

Use any given script with the option --help or check the source codes available in adaro-rl/pipelines to see how to use them.

Python

You can also use the scripts directly in Python. Here are some examples equivalent to the ones presented above in command lines:

import adaro_rl.zoo as zoo
config = zoo.configs['HalfCheetah-v5']
adaro_rl.train(config=config)
adaro_rl.test(config=config)
adaro_rl.online_attack(config=config, attack_name='FGM_D', eps=0.1, norm=2)

Examples

Example scripts and jupyter notebooks are available in examples folder of the repository. You will find there :

• Attacks examples and examples/attacks.py: a jupyter notebook and a python script showing the usages of the attacks available.
• Pipelines examples and examples/pipelines.sh: a jupyter notebook and a bash script showing the usages of the high level pipelines available.

🛠️ Documentation

• docs/ contains the documentation the library

To build the HTML documentation:

cd docs
make html

The output will be in:

docs/build/html/index.html

Open this file in a browser to view the docs locally.