πŸ’‘ Guideline

Use python 3.10

git clone git@github.com:IRT-SystemX/adaro-rl.git
pip install -e adaro-rl

πŸ“¦ Content of the Library

adaro_rl/
β”œβ”€β”€ attacks/
β”œβ”€β”€ pipelines/
β”œβ”€β”€ wrappers/
β”œβ”€β”€ zoo/
└── __init__.py

  • src/adaro_rl/attacks/ contains the implementation of many adversarial attacks in a unified framework.

  • src/adaro_rl/pipelines/ contains the implementation of scripts to train, test, attack, and adversarially train agents.

  • src/adaro_rl/wrappers/ contains the implementations of wrappers of gymnasium environments to apply attacks from different perspectives (the agent or the adversary) and with different supports for perturbation (observations or environment).

  • src/adaro_rl/zoo/ contains the implementation of tools and functionality to run the scripts.

πŸ›‘οΈ Adversarial Attacks

Available Methods

Random Attacks

  • RandomUniformAttack or RUA: generates perturbation with random uniform sampling.

  • RandomNormalAttack or RNA: generates perturbation with random normal sampling.

  • RandomSignedAttack or RSA: generates perturbation with random sampling between -1 and 1.

Fast Gradients Method Based Attacks

  • FGM_D for FastGradientMethod_DiscreteAction: generates perturbation with fast gradient method to attack discrete action policies.

  • FGM_C for FastGradientMethod_ContinuousAction: generates perturbation with fast gradient method to attack continuous action policies.

  • FGM_V for FastGradientMethod_V_Critic: generates perturbation with fast gradient method to attack value critics.

  • FGM_QC for FastGradientMethod_Q_Critic: generates perturbation with fast gradient method to attack the q critic of a q actor-critic agent.

  • FGM_QAC for FastGradientMethod_Q_ActorCritic: generates perturbation with fast gradient method to attack both models of a q actor-critic agent.

Fast Gradients Method Based Attacks

  • FGSM_D for the sign version of FGM_D.

  • FGSM_C for the sign version of FGM_C.

  • FGSM_V for the sign version of FGM_V.

  • FGSM_QC for the sign version of FGM_QC.

  • FGSM_QAC for the sign version of FGM_QAC.

Parameters of the Attacks

The common parameters for all attacks are the following:

  • eps defines the amount of perturbation applied with an attack.

  • norm defines the norm of the delta. Options: 0, 1, 2, … inf.

  • max_eps defines the maximum amount allowed on each dimension of the perturbation. It can be a number, the same for each dimension. It can also be an array, defining a specific amount of perturbation for each dimension. By default, max_eps is set as delta.

  • is_proportional_mask is an array of booleans that defines for each dimension, if the perturbation has to be applied proportionally to the value of the support on which the perturbation is applied. By default, is_prop_perturb_mask is None.

Other parameters for gradient attacks are the following:

  • target defines the target followed by the attack. Options: β€˜untargeted’, β€˜targeted’, β€˜min’, β€˜max’ or β€˜target_fct’. Default is β€˜untargeted’.

πŸ› οΈ Usage

Usage of Attacks Only

The first way to use the ADARO-RL library is to directly apply the attack within your infrastructure with your training and testing scripts.

from adaro_rl.attacks import make_attack
attack = make_attack(attack_name, **attack_kwargs)
perturbation = attack.generate_perturbation(observation)
adv_obs = observation + perturbation

  • Here, use one of the attack_name listed in adaro_rl.attacks.attack_names.

  • Add the attack_kwargs needed for the attack_name specified.

  • Use an observation of your environment|dataset.

Usage with the Scripts

The second way to use the ADARO-RL library is to use the application available in adaro_rl in command line. This enables to train, test, attack, and adversarial train agents, it is needed to choose a configuration of an existing use case provided in the adaro_rl.zoo, add your configurations in the zoo, or provide a new zoo module to use.

Command Lines

Here are some basic command line examples :

adaro_rl train --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo'
adaro_rl test --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo'
adaro_rl online_attack --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo' --attack-name 'FGM_D' --eps 0.01 --norm "2"

The different pipelines that can be executed by the application are the following:

  • train

  • test

  • online_attack

  • adversarial_train

Use any given script with the option --help or check the source codes available in adaro-rl/pipelines to see how to use them.

Python

You can also use the scripts directly in Python. Here are some examples equivalent to the ones presented above in command lines:

import adaro_rl.zoo as zoo
config = zoo.configs['HalfCheetah-v5']
adaro_rl.train(config=config)
adaro_rl.test(config=config)
adaro_rl.online_attack(config=config, attack_name='FGM_D', eps=0.1, norm=2)

Examples

Example scripts and jupyter notebooks are available in examples folder of the repository. You will find there :

  • Attacks examples and examples/attacks.py: a jupyter notebook and a python script showing the usages of the attacks available.

  • Pipelines examples and examples/pipelines.sh: a jupyter notebook and a bash script showing the usages of the high level pipelines available.