๐Ÿ“– Guidelines๏ƒ

To install and use the uqmodels library, it is recommended to create a Python virtual environment. You can do that with virtualenv, as follows:

Setting environement๏ƒ

pip install virtualenv
virtualenv -p <path/to/python3.9> myenv
source myenv/bin/activate

Installation๏ƒ

Once your virtual environment is activated, you can install the uqmodels library directly from Pypi by typing :

pip install uqmodels

This command will install the uqmodels package and all required dependencies.

Tutorial๏ƒ

Tutorials notebooks are provided here : examples

๐Ÿ”€ Description of main object.๏ƒ

UQModel define and manipulate several object in a format close of Scikit learn library.

  • UQEstimator are (fit/predict) procedure that follow scikitlearn standards.

    • fit(X,y)

    • predict(X)->pred,UQ

  • UQModel are (fit/predict&score) procedure that follow scikitlearn standards.

    • fit(X,y)

    • predict(X) -> pred,list_KPI

    • score(X,y) -> list_KPI

  • Processor are (fit/transform) procedure inspired from scikitlearn Transformer but without input/ouput constraint of form.

    • Preprocessor transform data (any form) into processed data (any form)

      • a specific kind of preprocessor that take (any form) and return (X,y) can be set-up witinh and UQModels.

    • Postprocessor transform UQ-modeling output (UQ,type_UQ,pred,y) into KPI (any form)

The chain of a UQModels then breaks down into

  • 1: (optional) A preprocessor that compute features from raw data

  • 2: (optional) A predictor that compute a prediction form features.

  • 3: A UQEstimator, that may compute both prediction, and have to compute UQmeasure.

  • 4: Predict-Post-Processor (optional) that exploit prediction and UQmeasure to compute UQ-KPI

  • 5: Score-Post-Processor (optional) that exploit prediction, UQmeasure and target to compute score-KPI

๐ŸŽฎ Usage๏ƒ

The uqmodels library provides several UQEstimators that aim to make UQ-regression, and residu-based anomaly detection. Theses estimators can be used as sklearn or tensorflow model, this mean you need to first import the necessary modules and then initialize and fit the predictor with your own data.

4 notebooks of example are provide on a synthetic mutlivariate time series dataset.

  • tutorial_UQModels.ipynb : Shows the basic use of RF-UQ as an UQEstimators to predict target and estimate UQmeasure, then combine it with an UQ-KPIprocessor to estimate confidence interval from the UQmeasre, and then directly encapsulates UQestimator and KPI-Processors for confidence interval, epistemic score and Anomaly-detection into an UQModels object.

  • example_UQestimators.ipynb : Shows an loop that execute serverals UQ-Estimators that produce differents kinds of UQmeasure (Variance,Quantile,โ€ฆ) in order to illustrate that UQ-KPI processors can handle several type of UQmeasure.

  • example_complex_UQModels.ipynb : Shows that itโ€™s possible to run and encapsulate in a UQModel more complex models with 3 probabilistic neural network MLP (mean & variance regression) under 3 UQ-overlay paradigm (MC-Dropout,Deep-ensemble,EDL), and 2 Deep neural newtork (LSTM-ED & Transform-ED) performing mutli-horizon mean & variance regression prediction with MC-Dropout based UQ-overlay.

  • illustration_Anom_with_UQ_and_ML_risk_aversion.ipynb : Illustrate on a toy example, how uncertainty decomposition can be used to build an residu based anomaly score considering distinctly uncertainty and ML-risk.


import uqmodels.modelization.ML_estimator.random_forest_UQ as RF_UQ
from uqmodels.UQModel import UQModel
import uqmodels.postprocessing.UQKPI_Processor as UQProc

# UQEstimator instanciation
UQEstimator_initializer = RF_UQ.RF_UQEstimator
UQEstimator_parameters = RF_UQ.get_params_dict()

# We can use UQEstimator as standards estimator :
UQEstimator = UQEstimator_initializer(**UQEstimator_parameters)
# Fit UQEstimator that produce regression and UQmeasure estimation.
UQEstimator.fit(X[train],y[train])
pred,UQ = UQEstimator.predict(X)

# Example of UQMOdels wrapping
import uqmodels.postprocessing.UQKPI_Processor as UQProc

#Specify Predict KPI-processors
list_predict_KPI_processors = [UQProc.UQKPI_Processor(),
                               UQProc.NormalPIs_processor(),
                               UQProc.Epistemicscorelvl_processor()]

#Specify Score KPI-processors
list_score_KPI_processors = [UQProc.Anomscore_processor()]

#Specify UQModels pipeline
RF_UQModel = UQModel(UQEstimator_initializer=UQEstimator_initializer,
                     UQEstimator_parameters=UQEstimator_parameters,
                     preprocessor = None, predictor=None,
                     list_predict_KPI_processors=list_predict_KPI_processors,
                     list_score_KPI_processors=list_score_KPI_processors,
                     random_state=0,name="RF_UQ",)

# Fit both UQEstimator and KPI-Processors
RF_UQModel.fit(X[train],y[train])

# Save fitted UQEstimator and KPI-Processors the RF_UQ at path "./model"
RF_UQModel.save('./model')

# New empty UQMOdel
RF_UQModel = UQModel()

# Load fitted UQEstimator and fitted KPI-Processors
RF_UQModel.load(path='model/'+name)

# Perform prediction and UQ-KPI processing
pred,(UQ,PIs,Elvl) = RF_UQModel.predict(X)

# Perform score processing from features and targets.
KPI_ANOM = RF_UQModel.score(X,y)

# KPI_ANOM is here an multivariate residu based anomaly score considering UQ

For more practical examples and detailed usage scenarios, please refer to the Examples directory in our Github repository. These examples help you to understand how to use uqmodels in your own projects.

Structure๏ƒ

UQModel is currently made up of 6 parts:

Root : in charge of base object, pipeline and UQModel definition๏ƒ

Root folder of UQModel library contains 2 python sources:

  • UQModel: define UQModel class (ML-dataset -> pred & UQ-KPI) UQmodel is a pipeline of modeling & postprocessing aim to provide prediction and complementary UQKPI. It is composed of a (optional) preprocessor, (optinals) predictor, a UQEstimator (see Modeling), and a list of UQKPI_Processors (see Post-processing).

  • Custom_UQModel : Implementation of Custom UQModel from the UQModel class

  • processing : define utility object for processing task (data -> data processed) & (Load & Save functionality for data and Processor/estimators).

    • Processor : Class that aim to process in a standards (Fit/Transform procedure) data with agnostic form.

    • Cache_manger : Class that define load/save/chech_if_cache functionality within a Processor (or an Estimator).

    • Data_loader : Agnostic class that take a load_API function to create a data_loader that take query and return data.

    • Pipeline : Class that instanciate a pipeline composed of Data_loader and list of Processor.

Modeling folder : in charge of uncertainty quantification (ML-dataset -> pred & UQmeasure)๏ƒ

  • UQEstimator : Define the UQEstimator class aimed to handle in an agnostic way estimator that perform uncertainty quantification, to force mutualization of the valorization of the UQmeasures produced.

    • ML_estimator : folder that contains functionality link to uncertainty quantification with/on machine learning model.

      • basline.py : common implementation of UQ-estimators from scikit estimator.

      • random_forest_UQ.py : modification of sklearn-RF to perform UQ-regression.

    • DL_estimator : folder that contains functionality link to uncertainty quantification with/on deep learning model.

      • Neural_network_UQ : Instanciate common procedure (fit,transform) for differents Neural network UQ-paradigm, and allow to instanciate MLP_MC-Dropout,MLP_Deep-Ensemble MLP-EDL that perform Gaussian probabilist regression (mu,sigma) and respective meta-modeling procedure (MC-Dropout, Deep_Ensemble or EDL)

      • loss, data_embedding,metalayers,utils : sources of loss, layers, and tools.

      • Lstm_ed,Transformer_ed : Naรฏve implementation of Encoder-Decoder models for multihorizon forecasting handling MC-Dropout.

Post-processing folder : in charge of postprocessing that include UQ & Anomaly residual-based๏ƒ

  • UQ_processing.py : Functionality link to UQ processing to intermediate form or to UQ-KPI form

  • anomaly_processing.py : Functionality link to Anomaly processing (with UQ) to produce anomaly score.

  • UQKPI_Processor : Class that define UQKPI_Processor in charge to tranform modeling output into specifics UQKPI as (Predictive intervals, Local model unreliability score, Anomaly score with UQ).

  • Custom_UQKPI_Processor : Custom UQKPI_Processor

Preprocessing folder : in charge of preprocessing (data row -> structured data -> ML-dataset):๏ƒ

  • Preprocessing.py : Functionality link to preprocessing or feature engenering

  • Preprocessor : Class that define Preprocessor in charge to transform data into preprocessed data.

  • structure.py : Class that aim to encapsulate metadata link to data structure.

Evaluation folder : (in progress) in charge of evaluation step๏ƒ

(Work in progress) : aim to provide metrics and functionality to faciliate evaluation step and to harmonise evaluation step and metrics definition to Abench library (agnostic benchmark purpose)

Visualization folder : (in progress) in charge of visualization๏ƒ

(Work in progress) : Refactoring needed. Contains various visualization tools

Data_generation folder : (in progress) in charge of provide synthetic dataset for example๏ƒ

(Work in progress) : Refactoring needed. Contains a data_generation process to generate gaussian time dependent time series.