uqmodels.evaluation package

Submodules

uqmodels.evaluation.base_metrics module

Metrics module for UQ method evaluation.

uqmodels.evaluation.base_metrics.NLL_loss(y, pred, sigma)[source]

uqmodels.evaluation.base_metrics.ace(y_true, y_pred_lower, y_pred_upper, alpha)[source]

uqmodels.evaluation.base_metrics.average_coverage(y_true, y_pred_lower, y_pred_upper)[source]

uqmodels.evaluation.base_metrics.interval_score(y_true, y_pred_lower, y_pred_upper, alpha)[source]

uqmodels.evaluation.base_metrics.mae(y_true, y_pred)[source]

uqmodels.evaluation.base_metrics.perf_pred(y_pred, y)[source]

uqmodels.evaluation.base_metrics.print_real_metrics_meta(res)[source]

uqmodels.evaluation.base_metrics.q_loss(y, pred, per)[source]

uqmodels.evaluation.base_metrics.quantile_loss(y, y_pred_lower, y_pred_upper, alpha)[source]

uqmodels.evaluation.base_metrics.real_metrics(y, pred_, bot, top, train, test, alpha=0.9, train_fit=None, verbose=0)[source]

uqmodels.evaluation.base_metrics.rmse(y_true, y_pred)[source]

uqmodels.evaluation.base_metrics.sharpness(y_pred_lower, y_pred_upper)[source]

uqmodels.evaluation.base_metrics.sharpness2(y_pred_lower, y_pred_upper)[source]

uqmodels.evaluation.evaluation module

uqmodels.evaluation.evaluation.compute_perf(Y_, pred, sigma, test)[source]

uqmodels.evaluation.evaluation.evaluate(y, output, list_metrics, list_sets=None, context=None, verbose=False)[source]

uqmodels.evaluation.evaluation.var_explain(y, pred, var_A, var_E, set_)[source]

uqmodels.evaluation.metrics module

class uqmodels.evaluation.metrics.Encapsulated_metrics[source]

Bases: ABC

Abstract Encapsulated Metrics class : Allow generic manipulation of metrics with output specifyied format

compute(y, output, sets, context, **kwarg)[source]

Compute metrics

Parameters:

output (array) – Model results
y (array) – Targets
sets (array list) – Sub-set (train,test)
context (array) – Additional information that may be used in metrics

class uqmodels.evaluation.metrics.Generic_metric(ABmetric, name='Metric', mask=None, list_ctx_constraint=None, reduce=True, **kwarg)[source]

Bases: Encapsulated_metrics

compute(y, output, sets, context, **kwarg)[source]

Compute metrics

Parameters:

output (array) – Model results
y (array) – Targets
sets (array list) – Sub-set (train,test)
context (array) – Additional information that may be used in metrics

uqmodels.evaluation.metrics.UQ_Gaussian_NLL(y, output, set_, mask, reduce, type_UQ='var', mode=None, **kwarg)[source]

Compute Neg likelihood by transform UQ into sigma using guassian assumption

Parameters:

y (np.array) – Targets/observation
output (np.array) – modeling output : (y,UQ)
set (list of mask) – subset specification
mask (bool array) – mask the last dimension
reduce (bool) – apply reduction
type_UQ (str, optional) – _description_. Defaults to “var”.
mode (_type_, optional) – _description_. Defaults to None.

Returns:

_description_

Return type:

val

uqmodels.evaluation.metrics.UQ_absolute_residu_score(y, output, set_, mask, reduce, type_UQ='var', mode=None, **kwarg)[source]

Compute absolute residu score from UQ,pred,y

Parameters:

y (np.array) – Targets/observation
output (np.array) – modeling output : (y,UQ)
set (list of mask) – subset specification
mask (bool array) – mask the last dimension
reduce (bool) – apply reduction
type_UQ (str, optional) – _description_. Defaults to “var”.
mode (_type_, optional) – _description_. Defaults to None.

Returns:

val

Return type:

val

uqmodels.evaluation.metrics.UQ_average_coverage(y, output, set_, mask, reduce, type_UQ='var', alpha=0.045, mode='UQ', **kwarg)[source]

Compute data coverage by transform UQ into (1-alpha)% coverage PIs

Parameters:

y (np.array) – Targets/observation
output (np.array) – modeling output : (y,UQ)
set (list of mask) – subset specification
mask (bool array) – mask the last dimension
reduce (bool) – apply reduction
type_UQ (str, optional) – _description_. Defaults to “var”.
alpha (float, optional) – _description_. Defaults to 0.045.
mode (str, optional) – _description_. Defaults to ‘UQ’.

Returns:

_description_

Return type:

_type_

uqmodels.evaluation.metrics.UQ_dEI(y, output, set_, mask, reduce, type_UQ='var_A&E', **kwarg)[source]

disentangled epistemic indicator from pred,UQ that provide insight about model unreliability

Parameters:

y (np.array) – Targets/observation
output (np.array) – modeling output : (y,UQ)
set (list of mask) – subset specification
mask (bool array) – mask the last dimension
reduce (bool) – apply reduction

Returns:

array of metrics values

Return type:

val

uqmodels.evaluation.metrics.UQ_heteroscedasticity_ratio(y, output, set_, mask, reduce, type_UQ='var', mode=None, **kwarg)[source]

Compute ratio by transform UQ into sigma using guassian assumption

Parameters:

y (np.array) – Targets/observation
output (np.array) – modeling output : (y,UQ)
set (list of mask) – subset specification
mask (bool array) – mask the last dimension
reduce (bool) – apply reduction
type_UQ (str, optional) – _description_. Defaults to “var”.
mode (_type_, optional) – _description_. Defaults to None.

Returns:

_description_

Return type:

val

uqmodels.evaluation.metrics.UQ_sharpness(y, output, set_, mask, reduce, type_UQ='var', **kwarg)[source]

Compute sharpness by transform UQ into 95% coverage PIs then compute size of PIs.

Parameters:

y (np.array) – Targets/observation
output (np.array) – modeling output : (y,UQ)
set (list of mask) – subset specification
mask (bool array) – mask the last dimension
reduce (bool) – apply reduction
type_UQ (str, optional) – _description_. Defaults to “var”.

Returns:

_description_

Return type:

_type_

uqmodels.evaluation.metrics.anom_score(y, output, set_, mask, reduce, type_output='all', min_A=0.08, min_E=0.02, **kwarg)[source]

uqmodels.evaluation.metrics.build_ctx_mask(context, list_ctx_constraint)[source]

uqmodels.evaluation.metrics.calibrate_var(y, output, set_, mask, reduce, type_output='all', alpha=0.955, **kwarg)[source]

uqmodels.evaluation.metrics.confidence_score(y, output, set_, mask, reduce, type_output='all', min_A=0.08, min_E=0.02, **kwarg)[source]

uqmodels.evaluation.metrics.cov_metrics(y, y_lower, y_upper, **kwarg)[source]

uqmodels.evaluation.metrics.dEI(y, output, set_, mask, reduce, type_output='all', **kwarg)[source]

uqmodels.evaluation.metrics.mae(y, output, set_, mask, reduce, **kwarg)[source]

uqmodels.evaluation.metrics.rmse(y, output, set_, mask, reduce, **kwarg)[source]

Root mean square error metrics

Parameters:

y (np.array) – Targets/observation
output (np.array) – modeling output : (y,UQ)
set (list of mask) – subset specification
mask (bool array) – mask the last dimension
reduce (bool) – apply reduction

Returns:

rmse values

Return type:

val

uqmodels.evaluation package

Submodules

uqmodels.evaluation.base_metrics module

uqmodels.evaluation.evaluation module

uqmodels.evaluation.metrics module

Module contents