Configuring Model Profiling

Robust Intelligence profiles the model in order to determine the tests to run on the model. With large datasets, this profiling can take a long time. Configuration options to alter the behavior of the profiling can shorten processing time.

By default, Robust Intelligence attempts to infer optimal values for all of these options. Manually set these parameters only when Robust Intelligence is not selecting appropriate values.

Template

Specify this configuration in the AI Stress Testing Configuration JSON file, under the "model_profiling" parameter of the "profiling_config" dictionary.

{
    #...,
    "model_profiling": {
        "nrows_for_summary": 1,
        "nrows_for_feature_importance": 2,
        "metric_configs_json": '{"foo": "bar"}',
        "impact_metric": "foo",
        "impact_label_threshold": 0.8,
        "drift_impact_metric": "foo",
        "subset_summary_metric": "foo",
        "num_feats_for_subset_summary": 3,
        "custom_metrics": {...},
    }
}

Arguments

Argument	Type	Description
nrows_for_summary	int	Default is null. The number of rows to use for calculating summary metrics of model. Specifying a large number of rows can affect performance.
nrows_for_feature_importance	int	Default is null. The number of rows to use when calculating feature importance of the model. Specifying a large number of rows can affect performance. This setting is ignored when feature importance is configured.
metric_configs_json	string	Default is null. The parameters to configure each metric used during testing, specified as a JSON string. For instance, to configure NDCG to accumulate only to a specific rank k=50, specify {"normalized_discounted_cumulative_gain": {"k": 50}}.
impact_metric	string	Default is null. The metric to use when computing model impact for abnormal input and transformation tests.
impact_label_threshold	float	Default is 0.8. When the fraction of labeled rows in the evaluation data falls below this threshold, Average Prediction is used for impact_metric and drift_impact_metric.
drift_impact_metric	string	Default is null. The metric to use when computing model impact for drift tests.
subset_summary_metric	string	Calculated by taking the difference between the worst subset degradation and the overall degradation of the configured metric.
num_feats_for_subset_summary	Optional int64	Number of features over which the subset performance degradation summary metric is aggregated.
custom_metrics	dict	Default is null. Specifies the configuration for user-defined custom metrics that can be used in Robust Intelligence tests.

For more information on how to specify custom metrics, see Custom Metrics.