Configuring Model Profiling

Robust Intelligence profiles the model in order to determine the tests to run on the model. With large datasets, this profiling can take a long time. Configuration options to alter the behavior of the profiling can shorten processing time. To improve performance of subsequent test runs, profiles are cached and reused; see the performance section for information about cached profiles.

By default, Robust Intelligence attempts to infer optimal values for all of these options. Manually set these parameters only when Robust Intelligence is not selecting appropriate values.

Model profiling template

Specify this configuration in the AI Stress Testing Configuration JSON file, under the model_profiling parameter of the profiling_config dictionary.

{
    #...,
    "model_profiling": {
        "nrows_for_summary": "1",
        "nrows_for_feature_importance": "2",
        "metric_configs_json": '{"foo": "bar"}',
        "impact_metric": "foo",
        "impact_label_threshold": 0.8,
        "drift_impact_metric": "foo",
        "subset_summary_metric": "foo",
        "num_feats_for_subset_summary": "3",
        "custom_metrics": {...},
    }
}

Model profiling arguments

The properties of the model_profiling configuration in the profiling_config are as follows:

Argument Type Description
nrows_for_summary string Default is null. The number of rows to use for calculating summary metrics of model. Specifying a large number of rows can affect performance.
nrows_for_feature_importance string Default is null. The number of rows to use when calculating feature importance of the model. Specifying a large number of rows can affect performance. This setting is ignored when feature importance is configured.
metric_configs_json string Default is null. The parameters to configure each metric used during testing, specified as a JSON string. For instance, to configure NDCG to accumulate only to a specific rank k=50, specify {"normalized_discounted_cumulative_gain": {"k": 50}}.
impact_metric string Default is null. The metric to use when computing model impact for abnormal input and transformation tests.
impact_label_threshold float Default is 0.8. When the fraction of labeled rows in the evaluation data falls below this threshold, Average Prediction is used for impact_metric and drift_impact_metric.
drift_impact_metric string Default is null. The metric to use when computing model impact for drift tests.
subset_summary_metric string Calculated by taking the difference between the worst subset degradation and the overall degradation of the configured metric.
num_feats_for_subset_summary Optional string Number of features over which the subset performance degradation summary metric is aggregated.
custom_metrics dict Default is null. Specifies the configuration for user-defined custom metrics that can be used in Robust Intelligence tests.

For more information on how to specify custom metrics, see Custom Metrics.