AI Stress Testing

This tutorial will guide you through how to get started with RIME by running AI Stress Testing on a model trained over a slightly modified version of the Adult Census Income dataset.

Running Stress Testing on the Income Example

Stress Testing with a Model and Dataset

For the basic example, we will run AI Stress Testing with a model. If you do not want to run RIME with the model, and instead want to run in a different mode (e.g. using prediction logs instead), see the end of this document for different workflows.

To kick off a run of AI Stress Testing:

rime-engine run-stress-tests --config-path examples/income/stress_tests_model.json

After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project. For a full guide to what you are seeing, please see our UI Documentation.

For additional command line options, please see the CLI Reference.

Stress Testing with Prediction Logs

Similar to the above steps, except the CLI command to run RIME with just prediction logs is:

rime-engine run-stress-tests --config-path examples/income/stress_tests_prediction_logs.json

Note that the command is exactly the same EXCEPT for the --config-path provided.

Stress Testing with Compliance

This will run a specific suite of tests geared towards bias/fairness. Similar to the above steps - to kick off a Compliance only run:

rime-engine run-stress-tests --config-path examples/income/stress_tests_compliance.json

The command is exactly the same, but inside the config file the categories list contains Compliance and data_info contains protected_features. The tests will be run over these features.

Running Stress Testing on your own Model and Datasets

This guide will cover how to run AI Stress Testing on your own data and model.

Define a Python Model File in the Correct Format

A model is not required for AI Stress Testing, but providing one will produce better results. RIME expects models to be provided in a specific format; see How to Create a Python Model File for step-by-step instructions.

Collect Data in the Correct Format

For AI Stress Testing, RIME expects that you will upload two datasets: a reference dataset (typically the training data) and an evaluation dataset (typically the validation, testing, or other data). Currently RIME expects each dataset to be passed in as a csv or parquet file where each column is a separate feature.

You can optionally provide labels for each datapoint. These should be passed in as a column in both datasets. Providing labels allows RIME to surface model performance metrics across our tests.

NOTE: in the multi-class classification scenario, each label should be a nonnegative integer i where the corresponding prediction for label i should be found in the ith dimenson of the prediction vector.

You can optionally provide predictions for each datapoint. If the model task is multi-class classification, they must be specified in a separate csv or parquet file as a dataframe where the ith column represents the predicted probabilities for the ith class/label. For all other model tasks, they should be provided as a column in both datasets. In the absence of a model, providing predictions will produce better results.

Create Configuration

With your data and model ready, you can now create a configuration file for AI Stress Testing. For a detailed reference on what the configuration should look like, see AI Stress Testing Configuration Reference.

Run the CLI

You can run the CLI, pointing it to the configuration file created in the above step, with:

rime-engine run-stress-tests --config-path <PATH-TO-CONFIGURATION>

After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project. For a full guide to what you are seeing, please see our UI Documentation.

Caching Test Results for Future Uploading

In some cases, you may want to cache the test results - if you need to restart the server, then you will be able to upload the results without needing to rerun stress testing. To do this, specify a --save-path when running stress testing:

rime-engine run-stress-tests --config-path <PATH-TO-CONFIGURATION> --save-path <SAVE-PATH>

This will cache results in a directory specified in --save-path. To directly re-upload the results, just do:

rime-engine upload --save-path <SAVE-PATH>

Troubleshooting

If you run into errors running this walkthrough, please reference the RIME Python Package section of our FAQ. Additionally, your RI representative will be happy to assist–feel free to reach out!