Validating Your Model with AI Stress Testing

This tutorial will guide you through validating CV models with AI Stress Testing.

All examples are available in the rime_trial/ bundle provided during installation.

Please ensure that the extra RIME CV dependencies have been installed from the cv_requirements.txt file from installation. If you run into a ModuleNotFoundError at any point during this walkthrough, it is likely that you need to install the RIME CV Extras!

pip install -r cv_requirements.txt

Running Stress Testing on an Image Classification Example

This example uses an image classification model trained on a slightly modified version of the Animals with Attributes 2 dataset.

To kick off a run of AI Stress Testing:

rime-engine run-images --config-path images_examples/classification/awa2/stress_test_config.json

After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project.

If you explore the test config in images_examples/classification/awa2/stress_tests_config.json you’ll see that we’ve configured a few parameters to specify the data, model, and other task-specific information:

{
    "run_name": "Image Classification",
    "data_info": { ... },
    "data_profiling_info": {
        "class_names": ["antelope", "grizzly+bear", ...,  "cow", "dolphin"]
    },
    "model_info": {
        "path": "images_examples/classification/awa2/awa2_cpu.py"
    },
    "model_task": "Image Classification"
}

For a full reference on the configuration file see the CV Configuration Reference.

For additional command line options, please see the CLI Reference.

Running Stress Testing on an Object Detection Example

This example uses an object model trained on a slightly modified version of the Pascal VOC dataset.

To kick off a run of AI Stress Testing:

rime-engine run-images --config-path images_examples/detection/voc/stress_test_config.json

After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project.

If you explore the test config in images_examples/detection/voc/stress_test_config.json you’ll see that we’ve configured a few parameters to specify the data, model, and other task-specific information.

{
    "run_name": "Object Detection",
    "data_info": { ... },
    "data_profiling_info": {
        "class_names": ["aeroplane", "bicycle", ...,  "train", "tvmonitor"]
    },
    "model_info": {
        "path": "images_examples/detection/voc/model.py"
    },
    "model_task": "Object Detection"
}

For a full reference on the configuration file see the CV Configuration Reference.

For additional command line options, please see the CLI Reference.

Running Stress Testing on Your Own Model and Datasets

Define a Python Model File

Please refer to How to Create a CV Model FIle for step-by-step instructions on creating a model interface for RI.

Gather Datasets

1. Prepare Input Data

For CV data, RI uses .json (or .jsonl) files containing metadata for each image. For a detailed specification of data formatting, see Input Data Format.

2. Specify Prediction Logs (Recommended)

Because model inference is usually the most time-consuming part of the testing framework, we recommend specifying cached prediction logs.

This can be done within the datafile itself (i.e., by adding a "probabilities" key to each data sample) or by specifying them in separate files. The actual specification for prediction data is described in Prediction Cache Data Format.

In the image classification example above, we specified model predictions in the datafile itself (see images_examples/classification/awa2/data/{train|test}_inputs_trial.json).

An example prediction (see the "probabilities" key) can be viewed from this file by running the following command from your terminal:

cat images_examples/classification/awa2/data/train_inputs_trial.json | jq '.[0]'

However, if you do not wish to create a separate prediction log, RI can call your model during a test run and infer its performance using a subsample of the provided datasets.

Create Configuration

With your data and model ready, you can now create a configuration file. Examples of these can be found in the rime_trial/ bundle (the ones used for these examples are under images_examples/).

For a detailed reference on what the configuration should look like, see AI Stress Testing Configuration Reference.

Conclusion

Congratulations! You’ve successfully used RI to test out the various CV models.

Once again, we strongly recommended that you run RI using precomputed predictions. This will greatly improve both the RI runtime and the test suite result quality.

Model inference tends to be the most computationally expensive part of each RI run. While access to the model is still required for some tests due to design constraints (e.g., the use of randomness, iterative attacks, etc.), providing predictions in advance can help RI avoid redundant computation so each run is fast and focused.

Troubleshooting

If you run into issues, please refer to our Troubleshooting page for help! Additionally, your RI representative will be happy to assist — feel free to reach out!