AI Stress Testing

This tutorial will get you started using the RIME CV CLI to stress test an image classification model using the Animals with Attributes 2 Dataset. If you are interested in learning more about RIME for CV, reach out to your RI trial representatives.

Setup

Please ensure that the extra RIME CV dependencies have been installed from the cv_requirements.txt file from installation. If you run into a ModuleNotFoundError at any point during this walkthrough, it is likely that you need to install the RIME CV Extras!

pip install -r cv_requirements.txt

Running Stress Testing on the Image Classification Example

In this tutorial, we will cover pre-production testing of an animal classification model using the RIME CV automated test suite.

Then, to kick off a CV Test run, execute the following command in your local terminal:

rime-engine run-images --config-path images_examples/classification/awa2/stress_test_config.json

If you explore the test config in images_examples/classification/cv/stress_tests_config.json you’ll see that we’ve configured a few parameters to specify the data, model, and other task-specific information. After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project. For a full guide to what you are seeing, please see our UI Documentation.

For a full reference on the configuration file see the CV Configuration Reference.

For additional command line options, please see the CLI Reference.

Running Stress Testing on the Object Detection Example

In this tutorial, we will cover pre-production testing of an object detection model trained on Pascal VOC using the RIME CV automated test suite.

Then, to kick off a CV Test run, execute the following command in your local terminal:

rime-engine run-images --config-path images_examples/detection/voc/stress_test_config.json

If you explore the test config in images_examples/detection/voc/stress_test_config.json you’ll see that we’ve configured a few parameters to specify the data, model, and other task-specific information. After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project. For a full guide to what you are seeing, please see our UI Documentation.

For a full reference on the configuration file see the CV Configuration Reference.

For additional command line options, please see the CLI Reference.

Running Stress Testing on your own Model and Datasets

To run RIME using your own data and model, please consult the CV Data Guide guide for the expected data format and How to Create a CV Model File for step-by-step instructions on how to connect your model to the testing framework.

Model inference is usually the most time-consuming part of the testing framework, which is why we recommend specifying cached prediction logs. This can be done by providing predictions within the datafile itself (by adding the “probabilities” key to each data sample) or by specifying them in a separate file. In the classification example above, we provide the prediction within the datafile itself (see images_examples/classification/awa2/data/train_inputs_trial.json). For an example of providing predictions within the datafile itself for object detection, see images_examples/detection/voc/stress_test_config.json.

However, if you do not wish to create a prediction log beforehand, RIME can call your model during a test run and infer its performance using a subsample of the provided datasets.

Conclusion

Congratulations! You’ve successfully used RIME to test out the various CV models. Assuming the RIME backend was launched successfully, you can navigate to the RIME web interface and view the test results.

Once again, we strongly recommended that you run RIME using precomputed predictions. This will greatly improve both the RIME runtime and the test suite result quality. Model inference tends to be the most computationally expensive part of each RIME run. While access to the model is still required for some tests due to design constraints (e.g., the use of randomness, iterative attacks, etc.), providing predictions in advance can help RIME avoid redundant computation so each run is fast and focused.

Troubleshooting

If you run into errors running this walkthrough, please reference the RIME Python Package section of our FAQ. Additionally, your RI representative will be happy to assist–feel free to reach out!