# Validating Your Model with AI Stress Testing

This tutorial will guide you through validating CV models with RIME AI Stress Testing.

All examples are available in the `rime_trial/` bundle provided during installation.

{{ cv_setup_extra_note }}

## Running Stress Testing on an Image Classification Example

This example uses an image classification model trained on a slightly modified version of the <a target="_blank" rel="noopener" href="https://cvml.ist.ac.at/AwA2/">Animals with Attributes 2</a> dataset.

To kick off a run of AI Stress Testing:
```bash
rime-engine run-images --config-path images_examples/classification/awa2/stress_test_config.json
```

{{ tabular_ui_redirect }}

If you explore the test config in `images_examples/classification/awa2/stress_tests_config.json` you'll see that we've configured a few parameters to specify the data, model, and other task-specific information:
```
{
    "run_name": "Image Classification",
    "data_info": { ... },
    "data_profiling_info": {
        "class_names": ["antelope", "grizzly+bear", ...,  "cow", "dolphin"]
    },
    "model_info": {
        "path": "images_examples/classification/awa2/awa2_cpu.py"
    },
    "model_task": "Image Classification"
}
```

{{ cv_config_note }}

{{ cli_note }}

## Running Stress Testing on an Object Detection Example

This example uses an object model trained on a slightly modified version of the <a target="_blank" rel="noopener" href="https://paperswithcode.com/dataset/pascal-voc">Pascal VOC</a> dataset.

To kick off a run of AI Stress Testing:
```bash
rime-engine run-images --config-path images_examples/detection/voc/stress_test_config.json
```

{{ tabular_ui_redirect }}

If you explore the test config in `images_examples/detection/voc/stress_test_config.json` you'll see that we've configured a few parameters to specify the data, model, and other task-specific information.
```
{
    "run_name": "Object Detection",
    "data_info": { ... },
    "data_profiling_info": {
        "class_names": ["aeroplane", "bicycle", ...,  "train", "tvmonitor"]
    },
    "model_info": {
        "path": "images_examples/detection/voc/model.py"
    },
    "model_task": "Object Detection"
}
```

{{ cv_config_note }}

{{ cli_note }}

---

## Running Stress Testing on Your Own Model and Datasets

### Define a Python Model File

Please refer to [How to Create a CV Model FIle](/for_data_scientists/reference/cv/specify_model_cv.md) for step-by-step instructions on creating a model interface for RIME.

### Gather Datasets

#### 1. Prepare Input Data

For CV data, RIME uses `.json` (or `.jsonl`) files containing metadata for each image. For a detailed specification of data formatting, see [Input Data Format](/for_data_scientists/reference/cv/task_data_format.md).

#### 2. Specify Prediction Logs (Recommended)

Because model inference is usually the most time-consuming part of the testing framework, we recommend specifying cached prediction logs.

This can be done within the datafile itself (i.e., by adding a `"probabilities"` key to each data sample) or by [specifying them in separate files](/for_data_scientists/reference/cv/prediction_info). The actual specification for prediction data is described in [Prediction Cache Data Format](/for_data_scientists/reference/cv/task_prediction_cache_format.md).

In the image classification example above, we specified model predictions in the datafile itself (see `images_examples/classification/awa2/data/{train|test}_inputs_trial.json`).

An example prediction (see the `"probabilities"` key) can be viewed from this file by running the following command from your terminal:
```
cat images_examples/classification/awa2/data/train_inputs_trial.json | jq '.[0]'
```

However, if you do not wish to create a separate prediction log, RIME can call your model during a test run and infer its performance using a subsample of the provided datasets.

### Create Configuration

With your data and model ready, you can now create a configuration file. Examples of these can be found in the `rime_trial/` bundle (the ones used for these examples are under `images_examples/`).

For a detailed reference on what the configuration should look like, see [AI Stress Testing Configuration Reference](/for_data_scientists/reference/cv/stress_testing.md).

## Conclusion

Congratulations! You've successfully used RIME to test out the various CV models.

Once again, we strongly recommended that you run RIME using precomputed predictions. This will greatly improve both the RIME runtime and the test suite result quality.

Model inference tends to be the most computationally expensive part of each RIME run. While access to the model is still required for some tests due to design constraints (e.g., the use of randomness, iterative attacks, etc.), providing predictions in advance can help RIME avoid redundant computation so each run is fast and focused.

## Troubleshooting
{{ troubleshooting_python_package_redirect }}