AI Firewall Continuous Tests

This tutorial will guide you through how to get started with RIME by running AI Firewall Continuous Tests on a text classification model trained on (a slightly modified version of) the arXiv dataset to categorize academic papers.

Setup

NLP Extras

Please ensure that the extra RIME NLP dependencies have been installed from the nlp_requirements.txt file from installation. If you run into a ModuleNotFoundError at any point during this walkthrough, it is likely that you need to install the RIME NLP Extras!

pip install -r nlp_requirements.txt

AI Firewall Continuous Tests vs. AI Firewall Continuous Tests Trial

We offer two workflows for firewall monitoring. In the primary workflow (“RIME Firewall Continuous Tests”), the user selects their desired stress test run (corresponding to the model being deployed into production) from the UI to create a firewall. From there the user periodically uploads incoming production data to RIME.

We also offer a simpler trial workflow (“RIME Firewall Continuous Tests Trial”) to demonstrate firewall capabilities without requiring the user to upload a stress test run or create a firewall. This tutorial will provide examples for both workflows.

Running Firewall Continuous Tests on the ArXiv Example

Running AI Firewall Continuous Tests Trial

To kick off a run of RIME Firewall Continuous Tests Trial:

rime-engine run-firewall-nlp-trial --config-path nlp_examples/classification/arxiv/firewall_trial_config.json

After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project. For a full guide to what you are seeing, please see our UI Documentation.

Running AI Firewall Continuous Tests

To kick off a run of RIME Firewall Continuous Tests:

Kick off a Stress Testing run (the reference dataset and model from this run will be used for firewall continuous tests).
```
rime-engine run-nlp --config-path nlp_examples/classification/arxiv/stress_tests_config_no_model.json
```
Go to the RIME web client, click on the “Default Project” and navigate to the “AI Firewall” tab on the left . You should see the following page.
Click on “Configure AI Firewall” and fill out the details. The “Name” field can be a name of your choice. The “AI Stress Test” field should be “”ArXiv Topic Classification” (this refers to the AI Stress Testing run that we just ran). Finally, the “Bin Size” field should be “1 Day”. Once you’ve filled everything out, you should record the “Firewall ID” (you will need this later to upload data and predictions to monitor) and click “Done”.
Using the Firewall ID we just generated, we can now upload data and predictions and run AI Firewall Continuous Tests. If you forgot to record the firewall ID in the previous step, you can find it by clicking the on the right.
```
rime-engine run-firewall-nlp --config-path nlp_examples/classification/arxiv/firewall_config.json --firewall-id <FIREWALL-ID>
```
Once this is done you can check the RIME interface and see that we’ve uploaded data from June 18, 2007 to June 25, 2007. To upload data for the next week, open the nlp_examples/classification/arxiv/firewall_config.json file and update the data paths to be:
```
{
 "eval_path": "nlp_examples/classification/arxiv/data/val_2.json.gz",
 "eval_pred_path": "nlp_examples/classification/arxiv/data/val_preds_2.json.gz"
}
```
Repeat the command from step 4 to upload the new data.
Repeat step 6 for data files val_3.json.gz/val_3_preds.json.gz through val_9.json.gz/val_9_preds.json.gz

After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project. For a full guide to what you are seeing, please see our UI Documentation.

Running Firewall Continuous Tests on your own Data and Model

This guide will cover how to run AI Firewall Continuous Tests on your own data and model.

Collect Data in the Correct Format

For AI Firewall Continuous Tests, RIME expects one reference dataset and evaluation datasets. For the Trial Version only a single evaluation dataset is needed. The evaluation datasets REQUIRE a timestamp attribute and will be split into slices based on that attribute. RIME will run tests comparing each slice to the reference dataset.

AI Firewall Continuous Tests REQUIRES predictions for each datapoint. These should be passed in as an attribute in all datasets.

Create configuration

With your data ready, you can now create a configuration file for AI Firewall Continuous Tests. For a detailed reference on what the configurations should look like, see AI Firewall Continuous Tests Trial Configuration Reference and AI Firewall Continuous Tests Configuration Reference.

Run the CLI

You can run the AI Firewall Continuous Tests Trial through the CLI, pointing it to the configuration file created in the above step, with:

rime-engine run-firewall-nlp-trial --config-path <PATH-TO-CONFIGURATION>

For AI Firewall Continuous Tests, you will need to obtain a Firewall ID. Please see the example above for detailed instructions on how to do so. You can then run the CLI, pointing it to your Firewall ID and the configuration file created in the above step, with:

rime-engine run-firewall-nlp --firewall-id <FIREWALL-ID> --config-path <PATH-TO-CONFIGURATION>

After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project. For a full guide to what you are seeing, please see our UI Documentation.

For additional command line options, please see the CLI Reference.

Troubleshooting

If you run into errors running this walkthrough, please reference the RIME Python Package section of our FAQ. Additionally, your RI representative will be happy to assist–feel free to reach out!