Running AI Stress Testing on the Fraud Example with RIME SDK

This tutorial will guide you through how to get started with RIME SDK by running AI Stress Testing on a model trained over a slightly modified version of the IEEE-CIS Fraud Detection dataset.

Before you begin, make sure:

You have completed the installation
You are in the Python environment with the RIME Python SDK installed
You have downloaded the example datasets and configuration files

Setup

The RIME SDK reads data and model from AWS S3. To begin using the RIME SDK, please first upload the fraud folder (found inside zip file downloaded above) to an S3 bucket that you have access to.

Running Stress Testing on the Fraud Example

For the basic example, we will run AI Stress Testing with a model. If you do not want to run RIME with the model, and instead want to run in a different mode (e.g. using prediction logs instead), see the end of this document for different workflows. The model provided in the examples folder is a Catboost model and we will need to install extra dependencies (in order to call that model). To do this, please run:

pip install -r workshop_requirements.txt

To begin, initialize the RIMEClient and point it to the location of your RIME backend.

from rime_sdk import RIMEClient

rime_client = RIMEClient("rime-backend.<YOUR_ORG_NAME>.rime.dev", "<YOUR_API_TOKEN>")

Next, start python and run the following:

import json

project = rime_client.create_project("FooBar's Test Project", "DESCRIPTION")

# Load the configuration file
f = open("examples/fraud/stress_tests_with_model.json")
config = json.load(f)
# TODO: update the S3 path below with the actual bucket name
config["data_info"]["ref_path"] = "s3://path/to/examples/fraud/train.csv"
config["data_info"]["eval_path"] = "s3://path/to/examples/fraud/val.csv"
config["model_info"]["path"] = "s3://path/to/examples/fraud/model.py"
config["tests_config_path"] = "s3://path/to/examples/fraud/test_config.json"

# Start the tests
job = rime_client.start_stress_test(test_run_config=config, project_id=project.project_id)
job.get_status(poll_rate_sec=1.0, verbose=True, wait_until_finish=True)

This will start running the tests and wait for them to finish. Once they finish, you will be able to see the results through the RIME interface on your web browser.

Stress Testing with Prediction Logs

Similar to the above steps, but instead the command you will want to run is:

# Load the configuration file
f = open("examples/fraud/stress_tests_no_model.json")
config = json.load(f)
config["data_info"]["ref_path"] = "s3://path/to/examples/fraud/train_with_preds.csv"
config["data_info"]["eval_path"] = "s3://path/to/examples/fraud/val_with_preds.csv"

# Start the tests
job = rime_client.start_stress_test(test_run_config=config, project_id=project.project_id)
job.get_status(poll_rate_sec=1.0, verbose=True, wait_until_finish=True)

Note that the command is exactly the same EXCEPT for loading and modifying the config.

Stress Testing with Data Only

Similar to the above steps, but instead the command you will want to run is:

# Load the configuration file
f = open("examples/fraud/stress_tests_data_only.json")
config = json.load(f)
config["data_info"]["ref_path"] = "s3://path/to/examples/fraud/train.csv"
config["data_info"]["eval_path"] = "s3://path/to/examples/fraud/val.csv"

# Start the tests
job = rime_client.start_stress_test(test_run_config=config, project_id=project.project_id)
job.get_status(poll_rate_sec=1.0, verbose=True, wait_until_finish=True)

Note that the command is exactly the same EXCEPT for loading and modifying the config.

Stress Testing for RIME NLP

To specify an NLP run, add data_type="nlp" to the call to start_stress_test(). For example:

# Load the configuration file
f = open("nlp_examples/classification/emotion_recognition/stress_tests_config.json")
config = json.load(f)
config["data_info"]["ref_path"] = "s3://path/to/nlp_examples/fraud/emotion_recognition/data/train.json"
config["data_info"]["eval_path"] = "s3://path/to/nlp_examples/fraud/emotion_recognition/data/test.json"

# Start the tests
job = rime_client.start_stress_test(test_run_config=config, project_id=project.project_id, data_type="nlp")
job.get_status(poll_rate_sec=1.0, verbose=True, wait_until_finish=True)