Running AI Firewall on the Fraud Example with the RIME SDK

This tutorial will guide you through how to get started with the RIME SDK by running AI Firewall on a model trained over a slightly modified version of the IEEE-CIS Fraud Detection dataset.

Before you begin, make sure:

  • You have completed the installation

  • You are in the Python environment with the RIME Python SDK installed

  • You have downloaded the example datasets and configuration files

Setup

The RIME SDK reads data and model from AWS S3. To begin using the RIME SDK, please first upload the fraud_continuous_testing folder (found inside zip file downloaded above) to an S3 bucket that you have access to.

Running AI Firewall on the Fraud Example

For the basic example, we will run AI Firewall with a model. If you do not want to run RIME with the model, and instead want to run in a different mode (e.g. using prediction logs instead), see the end of this document for different workflows. The model provided in the examples folder is a Catboost model and we will need to install extra dependencies (in order to call that model). To do this, please run:

pip install -r workshop_requirements.txt

To begin, initialize the RIMEClient and point it to the location of your RIME backend.

from rime_sdk import RIMEClient

rime_client = RIMEClient("rime-backend.<YOUR_ORG_NAME>.rime.dev", "<YOUR_API_TOKEN>")

Next, start python and run the following:

import json

project = rime_client.create_project("FooBar's Test Project", "DESCRIPTION")

# Load the configuration file
f = open("examples/fraud_continuous_testing/trial_config.json")
config = json.load(f)

config["data_info"]["ref_path"] = "s3://path/to/examples/fraud_continuous_testing/ref.csv"
config["data_info"]["eval_path"] = "s3://path/to/examples/fraud_continuous_testing/eval_2021_04_01_to_2021_05_01.csv"

# Start the tests
job = rime_client.start_firewall_from_reference(test_run_config=config, project_id=project.project_id)
job.get_status(poll_rate_sec=1.0, verbose=True, wait_until_finish=True)

This will start running the tests and wait for them to finish. Once they finish, you will be able to see the results through the RIME interface on your web browser.

Running AI Firewall on an Incremental Batch of Data

Similar to the above steps, but instead the command you will want to run is:

# Load the configuration file
f = open("examples/fraud_continuous_testing/stress_testing_config.json")
ot_config = json.load(f)
ot_config["data_info"]["ref_path"] = "s3://path/to/examples/fraud_continuous_testing/ref.csv"
ot_config["data_info"]["eval_path"] = "s3://path/to/examples/fraud_continuous_testing/eval_2021_04_01_to_2021_04_29.csv"

# Start the stress test
job = rime_client.start_stress_test(test_run_config=ot_config, project_id=project.project_id)
job.get_status(poll_rate_sec=1.0, verbose=True, wait_until_finish=True)
ot_test_run_id = job.get_test_run_result()['metadata.id'][0]

# Create Firewall
firewall = rime_client.create_firewall("fw sdk test", 86400, ot_test_run_id, project.project_id)

# Run Firewall on an Incremental Batch of Data
incremental_config = {
    "eval_path": "s3://path/to/examples/fraud_continuous_testing/eval_2021_04_30_to_2021_05_01.csv",
    "timestamp_col": "timestamp"
}
job = firewall.run_firewall_incremental_data(incremental_config)