Creating a new Continuous Test

Robust Intelligence allows you to create Continuous Testing to monitor your model for vulnerabilities over time. You can create a Continuous Test using the Python SDK.

SDK

1. Create a project using the following SDK command.

   project = rime_client.create_project(
       name="foo", description="bar", model_task="MODEL_TASK_BINARY_CLASSIFICATION"
   )
   

2. Register a reference dataset. Depending on where you are looking to pull data from you may need to create a data integration.

   reference_id = project.register_dataset(
       name=DATASET_NAME,
       data_config={
           "connection_info": {
               "data_file": {
                   "path": "s3://path/to/data/file",
               },
           },
           "data_params": {
               "label_col": LABEL_COL,
               "timestamp_col": TIMESTAMP_COL,
           },
       },
       integration_id=INTEGRATION_ID,
   )
   

   The SDK returns the ID of the reference dataset. There are additional options available to further customize how Continuous Testing ingests your data.

3. Register a model. You may optionally provide additional model information to allow Robust Intelligence to query the model during testing.

   model_id = project.register_model(
       name="baz"
   )
   

   The SDK returns the model ID. There are additional options available to further customize how Continuous Testing ingests your data.

   If you do not provide the information to query the model, you must register predictions of the model and dataset.

   project.register_predictions(
       dataset_id=reference_id
       model_id=model_id
       pred_config={
           "connection_info": {
               "data_file": {
                   "path": "s3://path/to/predictions/file",
               },
           },
           "pred_params": {"pred_col": PREDICTION_COL},
       },
       integration_id=INTEGRATION_ID,
   )
   

4. Create a new Continuous Test instance in your project. You may choose to create a Continuous Test with scheduled continuous testing parameters or leave those parameters blank. If Scheduled CT is set, Robust Intelligence will run the Continuous Test automatically for you (at an interval defined by the bin_size you set) by utilizing the data integrations you specify. Otherwise, you will need to manually run the Continuous Test over an evaluation dataset that you provide. The steps below show how to start a manual run.

   project.create_ct(
       model_id=model_id, ref_data_id=reference_id, bin_size=timedelta(hours=3)
   )
   

5. Register an evaluation dataset. This dataset must have the timestamp_col field in the data_params dictionary.

   evaluation_id = project.register_dataset(
       name="bar",
       data_config={
           "connection_info": {
               "data_file": {
                   "path": "s3://path/to/data/file",
               },
           },
           "data_params": {
               "label_col": LABEL_COL,
               "timestamp_col": TIMESTAMP_COL,
           },
       },
       integration_id=INTEGRATION_ID,
   )
   

   The SDK returns the ID of the evaluation dataset. There are additional options available to further customize how Continuous Testing ingests your data.

6. Register a prediction set if the information to query the model is not provided.

   project.register_predictions(
       dataset_id=evaluation_id
       model_id=model_id
       pred_config={
           "connection_info": {
               "data_file": {
                   "path": "s3://path/to/predictions/file",
               },
           },
           "pred_params": {"pred_col": PREDICTION_COL},
       },
       integration_id=INTEGRATION_ID,
   )
   

7. Issue the following command to start the Continuous Test, specifying the configuration created in the previous step and the unique ID of your Continuous Test. Use the argument override_existing_bins to specify whether this Continuous Test should override the results of previous time bins for which other Continuous Tests have run.

   Here, ct is your Continuous Test instance, which you can get using the Project.get_ct() method.

   ct_job = ct.start_continuous_test(
      eval_data_id=evaluation_id,  override_existing_bins=False,
   )
   ct_job.get_status(verbose=True, wait_until_finish=True)
   

   Anytime new data has been gathered and you want to re-run your test, repeat steps 5-7 to start a new Continuous Test with your latest evaluation dataset and prediction set.

   If you want to automate your Continuous Test so that it runs at a regular interval, set its schedule as shown in the section Scheduled Continuous Testing.