Creating a new Continuous Test

Robust Intelligence allows you to create a Continuous Test to monitor the performance of a model over time. You can create a Continuous Test using the Python SDK. Continuous Tests run on a Firewall.

SDK

1. Create a project using the following SDK command.

   project = rime_client.create_project(
       name="foo", description="bar", model_task="MODEL_TASK_BINARY_CLASSIFICATION"
   )
   

2. Register a reference dataset. Depending on where you are looking to pull data from you may need to create a data integration: for all data sources except data collector, you will need to create a data integration.

   reference_id = project.register_dataset(
       name=DATASET_NAME,
       data_params={
           "connection_info": {"data_file": {"path": FILE_PATH}},
           "data_params": {"label_col": LABEL_COL},
       },
       integration_id=INTEGRATION_ID,
   )
   

   The SDK returns the ID of the reference dataset.

3. Register a model. You may optionally provide additional model information to allow Robust Intelligence to query the model during testing.

   model_id = project.register_model(
       name="baz"
   )
   

   The SDK returns the model ID. If you do not provide the information to query the model, you must register predictions of the model and dataset.

   project.register_predictions(
       dataset_id=reference_id
       model_id=model_id
       pred_config={
           "connection_info": {
               "data_file": {
                   "path": PREDICTION_FILE_PATH,
               },
           },
           "pred_params": {"pred_col": PREDICTION_COL}
       },
   )
   

4. Create a new Firewall object in the project. You may choose to create a Firewall with Scheduled CT parameters or leave those parameters blank. If Scheduled CT is set, the Firewall will run Continuous Tests automatically for you by pulling the data from the data sources you specify. Otherwise, you will need to manually run Continuous Tests over an evaluation dataset that you provide. This guide will provide an example of the latter case.

   project.create_firewall(
       model_id=model_id, ref_data_id=reference_id, bin_size=timedelta(hours=3)
   )
   

5. Register an evaluation dataset. This dataset must have the timestamp_col field in the data_params dictionary.

   evaluation_id = project.register_dataset(
       name="bar",
       data_config={
           "connection_info": {"data_file": {"path": EVALUATION_DATA_FILE_PATH}},
           "data_params": {
               "label_col": LABEL_COL,
               "timestamp_col": TIMESTAMP_COL,
           },
       },
   )
   

   The SDK returns the ID of the evaluation dataset.

6. Register a prediction set if the information to query the model is not provided.

   project.register_predictions(
       dataset_id=evaluation_id
       model_id=model_id
       pred_config={
           "connection_info": {
               "data_file": {
                   "path": PREDICTION_FILE_PATH,
               },
           },
           "pred_params": {"pred_col": PREDICTION_COL}
       },
   )
   

7. Issue the following command to start the Continuous Test, specifying the configuration created in the previous step and the unique ID of the firewall that contains the Continuous Test. Use the argument override_existing_bins to specify whether this Continuous Test should override the results of previous time bins for which other Continuous Tests have run.

   continuous_test = firewall.start_continuous_test(
      eval_data_id=evaluation_id,  override_existing_bins=False,
   )
   

8. Once new data has been gathered, repeat steps 5-7 to start a new Continuous Test with the new evaluation dataset and prediction set.