Continuous Tests

A continuous test evaluates a machine learning model’s resilience with regards to a specific vulnerability.

Creating a new Continuous Test from the SDK

This procedure presumes a model is already loaded into RIME and that all commands are issued in a Python environment.

1. (When a suitable project does not already exist) Create a project using the following SDK command.

   project = rime_client.create_project(
       name="foo", description="bar", model_task="MODEL_TASK_BINARY_CLASSIFICATION"`
   )
   

2. Register a reference dataset.

   reference_id = project.register_dataset(
       name=DATASET_NAME,
       data_params={
           "connection_info": {"data_file": {"path": FILE_PATH}},
           "data_params": {"label_col": LABEL_COL},
       },
       integration_id=INTEGRATION_ID,
   )
   

   The SDK returns the ID of the reference dataset.

3. Register an evaluation dataset.

   evaluation_id = project.register_dataset(
       name="bar", data_config=dict
   )
   

   The SDK returns the ID of the evaluation dataset.

4. Register a model.

   model = project.register_model(
       name="baz"
   )
   

   The SDK returns the model ID.

5. Register a prediction set.

   prediction = rime_project.register_predictions(
   dataset_id=DATASET_ID,
   model_id=MODEL_ID,
   pred_config={
       "connection_info": {
           "delta_lake": {
               # Unix timestamp equivalent to 02/08/2023
               "start_time": 1675922943,
               # Unix timestamp equivalent to 03/08/2023
               "end_time": 1678342145,
               "table_name": TABLE_NAME,
               "time_col": TIME_COL,
           },
       },
       "pred_params": {"pred_col": PREDS},
   },
   )
   

   The SDK returns the prediction set ID.

6. Create a new Continous Test in the project.

   project.create_firewall(
       model_id="model", ref_data_id="foo",bin_size=timedelta(hours=3)
   )
   

7. Issue the following command to start the Continuous Test, specifying the configuration created in the previous step and the unique ID of the project that contains the Continuous Test.

   continuous_test = firewall.start_continuous_test(
      eval_data_id="bar", override_existing_bins=False,
   )