Creating a New Stress Test

This procedure presumes a model.py file is already registered to the Robust Intelligence platform and that all commands are issued in a Python environment.

  1. If a suitable project does not already exist, create a project using the create_project command in the SDK.

    project = rime_client.create_project(
    name="project", description="$DESC", model_task='MODEL_TASK_BINARY_CLASSIFICATION'
)

    You can also access an existing project using its project ID:

    project = rime_client.get_project(MY_PROJECT_ID)

    Replace $DESC with a description of the project or modeling TASK_NATURAL_LANGUAGE_INFERENCE

    • MODEL_TASK_FILL_MASK

  2. Register a reference dataset.

    This example registers a dataset from a file.

    reference_id = project.register_dataset(
    name=DATASET_NAME,
    data_params={
        "connection_info": {"data_file": {"path": FILE_PATH}},
        "data_params": {"label_col": LABEL_COL},
    },
    integration_id=INTEGRATION_ID,
)

    The SDK returns the ID of the reference dataset.

    Note that it is best to register a dataset once and then re-use its dataset ID (called reference_id in this example) afterwards. The SDK commands project.has_dataset and project.get_dataset can be used to retrieve the dataset ID associated with a particular name.

  3. Register an evaluation dataset.

    evaluation_id = project.register_dataset(
    name=DATASET_NAME,
    data_params={
        "connection_info": {"data_file": {"path": FILE_PATH}},
        "data_params": {"label_col": LABEL_COL},
    },
    integration_id=INTEGRATION_ID,
)

    The SDK returns the ID of the evaluation dataset.

  4. Register a model.

    This example registers a model from a model.py file. The Specify a Model section has details on how to format model.py files.

    model_id = project.register_model_from_path(
        name=MODEL_NAME,
        remote_path=S3_PATH_TO_MODEL_PY
)

    This example registers a Hugging Face model.

           model_id = project.register_model(
           name=MODEL_NAME,
           model_config={
               "hugging_face": {
                   "model_uri": URI,
                   "kwargs": {
                       "tokenizer_uri": TOKENIZER_URI,
                       "class_map": MAP,
                       "ignore_class_names": True,
                   },
               }
           },
       )

    The SDK returns the model ID.

  5. Create a test_run_config that uses the registered reference and evaluation datasets and the model, specifying a name and model task in addition to the IDs of the datasets and model registered earlier in this procedure.

    config = {
    "data_info": {"ref_dataset_id": reference_id, "eval_dataset_id": evaluation_id},
    "model_id": model_id,
    "run_name": "My Stress Test Run",
}

    You can also specify predictions in the test run configuration.

  6. Issue the following command to start the stress test, specifying the test_run_config you created in the previous step and the unique ID of the project that contains the stress test.

    job = client.start_stress_test(
    test_run_config=config, project_id=project.project_id
)

Once the stress test has completed, you can query test run results to programmatically access the results of the stress test.