Custom Tests

In order to use the custom test functionality, you first need to define the custom test. This should be exposed in a Python file. We recommend using the custom tests from the pre-defined categories over the generic custom tests whenever possible. The structure of this file depends on the type of custom test you are writing Please see below for detailed descriptions of each type of custom test.

Once this python file exists, you must make it accessible to the RIME server. For example, you may have some internal way of managing RIME storage, or you might upload it through the SDK:

upload_path = client.upload_file("local/path/to/custom_test_file.py")

Then you should include the path in the test configuration for a given run. This is done with the custom_tests key within either the tests_config_path or tests_config. Each custom test config should also specify a custom_test_type key, which should be one of GENERIC or ABNORMAL_INPUTS (defaults to GENERIC). In addition, you can optionally specify a custom_test_category key which will control which category the custom test shows up under. If you do not specify anything, a default will be used based on the custom_test_type you provide.

Here is an example config:

"tests_config": {
    "custom_tests": [
        {
            "test_path": upload_path,
            "custom_test_category":  "Example Test Category 1"
        },
        {
            "custom_test_type": "ABNORMAL_INPUTS",
            "test_path": "s3://rime-blob-storage-bucket/path/to/custom_abnormal_inputs_test.py",
            "params": {
                "arg_1": 5,
                "arg_2": "example_argument_string"
            },
            "custom_test_category": "Example Test Category 2"
        }
    ]
}