Release Notes

For notes on upgrades and deprecations to the Robust Intelligence REST API, refer to the API changelog.

Unless noted otherwise, you must upgrade all Robust Intelligence components to take advantage of the changes listed here.

2.10

Unified versioning across AI Firewall and validation

Starting in the 2.10 release, the AI Firewall and Robust Intelligence validation (RIME) products share a common version number.

Option to choose which features which will be analyzed

The Data Profiling configuration within the Stress Testing configuration now has a new optional parameter features_to_profile, which if specified, allows the user to select the specific features which will be analyzed by Robust Intelligence, rather than relying on our default Smart Feature Sampling logic.

New DeleteFirewallInstance function to remove an AI Firewall instance

The new DeleteFirewallInstance API endpoint marks the specified AI Firewall instance with REQUESTED_DELETE status. The AI Firewall Agent waits for the AI Firewall instance to become inactive and then deletes it.

2.9, FW 0.6

AI Firewall now requires a control plane

Starting in RI v2.9/AI Firewall v0.6, the Robust Intelligence AI Firewall must be deployed with a Robust Intelligence control plane. See deployment.

AI Firewall credentials

Existing AI Firewall credentials will no longer work after you upgrade to Firewall v0.6:

Firewall API keys are no longer accepted as a Firewall credential in the v0.6 and later Firewall.
Firewall auth tokens created in v0.5 and earlier are not valid in the v0.6 and later Firewall.

To authenticate to the v0.6 and later Firewall, generate a new Firewall auth token as explained in “Get your Firewall Auth Token” in the AI Firewall User Guide.

2.8, FW 0.5

End of Life for Terraform modules

Official support for the deployment-related Terraform modules has concluded. Your Solutions Architect will guide you through migrating existing resources from these modules.

Internal Agent replaced during upgrade

The upgrade to Robust Intelligence v2.8 automatically replaces its internal agent. For SaaS and hybrid deployments, replacement is automatic, and no action is required. For self-hosted deployments, you must generate an Agent ID for the new rime-agent. Your Robust Intelligence Solutions team will guide you through the change.

Dedicated Validation and Firewall APIs

The REST APIs for the Robust Intelligence AI Validation and AI Firewall products are now separate. Open the REST API Reference from the UI and click API Docs for the AI Validation API endpoints, or Firewall API Docs for the AI Firewall API endpoints.

2.7

File scanning supports all model task types

File Scanning now lets you scan the following model types: generative, binary classification, and multiclass classification.

SDK class `Firewall` has been replaced with `ContinuousTest`

The Firewall class, deprecated since v2.3, has been removed from the Python SDK. Use the ContinuousTest class, instead. API and SDK classes that previuosly used the Firewall name were renamed in v2.3 with the name ct to better indicate their purpose for continuous testing:

ContinuousTest
Client.create_ct()
Client.has_ct()
Client.get_ct()
Client.delete_ct()
Client.get_ct_for_project()
Project.update_ct_categories()

2.6

Scheduled stress tests

This release introduces the ability to run model stress tests on a daily, weekly or other schedule that you set. Scheduled stress tests are the recommended way to test models on a recurring basis, rather than scheduled continuous tests, which remain supported as of v2.6. See Scheduled Stress Testing.

New generative test: Harmful SQL Generation

This test applies a harmful SQL generation attack that uses natural language prompts that may cause the model to generate harmful SQL queries. It will fail if the model generates SQL queries that expose table schemas or create, alter, or drop tables. See the Model tests reference.

2.5

Better setting and defaulting of memory and CPU preferences for scheduled continuous tests

In earlier versions of Robust Intelligence, scheduled continuous tests did not give you a way to specify the desired allocation of memory and CPU for the job. For a stress test run, it’s possible to specify the desired memory and CPU when you start the test run, but it’s not possible to set these values at runtime for a scheduled continuous test. For this reason, better control over and defaulting of memory and CPU settings was needed.

This has been fixed by setting scheduling continuous test runs to use the the project’s memory and CPU settings (in earlier versions, we used system defaults).

You can now specify the desired default CPU and memory preferences for a scheduled continuous test in its project. You provide these in the runtime information setting in the project. See Project.update_run_time_info in the SDK.

2.4

Incorrect memory and CPU settings for continuous test runs

In version 2.4.4 and earlier of Robust Intelligence, continuous test runs fail to default to the project’s memory and CPU settings. Instead, they use system defaults that might provide insufficient resources for the test run. This issue has been corrected in version 2.4.5 and later. This issue affects only scheduled continuous test runs and not stress test runs.

AWS Terraform Provider Upgraded to >= 4.47.0, < 5.0.0 for rime_eks_cluster Module

This change only affects customers upgrading self-hosted instances of Robust Intelligence. When using Terraform to upgrade to a 2.4 or higher version of the rime_eks_cluster Terraform module, you must now use a version of the AWS Terraform provider that is >= 4.47.0 and < 5.0.0.

AWS Load Balancer Controller Upgraded to 2.6.1

The AWS Loadbalancer Controller has been upgraded to version 2.6.1. This includes updates to its Custom Resource Definitions (CRDs).

You will need to remirror the image and update your deployment if using a private docker registry. This image can be found at docker.io/robustintelligencehq/aws-alb-ingress-controller:v2.6.1

Upgrade actions:

If you’re performing a fresh installation you do not need to do anything.
If you’re using RI Platform Helm charts you also do not need to take any different actions.
If you’re not using Helm though and upgrading to RI Platform version 2.4 from an earlier version follow these directions:

Update CRDs to the ones that are bundled with AWS Loadbalancer version 2.6.1:
```
$ kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=v0.0.150"
```

2.3

Firewall API deprecated and replaced with new `ContinuousTest` class and methods

In v2.3 of the SDK, the new ContinuousTest class replaces the Firewall class. The v1/firewall API and the associated Python SDK methods were deprecated as of v2.2 and will not be supported starting in v2.4. Please convert your code to use the new ContinuousTest class and methods.

The deprecated Firewall API and SDK classes and methods have been renamed to incorporate the name ct to better indicate that they’re used for running and managing continuous tests. The new and renamed classes and methods include:

ContinuousTest
Client.create_ct()
Client.has_ct()
Client.get_ct()
Client.delete_ct()
Client.get_ct_for_project()
Project.update_ct_categories()

File scanning

This release adds the ability to scan model artifacts for vulnerabilities. Once a model has been registered with Robust Intelligence, you can run a file scan on the model artifacts. The scan identifies any potentially dangerous imports in the model which pose risks during de-serialization. This is a critical step in securing your model supply chain. See File Scanning for details

Ingress-Nginx Changes

This release bumps the Ingress-Nginx version to v1.8.1. You will need to remirror the image and update your deployment if using a private docker registry. This image can be found at docker.io/robustintelligencehq/ingress-nginx-controller:v1.8.1.

AWS Load Balancer Controller Upgraded to 2.5.4

The AWS Loadbalancer Controller has been upgraded to version 2.5.4. This includes updates to its Custom Resource Definitions (CRDs).

You will need to remirror the image and update your deployment if using a private docker registry. This image can be found at docker.io/robustintelligencehq/aws-alb-ingress-controller:v2.5.4

Upgrade actions:

If you’re performing a fresh installation you do not need to do anything.
If you’re using RI Platform Helm charts you also do not need to take any different actions.
If you’re not using Helm though and upgrading to RI Platform version 2.3 from an earlier version follow these directions:

Update CRDs to the ones that are bundled with AWS Loadbalancer version 2.5.4:
```
$ kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=v0.0.142"
```

2.2

Cleaner separation of continuous test results from monitored event results

Version 2.2 simplifies the way Robust Intelligence shows Continuous Test results. Continuous Tests are tests that run on a regular schedule to track model risk and compliance. These now appear in the Continuous Testing sections of the UI and API. In contrast, we now handle scheduled model performance tests and scheduled custom tests as Monitors. These results now appear in the Monitors section of the UI.

Model performance results are no longer used in calculating the risk severity measures that appear in Continuous Test results.

Firewall API deprecated

The v1/firewall API and the associated Python SDK methods have been deprecated. The endpoints and SDK methods will not be supported from Robust Intelligence v2.4 onwards.

2.1

Category field deprecated

The string category field is deprecated in favor of the test_category enum field. The string category field will be removed in v2.3. category is currently a sub-field of TestBatchResult, TestCase, and CategorySummaryMetric.

Python 3.7 no longer supported

Robust Intelligence now supports Python 3.8 through Python 3.11. As of Robust Intelligence 2.1, projects using Python 3.7 are no longer supported. Projects based on Python 3.7 should be migrated to Python 3.8 or later.

delta_lake connection_info renamed

The delta_lake connection_info has been renamed to databricks. This loads a Databricks Delta Lake Table.

Before:

{
    "connection_info": {
        "delta_lake": {
            "table_name": "Table",
        }
    },
    "data_params": ...,
}

After:

{
    "connection_info": {
        "databricks": {
            "table_name": "Table",
        }
    },
    "data_params": ...,
}

2.0 GA

global_exclude_columns parameter has moved

The location in the Test Suite Configuration to specify the global_exclude_columns parameter has changed.

Before:

stress_test_config = {
    # other config params
    "test_suite_config": {
        "individual_tests_config": {
            "global_exclude_columns": ["foo", "bar"]
        }
    }
}

After:

stress_test_config = {
    # other config params
    "test_suite_config": {
        "global_exclude_columns": ["foo", "bar"]
    }
}

Any existing projects with this parameter specified in the project’s Test Suite Configuration may need to be updated with this new version of the configuration.

The specification for the test categories that Robust Intelligence runs has changed, both at a project level and for individual Stress Tests.

Before:

# Update the project's Stress Testing, Continuous Testing categories
project.update_test_suite_config({
    "categories": [
      {
          "name": "Model Performance",
          "run_st": True,
          "run_ct": True
      },
      {
          "name": "Subset Performance",
          "run_st": True,
          "run_ct": False
      },
      {
          "name": "Drift",
          "run_st": False,
          "run_ct": True
      }
    ],
    # ...
})

# When running Stress Testing, user can override the project's Stress Testing categories:
stress_test_config = {
    # ... other params in stress test config
    "test_suite_config": {
        "categories": [
            {
                "name": "Bias and Fairness",
                "run_st": True,
                "run_ct": False
            }
        ],
      # ... other params in test suite config
    }
}

After:

# Update the project's Stress Testing, Continuous Testing categories
project.update_stress_testing_categories(
    ["TEST_CATEGORY_TYPE_MODEL_PERFORMANCE", "TEST_CATEGORY_TYPE_SUBSET_PERFORMANCE"]
)
project.update_continuous_testing_categories(
    ["TEST_CATEGORY_TYPE_MODEL_PERFORMANCE", "TEST_CATEGORY_TYPE_DRIFT"]
)

# When running Stress Testing, user can override the project's Stress Testing categories:
stress_test_config = {
    # ... other params in stress test config
    "categories": ["TEST_CATEGORY_TYPE_BIAS_AND_FAIRNESS"]
    "test_suite_config": {
      # ... other params in test suite config (if any)
    }
}

Release Notes

2.10

Unified versioning across AI Firewall and validation

Option to choose which features which will be analyzed

New DeleteFirewallInstance function to remove an AI Firewall instance

2.9, FW 0.6

AI Firewall now requires a control plane

AI Firewall credentials

2.8, FW 0.5

End of Life for Terraform modules

Internal Agent replaced during upgrade

Dedicated Validation and Firewall APIs

2.7

File scanning supports all model task types

SDK class Firewall has been replaced with ContinuousTest

2.6

Scheduled stress tests

New generative test: Harmful SQL Generation

2.5

Better setting and defaulting of memory and CPU preferences for scheduled continuous tests

2.4

Incorrect memory and CPU settings for continuous test runs

AWS Terraform Provider Upgraded to >= 4.47.0, < 5.0.0 for rime_eks_cluster Module

AWS Load Balancer Controller Upgraded to 2.6.1

2.3

Firewall API deprecated and replaced with new ContinuousTest class and methods

File scanning

Ingress-Nginx Changes

AWS Load Balancer Controller Upgraded to 2.5.4

2.2

Cleaner separation of continuous test results from monitored event results

Firewall API deprecated

2.1

Category field deprecated

Python 3.7 no longer supported

delta_lake connection_info renamed

2.0 GA

global_exclude_columns parameter has moved

Related Topics

SDK class `Firewall` has been replaced with `ContinuousTest`

Firewall API deprecated and replaced with new `ContinuousTest` class and methods