RI Bias and Fairness Income Classification Walkthrough

You are a data scientist at the Census Bureau. The data science team has been tasked with implementing a classification model to predict whether an individual’s income is above $50k USD. The primary goal of this project is to test the hypothesis that income inequality is not dependent on protected attributes. One could imagine such models being used downstream for various purposes, such as loan approval or funding allocation. A biased model could yield disadvantageous outcomes for protected groups. For instance, we may find that an individual with a specific race or race/gender combination causes the model to consistently predict a low income, causing a higher rate of loan rejection.

In this Notebook Walkthrough, we will walkthrough our core product of AI Stress Testing in a fairness setting. RIME AI Stress Testing allows you to test the developed model and datasets. With this bias and fairness setting, you will be able to verify your AI model for compliance-related issues.

Latest Colab version of this notebook available here

Install Dependencies, Import Libraries and Download Data

Run the cell below to install libraries to receive data, install our SDK, and load analysis libraries.

!pip install rime-sdk &> /dev/null

import pandas as pd
from pathlib import Path
from rime_sdk import Client

!pip install https://github.com/RobustIntelligence/ri-public-examples/archive/master.zip
from ri_public_examples.download_files import download_files
download_files('tabular-2.0/income', 'income')

Establish the RIME Client

To get started, provide the API credentials and the base domain/address of the RIME service. You can generate and copy an API token from the API Access Tokens Page under Workspace settings. For the domian/address of the RIME service, contact your admin.

API_TOKEN = '' # PASTE API_KEY
CLUSTER_URL = ''# PASTE DEDICATED DOMAIN OF RIME SERVICE (eg: rime.stable.rbst.io)
client = Client(CLUSTER_URL, API_TOKEN)

Create a New Project

You can create projects in RIME to organize your test runs. Each project represents a workspace for a given machine learning task. It can contain multiple candidate models, but should only contain one promoted production model.

description = (
    "Use the Robust Intelligence platform to help ensure that your"
    " models are compliant and to facilitate easy communication"
    " between your data science and risk teams. Demonstration"
    " uses the Adult Census Income dataset"
    " (https://www.kaggle.com/datasets/uciml/adult-census-income)"
    " and a model trained to predict whether someones annual"
    " income exceeds $50,000."
)
project = client.create_project(
    name="Tabular Bias and Fairness Income Demo",
    description=description,
    model_task="MODEL_TASK_BINARY_CLASSIFICATION"
)

project.project_id

Go back to the UI to see the new ``Tabular Bias and Fairness Income Demo`` project.

Training an Income Model and Uploading the Model + Datasets

Let’s first take a lot at what the dataset looks like. We can observe that the data consists of a mix of categorical and numeric features.

pd.read_csv('income/data/ref.csv').head()

For this demo, we are going to use a pretrained CatBoostClassifier Model.

The model predicts whether a particular individual has an income of >50k.

We now want to kick off RIME Stress Tests, in a compliance setting, that will help us determine if the model is biased against protected attributes. In order to do this, we will upload this pre-trained model, the reference dataset the model was trained on, and the evaluation dataset the model was evaluated on to an S3 bucket that can be accessed by RIME.

upload_path = "ri_public_examples_income"

model_s3_dir = client.upload_directory(
    Path('income/models'), upload_path=upload_path
)
model_s3_path = model_s3_dir + "/model.py"

ref_s3_path = client.upload_file(
    Path('income/data/ref.csv'), upload_path=upload_path
)
eval_s3_path = client.upload_file(
    Path('income/data/eval.csv'), upload_path=upload_path
)

ref_preds_s3_path = client.upload_file(
    Path("income/data/ref_preds.csv"), upload_path=upload_path
)
eval_preds_s3_path = client.upload_file(
    Path("income/data/eval_preds.csv"), upload_path=upload_path
)

Once the data and model are uploaded to S3, we can register them to RIME. In this bias and fairness setting, we require some additional information when registering datasets. Within the data_params parameter in the registering function, we include the protected features present in the data such that we can run our bias and fairness tests on those features. Once the datasets and models are registered, we can refer to these resources using their RIME-generated ID’s.

from datetime import datetime

dt = str(datetime.now())

# Note: models and datasets need to have unique names.
model_id = project.register_model_from_path(f"model_{dt}", model_s3_path)

ref_dataset_id = project.register_dataset_from_file(
    f"ref_dataset_{dt}", ref_s3_path, data_params={"label_col": "income",
                                                  "protected_features": ["sex", "race", "education", "age", "native.country"]}
)
eval_dataset_id = project.register_dataset_from_file(
    f"eval_dataset_{dt}", eval_s3_path, data_params={"label_col": "income",
                                                    "protected_features": ["sex", "race", "education", "age", "native.country"]}
)

project.register_predictions_from_file(
    ref_dataset_id, model_id, ref_preds_s3_path
)
project.register_predictions_from_file(
    eval_dataset_id, model_id, eval_preds_s3_path
)

Running a Stress Test with Bias and Fairness

AI Stress Tests allow you to test your data and model before deployment. They are a comprehensive suite of hundreds of tests that automatically identify implicit assumptions and weaknesses of pre-production models. Each stress test is run on a single model and its associated reference and evaluation datasets.

To run Stress Tests with the Bias & Fairness mode, there are two main changes to make. The first has been done already, namely specifying a set of protected_features in the data_param parameters of both datasets. The protected features are the specific features that you want Stress Tests to run over in order to test your model for signs of bias. Additionally, you will want to specify the Bias and Fairness Category in stress test config. This category does not run by default so specifying as such is necessary:

stress_test_config = {
        # rest of configuration ...
        "categories": ["TEST_CATEGORY_TYPE_BIAS_AND_FAIRNESS"]
}

Note how the “categories” field contains “Bias and Fairness”.

Below is a sample configuration of how to setup and run a RIME Stress Test.

stress_test_config = {
    "run_name": "Bias and Fairness Demo Run",
    "data_info": {
        "ref_dataset_id": ref_dataset_id,
        "eval_dataset_id": eval_dataset_id,
    },
    "model_id": model_id,
    "categories": ["TEST_CATEGORY_TYPE_BIAS_AND_FAIRNESS", "TEST_CATEGORY_TYPE_MODEL_PERFORMANCE", "TEST_CATEGORY_TYPE_SUBSET_PERFORMANCE"],
}

stress_job = client.start_stress_test(test_run_config=stress_test_config, project_id=project.project_id)
stress_job.get_status(verbose=True, wait_until_finish=True)

Wait for a couple minutes and your results will appear in the UI.

Compliance Stress Test Results

In the compliance setting, the stress tests are organized around a central “Compliance” tab. You can view the detailed results in the UI by running the below cell and redirecting to the generated link. This page shows granular results for a given AI Stress Test run.

test_run = stress_job.get_test_run()
test_run

Analyzing the Results

Below you can see a snapshot of the results.

Similar to running RIME Stress Tests in the default setting, we surface an overall distribution of test severities, model metrics, as well as key insights to the right. Here the insights are primarily focused on problematic features and subsets within the set of specified protected features.

We can take a look at a few tests, starting with the Demographic Parity test.

This test measures the ratio in minimum/maximum positive prediction rate between the subsets of a feature. When we look at the results of the Demographic Parity test over the “race” feature, we see that the positive prediction rate is far below the 80% threshold for various races. This means that for individuals in these groups, the model consistently predicts that they will have a low income. One could imagine that for downstream tasks such as loan approval that have a direct effect on the individual, such a model will lead to negative outcomes for protected subgroups.

Next we can take a look at the Discrimination By Proxy test.

In most if not all cases, the protected features are not directly used by the model. However, there may exist “proxy” features used by the model that have a high mutual information with the protected features; if the model depends heavily on these features, then the model will still exhibit biased predictions against protected subgroups. The “Discrimination By Proxy” test measures the mutual information between a protected feature and proxy features. Here we see that sex as a protected feature has a high mutual information with relationship, a proxy feature.

Next we can take a look at the Feature Independence Test.

This test is model-agnostic - it measures if there is a relationship between the protected (categorical) attribute and the label by running a Chi-Squared Test. Here we see that there does exist a statistical relationship between race, sex, and education and the label. This implies that even before the model is trained, we will want to carefully understand if the data and task itself is inherently “biased”, and decide if we want to adjust how we train the model to correct for these biases.

Next, we run the Intersectional Group Fairness Test.

This test runs over every pair of protected categorical attributes and analyzes the difference in positive prediction rate between every subset pair. Here we see that when the test is run over sex, education, the difference in positive prediction rate between Male, Masters and Female, HS-grad, is very high, indicating bias over protected attributes.

Programmatically Querying the Results

RIME not only provides you with an intuitive UI to visualize and explore these results, but also allows you to programmatically query these results. This allows customers to integrate with their MLOps pipeline, log results to experiment management tools like MLFlow, bring automated decision making to their ML practicies, or store these results for future references.

Run the below cell to programmatically query the results. The results are outputed as a pandas dataframe.

Access results at the a test run overview level

test_run_result = test_run.get_result_df()
test_run_result.to_csv("Income_Test_Run_Results.csv")
test_run_result

Access detailed test results at each individual test cases level.

test_case_result = test_run.get_test_cases_df()
test_case_result.to_csv("Income_Test_Case_Results.csv")
test_case_result.head()