Tabular Data/Model Setup

Python package exposing core functionality of RIME.

The RIME Library offers data scientists/developers a convenient way to use the full suite of RIME data/model tests directly within their python environment.

class rime.tabular.ModelTask(value)

Enum representing the different model tasks.

BINARY_CLASSIFICATION = 'Binary Classification'

Binary Classification Task

MULTI_CLASSIFICATION = 'Multi-class Classification'

Multi-class Classification Task

RANKING = 'Ranking'

Ranking Task

REGRESSION = 'Regression'

Regression Task

class rime.tabular.DataContainer(data_profile: DataProfile, df: Optional[DataFrame] = None, labels: Optional[Series] = None, preds: Optional[PredictionContainer] = None, query_ids: Optional[Series] = None, pred_profile: Optional[PredictionProfile] = None, label_profile: Optional[LabelProfile] = None, subset_profile: Optional[SubsetPerformanceProfile] = None, timestamps: Optional[Series] = None, is_binary_ranking: bool = False)

Class to store all data artifacts for a given dataset.

classmethod from_df(df: DataFrame, model_task: ModelTask, labels: Optional[Series] = None, preds: Optional[ndarray] = None, query_ids: Optional[Series] = None, ref_data: Optional[DataContainer] = None) → DataContainer

Load data container from df.

Parameters:

  • df (pd.DataFrame) – Input dataframe.

  • model_task (ModelTask) – Model task.

  • labels (Optional[pd.Series]) – Label vector.

  • preds (Optional[np.ndarray]) – Prediction array.

  • query_ids (Optional[pd.Series]) – Query id’s, for ranking only.

  • ref_data (Optional[rime.tabular.DataContainer]) – Reference data container (used for creating evaluation DataContainer).

classmethod from_data(parsed_data: ParsedData, model_task: ModelTask, profiling_config: ProfilingConfig, ref_data: Optional[DataContainer] = None, top_k_feats: Optional[List[str]] = None, metrics: Optional[List] = None) → DataContainer

Load data container from df.

Parameters:

  • parsed_data (ParsedData) – Tuple of required information.

  • model_task (ModelTask) – Model task.

  • profiling_config (ProfilingConfig) – Profiling config object.

  • ref_data (Optional[rime.tabular.DataContainer]) – Reference data container (used for creating evaluation DataContainer).

  • top_k_feats (Optional[List[str]]) – List of top_k_feats if smart sampling

  • metrics – Optional list of metrics to use.

property query_ids: Series

Safely access the preds container.

property timestamps: Series

Safely access the timestamps.

property is_binary_ranking: bool

Safely check the is_binary_ranking property.

property is_query_ids_none: bool

Notifies user if this has query_ids.

class rime.tabular.TabularRunContainer(ref_data: DataContainer, eval_data: DataContainer, model: Optional[TabularBlackBoxModel] = None, model_profile: Optional[ModelProfile] = None)

Class to store the testing state when a model is provided.

get_failing_row_details(index: int) → List[Detail]

Get failing row details.

classmethod from_model(ref_data: DataContainer, test_data: DataContainer, model: TabularBlackBoxModel, model_profiling_info: Optional[ModelProfilingInfo] = None) → TabularRunContainer

Make predictions with model if they do not already exist.

classmethod from_predict_dict_function(ref_data: DataContainer, test_data: DataContainer, predict_dict_func: Callable, model_task: ModelTask) → TabularRunContainer

Load model from predict_dict function.

Parameters:
classmethod from_predict_df_function(ref_data: DataContainer, test_data: DataContainer, predict_df_func: Callable, model_task: ModelTask, model_profiling_info: Optional[ModelProfilingInfo] = None) → TabularRunContainer

Load model from predict_df function.

property protected_feature_pairs: List[Column]

Return protected feature pair columns.

property protected_features: List[str]

Return protected feature column names.

property features_not_in_model: List[str]

Return feature names not included in model.

property common_columns: List[Column]

Return Common Columns between Ref and Test Data.

Removes protected feature pair columns, these are only used in a subset of tests.

property profiled_all_cols: bool

Return whether smart sampling was run.

property optional_impact_metric_name: Optional[Union[BaseMetricName, CustomMetricName]]

Return the impact metric name if it exists, otherwise None.

NLP Data/Model Setup

Python package exposing core NLP functionality for RIME.

The RIME NLP library provides tooling for the RIME testing suite over a number of natural language tasks.

class rime.nlp.Task(value)

Enumeration of supported NLP tasks.

NER = 'Named Entity Recognition'

Named Entity Recognition Task

CLASSIFICATION = 'Text Classification'

Text Classification Task

class rime.nlp.DataContainer(data: List[dict], preds: List[dict], preds_index: List[int], data_profile: DataProfile, perf_summary: Dict[MetricName, float], subsets_profiles: Dict[str, SubsetsInfo], contains_labels: bool)

Class to store all data artifacts for a given dataset.

Parameters:

  • data – Input data. Each item in the list is a single datapoint containing the model input, optional label(s), and other task-related metadata.

  • preds – Model predictions.

  • preds_index – Indices mapping the (possibly sampled) model predictions to the input data. preds[i] = model.predict_dict(data[preds_index[i])

  • data_profile – NLP task-specific profile of the data generated by the RIME profiler

  • perf_summary – performance summary generated by the NLP task’s model evaluator

  • subsets_profiles – Performance metrics for each feature subset

  • contains_labels – bool whether labels are present for all data points

property is_preds_none: bool

Notifies user if this has no preds.

property num_instances: int

Return the length of the dataset.

get_feature_profile(feature_name: str) → dict

Return the histogram profile for the given feature.

contains_feature_profile(feature_name: str) → bool

Return whether the profiled feature.

get_custom_feature(feature_name: str) → dict

Return custom feature.

contains_custom_feature(feature_name: str) → bool

Return whether custom feature is present.

preds_with_labels_at_indices(indices: Collection[int], return_difference: bool = True) → Tuple[List[dict], List[dict], Optional[List[dict]], Optional[List[dict]]]

Return preds and labels at the index intersection.

get_tabular_data_container(task: UnstructuredTask, profiling_config: ProfilingConfig, ref_data: Optional[DataContainer] = None) → DataContainer

Get tabular run container.

get(raw_data: dict) → dict

Get datapoint.

class rime.nlp.ModelEvaluator

Abstract base class for the model evaluator.

abstract classmethod avg_pred(data: List[dict], preds: List[dict]) → Optional[Pred_T]

Return the average prediction vector.

abstract static get_avg_pred_diff(base_avg_preds: Pred_T, avg_preds: Pred_T) → float

Return the floating point difference in observed values.

classmethod get_default_metrics() → List[MetricName]

Return the default metrics.

classmethod compute_labeled_impact(data: List[dict], preds: List[dict]) → Optional[float]

Return the impact metric performance.

classmethod performance(labels: List[dict], preds: List[dict], metrics: List[MetricName]) → dict

Return specified metrics for a set of entity model outputs and labels.

classmethod get_default_metric_name_for_feature(feature: Feature) → MetricName

Return the impact metric name for a given test.

Label and prediction subset tests only support accuracy-type metrics, since f1, recall, precision, etc. are misleading.

class rime.nlp.RunContainer(ref_data: UDC, eval_data: UDC, task: UMT, model: Optional[SingleTaskModel], model_evaluator: Type[ModelEvaluator], data_profiling_info: Optional[DataProfilingInfo] = None, model_profiling_info: Optional[BaseModelProfilingInfo] = None, **kwargs: Any)

Base NLP run container class.

get_failing_row_details(index: int) → List[Detail]

Get failing row details.

static get_tabular_model_profile(task: Task, ref_data: DataContainer, class_names: Optional[List[str]]) → Optional[ModelProfile]

Get tabular model profile.

class rime.nlp.SingleTaskModel(task: BaseTask, predict_dict: Callable[[dict], dict], **kwargs: Any)

Base class for a model that performs a single NLP task.