0.3.0 - 2023-08-14

• PyPI
• GitHub

⚙️ New Metrics Computation Mode

• An error_analysis mode that measures subgroup and group metrics for correct and incorrect predictions, in addition to default groups. For example, when a default computation mode measures metrics for sex_priv and sex_dis, an error_analysis mode measures metrics for (sex_priv, sex_priv_correct, sex_priv_incorrect) and (sex_dis, sex_dis_correct, sex_dis_incorrect). Therefore, a user can analyze how a model is stable or certain about its incorrect predictions.

• An example yaml file for the computation mode:

  dataset_name: COMPAS
  bootstrap_fraction: 0.8
  n_estimators: 50
  computation_mode: error_analysis
  sensitive_attributes_dct: {'sex': 1, 'race': 'African-American', 'sex&race': None}
  

🗃 New Benchmark Fair-ML Datasets

• LawSchoolDataset. A data loader for the Law School dataset that contains sensitive attributes among feature columns.

  • Target: binary classify whether a candidate would pass the bar exam or predict a student’s first-year average grade (FYA).
  • Source: https://github.com/tailequy/fairness_dataset/blob/main/experiments/data/law_school_clean.csv.
  • Broader description: https://arxiv.org/pdf/2110.00530.pdf.

• RicciDataset. A data loader for the Ricci dataset that contains sensitive attributes among feature columns.

  • Target: binary classify whether an individual obtains a promotion based on the exam results.
  • Source: https://github.com/tailequy/fairness_dataset/blob/main/experiments/data/ricci_race.csv.
  • Broader description: https://arxiv.org/pdf/2110.00530.pdf.

💠 Analyzers and Metrics

• New subgroup metrics:

  • Statistical Bias is a feature of a statistical technique or of its results whereby the expected value of the results differs from the true underlying quantitative parameter being estimated (ref).
  • Aleatoric Uncertainty is a mean entropy of ensemble (ref).
  • Overall Uncertainty is an entropy of mean prediction of ensemble (ref).

• Changed a reference group in a sensitive_attributes_dct: now a disadvantaged group is used as a reference to compute intersectional metrics. For example, if we need to compute metrics for sex & race group (sex -- [male, female], race -- [white, black]), then sex&race_dis would include records for black females, and sex&race_priv would include all other records in a dataset.