Skip to content

0.1.0 - 2023-02-06

🚀 Models Audit Pipeline

  • Developed an entire pipeline for auditing model stability and fairness with detailed reports and visualizations

  • Designed and implemented an extensible architecture split on components (User interfaces, MetricsComposer, etc.) that can be easily adapted to your needs

  • Enabled easy pipeline adaptability for different classification datasets

  • Added a feature to audit blind classifiers, which were trained on features without sensitive attributes, and use these sensitive attributes for analysis

👩‍💻 User Interfaces

  • Added three types of user interfaces:

    • Interface for multiple runs and multiple models
    • Interface for multiple models and one run
    • Interface for one model and one run

  • Added an ability to input arguments to interfaces via user-friendly config yaml files or direct arguments

🗃 Datasets and Preprocessing

  • Added built-in preprocessing techniques for raw classification datasets

  • Developed an ability to work with non-binary features

  • Enabled access to COMPAS and Folktables benchmark datasets via implemented data loaders

💠 Analyzers and Metrics

  • Added an ability to analyze intersections of sensitive attributes

  • Implemented a set of error and variance metrics:

    • 6 subgroup variance metrics
      • Mean
      • Std
      • IQR
      • Entropy
      • Jitter
      • Label Stability
    • 8 subgroup error metrics
      • TPR
      • TNR
      • PPV
      • FNR
      • FPR
      • Accuracy
      • F1
      • Selection-Rate
    • 5 group variance metrics
      • Label Stability Ratio
      • IQR Difference
      • Std Difference
      • Std Ratio
      • Jitter Difference
    • 5 group fairness metrics
      • Equalized Odds TPR
      • Equalized Odds FPR
      • Disparate Impact
      • Statistical Parity Difference
      • Accuracy Difference

📈 Reports and Visualizations

  • Added an ability to create predefined plots for result metrics

  • Developed a feature to make detailed summary reports with visualizations

😌 Convenience

  • Enabled smart saving of result metrics in files

  • In the multiple runs interface, a file with result metrics is saved each time when each run is completed. In such a way, if you get an error in one of the runs, the results of the previous runs will be saved.

  • Enabled library installation via pip

  • Created and hosted a website for detailed documentation with examples