3. Fairness#
Fairness class offers functionalities to explain how fair or unfair are the
classifications made by a (Deep) Machine Learning model on a set of features that we consider sensitive
(gender, age, ethnic group, religion, etc.).
These explanations are based on three fairness criterion (Independence, Separation, and Sufficiency) that are used to determine if a classification model is fair. It will be regarded fair if none of the sensitive features influence its predictions.
Fairness Scores and Fairness Category are calculated based on the definitions of each fairness criterion in order to explain and indicate which target class and which value of the sensitive feature is committing a possible injustice in classification.
Requirements#
To perform these calculations we need a dataset with the following characteristics:
‘N’ columns with features
Column with target class (‘y_true’)
Column with predictions class (‘y_predict’)
List of sensitive features to be examined
Below is an example of a dataset, taking [‘Gender’] as list of sensitive features to be examined:
Gender |
y_true |
y_predict |
|---|---|---|
MAN |
YES |
YES |
MAN |
YES |
YES |
WOMAN |
NO |
NO |
MAN |
NO |
YES |
WOMAN |
YES |
NO |
MAN |
YES |
NO |
MAN |
YES |
YES |
WOMAN |
YES |
YES |
MAN |
NO |
NO |
WOMAN |
NO |
NO |
Independence criterion#
We say that the random variables (Y, A) satisfy independence if the sensitive feature ‘A’ are statistically independent of the prediction ‘Y’.
That is, the probability of being classified by the model in each of the classes must be the same for two elements with different sensitive feature values (Demographic Parity).
Attention
The acceptance rate must be equal for each group (A=a, A=b)
We define the Independence Score as:
Hint
Taking the example dataset above, we calculate the independence score as:
Separation criterion#
We say the random variables (Y, A, T) satisfy separation if the sensitive characteristics ‘A’ are statistically independent of the prediction ‘Y’ given the target value ‘T’.
That is, the probability of being classified by the model in each of the classes is the same for two elements with different sensitive feature values, given that both belong to the same group (Equal opportunities).
Attention
The probability of predicting a true positive or a false positive for each group (A=a, A=b) must be the same.
We define the separation score as:
Hint
Taking the example dataset above, we calculate the separation score as:
Sufficiency criterion#
We say the random variables (Y, A, T) satisfy sufficiency if the sensitive characteristics ‘A’ are statistically independent of the target value ‘T’ given the prediction ‘Y’.
That is, the probability of being classified by the model in each of the classes is the same for two elements with different sensitive feature values, since the prediction includes them in the same group (Predictive Parity).
Attention
The probability that two elements with different values of the sensitive feature are well classified must be the same.
We define the sufficiency score as:
Hint
Taking the example dataset above, we calculate the sufficiency score as:
Fairness Category#
In order to make the equity results easier to grasp, some categories have been established that identify the features and values on which You might be committing discrimination with a color code from green to red (similar to that used in other industries like Nutriscore or energy efficiency).
Below are the categories and theirs score ranges:
Category |
Range Score |
|---|---|
A+ |
0.0 <= score <= 0.02 |
A |
0.02 < score <= 0.05 |
B |
0.05 < score <= 0.08 |
C |
0.08 < score <= 0.15 |
D |
0.15 < score <= 0.25 |
E |
0.25 < score <= 1.0 |
Each of those categories has the following color designation:
Example#
To demonstrate how to apply the Fairness class’s features, consider the following
dataset, which has two features (gender and title) and two columns, one with the true class (y_true)
and another with the model’s prediction (y_predict).
>>> import pandas as pd
>>> df = pd.DataFrame({'gender': ['MAN', 'MAN', 'WOMAN', 'MAN', 'WOMAN', 'MAN', 'MAN', 'WOMAN', 'MAN', 'WOMAN'],
... 'title': ['Mr', 'Mr', 'Mrs', 'Mr', 'Mrs', 'Mr', 'Mr', 'Mrs', 'Mr', 'Mrs'],
... 'y_true': ['YES', 'YES', 'NO', 'NO', 'YES', 'YES', 'YES', 'YES', 'NO', 'NO'],
... 'y_predict': ['YES', 'YES', 'NO', 'YES', 'NO', 'NO', 'YES', 'YES', 'NO', 'NO']},
... columns=['gender', 'title', 'y_true', 'y_predict'])
>>> df
gender title y_true y_predict
0 MAN Mr YES YES
1 MAN Mr YES YES
2 WOMAN Mrs NO NO
3 MAN Mr NO YES
4 WOMAN Mrs YES NO
5 MAN Mr YES NO
6 MAN Mr YES YES
7 WOMAN Mrs YES YES
8 MAN Mr NO NO
9 WOMAN Mrs NO NO
fit() method in the Fairness
class conducts all Fairness computations (score and category). df,sensitive_cols, target_col, and predict_col
parameters are required by the fit() method:
>>> from xaiographs import Fairness
>>> f = Fairness()
>>> f.fit(df=df,
... sensitive_cols=['gender'],
... target_col='y_true',
... predict_col='y_predict')
We retrieve the global Fairness criteria results by accessing the various attributes (properties). We get a DataFrame
with the scores and categories of the fairness criterion for each of the sensitive features when we use
the fairness_global_info attribute:
>>> f.fairness_global_info
sensitive_feature independence_global_score independence_category separation_global_score separation_category sufficiency_global_score sufficiency_category
0 gender 0.416667 E 0.375 E 0.216667 D
The attributes independence_info,
separation_info, and
sufficiency_info
are accessible for more information on the calculation of Fairness criteria:
>>> f.independence_info
sensitive_feature sensitive_value target_label independence_score independence_category
0 gender MAN | WOMAN YES 0.416667 E
1 gender MAN | WOMAN NO 0.416667 E
>>> f.separation_info
sensitive_feature sensitive_value target_label separation_score separation_category
0 gender MAN | WOMAN YES 0.250000 D
1 gender MAN | WOMAN NO 0.500000 E
>>> f.sufficiency_info
sensitive_feature sensitive_value target_label sufficiency_score sufficiency_category
0 gender MAN | WOMAN YES 0.250000 D
1 gender MAN | WOMAN NO 0.166667 D
fairness_info attribute is available to obtain all
the information of all the Fairness criteria, which returns all the scores and categories as well as the weight of
each feature value:
>>> f.fairness_info
sensitive_feature sensitive_value is_binary_sensitive_feature target_label independence_score independence_category independence_score_weight separation_score separation_category separation_score_weight sufficiency_score sufficiency_category sufficiency_score_weight
0 gender MAN | WOMAN True YES 0.416667 E 0.5 0.25 D 0.5 0.25 D 0.6
1 gender MAN | WOMAN True NO 0.416667 E 0.5 0.50 E 0.5 0.166667 D 0.4
We have included a table to help you comprehend these recent results:
Column Name |
elem 0 |
elem 1 |
|---|---|---|
sensitive_feature |
gender |
gender |
sensitive_value |
MAN - WOMAN |
MAN - WOMAN |
is_binary_sensitive_feature |
True |
True |
target_label |
YES |
NO |
independence_score |
0.416667 |
0.416667 |
independence_category |
E |
E |
independence_score_weight |
0.5 |
0.5 |
separation_score |
0.25 |
0.50 |
separation_category |
D |
E |
separation_score_weight |
0.5 |
0.5 |
sufficiency_score |
0.25 |
0.166667 |
sufficiency_category |
D |
D |
sufficiency_score_weight |
0.6 |
0.4 |
The Fairness class also offers complementary information such as:
Confusion Matrix
>>> f.confusion_matrix
y_predict NO YES
y_true
NO 3 1
YES 2 4
Matrix of correlations (pearson) between features:
>>> f.correlation_matrix
gender title
gender NaN 1.0
title NaN NaN
List of highly correlated features (\(pearson \geq 0.9\)) for proxy feature detection.
>>> f.highest_correlation_features
feature_1 feature_2 correlation_value is_correlation_sensible
0 title gender 1.0 True