post on the Confusion Matrix, we applied the logistic regression algorithm to the Breast Cancer Wisconsin dataset to classify whether the tumor is malignant or benign.
We evaluated the classification model using various metrics like accuracy, precision, etc.
Now, in binary classification models, we have another way to evaluate the model, and that is ROC AUC.
In this blog, we will discuss why we have another metric and when it should be used.
To understand ROC AUC in detail, we will consider the IBM HR Analytics dataset.
In this dataset, we have information about 1,470 employees such as their age, job role, gender, monthly income, job satisfaction, etc.
In total, there are 34 features describing each employee.
We also have a target column, ‘Attrition’, which is ‘Yes’ if the employee left the company and ‘No’ if the employee stayed.
Let’s have a look at the class distribution of the target column.
From the above class distribution, we can observe that the dataset is imbalanced.
Now, we need to build a model based on this data to classify employees according to whether they will stay in the company or not.
As this is a binary classification (Yes/No) task, let’s use the logistic regression algorithm on this data.
Code:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix, classification_report
# Load the dataset
df = pd.read_csv("C:/HR-Employee-Attrition.csv")
# Drop non-informative columns
df.drop(['EmployeeNumber', 'Over18', 'EmployeeCount', 'StandardHours'], axis=1, inplace=True)
# Encode the target column
df['Attrition'] = df['Attrition'].map({'Yes': 1, 'No': 0})
# One-hot encode categorical features
df = pd.get_dummies(df, drop_first=True)
# Split features and target
X = df.drop('Attrition', axis=1)
y = df['Attrition']
# Train-test split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42
)
# Feature scaling
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Train logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train_scaled, y_train)
# Predict on test data
y_pred = model.predict(X_test_scaled)
# Predict probabilities for the positive class
y_prob = model.predict_proba(X_test_scaled)[:, 1]
# Confusion matrix and classification report
conf_matrix = confusion_matrix(y_test, y_pred)
report = classification_report(y_test, y_pred)
# Display results
print("Confusion Matrix:\n", conf_matrix)
print("\nClassification Report:\n", report)
Confusion Matrix and Classification Report
From the above classification report, we observe that the accuracy is 86%. However, the recall for ‘1’ (attrition = Yes, meaning the employee left the job) is 0.34, indicating that the model correctly identified only 34% of employees who left the job.
The recall for ‘0’ (attrition = No, meaning the employee stayed in the job) is 0.96, indicating that the model correctly identified 96% of employees who stayed.
This happens due to an imbalanced dataset. Accuracy can be misleading here.
Does this mean we need to change our algorithm? No.
We need to change the way we evaluate our model, and the best way to evaluate classification models with an imbalanced dataset is ROC AUC.
Now we know that there is another method to evaluate classification models, i.e., ROC AUC. But before exploring ROC AUC, let’s have a clear idea of what has happened so far.
We applied Logistic Regression on the IBM HR dataset, and the model gave us a probability score for each employee, representing how likely they are to leave the job.
When we generate a confusion matrix and classification report, they are based on a threshold, which by default is 0.5.
If the predicted probability is greater than 0.5, the employee is considered to have left the job; if the probability is smaller than 0.5, the employee is considered to have stayed.
From this, we obtained an accuracy of 86%, but recall was only 34%. We observed that accuracy is misleading, so we decided to evaluate the model using ROC AUC.
ROC AUC
First, we will discuss the Receiver Operating Characteristic (ROC) curve.
We get the ROC curve by plotting the True Positive Rate versus False Positive Rate.
We already know that the classification report is based on a single threshold, But the ROC curve is generated by calculating True Positive Rate (TPR) and False Positive Rate (FPR) at all possible thresholds and then plotting them.
Let’s take a sample dataset and see how we generate the ROC curve from it.
Now, for the above data, we calculate TPR and FPR at the possible thresholds and then plot them.
What are Possible Thresholds?
To generate the ROC curve, we don’t need to calculate the TPR and FPR at every value between 0 and 1.
Instead, we use the predicted probabilities from the dataset and one value above the maximum predicted probability (so all predictions are negative, starting the curve at (0,0)) and one below the minimum predicted probability as thresholds (so all predictions are positive, ending the curve at (1,1)).
Why not every number between 0 and 1 as Threshold?
Consider our sample data. We have a predicted probability of 0.6592, and we will use that as a threshold to calculate TPR and FPR.
Now, between 0.6592 and 0.8718 the TPR and FPR remain the same, and they only change once the threshold crosses a predicted probability.
That is why we use unique predicted probabilities as thresholds to generate the ROC curve.
Now, based on our sample data, let’s generate the ROC curve and see what we can observe.
To generate the ROC curve, we need to calculate the TPR and FPR.
\[
\text{True Positive Rate (TPR)} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Negatives (FN)}}
\]
The True Positive Rate (TPR) is also called Recall.
\[
\text{False Positive Rate (FPR)} = \frac{\text{False Positives (FP)}}{\text{False Positives (FP)} + \text{True Negatives (TN)}}
\]
The thresholds we are going to use for this sample data to calculate TPR and FPR are {1, 0.9799, 0.9709, 0.8737, 0.8718, 0.6592, 0.6337, 0}.
Let’s calculate TPR and FPR at each threshold.
\[
\begin{aligned}
\mathbf{At\ threshold\ 0.9799:} & \\[4pt]
\mathrm{True\ Positives\ (TP)} &= 1,\quad \mathrm{False\ Negatives\ (FN)} = 2,\\
\mathrm{False\ Positives\ (FP)} &= 0,\quad \mathrm{True\ Negatives\ (TN)} = 3\\[6pt]
\mathrm{TPR} &= \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} = \frac{1}{1+2} = \frac{1}{3} \approx 0.33\\[6pt]
\mathrm{FPR} &= \frac{\mathrm{FP}}{\mathrm{FP}+\mathrm{TN}} = \frac{0}{0+3} = 0\\[6pt]
\Rightarrow (\mathrm{FPR},\,\mathrm{TPR}) &= (0,\,0.33)
\end{aligned}
\]
This way, we calculate TPR and FPR at each threshold.
Now, let’s plot TPR versus FPR to obtain the ROC curve.
This is how the ROC curve is generated. Since we considered only a 6-point sample, it is difficult to observe and interpret the curve clearly. The main goal here is to understand how a ROC curve is generated.
Now we need to interpret the ROC curve, and for that we will generate the ROC curve using Python on our dataset.
Code:
# Compute ROC curve and AUC
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
roc_auc = auc(fpr, tpr)
# Print AUC
print(f"AUC: {roc_auc:.2f}")
# Plot ROC curve
plt.figure(figsize=(6,6))
plt.plot(fpr, tpr, label=f"ROC curve (AUC = {roc_auc:.2f})", linewidth=2)
plt.plot([0,1], [0,1], 'k--', label="Random guess (AUC = 0.5)")
plt.xlim([0,1])
plt.ylim([0,1.05])
plt.xlabel("False Positive Rate (FPR)")
plt.ylabel("True Positive Rate (TPR)")
plt.title("ROC Curve - Logistic Regression (HR Dataset)")
plt.legend(loc="lower right")
plt.grid(True)
plt.show()
Plot:
Let’s see what we can interpret from the ROC curve alone, irrespective of AUC, since we will discuss AUC later.
In the above plot, the y-axis represents the True Positive Rate, which means how many actual positives the model correctly identifies, and the x-axis represents the False Positive Rate, which means how many false positives it generates.
In the ROC curve, we observe how the model behaves by varying thresholds. We want the True Positive Rate to be as high as possible while keeping the False Positive Rate low, which means the curve should rise toward the top-left corner.
If the curve is near or along the diagonal, the model is essentially making random guesses, and its performance is not satisfactory.
If the curve lies below the diagonal, the model’s performance is very poor.
This way, we can get an idea of the model’s performance across various thresholds.
Now let’s discuss about Area Under the Curve (AUC).
We have already seen that, for our data AUC is 0.81.
An AUC of 0.81 means that if you pick one employee who left and another who stayed, there is an 81% chance that the model assigns a higher probability to the employee who left.
Now, let’s use the sample dataset to understand how AUC is calculated.
Now once again we go back to the ROC curve that we generated using our sample data.
The shaded regions in the above plot represents the AUC.
Now let’s proceed with calculation of AUC.
From point (0.00, 0.33) to (0.33, 0.33), the area under the curve is represented by the orange rectangle.
From point (0.33, 0.33) to (0.67, 0.33), the area under the curve is represented by the green rectangle.
From point (0.67, 0.33) to (1.00, 0.33), the area under the curve is represented by the red rectangle.
Now to find the AUC, we need to calculate the areas of rectangles and add them.
$$
\text{Orange Rectangle: } l \times b = 0.33 \times 0.33 = 0.11
$$
$$
\text{Green Rectangle: } l \times b = 0.34 \times 0.33 = 0.11
$$
$$
\text{Red Rectangle: } l \times b = 0.33 \times 0.33 = 0.11
$$
$$
\text{Total AUC} = 0.11 + 0.11 + 0.11 = 0.33
$$
This way we calculate AUC.
In the above sample, we can also find the area without dividing it into point-by-point segments, but in the real world we don’t get to see such ROC curves.
Now let’s consider an example ROC curve that is similar to real-world cases and calculate the AUC.
Now let’s find AUC for this ROC curve.
Here we have three segments. Two of them are trapezoids and one looks like a triangle. However, we don’t use separate formulas for each shape, since rectangles and triangles are rarely formed.
We only use the trapezoid area formula.
$$
\text{Area} = \tfrac{1}{2} \times (y_1 + y_2) \times (x_2 – x_1)
$$
Now, using this formula, let’s find the AUC.
$$
\text{Segment 1: } (0.0,0.0) \;\rightarrow\; (0.2,0.4) \\
\text{Area} = \tfrac{1}{2} \times (0.0 + 0.4) \times (0.2 – 0.0) = 0.04
$$
Here, the trapezoid formula automatically reduces to the formula for the area of a triangle.
$$
\text{Segment 2: } (0.2,0.4) \;\rightarrow\; (0.6,0.8) \\
\text{Area} = \tfrac{1}{2} \times (0.4 + 0.8) \times (0.6 – 0.2) = 0.24
$$
$$
\text{Segment 3: } (0.6,0.8) \;\rightarrow\; (1.0,1.0) \\
\text{Area} = \tfrac{1}{2} \times (0.8 + 1.0) \times (1.0 – 0.6) = 0.36
$$
$$
\text{Total AUC} = 0.04 + 0.24 + 0.36 = 0.64
$$
This is how the AUC is calculated. We now understand how we got an AUC of 0.81 for our HR dataset.
But there is also a second method to calculate AUC.
Again, we go back to our sample dataset.
Positives (1’s): [0.9799, 0.6592, 0.6337]
Negatives (0’s): [0.9709, 0.8737, 0.8718]
Here we have total 9 positive-negative pairs.
Now we compare each positive with each negative to see whether the positive is ranked higher or the negative.
$$
0.9799 \;(\text{positive}) > 0.9709 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked higher}
$$
$$
0.9799 \;(\text{positive}) > 0.8737 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked higher}
$$
$$
0.9799 \;(\text{positive}) > 0.8718 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked higher}
$$
$$
0.6592 \;(\text{positive}) < 0.9709 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked lower}
$$
$$
0.6592 \;(\text{positive}) < 0.8737 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked lower}
$$
$$
0.6592 \;(\text{positive}) < 0.8718 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked lower}
$$
$$
0.6337 \;(\text{positive}) < 0.9709 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked lower}
$$
$$
0.6337 \;(\text{positive}) < 0.8737 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked lower}
$$
$$
0.6337 \;(\text{positive}) < 0.8718 \;(\text{negative}) \;\;\;\Rightarrow\;\; \text{positive ranked lower}
$$
$$
\text{Correctly ranked pairs} = 3, \quad \text{Total pairs} = 9, \quad \text{AUC} = \tfrac{3}{9} = 0.33
$$
This is called as Ranking Method for AUC.
We obtained the same value of 0.33 using both methods for sample data.
We can understand that
$$
\text{AUC} = \frac{\text{Number of correctly ranked pairs}}{\text{Total number of pairs}}
$$
We can interpret AUC as the probability that a randomly chosen positive is ranked higher than a randomly chosen negative.
Now that we have an idea of how to generate the ROC curve and calculate AUC, let’s discuss the significance of ROC-AUC.
We used ROC-AUC when we found that accuracy is misleading. But instead of ROC-AUC, we may ask: why not run a loop over all threshold values, calculate accuracy and other metrics, and then select the best threshold?
Yes, that is possible. However, when we compare two models, we cannot compare them based on their best thresholds, since different models may have different best thresholds.
ROC-AUC gives us a single number that summarizes a model’s performance and allows comparison across different models.
Another point is that the best threshold depends on the metric we choose.
The “best” threshold changes depending on whether we optimize for accuracy, precision, recall, or F1-score. ROC-AUC is threshold-independent, making it a more general measure of model quality.
Finally, ROC-AUC captures the ranking ability of the model, which makes it especially useful for imbalanced datasets.
Dataset
The IBM HR Analytics Employee Attrition dataset used in this article is from Kaggle, which is licensed under CC0 (Public Domain), making it safe for use in this analysis and publication.
I hope you found this article helpful.
Feel free to share your thoughts.
Thanks for reading!