AUC can be intuitively understood as: “the probability that the classifier will assign a higher score to a randomly chosen positive example than to a randomly chosen negative example.” – Wikipedia
Yeah ok nice but what does that really mean? Actually the previous intuition is a bit tricky to understand. So let’s try to understand it.
Suppose we have a binary classification problem scenario as the following: we have a dataset with instances that have either or as labels. You divide the dataset into two parts: 1- training set, 2-test test. Next you train a classifier with the training set.
Now we want to test the performance of the classifier. Normally your test set has the following form: is the instances matrix and is a vector that says if each instance is or . You feed into the classifier and let it classify each instance and ask it to give a confidence score for each class. For example, if the instance number 15 in has a true label then your classifier will probably give the following confidence scores: for class and for class . So after feeding into the classifier each instance will have two confidence scores. Let’s assume that and are vectors that hold the confidence scores for all the nodes, so holds the scores of all instances for class and for class .
Let’s recall what we have until this point: , , , . Now when you compute the AUC you only use and for the calculations (why? later my friend). Again holds the true labels and holds the confidence or the probability of each instance to be of class (or the positive class). Let’s assume you have a perfect classifier that classifies everything correctly. Then now notice that in all instances that have true labels as should have really high scores, while those that have true labels as should have really low scores.
Now suppose you pick two instances from the test set at random, that has a true label and has a true label . After that you check the scores of these two from . You will see that might have and have . Notice that has a higher score than because is truly positive and is truly negative. Now suppose you keep picking positive and negative instances at random from and you make a comparison to check if the truly positive instance has a higher score than the truly negative instance. Since our classifier is perfect it will always give higher scores to truly positive instances than to truly negative instances. So its AUC will be .
Let’s consider if our classifier is not perfect, then some instances that are truly positive in reality will have low scores in and some instances that are truly negative in reality will have high scores in . In this case our comparisons will have some misses, for example: given two randomly picked positive and negative instances from , the positive instance will have a lower score than the negative instance. This way the AUC of the imperfect classifier will be bad.
Actually suppose you made comparisons, of the comparisons are correct and of them are incorrect, then you can compute the AUC using the following formula:
Now go back and read the intuition at the beginning of the post and hopefully you will fully understand it. (Always keep in your mind that we only use a probability vector for the positive class to compute the AUC)
p.s. this post was written very quickly without editing, so I hope it is not confusing or has serious mistakes.