[AIRG] AIRG 11/14 AUC Analogs for Multi-Class Models

[Ross Kleiman <rkleiman@xxxxxxxxxxx>]

Hi AIRG,

This Wednesday, 11/14, I will be presenting a survey on current methodologies for extending the area under the receiver operating characteristic curve (AUC) to multi-class problems and some recent work I have done in this area.

Title: ðððð: A Performance Metric for Multi-Class Models

Presenter:  Ross Kleiman

Time: Wednesday, November 14th, 4pm

Location: CS 3310

AUC is the most commonly used performance measure for binary classification models. Extending AUC to problems with greater than 2 classes (multi-class tasks) has resulted in two approaches each with their own flaws. The first approach extends the receiver operating characteristic (ROC) curve to a ROC "surface". There is no agreed-upon way to construct such a ROC surface and regardless of construction, computing its volume is a combinatorially complex problem. An alternative approach is to perform multiple pairwise AUC calculations and compute an average amongst these aggregated separability measures. While fast, this approach sacrifices many of the desirable properties of the 2-class AUC measure as it can result in scores much less than 1, even when the model predictions are all correct and perfectly separable. In our work, we take an alternative approach and derive a multi-class AUC extension by using its equivalence to the Mann-Whitney U-statistic (the probability a randomly selected positive instance will be ranked higher than a randomly selected negative instance by the model). We extend the concept of ranking binary predictions to ranking multi-class predictions (which are categorical distributions). We present a measure that is fast to compute and still maintains the desirable properties of the two-class AUC. We call this measure AUC "mu" as "mu" is an acronym for multi-class U-statistic.

Cheers,

Ross