Re: [AIRG] AIRG 11/14 AUC Analogs for Multi-Class Models

[Aubrey Barnard <barnard@xxxxxxxxxxx>]

AIRG,

Come learn about evaluating performance in multi-class settings at AIRG 
today. Ross Kleiman will survey existing methods, explain their 
trade-offs, and propose a solution.

4pm, CS 3310
No paper; just show up. (This is an important aspect of methodology.)

I hope to see you there!

Aubrey


On 11/12/18 9:02 PM, Ross Kleiman wrote:
> Hi AIRG,
> 
> 
> This Wednesday, 11/14, I will be presenting a survey on current 
> methodologies for extending the area under the receiver operating 
> characteristic curve (AUC) to multi-class problems and some recent work 
> I have done in this area.
> 
> 
> Title: ðððð: A Performance Metric for Multi-Class Models
> 
> Presenter:Â Ross Kleiman
> 
> Time: Wednesday, November 14th, 4pm
> 
> Location: CS 3310
> 
> 
> AUC is the most commonly used performance measure for binary 
> classification models. Extending AUC to problems with greater than 2 
> classes (multi-class tasks) has resulted in two approaches each with 
> their own flaws. The first approach extends the receiver operating 
> characteristic (ROC) curve to a ROC "surface". There is no agreed-upon 
> way to construct such a ROC surface and regardless of 
> construction,Âcomputing its volume is a combinatorially complex problem. 
> An alternative approach is to perform multiple pairwise AUC calculations 
> and compute an average amongst these aggregated separability measures. 
> While fast, this approach sacrifices many of the desirable properties of 
> the 2-class AUC measure as it can result in scores much less than 1, 
> even when the model predictions are all correct and perfectly separable. 
> In our work, we take an alternative approach and derive a multi-class 
> AUC extension by using its equivalence to the Mann-Whitney U-statistic 
> (the probability a randomly selected positive instance will be ranked 
> higher than a randomly selected negative instance by the model). We 
> extend the concept of ranking binary predictions to ranking multi-class 
> predictions (which are categorical distributions). We present a measure 
> that is fast to compute and still maintains the desirable properties of 
> the two-class AUC. We call this measure AUC "mu" as "mu" is an acronym 
> for multi-class U-statistic.
> 
> 
> Cheers,
> 
> Ross
> 
> 
> 
> _______________________________________________
> AIRG mailing list
> AIRG@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/airg
>