Hyper-parameters’ elicitation for cautious classifier : first proposal

In the presence of uncertainty, cautious prediction methods are essential to avoid serious errors when using machine learning in sensitive domains, e.g., medicine, autonomous vehicles, etc. These methods involve the use of utility or loss functions in the decision and/or evaluation stages. The utility function, which is inherently subjective, extends the classical and inherently objective accuracy metric to comparisons of subsets. These  functions are parameterized to express the user’s disposition for imprecision. This article presents preliminary work addressing the underexplored problem of eliciting these parameters from the user. We show how the user is guided through successive comparisons toward the optimal value.