# Assessing Learners¶

We assemble various notions of error that are useful in machine learning.

Throughout these notes, we deal with

## To-Do¶

- Notions of Error
Conditional Generalization/Test Error

Expected Generalization/Test Error

Empirical Error

In sample error

- optimism
statements about covariance and optimism

- Is learning possible?
VC dimension in relating training/in sample error pg 239 of ESL

## Binary Classifiers¶

The accuracy is the probability your answer is correct when randomly drawing a sample from your observed data.

The notion below require a designation of positive and negative classes. That is, a boolean structure on the labels.

The sensitivity of the classifiers is the accuracy when one discards all the observations of the individuals who were not positive “in reality”.

The specificity of the classifiers is the accuracy when one discards all but those individual who are “in reality” negative.

Everything here is relative. In many cases of interest, “in reality” can not be given any falsifiable meaning.

In practice, “in reality” means “according to some other learner”. In the case above, the learner is the nonparametric, unregularized MLE, aka the “empirical distribution”.

This is something to keep in mind about these statistical assessments: they are always implicitly relative. Fortunately, hypothesis testing is tailored made for the relative situation.

## Notions of Error¶

Definition

Conditional Generalization error

Definition

Generalization Error

Definition

kdf O

Definition

kdf O

## Optimism¶

Definition

definition of optimism