• We want to learn the relationship Y ~ X, where X has $p$ components.

• We assume a general form like $$Y=f\left(X\right)+ϵ$$

• $f$ is a fixed function, but we don't know what it looks like.

• We want an estimate $\hat{f}$ for it.

Prediction (Machine Learning, AI)

• generate $\hat{Y}=\hat{f}\left(X\right)$

• $\hat{f}$ is a black box

• We don't know or care why it works as long as the prediction is good

• There are two (!) Errors:

  1. Reducible error $\hat{f}$
  2. Irredicuble error $ϵ$

• We can work to improve the Reducible error

• The Irreducible error is a feature of the DGP, hence, nature. Life. Karma. Measurement incurs error.

Nonlinear Models

• More nonlinear models are able to get closer to the data.

• Hence, they are good predictors

• But hard to interpret

Training Data

1. $n$ data points $i=1,...,n$

2. $y_i$ is $i$'s response

3. $X_i=(x_{i1},\dots ,x_{ip})$ are predictors

4. Data: $(X_1,y_1),\dots ,(X_n,y_n)$

(Up until now, training data was the only data we have encountered!)

Procedure

1. We make a parametric assumption, i.e. we write down how we think $f$ looks like. E.g. $$Y={\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_p{x}_p$$ Here we only have to find $p+1$ numbers!

2. We train the model, i.e. we choose the $\beta$'s. We are pretty good at that -> OLS ✌️

• We make a no explicit assumption about functional form.

• We try to get as close as possible to the data points.

• We try to do that under some contraints like:

  • Not too rough
  • Not too wiggly

Smooth, not wiggly

• You can see that the researcher has an active choice to make here: how smooth?

• Parameters which guide choices like that are called tuning parameters.

• As $\hat{f}$ becomes too variable, we say there is overfitting: The model tries too hard to fit patterns in the data, which are not part of the true $f$!

• This graph offers a nice classification of statistical learning methods in flexibility vs interpretability space.

• Sometimes it's obvious what the right choice is for your application.

• But often it's not. It's a more complicated tradeoff than the picture suggests.

• (It's a very helpful picture!)

• We will only be touching upon a small number of those. They are all nicely treated in the ISLR book though!

Supervised Learning

• We have measures of input $x$ and output $y$

• We could predict new $y$'s

• Or infer things about Y ~ X

• Regression or Classification are typical tasks

• We know the mean squared error (MSE) already: $$MSE=\frac{1}{n}\sum _{i=1}^n(y_i-\hat{f}(x_i){)}^2$$

• We encountered the closely related sum of squared residuals (SSR): $$SSR=\sum _{i=1}^n(y_i-\hat{f}(x_i){)}^2$$

• As we know, OLS minimizes the SSR. (minimizing SSR or MSE yields the same OLS estimates.)

Training

• We have a training data set $$\left\{\right(y_1,x_1),\dots ,(y_n,x_n\left)\right\}$$

• we use those $n$ observations to find the function $q$ that minimizes the Training MSE: $$\hat{f}=\arg \underset{q}{min}MSE=\frac{1}{n}\sum _{i=1}^n(y_i-q(x_i){)}^2$$

• In many cases we don't have a true test data set at hand.

• Most methods therefore try to minimize the training MSE. (OLS does!)

• At first sight this seems really reasonable.

Variance

• How much would $\hat{f}$ change if we estimated it using a different data set?

• Clearly we expect some variation when using different samples (sampling variation), but not too much.

• Flexibel models: moving just a single data point will result in a large change in $\hat{f}$.