3. Regression - Linear

Simple Linear Regression

Simple Linear Regression is where we only have one predictor variable. When we have a combination of predictor variables, then the prediction method is called multiple regression.

Link to original

Linear Assumption

When working with linear regression models, we take the linear assumption: the relationship between the input and output is linear. It does not support anything else.

Link to original

We must also not have too noisy data as outliers in data sets can be problematic.

Simple Linear Regression

The ‘best fitting’ line is called the regression line. It shows the predicted score on $y$ for each possible value of $x$.

Vertical dashed lines represent the errors of prediction. Prediction error is also called residual. The error of prediction is the difference in $y$-value between the point and the regression line.

We calculate the best fitting line by the sum of squared errors, i.e. the line with the smallest SSE is the ‘best fit’ for the data.

$$\text{sum of squared errors}=\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2$$

Computing Regression Function

Now that we have the best fitting line, we can produce the function (predictive model). The formula is: $\hat{y}={\beta }_0+{\beta }_{1}x$ Where ${\beta }_0$ is the y-intercept. Where ${\beta }_1$ is the slope of the line.

Minimising the SSE:

It is not efficient to compute the sum of the squared prediction errors for an infinite number of possible lines, so in reality we use two formulas for the intercept and slope to find the equation of the line that minimises the SSE:

$${\beta }_1=\frac{{\sum }_{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{{\sum }_{i=1}^n(x_i-\overline{x}{)}^2}$$

And then:

$${\beta }_0=\overline{y}-{\beta }_1\overline{x}$$