Statistics Least Squares Regression

Hi and welcome to www.educator.com.0000

In the previous lesson we learned about conceptually the idea of regression.0003

In this lesson of squares regression we are going to talk about how to actually calculate a regression line and find it.0007

Here is the roadmap and we are going to talk about what it means to best fit the data, and what does it mean for a line to best fit the data.0016

We are going to talk about sum of squared errors and y that conflict is important for regression.0025

We are going to talk about sum quantitative properties of the regression line.0030

We know conceptually what it means but once we do have a regression line there are sum rules that the regression line conforms to.0034

We are going to talk about how to actually find the slope and the intercept of the regression line,0042

What does it mean to best fit the data?0050

Well you can think about it like this, there are any number of lines that you could drop through a set of data.0053

We could draw that one, we could draw this one, we can draw this one, we could draw that one.0060

There are an infinite number of possible ones, but our goal is a regression line that is in the middle of all of these data points.0067

When it is in the middle that is what we mean by best fitting line.0077

You can think of this fit as roughly being equal to the concept of in the middle and the difference between all of these lines0082

and the true regression line is that the best fitting line is roughly in the middle.0106

How do we find the best fitting line?0114

Quantitatively what it means to best fit the line means that this line had the lowest sum of squared errors.0117

Because of that the regression line is also called the least squares line.0128

That is y it is called the least squares method.0135

Even though in the middle and best fit are good conceptual ideas they are not quantitative ideas.0138

This is the quantitative definition of what the best fitting line is.0146

Let us talk about what error is.0151

We had a particular word for error and that word is the residual.0154

And that residual is the difference between y and the predicted y from our best fitting line.0160

Having the lowest sum of squared errors is having the lowest SSE is really having a sum of all the squared residuals.0170

Residuals square and another way to write that in y – y hat².0185

This is our quantitative measure of how good our line is.0199

Now there is one of a Catch-22 here.0206

We have to have the line before we could figure out whether it has the lowest SSE but the question is how do we find that line?0208

First, before we go on let us talk about y we need to square these residuals?0220

Remember when we talked about what it means to be in the middle?0226

It means that the distances on the positive side or the point above the line and the points below the line,0228

the negative distances and these should all balance out.0235

If you have a bunch of positive and a bunch of negative and you add them together you should get 0.0239

Here is the tricky part that sum of the residuals, y – y hat.0246

The sum of the residual period that should be roughly equal to 0 for the best fitting line.0255

Because of that we want to square these distances.0263

I will write the squared in red because that means that this value, the sum of squared errors should be greater than 0.0272

We definitely want to square it.0283

These other mathematical properties that will be able to take these seventh feature.0285

We know what it means to quantitatively be the regression line.0295

It means having the lowest sum of squared errors but there is other quantitative properties that come along.0299

One important property to note is that this line, the regression line also contains point of averages.0304

The average of all your x and the average of all your y.0313

The average of variable 1 and the average of variable 2.0317

This point is often also called the center of mass.0321

It is really easy to find this point you just take the average of your x and take the average of your y.0325

x bar and y bar is your point of average.0333

You can also think of it as the center of mass because if we think of all your points, the scatter plot as long like a object, this is the center of that mass.0339

We already know that this line has the lowest SSE of any other line that also contains point of averages.0352

The sum of the residuals when you do not square it that should be approximately 0.0362

And because the sum of the residual is 0, the mean of the residuals is also 0 because the mean is the sum divided by the number of points.0371

If the sum is 0 it means it will be 0 and the variation of the residual is as small as possible.0379

It is smaller than other lines.0389

One way to quantify variation is something like standard deviation.0391

The residual have the smallest standard deviation than any other line.0396

Those are very important quantitative properties that we need to know.0404

This sounds like a wonderful, magical line.0411

How do we find such a line?0413

You might be thinking that this is sounding pretty hard and maybe we have to find the SSE0416

for a whole bunch of different line equations and find the one with the lowest SSE.0424

That is actually problematic.0431

It is a good idea.0433

It is a good conceptual idea but it is problematic, and here is y.0435

There are an infinite number of lines.0439

You can just change the y intercept by .0001 and get a totally different line.0447

You can change the slope but a tiny, tiny amount and get a totally different line.0454

There is an infinite number of lines that we would have to test.0459

Infinite number of potential lines.0465

We can find the SSE of infinite number of line.0473

That is just not an option for us.0479

Thank you to our hero Carl Gauss he was a mathematician and all kinds of German guy, and he helped us out a lot in statistics.0483

Carl Gauss invented this method called the method of least squares and through Carl method we could easily find the slope.0498

Here is how we do it.0508