Statistics Transformations of Data

Welcome www.educator.com.0000

We are going to be talking transformations of data today.0002

First we are going to talk about why we even transform data then we are going to talk about0007

two different broad types of transformations, shape preserving and shape changing transformation.0012

Then we will talk about some common shape changing transformations that you might need to know.0017

Some of them you already know.0024

First y transform.0028

One of the big reasons to transform data specially in the shape changing way is that all the stuff with a regression and correlation,0030

and all the stuff we have been learning works for linear patterns.0040

If the pattern is not linear even if you can still fit it to a regression line and you still can find the correlation it probably is not the best way to go.0044

Because for instance, in this graph you could see this has a distinct sort of curvy shape.0054

A simple linear regression one that account for a lot of this variation not as well as a curved line work.0060

Sometimes the transformation might make a nonlinear pattern more linear, thus making regression and correlation more useful.0074

All of a sudden you can use regression and correlation and it will account for a lot of the data.0086

That might be one reason to do that.0089

Let us look at this data for example.0090

This is data that we have looked at before from www.dotMinder.org, where it shows the income per person and GDP per capita.0093

It takes all that stuff that your country buys and sells and a divided by how many people you have.0102

It shows life expectancy here and notice that it says lin, this is a linear graph.0110

It is just showing you even intervals, the distance between 10,000 and 20,000 is the same as the distance between 60,000 and 70,000.0120

Same here, the distance between 55 and 60 years old is the same as the distance between 80 and 85.0133

But one of the issues with this is that it has a distinctly curved shape.0139

And primarily, it is that a lot of countries are very poor in terms of GDP per capita.0144

They are very poor and they are all put together over on this side.0152

Most countries make less than about 15,000 per person.0158

They are also squished over on this side.0165

It would be nice if we could somehow stroke out this part and squish those down0172

because these countries are probably very similar because they are rich countries.0179

A lot of them are Europe and this part of the United States because of that this might be nice.0186

One way we could do that is we could do a log transformed.0194

Instead of giving us the income per person we can look at it at a logarithmic scale.0198

If you remember logs, logs is a lot like this log this 10 that means 10 to that power, 10 to the nth power will give you that number x.0205

We have log 10(x).0228

10^y will give us x.0231

Instead of plotting the actual x it is asking maybe we could transform this so that it is giving us just the exponents.0237

The way you could do this is to show this in a logarithmic scale and now the first parts of these are stretched out.0255

The distance between 401,000 is big and that is bigger than the distance between 20,000 and 40,000.0268

That is our logarithmic scale or exponential scale.0277

Here we see the same data except now we are looking at plotting by the log of these incomes.0281

Here what we see is a more linear pattern.0294

Before we saw a curved pattern but now we see roughly more of a linear pattern.0299

That is one reason why transformations are very useful.0308

There are two kinds of broad transformations that you should know.0312

One is shaped preserving transformation and the other shape changing transformation.0320

Shape preserving transformations are something like you do not actually do the distributions of shape.0325

The shape looks the same.0332

When we look at a scatter plot, the scatter plot will look exactly the same.0334

If it is linear it will stay linear.0339

Shape changing means that if it is linear we will make it curvilinear.0342

If it is curvilinear we will make it more linear.0347

Those are shape changing transformations.0351

In order to be shape preserving, this means that these are any linear transformations.0353

Remember, the equation for a line is y = mx + b.0360

This is the classic formula for a line.0369

Anything if you add a constant or you multiply a value by a constant those are called linear transformation.0372

Shape changing transformation are anything that is non linear.0379

Now you need to do something more than adding by a constant or multiplying by a constant.0387

That might be changing x into x² or taking the square root of x or adding in other variables.0396

Adding in another variable here.0408

These are non linear transformations.0412

This is anything you do beyond just adding or subtracting or multiplying and dividing by a constant.0418

Here are some common shape preserving transformations.0427

These are the ones that do not change the shape at all.0433

When you add and subtract the constant that is fine.0437

If you multiply or divide a constant that is fine.0440

Converting units is often a common shape preserving transformation.0443

For instance, we collected our data in feet, but we want to see it in inches or something like we looked at minute, but we really wanted in hour.0450

There we are just multiplying like a constant here where you multiplying by 12.0461

Here we are dividing by 60.0468

Those are shape preserving, you will have the same shape.0470

Another shape preserving transformation is standardization or finding that z scores.0475

When we find the z scores, the z scores will have the same shape as your raw scores because we are subtracting by constant.0481

Subtracting by x bar or the mean and dividing by constant.0489

You could do combinations of these two things and still have a shape preserving transformation.0495

Another common shape preserving transformation that you might want to know are transformations from frequency to relative frequency.0500

So that is also shape preserving where you might have raw number of people that you might also want to have proportion from the total.0515

That is another way, because remember finding relative frequency is often just dividing by a constant.0528

Those are shape preserving transformations that you have already seen.0535

The shape changing transformations, the most common ones used are power transformation and log transformations.0539

Power transformations or anything where you raise your y x by some power.0549

For instance from y you change it into y² or y into the square root of y or dividing 1/y.0557

Raising it to the negative power.0571

Any of these and any combination of these is a power transform.0574

log transformed are finding the exponent.0579

Instead of raising it you have to find the exponent.0583

You could find the log of y and this will give you smaller numbers or you can find the natural log of numbers as well.0587

Any of these are possibilities.0596

You could also look at things like the e to y so that is just the inverse of this.0600

And also like some other constant to y so we could use exponential constant or do something else.0612

Although I have written y here and an oftentimes you might see y become transparent but it is also quite common to transform x as well.0620