Linear Algebra Orthonormal Bases in n-Space

Welcome back to Educator.com, welcome back to linear algebra.0000

In the previous lesson, we talked about transition matrices for a particular space, where we have 2 or more bases for that space.0004

There we said that one basis is as good as another. As it turns out that is true.0015

One basis is not necessarily better than another intrinsically, however as it turns out, there are certain bases... one particular basis in particular that is better computationally.0019

It just makes our life a lot easier. That is called an orthonormal basis.0029

So, that is what we are going to talk about today. We are going to introduce the concept, and then we will talk about how to take a given basis and turn it into an orthonormal basis by something called the Gram Schmidt orthonormalization process.0034

It can be a little bit computationally intensive, and notationally intensive, but there is nothing particularly strange about it.0047

It is all things you have seen. Essentially it is all just arithmetic, and a little bit of the dot product.0055

So, let us just jump in and see what we can do.0059

Okay. Let us talk about the standard basis in R2 and R3, the basis that we are accustomed to.0063

If I take R2, I can have a basis (1,0), that is one vector, and the other vector is (0,1)... okay?0071

Also known as i and j, also known as e1 and e2, again just different symbols for the same thing.0084

R3 is the same thing. R3... we have... (1,0,0), (0,1,0), and (0,0,1), as a basis.0095

Three vectors, three dimensional vector space, you also know them as i,j, k, and we have also referred to them as e1, e2, e3.0108

Now, what is interesting about these particular bases, notice, let us just deal with R3... vector a and vector 2 are orthogonal meaning that their dot product is 0, perpendicular.0122

1 and 3 are orthogonal, 2 and 3 are orthogonal. Not only that, they are not just orthogonal, mutually orthogonal, but each of these has a length of 1.0136

So, this is what we call orthonormal, that the vectors are mutually orthogonal, and they have a length of 1.0146

As it turns out, this so called natural basis works out really, really, well computationally.0154

We want to find a procedure... is there a way where given some random basis, or several random bases, can we choose among them and turn that basis into something that is orthonormal, where all of the vectors have a length of 1 and all of the vectors are mutually orthogonal.0160

As it turns out, there is. The Gram Schmidt orthonormalization process. A beautiful process, and we will go through it in just a minute.0175

Let us just start off with some formal definitions first.0183

So, we have a set s, which is the vectors u1, u2... all the way to uN is orthogonal, if any two vectors in s are orthogonal.0188

What that means mathematically is that u_i · u_j, so u₁ · u₃ = 0.0222

The dot product of those two vectors is equal to 0. That is the definition of orthogonality. Perpendicularity if you will.0231

Now, the set is orthonormal if each vector also has a length norm of 1.0239

Mathematically, that means u_i dotted with itself gives me 1.0262

Let us just do an example. Let us take u₁ = the vector (1,0,2), u₂ = the vector (-2,0,1), and u₃ = the vector (0,1,0).0269

Okay. So, as it turns out, the set u1, u2, u3, well if I do the dot product of u1, u2... u1, u3... u2, u3, I get 0 for the dot product.0293

So, this set is orthogonal.0311

Now, let us calculate some norms. Well, the norm of u₁ is equal to this squared + this squared + this squared under the radical sign.... is sqrt(5).0316

The norm for u2, that is our symbol for norm, is equal to this squared + this squared + this squared... also sqrt(5).0334

And... the norm for u3 is 1. So, this one is already a unit vector, these two are not.0348

So, since I have the norm, how do I create a vector that is length 1. I take the vector and I multiply it by the reciprocal of its norm, or I divide it by its norm, essentially.0353

So, we get the following. If I have the set u1/sqrt(5), u2/sqrt(5), and u3... this set is orthonormal.0369

Again, radical sign in the denominator, it is not a problem, it is perfectly good math. In fact, I think it is better math than simplification.0388

What they call rationalizing the denominator, or something like that... I like to see where my numbers are.0399

If there is a radical 5 in the denominator, I want it to be there. I do not want it to be u2sqrt(5)/5, that makes no sense to me, personally, but it is up to you though.0405

Okay. Quick little theorem here.0416

If s is an orthogonal set, or orthonormal, then s is linearly independent.0428

So, if I have a set that I do not know is a basis, I know this theorem says that they are linearly independent.0452

So, the particular space that they span, it is a basis for that space.0460

Okay. Let us go ahead and do this. Like we said before, using an orthonormal basis can actually improve the computation.0466

It just makes the computational effort a little bit less painful.0476

So, now if we have s, u1, u2, all the way to u_n...0481

If this set is a basis for some vector space v and u is some random vector in v, then we know that we can express u as linear combination... c1u1 + c2u2 + cNuN.0498

u's, v's, w's, all kinds of stuff going on. Okay, so what we did before was we just solved this linear system.0528

We found c1, c2, all the way to cN, you now, Gauss-Jordan elimination, reduced row echelon form.0536

Well, here is what is interesting. If s is orthonormal, well still if it is orthonormal, when it is orthogonal it is still a basis so you still get this property.0545

You know -- u is still that, but it is a really, really simple way to find the cN without finding the linear system.0562

What you end up with is the following. Each c_i is equal to the vector u dotted with u_i.0570

For example, if I wanted to find the second coefficient, I would just take the vector u and I would dot it with the second vector in the basis.0582

That is fantastic. It is just a simple dot product. For vectors that are -- you know -- maybe 2-space, 3-space, 4-space, maybe even 5-space, there is no solution, you do not have to worry about it.0592

You can just do the multiplication and the addition in your head. The dot product is really, really easy to find.0600

So, let us do an example of this. We will let s equal... okay... it is going to be a little strange because we are going to have some fractions here...0608

(2/3, -2/3, 1/3, 2/3, 1/3, -2/3, 1/3, 2/3, 2/3,), so this is our set s.0625

Okay, well, we want to be able to write some random vector v, which is equal to let us say (3,4,5) as a linear combo of the vectors in s.0650

So, we want c1, let us call it v1, the vectors in s, let us just call them v1, v2, v3... + c2v2 + c3v3.0675

So, we want to find c1, c2, c3. Well, as it turns out that basis, even though it looks a little strange, it actually ends up being orthonormal.0691

The length of each of those vectors is 1, and the mutual dot product of each of those is 0.0700

So, it is an orthonormal set. Therefore, c1 = v · v1.0708

Well, v is just 3, 4, 5, okay? I am going to dot it with v1, which was 2/3, -2/3, 1/3, and I get 1.0718

When I do c2, well that is just equal to v · v2. When I do that, I get 0.0737

If I do c3, that is just v, again, 3, 4, 5, dotted with v3, which was v3 up there in s, and this ends up being 7.0746

Therefore, v is equal to, well, 1 × v1 + 0 × v2 + 7v3.0759

There we go. I did not have to solve the linear system. I just did simple dot product. Simple multiplication and addition. Very, very nice.0771

One of the benefits of having an orthonormal basis. There are many benefits, trust me on this one.0781

Okay. So, now let us come down to the procedure of actually constructing our orthonormal basis.0786

So we are going to go through this very, very carefully. There is going to be a lot of notation, but again, the notation is not strange. You just have to make sure to know what is where.0792

The indices are going to be very, very important... so, take a look at it here, take some time to actually stare at the Gram Schmidt orthogonalization process in your textbook.0802

That is really the best way to sort of wrap your mind around it. Of course, doing examples, which we will do when you do problems, but just staring at something is really a fantastic way to understand what it is that is going on.0815

In mathematics, it is the details, the indices, the order of things. Okay.0826

Let us see. So, let us write it out as a theorem first.0835

You know, let me... let me go to a black ink... theorem...0841

Let w be a non-zero subspace. So again, when we can speak of a basis we can speak of a base for the entire space, it does not really matter.0857

In this case we are just going to express this theorem as a subspace.0870

Again, the whole space itself is a subspace of itself, so this theorem is perfectly valid in this case.0875

... be a subspace of rN with basis s = u1, u2, uN. N vectors, N space.0883

Then, there exists an orthonormal basis t, which is equal to... let us call it w1, w2, all the way to wN for w.0906

So, this theorem says that if I am given a basis for this subspace or a space itself, I can find... there exists an orthonormal basis.0934

Well, not only does one exist, as it turns out, this particular procedure constructs it for us. So, the proof of this theorem is the construction of that orthonormal basis itself.0945

That is the Gram Schmidt orthonormalization process. They call it the orthogonalization process, which is really what we are doing.0955

We are finding orthogonal vectors, but we know how to change vectors that are orthogonal to vectors that are orthonormal.0961

We just divide by their norms... nice and simple. Okay.0967

So, let us list the procedure so that we have some protocol to follow.0973

Procedure... okay... procedure for constructing an orthonormal basis t, which we said is going to be w1, w2... all the way to wN.0979

Constructing an orthonormal basis t... from basis s = u1, u2, all the way to uN.1014

I am going to change something here. I am going to not use w, I think I am going to use v instead.1038

I used u for s, so I am going to go back and choose... let me call them v so that we stay reasonably consistent... to vN.1047

Okay. So, we are given a basis s, we want to construct an orthonormal basis t from s, here is how we do it.1058

First things first. We let v1, the first vector in our basis t, our orthonormal, we let it equal u1.1068

We just take it straight from there. The first vector is the first vector.1076

Okay. Two. This is where the notation gets kind of intensive.1082

v_i = u_i - u_i · 1₁/v₁ · v₁ × the vector v₁ - u_i · v₂/v₂ · v₂ × the vector v₂... and so on.1086

Until we get to u_i · v_i-1/v_i-1 · v_i-1 × v_i-1. Okay.1134

Do not worry about this, it will... when we do the example, this will make sense. Again, this is just mathematical formulism to make sure that everything is complete1158

When we do the example, you will see what all of these i's and i-1's and v's mean.1165

Three. t*, when we have collected our v₁, v₂, that we go from the first two steps... is orthogonal.1173

We have created an orthogonal set.1194

Now, we want to take -- that is fine, we will go ahead and we will take for every v_i, we are going to divide it by its norm.1199

So, for each of these v₁, v₁, in this set which is orthogonal, we are going to divide each of these vectors by the norm of that vector.1231

Then, of course, what you get is the final set t, which is v₁/norm(v₁), v₂/norm(v₂), and so on and so forth, all the way to v_n, not v_i.../norm(v_n)/1242

This set is orthonormal. Let us just do an example and it will all make sense.1287

So, let us start here. Let us do our example in blue.1299

We will let s = u1, u2, u3 = 1, 1, 1, -1, 0, -1, -1, 2, 3.1308

This is our set. We want to transform s into an orthonormal basis, for R3. This is a basis for R3.1342

These are linearly independent. They span R3. Three vectors. We want to change this into an orthonormal basis.1353

We want each of the vectors in our basis to be orthogonal, mutually orthogonal, and we want them to have a length of 1. So, we will run through our procedure.1361

Okay. First thing we do, let v1 = u1. So the first thing I am going to do is I am going to choose by vector (1,1,1).1371

That is my first vector in my orthogonal set. Nice, we got that out of the way. Okay.1380

Two. Go back to the previous slide and check to see that number two thing with all of the indices going on. Here is what it looks like based on this number of vectors.1387

v₂ = u₂ - u₂ · v₁, which is this thing /v₁ · v₁ × vector v₁.1400

That is equal to... well, u₂ is (-1, 0, -1).1430

Now, when I take u₂, which is (-1, 0, -1), and I dot it with v₁, which is (1, 1, 1), I get -2.1440

When I take v₁ dotted with v₁, I get three. So it is -2/3 × v₁, which is the (1,1,1).1453

When I do that, I get -1/3, 2/3, -1/3... okay.1471

The next one v3, I have v₁, I have v₂, which is here. I need v₃, right? because I need 3 vectors.1489

So, v₃ = well, it is equal to -- go back to that number 2 in our procedure -- it is equal to u₃ - u₃ · v₁/v₁ · v₁ × v₁ - u₃ · v₂/v₂ · v₂, all × v₂.1498

Well if you remember that last entry in that number 2, it said v^i-1.1543

Well, since we are calculating v₃, 3 - 1 is 2, so that is it. We stop here. We do not have to go anymore.1547

That is what that symbolism meant, it tells us how many of these we get.1557

If we are calculating v₄, well, 4 - 1 is... so that is v_i, i is 4. It is 4 - 1, that means we go all the way up to this last entry, which is 3. So we would have three of these.1563

That is all this means. That is all that symbolism meant. Just follow the symbolism, and everything will be just fine.1578

okay. This actually ends up equaling... well u₃ is (-1,2,3).1584

When I do u₃ · v₁, which is (-1,2,3) · (1,1,1) over (1,1,1) · (1,1,1), I am going to end up with 4/3 × (1,1,1) - ... when I do u₃ · v₂, which is u₃ · v₂/v₂ · v₂.1597

I am going to get -2/6 × v₂ - (1,2,1)... okay?1627

I am going to end up with (-2,0,2). So, let me go to red.1644

(1,1,1), that... and that... so let us write our what we have got.1657

Our t*, our orthogonal set is this sub... interesting... 1, 1, 1, -1/3, 2/3, -1/3, and -2, 0, 2.1678

This set is orthogonal. Now, let us take a look at v₂ real quickly here.1706

v₂ = (-1/3, 2/3, -1/3), well, let me pull out the third... that is equal to 1/3 × (-1, 2, -1).1721

These vectors, if I just take the vector (-2, 2, 1), and if I take the vector 1/3 × that, which is this vector, they are vectors going in the same direction.1741

They are just different lengths of each other. So, because they are vectors going in the same direction, I do now need the fractional version of it. I can just drop the denominator from that, because again, we are going to be normalizing this.1754

We are going to be reducing it to a length of 1, so it does not matter whether I take this vector or this vector.1766

They are just different vectors in the same direction. Does that make sense?1772

So, I can rewrite my t*, my orthogonal set, as (1,1,1), (-1,2,-1), and (-2,0,2).1776

They are still orthogonal. This, the dot product of this and this is going to be 0. They dot product of this and this is going to be 0, it does not matter. They are in the same direction.1795

They are just different lengths, we are going to be normalizing it anyway. So, we want to make life easier for ourselves, so that is our orthogonal set.1806

Now, what we want to do is calculate the norms.1820

So, the norm of v₁ = sqrt(3). The norm of v₂ = sqrt(6), and the norm of v₃, right here, is equal to sqrt(8).1827

Therefore, our final orthonormal... we have 1/sqrt(3), 1/sqrt(3), 1/sqrt(3)... -1/sqrt(6), 2/sqrt(6), -1/sqrt(6) and... -2/sqrt(8), 0, 2/sqrt(8)...1855

Again, I may have missed some minus signs or plus signs... it is procedure that is important.1897

This is orthonormal. In this case, the length matters. In the previous, they were orthogonal and so we dropped the denominator from the vectors that we found because again, they are vectors in the same direction.1907

The same direction, they are still going to be orthogonal. So, I can make it easy for myself by not having to deal with fractions. But, in this case, we are normalizing it. We are taking that orthogonal set and we are dividing each of those vectors by its own norm to create vectors.1925

Each of these has a length of 1. Any two of these are mutually orthogonal. Their dot product equals 0. This is an orthonormal basis for our three.1940

It is just as good of a basis as our standard basis, the i, j, k. Computations are easy, and who knows, there might be some problem where this basis is actually the best basis to work with.1953

Again, it just depends on frames of reference.1964

Okay. Thank you for joining us at Educator.com.1968