Linear Algebra Orthogonal Complements, Part II

Welcome back to Educator.com and welcome back to linear algebra.0000

In our last lesson, we introduced the notion of an orthogonal complement, and this time we are going to continue talking about orthogonal complements.0004

We are going to be talking about these 4 fundamental subspaces that are actually associated with any random matrix, and we are going to talk about the relationships that exist between these spaces.0013

Then we are going to talk about something called a projection. The projection is a profoundly, profoundly important concept.0023

It shows up in almost every area of physics and engineering and mathematics in ways that you would not believe.0033

As it turns out, those of you who are engineers and physicists... one of the tools in your tool box that is going to be almost the primary tool for many years to come is going to be the idea of something called Fourier series.0042

If you have not been introduced to it yet, you will more than likely be introduced to it sometime this year... and Fourier series actually is an application of projection.0052

Essentially what you are doing is you are taking a function and what you are doing is you are projecting that function onto -- how shall I say this -- you are projecting it onto an infinite dimensional vector space on the individual axes which are the trigonometric functions.0061

Let us say, for example, I have a function 5x. I can actually project that function onto the cos(x) axis, onto the sin(x) axis, onto the cos(2x) axis, onto the sin(2x) axis, so on and so forth.0080

I can actually represent that function in terms of cosine functions.0096

Now, you will learn it algebraically, but really what you are doing is you are actually doing a projection.0100

You are projecting a function onto other functions, and it is really quite extraordinary.0105

When you see it that way, the entire theory of Fourier series becomes open to you, and more than that, the entire theory of orthogonal polynomials becomes open to you.0111

That is going to connect to our topic that we discuss in the next lesson, which is Eigenvectors and Eigenvalues.0120

So, linear algebra really brings together all areas of mathematics. Very, very central. Okay. Let us get started.0126

Let us see. So, let us go ahead and talk about our four fundamental vector spaces associated with a matrix a.0135

So, if a is an m by m matrix, then, there are 4 spaces associated with a.0143

You actually know of all of these spaces. We have talked about them individually, now we are going to bring them together.0173

One is the null space of a, which is, if you remember, is the solution space for the equation ax = 0.0178

Let me put that in parentheses here. It is the solution space, the set of all vectors x, such that the matrix a × x = 0. Just a homogeneous system.0189

Two, we have something called the row space of a. Well, if you remember, the row space of a if I take the rows of a, just some random m by m matrix, they actually form a series of vectors... m vectors in RN.0200

The space that is spanned by those vectors, that is the row space.0221

Then we have the null space of a transpose. So, if I take a and just flip it along its main diagonal and then I solve this equation for this set of vectors, x such that a transpose × x = 0, I get its null space.0229

It is also a subspace, and the row space is a subspace. All of these are subspaces.0246

Oops -- it would be nice if I could actually count properly... 1, 2, 3.0251

Now, our fourth space is going to be, well, you can imagine... it is going to be the column space.0257

Again, the column space if I take the individual columns of the matrix, they are vectors in RM, and they form a space... the span of those vectors form a space.0264

Now, they do not all have to be linearly independent. I can take those vectors, remember from an old discussion and I can find a basis... so the basis might be fewer vectors but they still span the same space.0278

Okay. So, let us start with a theorem here, which is an incredibly beautiful theorem.0290

As you can figure it out, linear algebra is full of unbelievably beautiful theorems... beautiful and very, very practical.0299

If a is m by n, then, this is kind of extraordinary, the null space of a is the orthogonal complement of the row space of a.0306

That is kind of amazing. Think about what that means for a second.0337

If I just have this rectangular array of numbers, 5 by 6 and I just throw some numbers in there... when I solve the equation ax = 0, the homogeneous system associated with that matrix, I am going to get a subspace, the null space.0344

As it turns out, if I take that matrix and I turn it into reduced row echelon, the non-zero rows form a basis for the column space. That is how we found the basis for the column space.0359

Those two subspaces, they are orthogonal to each other. That is extraordinary. There is no reason to believe why that should be the case, and yet there it is.0370

B, the complement of that, the null space of a transpose is the orthogonal complement to the column space.0380

It is the orthogonal complement of the column space of a.0399

So, that is the relationship. The null space of a given matrix a, its null space and its column space are orthogonal complements.0407

If I take the transpose of a, the null space of the transpose and the column space of the original a, which ends up being the row space of a transpose because I have transposed it... those two are orthogonal complements.0416

Let us do an example and see if this... if we can make sense of some of it just by seeing some numbers here.0433

So, let us go... let us let a equal, it is going to be a big matrix here, and again we do not worry about big matrices because we have our math software.0441

1, -2, 1, 0, 2... now I am not going to go through all of the steps.0450

You know, this reduced row echelon, solving homogeneous systems, all of this stuff, I am going to give you the final results.0457

At this point, I would like to think that you are reasonably comfortable either with the computational procedure manually, or you are using mathematical software yourself. I just do this in math software, myself.0463

1, -1, 4, 1, 3, -1, 3, 2, 1, -1, 2, -3, 5, 1, 5... Okay.0473

So, this is our matrix a. Our task is to find the four fundamental spaces associated with this matrix and confirm the theorem.0488

So, this random rectangular array of numbers, something really, really amazing emerges from this. There are spaces that are deeply, deeply interconnected.0516

So, let us see what happens here. Okay. When we take a, so the first thing we want to do is we want to find the row space of a.0527

Let us go ahead and do that. So, row space of a.0536

When I take a, and I reduce it to reduced row echelon form, I get the following matrix, 1, 0, 7, 2, 4, 0, 1, 3, 1, 1... and I get 0's in the other 2 rows.0540

Well, what that means, basically what that tells me is that my row space, if I take these -- let me go red -- if I take that vector and that vector, they form a basis for the row space.0560

So, my row space... basis for the row space and I am going to write these as column vectors, is 1, 0, 7, 2, 4... and 0, 1, 3, 1, 1...0577

So, the dimension is 2. There you go. Also, I have my row space is 2-dimensional. Okay.0600

Now, I need to find the null space. Well, I can almost guess this. The theorem says that the row space and the null space are going to be orthogonal complements.0610

Well, I know that the orthogonal complement, or some subspace + the direct some of some subspace plus its orthogonal complement gives me the actual space itself.0620

In this case, I am talking about 1, 2, 3, 4, 5... I am talking about R5.0630

Well, if I already have a dimension 2, I know that the dimension of my orthogonal complement is going to be 3 and so I am hoping that when I actually do the calculation I end up with 3 vectors.0637

Let us see what happens. The null space, well the null space is the set of all vectors x such that a(x) = 0.0648

I solve a homogeneous system and I get my basis... I am not going to actually show this one.0662

So, my basis for null of a... I tend to symbolize it like that... is equal... set notation... I have the vector (-7, -3, 1, 0, 0).0669

I end up with (-2, -1, 0, 1, 0) and the presumption here is that you are comfortable doing this, the reduced row echelon, solving the homogeneous system, putting it into a form that you can actually read off your vectors.0686

(-4, -1, 0, 0, 1). Well, there you go. We have a dimension equals 3.0703

So, the dimension 2 + the dimension 3 = 5. The row space was a series of vectors in R5, so our dimensions match.0711

Now the question is, I need to basically check that this is the case... I need to check that each of these vectors is orthogonal to the 2 vectors that I found.0721

As it turns out, they are orthogonal. When you actually take the dot product of each of these with each of the other ones, you are going to end up with 0.0731

So, this confirms our theorem. The first part of the theorem. The row space of a and the null space of a.0739

Okay. So, now let us take our column space. So, we are going to take a transpose.0748

Now, let me actually write out a transpose, I would like you to see it. It is (1,1,-1,2)... this is going to be R4, okay? -2.0760

Now, the column space of a, what I have done here is I have actually transpose a. I have turns the rows in columns and the columns into rows.0776

So, now the columns of a are written as rows. That is why I am doing it this way. Okay?0783

-2,-1,3,-3,1,4,2,5,0,1,1,1... and I have 2,3,-1, and 5. Okay.0790

So, I have 5 vectors in R4. So, here we are talking about R4.0814

Alright. Now, when I subject this to reduced row echelon form, I am going to end up with some non-zero columns.0823

That is going to be a basis for my column space.0833

I get 1, 0, -2, 1, 0, 1, 1, 1, and 0's everywhere else, 0, 0, 0, 0... 0, 0, 0, 0.0838

These first 2 actually form a basis for my column space.0850

So, let me write that down... basis for my column space equals the set of vector 1, 0, -2, 1, 1, 0, -2, 1... I think it is always best to write them in vertical form, and 0, 1, 1, 1.0856

Not good -- we would like them to be clear... 1, 1, 1... there you go. That forms a basis for our column space.0869

Well, the dimension is 2. You know your spaces are 4... 4 - 2 is 2.0890

We are going to expect that are homogeneous system, our null space of a transpose is going to be 2-dimensional. We should have 2 vectors.0897

Well, let us confirm. As it turns out, when we solve a transpose × x = the 0 vector... basis for null -- love this stuff, it is great -- a transpose equals... again, I am just going to give the final answer... 2, -1, 1, 0.0905

That is one vector, and the second vector is... these are basis vectors for our subspace... 0, 1.0938

Sure enough, we end up with a dimension 2. So, our dimensions match. Now we just need to check that any vector in here × any vector in what we just got ,the column space are orthogonal.0945

It turns out that they are. If you do the dot product of those, you are going to end up with 0.0956

So, sure enough, once again, row space a, okay? is going to be the orthogonal complement of the null space of a.0961

The column space of a is the orthocomplement of null space of a transpose.0982

Simply by virtue of rectangular array of numbers, you have this relationship where these spaces are deeply interconnected.0995

It is really rather extraordinary... and they add up to the actual dimension of the space. Okay.1001

So, let us talk about projections and some applications. So, projections are very, very, very important.1011

Recall, if you will that w is a subspace, I am just going to write ss for subspace of RN, then, w direct sum plus w perp is equal to RN.1019

We sort of have been hammering that point. That is some subspace and some orthogonal complement, the dimensions add up to n.1042

When you add them you actually get this space, RN. Okay.1048

That was one of the proofs that we discussed... one of the theorems that we had in the last lesson.1053

Now, we did not go through a proof of that, and I certainly urge you, with each of these theorems, to at least look at the proofs, because a lot of the proofs are constructive in nature and they will give you a clue as to why things are the way that they are.1059

So, in the proof of the theorem, it is shown that if w, if that subspace has an orthonormal basis, remember an orthonormal basis is well, all of the vector are of length 1, and they are all mutually orthogonal.1070

We had that Gram Schmidt orthonormalization process where we take a basis and we can actually turn it into an orthonormal basis by first making it orthogonal and then dividing by the norms of each of those vectors.1104

So, if it has an orthonormal basis, let us say w1, w2, all the way to w_k, we do not know how many dimensions it is.1116

v is any vector in RN, then there exists a unique... there exists unique vectors w from the subspace w and u from the subspace w perp, such that the vector v can be written as w + u.1132

Well, we know this already. Essentially what we are saying is that if we take any vector in RN, I can represent it uniquely as some vector from the subspace w + some vector in its orthogonal complement.1171

Okay. Here is the interesting part. Also... we will write it as an also... this particular w, let me actually circle it in blue, there is a way to find it.1185

Here is how we find it... w = the vector v · w1 × w1 + the vector v · w2 × w2 + the vector v · wk, as many vectors as there are in the basis, × wk.1196

This is called the projection. This is called the projection of v onto the vector space w.1230

It is symbolized as proj... as a subscript we write the w... that is the subspace w... and this is the vector v. Okay.1250

We definitely need to investigate what it is that this looks like. When we do a projection -- let me draw this out so that you see what this looks like.1262

So, we are going to be working in R3. So, let me draw a plane here.1270

Let me draw a vector... this is going to be our w vector, then this is going to be our u vector.1287

Let me make v... let me make it blue. So, v, once again, let us remind ourselves... v is any vector in this particular case, it will just be vector RN.1300

You know what, since we are dealing in 3, let me be specific. R3.1315

w is our vector in the subspace, which is a 2-dimensional subspace in 2, so this plane here... that is the subspace w, and the u is in the subspace w perp.1323

So, here is what we are doing. Well, we said that this particular... so v can be written as something from w + something from u, because we know that RN is equal to the direct sum of w and w perp.1347

So some vector from w, some vector from here, so that is a vector in w, that is a vector in w perp.1368

Well, when we add them together, we get v. This is just standard vector addition.1374

Here is what is really interesting. If we have a basis for this subspace, if we actually project v, project means shine a light on v so that you have a shadow, a shadow of v on this subspace... that is what the projection means.1380

That is where you get that v · w1 × w1 + v · w2... when you do that, what we just wrote down for the projection, you actually end up finding w.1397

Okay. Now, we had also written since v is equal to w + u... as it turns out if I wanted to find u, well, just move that over.1412

Equals v - w. That is it. This is really, really great. So, let us do a problem and I think all of this will start to make sense.1425

So, let us go... example here... we will let w be a subspace of R3 with an orthonormal basis... I often just write ortho for orthonormal basis.1437

Again, orthonormal bases, they tend to have fractions in them because they are of length 1.1459

We do not always want to use (0,0,1), we want something to be reasonably exciting.1465

Let us go... oops -- these lines are making me crazy -- 2/sqrt(14), 3/sqrt(14), 1/sqrt(14)... one vector.1473

The other vector is 1/sqrt(2),0,-1/sqrt(2)... so these are orthonormal.1493

It is an orthonormal basis for the subspace R3, there is 2 vectors in it, so our subspace w has dimension 2, which means that our orthogonal complement w perp has dimension 1. 2 + 1 has to equal 3.1501

Okay. We will also let v, the vector in R3 equal to some random 4, 2, 7... this is what we want to find.1516

We want to find the projection of v onto w, and the vector... and we want to find -- this has to stop, why does this keep happening?1529

So, we want to find the projection of v into this subspace, and we want to find the vector u that is orthogonal to every vector in w.1549

In other words, we want to find w perp. Okay. So, how can we do that. Switch the page here...1572

Well we know that from our formula, from our theorem, that our w is equal to the projection onto w of v, which is what we wanted.1584

That is going to equal v · w1, one of the vectors in the basis × one of the vectors in the basis... plus v · w2, the second vector in the basis × that vector in the basis.1597

Again, I am going to let you work these out. So take v · w1, you are going to get a scalar, multiply it by w1, it is going to give you a vector.1617

You are going to add to that v · w2, which is a scalar × w2, which is a vector. When you add two vectors together you are going to get a vector.1626

So, you will end up with something like this. 21/sqrt(14) × 2/sqrt(14), 3/sqrt(14), 1/sqrt(14), + -3/sqrt(2) ×, well 1/sqrt(2), 0, -1/sqrt(2). That is what this is.1634

Then when you put those together, you are going to end up with 21/14, 63/14, 42/14, if I have done my arithmetic correctly.1666

So, the projection of v onto the subspace w is this vector right here.1684

That is what that means. If I take v and if I take the shadow of v on that subspace, I am going to get a vector.1692

It is nice. Okay. Now, well we know that v is equal to w, this is w by the way. That is the projection.1701

plus u, well we have v, we just found w, so now we want to find u.1714

Well, u is just equal to v - w, when I take v - w, that is equal to, well, (4,2,7) - what I just found... 21/14, 63/14, 42/14.1720

I am going to get 35/14, - 35/14, and 56/14. So, my vector v in R3 is a sum of that vector + that vector.1746

It is pretty extraordinary, yeah? Again, this idea of a projection. All you are doing is you are taking a random vector and onto another space you are just shining the light. you are just taking the shadow.1772

The shadow means you are taking the perpendicular... you are dropping a perpendicular from the end of that vector onto there, and this vector that you get, whatever it is, that is the projection.1787

That is all the projection means. Perpendicular.1800

As we know, the perpendicular from a point down to something else is the shortest distance from that object to that something else.1805

So, let us draw the picture one more time, so that we are clear about what it is we are doing.1815

We had w, we had u, I will put v here, that is v, this is -- oops, we wanted this in red.1825

This is u, this is w, vector v is equal to w + u, u is equal to the vector v - the vector w. That is what this picture says.1841

So, the distance from v, from the vector v to the subspace w... this is a capital W... is, well, the distance from v to the subspace w, the perpendicular distance, well it is equal to the norm of u.1863

Well, the norm of u equals the norm of the projection of v onto the subspace w, which is equal to the norm of v - ... no, I am sorry, that is not correct, getting a little ahead of myself here.1898

Vector u, the norm of u is the norm of this thing, which is v - the projection of v onto the subspace w.1923

In 3-space, it makes sense because you are used to seeing 3-space and distance.1938

Well, the distance from this point to this point is just the distance of the vector u, which you can calculate here. That is just the norm, and you know you found w by that formula that we just used which is the projection of v onto the subspace w.1944

This is the subspace w, so we project it on here, we get w, here is what may not make sense. What if you are dealing with a 14 dimensional space?1960

Let us say that your vector in R-14, you project it onto a subspace which is say 3 dimensional.1970

How do you talk about a distance in that case? Well, again, distance is just an algebraic property.1977

So, in some sense, you have this distance of a vector in a 14-dimensional space to its 3-dimensional subspace.1982

There is a distance "defined," and that distance is precisely the projection of that 14-dimensional vector, of that vector in 14-dimensional space onto the 3-dimensional subspace.1994

Again, this is the power of mathematics. We are not limited by reality. We are not limited by our senses. We are, in fact, not limited at all as long as the math supports it. As long as the algebra is correct. We are not limited by time or space.2007

Okay. Thank you very much for joining us here at Educator.com to finish up our discussion of orthogonal complements. We will see you next time.2023