Inner product spaces & orthogonality
The idea
An inner product equips a vector space with geometry: the dot product u · v defines lengths via |v| = √(v · v) and declares u and v orthogonal when u · v = 0 — the n-dimensional generalization of perpendicular, with the Pythagorean theorem you have trusted since school holding in any dimension. Orthogonality is what makes bases comfortable: coordinates against an orthogonal basis are computed by independent dot products instead of by solving a coupled linear system.
The workhorse construction is projection: the component of b along a nonzero vector a is proj = ((a · b)/(a · a)) a, the closest point to b on the line through a. The leftover b − proj is orthogonal to a by construction — that residual is the error of the best approximation, and the same decomposition scaled up gives least-squares fitting and Gram-Schmidt orthogonalization.
Mind the misconception that orthogonal means unrelated or independent in some loose sense. Orthogonality is a precise metric statement — inner product exactly zero — and it is stronger than linear independence: orthogonal nonzero vectors are always independent, but independent vectors are usually not orthogonal.
Worked example
In R³, find the projection of b = (1, 2, 2) onto a = (2, 1, 0), and verify that the residual b − proj is orthogonal to a.
- Compute the two needed dot products: a · b = 2(1) + 1(2) + 0(2) = 4, and a · a = 4 + 1 + 0 = 5.
- Form the projection: proj = (4/5) a = (8/5, 4/5, 0) — the multiple of a closest to b, scaled by how strongly b aligns with a relative to a's own length squared.
- Compute the residual: b − proj = (1 − 8/5, 2 − 4/5, 2 − 0) = (−3/5, 6/5, 2).
- Verify orthogonality: a · (b − proj) = 2(−3/5) + 1(6/5) + 0(2) = −6/5 + 6/5 = 0, exactly as the construction promises.
- Interpret the split: b decomposes into a part along a with length |proj| = (4/5)√5 ≈ 1.79 and a perpendicular part of length √(9/25 + 36/25 + 4) ≈ 2.41 — the second is the distance from b to the line through a, the irreducible error of approximating b by multiples of a.
Answer. The projection is (8/5, 4/5, 0), and the residual (−3/5, 6/5, 2) is orthogonal to a since their dot product is 0.
Check your understanding
- Why does the projection formula divide by a · a, and what goes wrong dimensionally without it?
- How does the orthogonality of the residual capture the idea that the projection is the closest point on the line?
- Why are orthogonal nonzero vectors automatically linearly independent, while the converse fails?
- Where does repeated projection-and-subtraction appear in Gram-Schmidt, and what does it produce?
Build the foundations first
Inner product spaces & orthogonality builds on these concepts. If any feel shaky, start there.