Linear Algebra/Inner product spaces

Recall that in your study of vectors, we looked at an operation known as the dot product, and that if we have two vectors in $\mathbb {R} ^{n}$, we simply multiply the components together and sum them up. With the dot product, it becomes possible to introduce important new ideas like length and angle. The length of a vector $\mathbf {a}$, is just $\|\mathbf {a} \|={\sqrt {\mathbf {a} \cdot \mathbf {a} }}$. The angle between two vectors, $\mathbf {a}$ and $\mathbf {b}$, is related to the dot product by

$\cos {\theta }={\frac {\mathbf {a} \cdot \mathbf {b} }{\|\mathbf {a} \|\cdot \|\mathbf {b} \|}}$

It turns out that only a few properties of the dot product are necessary to define similar ideas in vector spaces other than $\mathbb {R} ^{n}$, such as the spaces of $m\times n$ matrices, or polynomials. The more general operation that will take the place of the dot product in these other spaces is called the "inner product".

The inner product

Say we have two vectors:

$\mathbf {a} ={\begin{pmatrix}2\\1\\4\end{pmatrix}},\mathbf {b} ={\begin{pmatrix}6\\3\\0\end{pmatrix}}$

If we want to take their dot product, we would work as follows

$\mathbf {a} \cdot \mathbf {b} =a_{1}b_{1}+a_{2}b_{2}+a_{3}b_{3}=(2)(6)+(1)(3)+(4)(0)=15$

Because in this case multiplication is commutative, we then have $\mathbf {a} \cdot \mathbf {b} =\mathbf {b} \cdot \mathbf {a}$.

But then, we observe

$\mathbf {v} \cdot (\alpha \mathbf {a} +\beta \mathbf {b} )=\alpha \mathbf {v} \cdot \mathbf {a} +\beta \mathbf {v} \cdot \mathbf {b}$

much like the regular algebraic equality $v(aA+bB)=avA+bvB$. For regular dot products this is true since, for $\mathbb {R} ^{3}$, for example, one can expand both sides out to obtain

${\begin{matrix}(\alpha v_{1}a_{1}+\beta v_{1}b_{1})+(\alpha v_{2}a_{2}+\beta v_{2}b_{2})+(\alpha v_{3}a_{3}+\beta v_{3}b_{3})=\\(\alpha v_{1}a_{1}+\alpha v_{2}a_{2}+\alpha v_{3}a_{3})+(\beta v_{1}b_{1}+\beta v_{2}b_{2}+\beta v_{3}b_{3})\end{matrix}}$

Finally, we can notice that $\mathbf {v} \cdot \mathbf {v}$ is always positive or greater than zero - checking this for $\mathbb {R} ^{3}$ gives this as

$\mathbf {v} \cdot \mathbf {v} =v_{1}^{2}+v_{2}^{2}+v_{3}^{2}$

which can never be less than zero since a real number squared is positive. Note $\mathbf {v} \cdot \mathbf {v} =0$ if and only if $\mathbf {v} =0$.

In generalizing this sort of behaviour, we want to keep these three behaviours. We can then move on to a definition of a generalization of the dot product, which we call the inner product. An inner product of two vectors in some vector space $V$, written $\langle \mathbf {x} ,\mathbf {y} \rangle$ is a function $V\times V\to \mathbb {R}$, which obeys the properties

• $\langle \mathbf {x} ,\mathbf {y} \rangle =\langle \mathbf {y} ,\mathbf {x} \rangle$
• $\langle \mathbf {v} ,\alpha \mathbf {a} +\beta \mathbf {b} \rangle =\alpha \langle \mathbf {v} ,\mathbf {a} \rangle +\beta \langle \mathbf {v} ,\mathbf {b} \rangle$
• $\langle \mathbf {a} ,\mathbf {a} \rangle \geq 0$ with equality if and only if $\mathbf {a} =0$.

The vector space $V$ and some inner product together are known as an inner product space.

The dot product in $\mathbb {C} ^{n}$

Given two vectors $\mathbf {a} =a_{1}{\vec {e}}_{1}+a_{2}{\vec {e}}_{2}+\dots +a_{n}{\vec {e}}_{n}\in \mathbb {C} ^{n}$ and $\mathbf {b} =b_{1}{\vec {e}}_{1}+b_{2}{\vec {e}}_{2}+\dots +b_{n}{\vec {e}}_{n}\in \mathbb {C} ^{n}$, the dot product generalized to complex numbers is:

$\mathbf {a} \cdot \mathbf {b} =\sum _{i=1}^{n}a_{i}^{*}b_{i}=a_{1}^{*}b_{1}+a_{2}^{*}b_{2}+\dots +a_{n}^{*}b_{n}$

where $z^{*}$ for an arbitrary complex number $z=c+di$ is the complex conjugate: $z^{*}=c-di$.

The dot product is "conjugate commutative": $\mathbf {a} \cdot \mathbf {b} =(\mathbf {b} \cdot \mathbf {a} )^{*}$. One immediate consequence of the definition of the dot product is that the dot product of a vector with itself is always a non-negative real number: $\mathbf {a} \cdot \mathbf {a} \geq 0$.

$\mathbf {a} \cdot \mathbf {a} =0$ if and only if $\mathbf {a} ={\vec {0}}$

The Cauchy-Schwarz Inequality for $\mathbb {C} ^{n}$

Cauchy-Schwarz Inequality

Given two vectors $\mathbf {a} ,\mathbf {b} \in \mathbb {C} ^{n}$, it is the case that $|\mathbf {a} \cdot \mathbf {b} |\leq |\mathbf {a} ||\mathbf {b} |$

In $\mathbb {R} ^{n}$, the Cauchy-Schwarz inequality can be proven from the triangle inequality. Here, the Cauchy-Schwarz inequality will be proven algebraically.

To make the proof more intuitive, the algebraic proof for $\mathbf {a} ,\mathbf {b} \in \mathbb {R} ^{n}$ will be given first.

Proof for $\mathbf {a} ,\mathbf {b} \in \mathbb {R} ^{n}$

$|\mathbf {a} \cdot \mathbf {b} |\leq |\mathbf {a} ||\mathbf {b} |$ follows from $|\mathbf {a} \cdot \mathbf {b} |^{2}\leq |\mathbf {a} |^{2}|\mathbf {b} |^{2}$ which is equivalent to

$\left(\sum _{i=1}^{n}a_{i}b_{i}\right)^{2}\leq \left(\sum _{i=1}^{n}a_{i}^{2}\right)\left(\sum _{j=1}^{n}b_{j}^{2}\right)$

expanding both sides gives:

$\sum _{i=1}^{n}\sum _{j=1}^{n}a_{i}b_{i}a_{j}b_{j}\leq \sum _{i=1}^{n}\sum _{j=1}^{n}a_{i}^{2}b_{j}^{2}$

$\iff \sum _{i=1}^{n}\sum _{j=1}^{n}(a_{i}b_{j})(a_{j}b_{i})\leq \sum _{i=1}^{n}\sum _{j=1}^{n}(a_{i}b_{j})^{2}$

"Folding" the double sums along the diagonal, and cancelling out the diagonal terms which are equivalent on both sides, gives:

$\iff \sum _{i=2}^{n}\sum _{j=1}^{i-1}2(a_{i}b_{j})(a_{j}b_{i})\leq \sum _{i=2}^{n}\sum _{j=1}^{i-1}((a_{i}b_{j})^{2}+(a_{j}b_{i})^{2})$

$\iff 0\leq \sum _{i=2}^{n}\sum _{j=1}^{i-1}(a_{i}b_{j}-a_{j}b_{i})^{2}$

The above inequality is clearly true, therefore the Cauchy-Schwarz inequality holds for $\mathbf {a} ,\mathbf {b} \in \mathbb {R} ^{n}$.

Proof for $\mathbf {a} ,\mathbf {b} \in \mathbb {C} ^{n}$

Note that $|\mathbf {a} \cdot \mathbf {b} |\leq |\mathbf {a} ||\mathbf {b} |$ follows from $|\mathbf {a} \cdot \mathbf {b} |^{2}\leq |\mathbf {a} |^{2}|\mathbf {b} |^{2}$ which is equivalent to $(\mathbf {a} \cdot \mathbf {b} )^{*}(\mathbf {a} \cdot \mathbf {b} )\leq (\mathbf {a} \cdot \mathbf {a} )(\mathbf {b} \cdot \mathbf {b} )$. Expanding both sides yields:

$\left(\sum _{i=1}^{n}a_{i}^{*}b_{i}\right)^{*}\left(\sum _{i=1}^{n}a_{i}^{*}b_{i}\right)\leq \left(\sum _{i=1}^{n}|a_{i}|^{2}\right)\left(\sum _{i=1}^{n}|b_{i}|^{2}\right)$

$\iff \sum _{i=1}^{n}\sum _{j=1}^{n}(a_{i}^{*}b_{i})^{*}(a_{j}^{*}b_{j})\leq \sum _{i=1}^{n}\sum _{j=1}^{n}|a_{i}|^{2}|b_{j}|^{2}$

$\iff \sum _{i=1}^{n}\sum _{j=1}^{n}(a_{j}b_{i})^{*}(a_{i}b_{j})\leq \sum _{i=1}^{n}\sum _{j=1}^{n}|a_{i}b_{j}|^{2}$

"Folding" the double sums along the diagonal, and cancelling out the diagonal terms which are equivalent on both sides, gives:

$\iff \sum _{i=2}^{n}\sum _{j=1}^{i-1}((a_{j}b_{i})^{*}(a_{i}b_{j})+(a_{i}b_{j})^{*}(a_{j}b_{i}))\leq \sum _{i=2}^{n}\sum _{j=1}^{i-1}(|a_{i}b_{j}|^{2}+|a_{j}b_{i}|^{2})$

$\iff \sum _{i=2}^{n}\sum _{j=1}^{i-1}2\Re ((a_{i}b_{j})^{*}(a_{j}b_{i}))\leq \sum _{i=2}^{n}\sum _{j=1}^{i-1}(|a_{i}b_{j}|^{2}+|a_{j}b_{i}|^{2})$

Given complex numbers $z$ and $w$, it can be proven that $2\Re (z^{*}w)\leq |z|^{2}+|w|^{2}$ (this is similar to $2xy\leq x^{2}+y^{2}$ for real numbers). The above inequality holds, and therefore the Cauchy-Schwarz inequality holds for complex numbers.