Linear Algebra/Spectral Theorem

Given a Hermitian matrix $A$, $A$ is always diagonalizable. It is also the case that all eigenvalues of $A$ are real, and that all eigenvectors are mutually orthogonal. This is given by the "Spectral Theorem":

The Spectral Theorem

Given any $n\times n$ Hermitian matrix $A$, there exists an $n\times n$ unitary matrix $U$, and an $n\times n$ diagonal matrix of real values $\Lambda$ such that $A=U\Lambda U^{H}$

The columns of $U={\begin{pmatrix}{\vec {u}}_{1}&{\vec {u}}_{2}&\dots &{\vec {u}}_{n}\end{pmatrix}}$ are the eigenvectors of $U$, and the diagonal entries of $\Lambda ={\begin{pmatrix}\lambda _{1}&0&\cdots &0\\0&\lambda _{2}&\cdots &0\\\vdots &\vdots &\ddots &\vdots \\0&0&\cdots &\lambda _{n}\end{pmatrix}}$ are the corresponding eigenvalues.

In essence $A$ can be decomposed into a "spectrum" of rank 1 projections: $A=\sum _{i=1}^{n}\lambda _{i}({\vec {u}}_{i}{\vec {u}}_{i}^{H})$

The spectral theorem can in fact be proven without the need for the characteristic polynomial of $A$, or any of the derivative theorems.

Proof of the Spectral Theorem

The proof will proceed by using induction on $n$.

Base Case $n=1$

When $A={\begin{pmatrix}a_{1,1}\end{pmatrix}}$, it must be the case that $a_{1,1}$ is real, or else $A$ is not Hermitian. The spectral decomposition is then simply $A={\begin{pmatrix}1\end{pmatrix}}{\begin{pmatrix}a_{1,1}\end{pmatrix}}{\begin{pmatrix}1\end{pmatrix}}^{H}$

Inductive Case $n\geq 2$

Let ${\vec {e}}_{i}$ denote the $i^{\text{th}}$ standard basis vector of $\mathbb {C} ^{n}$. Let $\mathbf {0} _{k\times m}$ denote a $k\times m$ matrix of 0s.

Let ${\vec {u}}_{1}$ be a unit length vector that maximizes ${\vec {u}}_{1}^{H}A{\vec {u}}_{1}$ (recall that ${\vec {u}}_{1}^{H}A{\vec {u}}_{1}$ is always real), and let $\lambda _{1}={\vec {u}}_{1}^{H}A{\vec {u}}_{1}$. Let $U_{1}$ be a unitary matrix where the first column is ${\vec {u}}_{1}$: $U_{1}{\vec {e}}_{1}={\vec {u}}_{1}$.

$A=U_{1}A_{1}U_{1}^{H}$ where $A_{1}=U_{1}^{H}AU_{1}$. It will now be shown that $A_{1}$ has the form $A_{1}={\begin{pmatrix}\lambda _{1}&\mathbf {0} _{1\times (n-1)}\\\mathbf {0} _{(n-1)\times 1}&A_{\text{reduced}}\end{pmatrix}}$

${\vec {e}}_{1}^{H}A_{1}{\vec {e}}_{1}={\vec {u}}_{1}^{H}A{\vec {u}}_{1}=\lambda _{1}$ so that the (1,1) entry of $A_{1}$ is $\lambda _{1}$. It will now be shown that the first row and column of $A_{1}$ is filled with 0s except for the first entry. For an arbitary $i=2,3,\dots ,n$, consider the parameterized unit vector ${\vec {u}}_{t}(t)=(\cos t){\vec {e}}_{1}+(\sin t){\vec {e}}_{i}$.

${\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)=((\cos t){\vec {e}}_{1}+(\sin t){\vec {e}}_{i})^{H}A_{1}((\cos t){\vec {e}}_{1}+(\sin t){\vec {e}}_{i})$ $=\lambda _{1}\cos ^{2}t+(a_{(i,1)}+a_{(1,i)})\cos t\sin t+a_{(i,i)}\sin ^{2}t$ $=\lambda _{1}\cos ^{2}t+2\Re (a_{(i,1)})\cos t\sin t+a_{(i,i)}\sin ^{2}t$ where $a_{(i,j)}$ denotes the $(i,j)$ entry of $A_{1}$ ($\Re$ and $\Im$ denote the real and imaginary components respectively).

${\frac {d}{dt}}({\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)){\bigg |}_{t=0}=2\Re (a_{(i,1)})$. Unit vector ${\vec {u}}={\vec {u}}_{1}$ maximizes ${\vec {u}}^{H}A{\vec {u}}$ which implies that ${\vec {u}}={\vec {u}}_{t}(0)={\vec {e}}_{1}$ is the unit vector that maximizes ${\vec {u}}^{H}A_{1}{\vec {u}}$. Therefore ${\frac {d}{dt}}({\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)){\bigg |}_{t=0}=0$ which gives $\Re (a_{(i,1)})=0$.

Now consider the parameterized unit vector ${\vec {u}}_{t}(t)=(\cos t){\vec {e}}_{1}+i(\sin t){\vec {e}}_{i}$.

${\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)=((\cos t){\vec {e}}_{1}+i(\sin t){\vec {e}}_{i})^{H}A_{1}((\cos t){\vec {e}}_{1}+i(\sin t){\vec {e}}_{i})$ $=\lambda _{1}\cos ^{2}t+(-ia_{(i,1)}+ia_{(1,i)})\cos t\sin t+a_{(i,i)}\sin ^{2}t$ $=\lambda _{1}\cos ^{2}t+2\Im (a_{(i,1)})\cos t\sin t+a_{(i,i)}\sin ^{2}t$

${\frac {d}{dt}}({\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)){\bigg |}_{t=0}=2\Im (a_{(i,1)})$ so ${\frac {d}{dt}}({\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)){\bigg |}_{t=0}=0\implies \Im (a_{(i,1)})=0$. Therefore $a_{(i,1)}=a_{(1,i)}=0$. It has been proven that the first row and column of $A_{1}$ is filled with 0s except for the first entry.

$A_{1}$ has the form $A_{1}={\begin{pmatrix}\lambda _{1}&\mathbf {0} _{1\times (n-1)}\\\mathbf {0} _{(n-1)\times 1}&A_{\text{reduced}}\end{pmatrix}}$, and by an inductive argument, $A_{\text{reduced}}$ has the spectral decomposition $U_{r}\Lambda _{r}U_{r}^{H}$.

Therefore $A=U\Lambda U^{H}$ where $U=U_{1}{\begin{pmatrix}1&\mathbf {0} _{1\times (n-1)}\\\mathbf {0} _{(n-1)\times 1}&U_{r}\end{pmatrix}}$ and $\Lambda ={\begin{pmatrix}\lambda _{1}&\mathbf {0} _{1\times (n-1)}\\\mathbf {0} _{(n-1)\times 1}&\Lambda _{r}\end{pmatrix}}$