Calculus/Derivatives of multivariate functions

The matrix of a linear transformation

Theorem

A linear transformation $L:\mathbb {R} ^{n}\to \mathbb {R} ^{m}$ amounts to multiplication by a uniquely defined matrix; that is, there exists a unique matrix $A\in \mathbb {R} ^{m\times n}$ such that

$\forall {\vec {v}}\in \mathbb {R} ^{n}:L({\vec {v}})=A{\vec {v}}$

Proof

We set the column vectors

${\begin{pmatrix}a_{1,j}\\a_{2,j}\\\vdots \\a_{n,j}\end{pmatrix}}:=L({\vec {e}}_{j})$

where $\{{\vec {e}}_{1},\ldots ,{\vec {e}}_{n}\}$ is the standard basis of $\mathbb {R} ^{n}$ . Then we define from this

$A:={\begin{pmatrix}a_{1,1}&\cdots &a_{1,n}\\\vdots &\ddots &\vdots \\a_{n,1}&\cdots &a_{n,n}\end{pmatrix}}$

and note that for any vector ${\vec {v}}=(v_{1},\ldots ,v_{n})^{t}$ of $\mathbb {R} ^{n}$ we obtain

$A{\vec {v}}=A\left(\sum _{j=1}^{n}v_{j}{\vec {e}}_{j}\right)=\sum _{j=1}^{n}Av_{j}{\vec {e}}_{j}=\sum _{j=1}^{n}v_{j}L({\vec {e}}_{j})=L\left(\sum _{j=1}^{n}v_{j}{\vec {e}}_{j}\right)=L({\vec {v}})$

Thus, we have shown existence. To prove uniqueness, suppose there were any other matrix $B\in \mathbb {R} ^{m\times n}$ with the property that $\forall {\vec {v}}\in \mathbb {R} ^{n}:L({\vec {v}})=B{\vec {v}}$ . Then in particular,

$B{\vec {e}}_{j}=L({\vec {e}}_{j})$

which already implies that $A=B$ (since all the columns of both matrices are identical).$\Box$

How to generalise the derivative

It is not immediately straightforward how one would generalize the derivative to higher dimensions. For, if we take the definition of the derivative at a point $x_{0}$

$\lim _{h\to 0}{\frac {f(x_{0}+h)-f(x_{0})}{h}}$

and insert vectors for $h$ and $x_{0}$ , we would divide the whole thing by a vector. But this is not defined.

Hence, we shall rephrase the definition of the derivative a bit and cast it into a form where it can be generalized to higher dimensions.

Theorem

Let $f:\mathbb {R} \to \mathbb {R}$ be a one-dimensional function and let $x_{0}\in \mathbb {R}$ . Then $f$ is differentiable at $x_{0}$ if and only if there exists a linear function $l:\mathbb {R} \to \mathbb {R}$ such that

$\lim _{h\to 0}{\frac {{\Big |}f(x_{0}+h)-{\big (}f(x_{0})+l(h){\big )}{\Big |}}{|h|}}=0$

We note that according to the above, linear functions $l:\mathbb {R} \to \mathbb {R}$ are given by multiplication by a $1\times 1$-matrix, that is, a scalar.

Proof

First assume that $f$ is differentiable at $x_{0}$. We set $l(h):=f'(x_{0})\cdot h$ and obtain

${\frac {{\Big |}f(x_{0}+h)-{\big (}f(x_{0})+l(h){\big )}{\Big |}}{|h|}}=\left|{\frac {f(x_{0}+h)-f(x_{0})}{h}}-f'(x_{0})\right|$

which converges to 0 due to the definition of $f'(x_{0})$ .

Assume now that we are given an $l:\mathbb {R} \to \mathbb {R}$ such that

$\lim _{h\to 0}{\frac {{\Big |}f(x_{0}+h)-{\big (}f(x_{0})+l(h){\big )}{\Big |}}{|h|}}=0$

Let $c$ be the scalar associated to $l$ . Then by an analogous computation $f'(x_{0})=c$ .$\Box$

With the latter formulation of differentiability from the above theorem, we may readily generalize to higher dimensions, since division by the Euclidean norm of a vector is defined, and linear mappings are also defined in higher dimensions.

Definition

A function $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ is called differentiable or totally differentiable at a point $x_{0}\in \mathbb {R} ^{m}$ if and only if there exists a linear function $L:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ such that

$\lim _{{\vec {h}}\to 0}{\frac {{\Big \|}f(x_{0}+{\vec {h}})-{\big (}f(x_{0})+L({\vec {h}}){\big )}{\Big \|}}{\|{\vec {h}}\|}}=0$

We have already proven that this definition coincides with the usual one in the one-dim. case (that is $m=n=1$).

We have the following theorem:

Theorem

Let $S\subseteq \mathbb {R} ^{m}$ be a set, let $x_{0}\in {\overset {\circ }{S}}$ be an interior point of $S$ , and let $f:S\to \mathbb {R} ^{m}$ be a function differentiable at $x_{0}$ . Then the linear map $L$ such that

$\lim _{{\vec {h}}\to 0}{\frac {{\Big \|}f(x_{0}+{\vec {h}})-{\big (}f(x_{0})+L({\vec {h}}){\big )}{\Big \|}}{\|{\vec {h}}\|}}=0$

is unique; that is, there exists only one such map $L$ .

Proof

Since $x_{0}$ is an interior point of $S$, we find $r>0$ such that $B_{r}(x_{0})\subseteq S$ . Let now $K:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be any other linear mapping with the property that

$\lim _{{\vec {h}}\to 0}{\frac {{\Big \|}f(x_{0}+{\vec {h}})-{\big (}f(x_{0})+K({\vec {h}}){\big )}{\Big \|}}{\|{\vec {h}}\|}}=0$

We note that for all vectors of the standard basis $\{e_{1},\ldots ,e_{n}\}$ , the numbers $\lambda e_{j}$ for $0\leq \lambda <r$ are contained within $S$ . Hence, we obtain by the triangle inequality

${\Big \|}L({\vec {e}}_{j})-K({\vec {e}}_{j}){\Big \|}={\frac {{\bigl \|}L(\lambda {\vec {e}}_{j})-K(\lambda {\vec {e}}_{j}){\bigr \|}}{\|\lambda {\vec {e}}_{j}\|}}\leq {\frac {{\Big \|}f(x_{0}+\lambda {\vec {e}}_{j})-{\big (}f(x_{0})+L(\lambda {\vec {e}}_{j}){\big )}{\Big \|}}{\|\lambda {\vec {e}}_{j}\|}}+{\frac {{\Big \|}f(x_{0}+\lambda {\vec {e}}_{j})-{\big (}f(x_{0})+K(\lambda {\vec {e}}_{j}){\big )}{\Big \|}}{\|\lambda {\vec {e}}_{j}\|}}$

Taking $\lambda \to 0$ , we see that $L({\vec {e}}_{j})=K({\vec {e}}_{j})$ . Thus, $L$ and $K$ coincide on all basis vectors, and since every other vector can be expressed as a linear combination of those, by linearity of $L$ and $K$ we obtain $L=K$ .$\Box$

Thus, the following definition is justified:

Definition

Let $f:S\to \mathbb {R} ^{n}$ be a function (where $S\subseteq \mathbb {R} ^{m}$ is a subset of $\mathbb {R} ^{m}$), and let $x_{0}$ be an interior point of $S$ such that $f$ is differentiable at $x_{0}$ . Then the unique linear function $L$ such that

$\lim _{{\vec {h}}\to 0}{\frac {{\Big \|}f(x_{0}+{\vec {h}})-{\big (}f(x_{0})+L({\vec {h}}){\big )}{\Big \|}}{\|{\vec {h}}\|}}=0$

is called the differential of $f$ at $x_{0}$ and is denoted $f(x_{0}):=L$ .

Directional and partial derivatives

We shall first define directional derivatives.

Definition

Let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function, and let ${\vec {v}}\in \mathbb {R} ^{m}$ be a vector. If the limit

$\lim _{h\to 0}{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{h}}$

exists, it is called directional derivative of $f$ in direction ${\vec {v}}$ . We denote it by $D_{\vec {v}}f(x_{0})$ .

The following theorem relates directional derivatives and the differential of a totally differentiable function:

Theorem

Let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function that is totally differentiable at $x_{0}$, and let ${\vec {v}}\in \mathbb {R} ^{m}\setminus \{0\}$ be a nonzero vector. Then $D_{\vec {v}}f(x_{0})$ exists and is equal to $f'(x_{0}){\vec {v}}$ .

Proof

According to the very definition of total differentiability,

$\lim _{h\to 0}\left\|{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{|h|\cdot \|{\vec {v}}\|}}-{\frac {f'(x_{0}){\vec {v}}}{|h|\cdot \|{\vec {v}}\|}}\right\|=0$

Hence,

$\lim _{h\to 0}\left\|{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{|h|}}-{\frac {f'(x_{0}){\vec {v}}}{|h|}}\right\|=0$

by multiplying the above equation by $\|{\vec {v}}\|$ . Noting that

$\left\|{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{|h|}}-{\frac {f'(x_{0}){\vec {v}}}{|h|}}\right\|=\left\|{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{h}}-{\frac {f'(x_{0}){\vec {v}}}{h}}\right\|$

the theorem follows.$\Box$

A special case of directional derivatives are partial derivatives:

Definition

Let $\{{\vec {e}}_{1},\ldots ,{\vec {e}}_{m}\}$ be the standard basis of $\mathbb {R} ^{m}$ , let $x_{0}\in \mathbb {R} ^{m}$ and let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function such that the directional derivatives $D_{{\vec {e}}_{j}}f(x_{0})$ all exist. Then we set

${\frac {\partial f}{\partial x_{j}}}:=D_{{\vec {e}}_{j}}f(x_{0})$

and call it the partial derivative in the direction of $x_{j}$ .

In fact, by writing down the definition of $D_{{\vec {e}}_{j}}f(x_{0})$ , we see that the partial derivative in the direction of $x_{j}$ is nothing else than the derivative of the function $y\mapsto f(x_{0,1},\ldots ,x_{0,j-1},y,x_{0,j+1},\ldots ,x_{0,m})$ in the variable $y$ at the place $x_{0,j}$ . That is, for instance, if

$f(x,y,z)=x^{2}+4z^{3}+3xy$

then

${\frac {\partial f}{\partial x}}=2x+3y\ ,\ {\frac {\partial f}{\partial y}}=3x\ ,\ {\frac {\partial f}{\partial z}}=12z^{2}$

that is, when forming a partial derivative, we regard the other variables as constant and derive only with respect to the variable we are considering.

The Jacobian matrix

From the above, we know that the differential of a function $f'(x_{0})$ has an associated matrix representing the linear map thus defined. Under a condition, we can determine this matrix from the partial derivatives of the component functions.

Theorem

Let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function such that all partial derivatives exist at $x_{0}$ and are continuous in each component on $B_{r}(x_{0})$ for a possibly very small, but positive $r>0$ . Then $f$ is totally differentiable at $x_{0}$ and the differential of $f$ is given by left multiplication by the matrix

$J_{f}(x_{0}):={\begin{pmatrix}{\dfrac {\partial f_{1}}{\partial x_{1}}}&\cdots &{\dfrac {\partial f_{1}}{\partial x_{m}}}\\\vdots &\ddots &\vdots \\{\dfrac {\partial f_{n}}{\partial x_{1}}}&\cdots &{\dfrac {\partial f_{n}}{\partial x_{m}}}\end{pmatrix}}$

where $f=(f_{1},\ldots ,f_{n})$ .

The matrix $J_{f}(x_{0})$ is called the Jacobian matrix.

Proof

${\frac {{\Big \|}f(x_{0}+{\vec {h}})-(f(x_{0})+J_{f}(x_{0}){\vec {h}}){\Big \|}}{\|{\vec {h}}\|}}$	$={\frac {\left\|\displaystyle \sum _{j=1}^{n}f_{j}(x_{0}+{\vec {h}}){\vec {e}}_{j}-\sum _{j=1}^{n}\left(f_{j}(x_{0})+\sum _{k=1}^{m}h_{k}{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0})\right){\vec {e}}_{j}\right\|}{\|{\vec {h}}\|}}$
	$\leq \sum _{j=1}^{n}{\frac {\left\|f_{j}(x_{0}+h)-\left(f_{j}(x_{0})+\displaystyle \sum _{k=1}^{m}h_{k}{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0})\right)\right\|}{\|{\vec {h}}\|}}$

We shall now prove that all summands of the last sum go to 0.

Indeed, let $j\in \{1,\ldots ,n\}$ . Writing again ${\vec {h}}=(h_{1},\ldots ,h_{m})$ , we obtain by the one-dimensional mean value theorem, first applied in the first variable, then in the second and so on, the succession of equations

$f_{j}(x_{0}+h_{1}{\vec {e}}_{1})-f_{j}(x_{0})=\overbrace {(x_{0,1}+h_{1}-x_{0,1})} ^{=h_{1}}{\frac {\partial f_{j}}{\partial x_{1}}}(x_{0}+t_{1}{\vec {e}}_{1})$

$f_{j}(x_{0}+h_{1}{\vec {e}}_{1}+h_{2}{\vec {e}}_{2})-f_{j}(x_{0}+h_{1}{\vec {e}}_{1})=\overbrace {(x_{0,2}+h_{2}-x_{0,2})} ^{=h_{2}}{\frac {\partial f_{j}}{\partial x_{2}}}(x_{0}+h_{1}{\vec {e}}_{1}+t_{2}{\vec {e}}_{2})$

$\vdots$

$f_{j}(x_{0}+h_{1}{\vec {e}}_{1}+\cdots +h_{m}{\vec {e}}_{m})-f_{j}(x_{0}+h_{1}{\vec {e}}_{1}+\cdots +h_{m-1}{\vec {e}}_{m-1})=\overbrace {(x_{0,m}+h_{m}-x_{0,m})} ^{=h_{m}}{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0}+h_{1}{\vec {e}}_{1}+\cdots +h_{m-1}{\vec {e}}_{m-1}+t_{n}{\vec {e}}_{m})$

for suitably chosen $t_{k}\in [x_{0,k},x_{0,k}+h_{k}]$ . We can now sum all these equations together to obtain

$f_{j}(x_{0}+{\vec {h}})-f(x_{0})=\sum _{k=1}^{m}h_{k}{\frac {\partial f_{j}}{\partial x_{k}}}\left(x_{0}+\sum _{l=1}^{k-1}h_{l}{\vec {e}}_{l}+t_{k}{\vec {e}}_{k}\right)$

Let now $\delta >0$ . Using the continuity of the ${\frac {\partial f_{j}}{\partial x_{k}}}$ on $B_{r}(x_{0})$ , we may choose $\delta _{k}>0$ such that

$\left|{\frac {\partial f_{j}}{\partial x_{k}}}\left(x_{0}+\sum _{l=1}^{k-1}h_{l}{\vec {e}}_{l}+t_{k}e_{k}\right)-{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0})\right|<{\frac {\epsilon }{m}}$

for $|h_{k}|<\delta _{k}$ , given that ${\vec {h}}\in B_{r}(0)$ (which we may assume as ${\vec {h}}\to {\vec {0}}$). Hence, we obtain

${\frac {\left\|f_{j}(x_{0}+h)-\left(f_{j}(x_{0})+\displaystyle \sum _{k=1}^{m}h_{k}{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0})\right)\right\|}{\|{\vec {h}}\|}}\leq {\frac {\|{\vec {h}}\|\cdot m\cdot {\frac {\epsilon }{m}}}{\|{\vec {h}}\|}}$

and thus the theorem.$\Box$

Corollary

If $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ is continuously differentiable at $x_{0}\in \mathbb {R} ^{m}$ and ${\vec {v}}\in \mathbb {R} ^{m}\setminus \{0\}$ , then

$D_{\vec {v}}f(x_{0})=\sum _{j=1}^{m}v_{j}{\frac {\partial f}{\partial x_{j}}}(x_{0})$

Proof

$D_{\vec {v}}f(x_{0})=f'(x_{0})({\vec {v}})=J_{f}(x_{0}){\vec {v}}=\sum _{j=1}^{m}v_{j}{\frac {\partial f}{\partial x_{j}}}(x_{0})$$\Box$