Calculus/The chain rule and Clairaut's theorem

Theorem

Let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function and let $g:\mathbb {R} ^{n}\to \mathbb {R} ^{l}$ be another function. Assume that $g$ is differentiable at $x_{0}$ and that $f$ is differentiable at $g(x_{0})$ .

Then $f\circ g$ is differentiable at $x_{0}$ and

$(f\circ g)'(x_{0})=f'{\big (}g(x_{0}){\big )}\circ g'(x_{0})$

Proof

We prove that $f'(g(x_{0}))\circ g'(x_{0})$ is a valid differential of $f\circ g$ , thereby proving differentiability.

We begin by noting from the second triangle inequality, that

$\left|{\frac {{\big \|}g(x_{0}+\mathbf {h} )-g(x_{0}){\big \|}}{\|\mathbf {h} \|}}-{\frac {\|g'(x_{0})\mathbf {h} \|}{\|\mathbf {h} \|}}\right|\leq {\frac {{\Big \|}g(x_{0}+\mathbf {h} )-{\big [}g(x_{0})+g'(x_{0})\mathbf {h} {\big ]}{\Big \|}}{\|\mathbf {h} \|}}\to 0,\mathbf {h} \to 0$

and hence the boundedness of

${\frac {\|g'(x_{0})\mathbf {h} \|}{\|\mathbf {h} \|}}\leq {\frac {mn\max \limits _{1\leq i\leq n \atop 1\leq j\leq m}|a_{i,j}|\|\mathbf {h} \|}{\|\mathbf {h} \|}}$

implies that of

${\frac {{\big \|}g(x_{0}+\mathbf {h} )-g(x_{0}){\big \|}}{\|\mathbf {h} \|}}$

where $A=(a_{i,j})_{1\leq i\leq n \atop 1\leq j\leq m}$ is the matrix of $g'(x_{0})$ .

Now we note by the triangle inequality, that

${\begin{aligned}&{\frac {{\Big \|}(f\circ g)(x_{0}+\mathbf {h} )-{\big [}(f\circ g)(x_{0})+f'{\big (}g(x_{0}){\big )}g'(x_{0})\mathbf {h} {\big ]}{\Big \|}}{\|\mathbf {h} \|}}\\&\leq {\frac {{\bigg \|}f{\big (}g(x_{0}+h){\big )}-{\Big [}f{\big (}g(x_{0}){\big )}+f'{\big (}g(x_{0}){\big )}{\big [}g(x_{0}+h)-g(x_{0}){\big ]}{\Big ]}{\bigg \|}}{\|\mathbf {h} \|}}\\&+{\frac {{\bigg \|}f{\big (}g(x_{0}){\big )}+f'{\big (}g(x_{0}){\big )}{\big [}g(x_{0}+h)-g(x_{0}){\big ]}-{\Big [}(f\circ g)(x_{0})+f'{\big (}g(x_{0}){\big )}g'(x_{0})\mathbf {h} {\Big ]}{\bigg \|}}{\|\mathbf {h} \|}}\end{aligned}}$

We shall first treat the first summand, which is more difficult, but not so difficult still. We rewrite it as

${\frac {{\bigg \|}f{\big (}g(x_{0}+h){\big )}-{\Big [}f{\big (}g(x_{0}){\big )}+f'{\big (}g(x_{0}){\big )}{\big [}g(x_{0}+h)-g(x_{0}){\big ]}{\Big ]}{\bigg \|}}{{\big \|}g(x_{0}+h)-g(x_{0}){\big \|}}}\cdot {\frac {{\big \|}g(x_{0}+h)-g(x_{0}){\big \|}}{\|\mathbf {h} \|}}$

The latter factor is bounded due to the above considerations, and the first one converges to 0 as $h\to 0$ (and thus $g(x_{0}+h)\to g(x_{0})$ due to the same boundedness (multiply with $\|h\|$); in fact, differentiability thus implies continuity).

Now for the second summand, which, by elementary cancellation and linearity of differentials, equals

${\frac {{\bigg \|}f'{\big (}g(x_{0}){\big )}{\Big [}{\big [}g(x_{0}+h)-g(x_{0}){\big ]}-g'(x_{0})\mathbf {h} {\Big ]}{\bigg \|}}{\|\mathbf {h} \|}}\leq mn\max _{1\leq i\leq l \atop 1\leq j\leq n}|b_{i,j}|{\frac {{\Big \|}{\big [}g(x_{0}+h)-g(x_{0}){\big ]}-g'(x_{0})\mathbf {h} {\Big \|}}{\|\mathbf {h} \|}}$

where $B=(b_{i,j})_{1\leq i\leq l \atop 1\leq j\leq n}$ is the matrix of the differential of $f$ . This goes to 0 as $\mathbf {h} \to 0$ due to the definition of the differential of $f$.$\Box$

The gradient

The first application of the chain rule that we shall present has something to do with a thing called gradient, which is defined for functions $f:\mathbb {R} ^{n}\to \mathbb {R}$, that is, the image is one-dimensional (in the special case $n=2$ these functions look like "mountains" of a function on the plane $\mathbb {R} ^{2}$).

Definition

Let $f:\mathbb {R} ^{n}\to \mathbb {R}$ be differentiable. Then the column vector

$\nabla f(x):={\begin{pmatrix}{\dfrac {\partial f}{\partial x_{1}}}\\\vdots \\{\dfrac {\partial f}{\partial x_{n}}}\end{pmatrix}}$

is called the gradient.

Theorem:

Let $f,g:\mathbb {R} ^{n}\to \mathbb {R}$ be two functions totally differentiable at $x_{0}$. Since they both map to $\mathbb {R}$, their product is defined, and we have

$\nabla (fg)=f\nabla g+g\nabla f$

Proof:

Now one could compute this directly from the definition of the gradient and the usual one-dimensional product rule (which actually has the merit of not requiring total differentiability), but there is a clever trick using the chain rule, which I found in Terence Tao's lecture notes, on which I based my repetition of this part of mathematics.

We simply define $h:\mathbb {R} ^{n}\to \mathbb {R} ^{2},h(x):={\big (}f(x),g(x){\big )}$ and $i:\mathbb {R} ^{2}\to \mathbb {R} ,i(x,y):=x\cdot y$. Then the function $f\times g$ equals $i\circ h$. Now the differential of $i$ is given by the Jacobian matrix

$J_{i}(x,y)=(y\ x)$

and the differential of $h$ is given by the Jacobian matrix

$J_{h}(x)={\begin{pmatrix}{\dfrac {\partial f}{\partial x_{1}}}(x)&\cdots &{\dfrac {\partial f}{\partial x_{n}}}(x)\\{\dfrac {\partial g}{\partial x_{1}}}(x)&\cdots &{\dfrac {\partial g}{\partial x_{n}}}(x)\end{pmatrix}}$

Hence, the product rule implies that the differential of $i\circ h$ at $x$ is given by

${\begin{pmatrix}g(x)&f(x)\end{pmatrix}}{\begin{pmatrix}{\dfrac {\partial f}{\partial x_{1}}}(x)&\cdots &{\dfrac {\partial f}{\partial x_{n}}}(x)\\{\dfrac {\partial g}{\partial x_{1}}}(x)&\cdots &{\dfrac {\partial g}{\partial x_{n}}}(x)\end{pmatrix}}$

and from the definition of the gradient we see that the differential is nothing but the transpose of the gradient (and vice versa, as taking transpose is idempotent).$\Box$

Now we shall use the chain rule to generalize a well-known theorem from one dimension, the mean value theorem, to several dimensions.

Theorem:

Let $f:\mathbb {R} ^{n}\to \mathbb {R}$ be totally differentiable, and let $a\neq b\in \mathbb {R} ^{n}$. Then there exists $t\in [0,1]$ such that

$f(b)-f(a)={\bigl \langle }b-a,\nabla f{\bigl (}tb+(1-t)a{\bigr )}{\bigr \rangle }$

where $\langle \cdot ,\cdot \rangle$ is the standard scalar product on $\mathbb {R} ^{n}$.

Proof:

This is actually a straightforward application of the chain rule.

We set

$g(\lambda ):=f{\bigl (}(1-\lambda )a+\lambda b{\bigr )}$

thus $g(0)=f(a)$ and $g(1)=f(b)$. By the one-dimensional mean-value theorem,

$g(1)-g(0)=g'(t)$

for a suitable $t\in [0,1]$. Now by the chain rule,

$g'(t)={\bigl \langle }b-a,\nabla f{\bigl (}tb+(1-t)a{\bigr )}{\bigr \rangle }$.$\Box$

Clairaut's theorem

The next theorem shows that the order of differentiation does not matter, provided that the considered function is sufficiently differentiable. We will not need the general chain rule or any of its consequences during the course of the proof, but we will use the one-dimensional mean-value theorem.

Theorem (Clairaut's theorem):

Let $f:\mathbb {R} ^{n}\to \mathbb {R}$ be such that the partial derivatives up to order 2 exist and are continuous. Then

$\partial _{i}\partial _{j}f=\partial _{j}\partial _{i}f$.

Proof:

We begin with the following lemma:

Lemma:

$\partial _{j}\partial _{i}f(x)=\lim _{\delta \to 0}{\frac {f(x)-f(x+\delta e_{i})-f(x+\delta e_{j})+f{\bigl (}x+\delta (e_{i}+e_{j}){\bigr )}}{\delta ^{2}}}$

Proof: We first apply the fundamental theorem of calculus to obtain that the above limit equals

$\lim _{\delta \to 0}{\frac {1}{\delta ^{2}}}\left(\int \limits _{0}^{\delta }\partial _{i}f(x+se_{i})ds-\int \limits _{0}^{\delta }\partial _{i}f(x+\delta e_{j}+se_{i})ds\right)$

Using integration by substitution and linearity of the integral, we may rewrite this as

$\lim _{\delta \to 0}{\frac {1}{\delta }}\int \limits _{0}^{1}{\bigl (}\partial _{i}f(x+\delta se_{i})-\partial _{i}f(x+\delta e_{j}+\delta se_{i}){\bigr )}ds$

Now we apply the mean value theorem in one variable to obtain

$\partial _{i}f(x+\delta se_{i})-\partial _{i}f(x+\delta e_{j}+\delta se_{i})=\delta \partial _{j}\partial _{i}f(x+\delta se_{i}+t_{\delta }e_{j})$

for a suitable $t_{\delta }\in [0,\delta ]$. Hence, the above limit equals

$\lim _{\delta \to 0}\int \limits _{0}^{1}\partial _{j}\partial _{i}f(x+\delta se_{i}+t_{\delta }e_{j})ds$

This is the average of $\partial _{j}\partial _{i}f$ over a certain subset of $B_{\delta }(x)$ and therefore converges to $\partial _{j}\partial _{i}f(x)$ by the continuity of $\partial _{j}\partial _{i}f$ (you can prove this rigorously by using

$\partial _{j}\partial _{i}f(x)=\int \limits _{0}^{1}\partial _{j}\partial _{i}f(x)ds$

and subtracting the integrals and applying the triangle inequality for integrals).$\Box$

Now the expression of the lemma is totally symmetric in $i$ and $j$, which is why Clairaut's theorem follows.$\Box$