Understanding Jacobians

The Jacobian Matrix

Consider a function that maps reals to reals, $f:\mathbb{R}\to \mathbb{R}$. The linear approximation of this function is given by $$f(a+h)\approx f(a)+h{f}^{\prime }(a)$$ This is pretty simple to do, and follows from taylor’s expansion upto the first order.

Let’s try expanding this concept to vector spaces. For a function $f:{\mathbb{R}}^n\to \mathbb{R}$, it’s linear approximation is given by $$f(aa+hh)\approx f(aa)+hh\cdot \mathrm{\nabla }f(aa)$$ (bold type indicates vectors). This also follows from the taylor theorem for multivariable functions.

What do we do when we have a function mapping vector spaces to vector spaces? Consider the function $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$. What would be the approximation term here? $$f(aa+hh)\approx f(aa)+???$$

Let’s try to solve this by decomposing the function $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$ into $f_i:{\mathbb{R}}^n\to \mathbb{R}$, $i=1,2,\cdots ,m$. For each $f_i$, we get $$f_i(aa+hh)\approx f_i(aa)+hh\cdot \mathrm{\nabla }{f}_i(aa)$$

If we collect all these approximations into a vector by representing the term $hh\cdot \mathrm{\nabla }{f}_i(aa)$ as a matrix product, we get our approximation for $f$:

$$\left[\begin{array}{c}{f}_1(aa+hh)\\ f_2(aa+hh)\\ ⋮\\ f_m(aa+hh)\end{array}\right]\approx \left[\begin{array}{c}{f}_1(aa)\\ f_2(aa)\\ ⋮\\ f_m(aa)\end{array}\right]+\left[\begin{array}{c}{\mathrm{\nabla }}^T{f}_1(aa)\\ {\mathrm{\nabla }}^T{f}_2(aa)\\ ⋮\\ {\mathrm{\nabla }}^T{f}_m(aa)\end{array}\right]\left[\begin{array}{c}{h}_1\\ h_2\\ ⋮\\ h_n\end{array}\right]$$

I’ve written the functions out in matrix form for clarity. ${\mathrm{\nabla }}^T$ denotes the transpose of the gradient vector. In simpler terms, we can rewrite this as $$f(aa+hh)\approx f(aa)+\mathrm{J}{\mathrm{J}}_f(aa)hh$$

Here, $\mathrm{J}{\mathrm{J}}_f(aa)$ is called the Jacobian matrix, and when expanded, it looks something like this:

$$\mathrm{J}\mathrm{J}=\left[\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& \cdots & \frac{\partial f_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial f_m}{\partial x_1}& \cdots & \frac{\partial f_m}{\partial x_n}\end{array}\right]$$

The Jacobian Matrix thus, is an analog of the gradient vector for functions that map vector spaces to vector spaces. Everything that we can do using gradients can be done in a more general form using the Jacobian Matrix. Consider the condition for differentiability of a multivariate scalar function $f:{\mathbb{R}}^n\to \mathbb{R}$: for $f$ to be differentiable at $aa$, we have the condition $$\underset{||hh||\to 0}{lim}\frac{f(aa+hh)-f(aa)-hh\cdot \mathrm{\nabla }f(aa)}{||hh||}=0$$ For a function $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$, the condition would be: $$\underset{||hh||\to 0}{lim}\frac{||f(aa+hh)-f(aa)-\mathrm{J}{\mathrm{J}}_f(aa)hh||}{||hh||}=0$$

here, $||\cdot ||$ is the euclidean norm of the vector.

Relating Jacobian Matrices and Transformation Matrices

If you notice, the Jacobian Matrix need not be square; for the special case $n=m$, that is for $f:{\mathbb{R}}^n\to {\mathbb{R}}^n$, the Jacobian matrix is square and acts as a linear transformation between the two n-dimensional vector spaces, as shown below: $$\left[\begin{array}{c}d{f}_1\\ ⋮\\ d{f}_n\end{array}\right]=\left[\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& \cdots & \frac{\partial f_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial f_n}{\partial x_1}& \cdots & \frac{\partial f_n}{\partial x_n}\end{array}\right]\left[\begin{array}{c}d{x}_1\\ ⋮\\ d{x}_n\end{array}\right]$$

Therefore, this acts like a linear transformation between the infinitesimal elements in the space of $f$ and in the space of $x$. One more proof of scaling is involved before the use of jacobian matrices in integrals becomes clear.

Scaling factor and the Jacobian Determinant

Recall that for any linear transformation, the determinant of the transformation gives us the scaling factor, that is the ratio of the change in ‘volume’ occupied by the vector. This is also known as a dilation transformation (because just the size is involved, without worrying about orientation).

I’ll provide a proof for the statement highlighted above in ${\mathbb{R}}^3$, but extending it to ${\mathbb{R}}^n$ is easy enough (an exercise for the reader, as they say in math textbooks :)). Consider the unit cube centered at origin, ie having it’s 3 vectors as $(1,0,0),(0,1,0)$ and $(0,0,1)$. On applying a transformation to this cube, we get the vertices as the transformation matrix itself. Here’s an image that speaks a thousand words (taken from Hop’s Blog):

In algebra terms, we get

$$\left[\begin{array}{ccc}a& x& l\\ b& y& m\\ c& z& n\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]=\left[\begin{array}{ccc}a& x& l\\ b& y& m\\ c& z& n\end{array}\right]$$

The volume of a parallelepiped is given by the determinants of the 3 edge vectors, and hence the volume of the transformed cube is $$V=\left|\begin{array}{ccc}a& x& l\\ b& y& m\\ c& z& n\end{array}\right|$$ Which is the determinant of the linear transform that we started with. $◼$

This is the main principle that allows us to use the Jacobian in multiple integrals while changing variables: it scales up or down the size of the area or volume element we are using proportionately to the change of variables.

References:

1. What is the Jacobian Matrix, a good MSE thread on the Jacobian Matrix
2. Jacobian Matrix and Determinant Wikipedia page, of course
3. A Quora question on Jacobian matrices with another very nice answer

Personal comments:

I thoroughly enjoyed writing so much ‘hard’ mathematics on this blog after a long time (last proper math post was on 11th September of Last year, and other math notes in the interim were published on the MathNotes site). A lot of calc textbooks don’t go into detail on Jacobians, instead just using them like a gift of god that fell out of the sky. The bare minimum they would provide would be a diagram of domain transformation, and the cliched example of converting to polar integrals (the disc is transformed into a rectangle), but that would still not make intuitive sense: why the determinant? And why this weird matrix? were the questions that popped up in my head, and I hope I’ve done justice to those questions in this article.