Jacobian

The Jacobian matrix is a fundamental concept in the field of mathematics, particularly in vector calculus, and plays a critical role in deep learning, especially in backpropagation algorithms and optimization problems.

The Jacobian matrix, denoted as $\mathbf{J}$, is a matrix of all first-order partial derivatives of a vector-valued function. Consider a function $\mathbf{f}: \mathbb{R}^n \rightarrow \mathbb{R}^m$ that maps an $n$-dimensional vector to an $m$-dimensional vector. The Jacobian matrix of this function is defined as:

$$
\mathbf{J} =
\begin{bmatrix}
\frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\
\vdots & \ddots & \vdots \\
\frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n}
\end{bmatrix}
$$

Example

Let’s consider a function $\mathbf{f}: \mathbb{R}^2 \rightarrow \mathbb{R}^2$ defined by:

$$
\mathbf{f}(x, y) =
\begin{bmatrix}
x^2y \\
5x + \sin(y)
\end{bmatrix}
$$

The Jacobian matrix of $\mathbf{f}$ is:

$$
\mathbf{J} =
\begin{bmatrix}
\frac{\partial (x^2y)}{\partial x} & \frac{\partial (x^2y)}{\partial y} \\
\frac{\partial (5x + \sin(y))}{\partial x} & \frac{\partial (5x + \sin(y))}{\partial y}
\end{bmatrix} $$

$$
\begin{bmatrix}
2xy & x^2 \\
5 & \cos(y)
\end{bmatrix}
$$

Python Code Example

Let’s implement the above example in Python using the numpy and sympy libraries.

import numpy as np
import sympy as sp

x, y = sp.symbols('x y')
f1 = x**2 * y
f2 = 5*x + sp.sin(y)

# Jacobian Matrix
J = sp.Matrix([[f1.diff(x), f1.diff(y)], 
               [f2.diff(x), f2.diff(y)]])
J

Application in Deep Learning

In deep learning, the Jacobian matrix is extensively used in backpropagation. It helps in calculating gradients of the error with respect to the weights, which are then used to update the weights to minimize the error. This process is fundamental in training neural networks. Understanding the Jacobian is crucial for designing and improving deep learning models, especially in tasks involving complex architectures and loss functions.

Footnote: The concept of the Jacobian matrix has been pivotal in the development of advanced optimization algorithms in deep learning, such as RMSprop and Adam, which rely on gradient-based optimization techniques.

Math for Deep Learning

/

Viewing the Jacobian as a stack of gradients gives us a clue as to what it represents. Recall, the gradient of a scalar field, a function accepting a vector argument and returning a scalar, points in the direction of the maximum change in the function. Similarly, the Jacobian gives us information about how the vector-valued function behaves in the vicinity of some point, $x_p$.
The Jacobian is to vector-valued functions of vectors what the gradient is to scalar-valued functions of vectors; it tells us about how the function changes for a small change in the position of $x_p$.