Partial and Mixed Partial Derivative

A partial derivative measures how a function’s output changes concerning one of its input variables while holding all other variables constant. Mathematically, the partial derivative of a function $f$ with respect to a variable $x_i$ is denoted as:

$$
\frac{\partial f}{\partial x_i}
$$

Let’s consider an example:

Example 1:

Suppose we have a function $f(x, y) = x^2 + 3xy – 2y^2$. We want to find $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$.

Calculation:

1. Partial derivative with respect to $x$ ($\frac{\partial f}{\partial x}$): Using the power rule, we get: $$
   \frac{\partial f}{\partial x} = 2x + 3y
   $$
2. Partial derivative with respect to $y$ ($\frac{\partial f}{\partial y}$): Applying the power rule again, we find: $$
   \frac{\partial f}{\partial y} = 3x – 4y
   $$

Mixed Partial Derivatives

Mixed partial derivatives involve taking partial derivatives of a function with respect to one variable and then with respect to another variable. It doesn’t matter in which order you take these derivatives. The notation for mixed partial derivatives is denoted as follows:

$$
\frac{\partial^2 f}{\partial x \partial y} \quad \text{and} \quad \frac{\partial^2 f}{\partial y \partial x}
$$

Example 2:

Let’s continue with our previous function $f(x, y) = x^2 + 3xy – 2y^2$. Now, we want to find $\frac{\partial^2 f}{\partial x \partial y}$ and $\frac{\partial^2 f}{\partial y \partial x}$.

Calculation:

1. Mixed partial derivative $\frac{\partial^2 f}{\partial x \partial y}$: First, find $\frac{\partial f}{\partial x}$ as calculated in Example 1. Then, take its partial derivative with respect to $y$: $$
   \frac{\partial^2 f}{\partial x \partial y} = \frac{\partial}{\partial y}(2x + 3y) = 3
   $$
2. Mixed partial derivative $\frac{\partial^2 f}{\partial y \partial x}$: Similarly, find $\frac{\partial f}{\partial y}$ as calculated in Example 1. Then, take its partial derivative with respect to $x$: $$
   \frac{\partial^2 f}{\partial y \partial x} = \frac{\partial}{\partial x}(3x – 4y) = 3
   $$

In both cases, we obtained the same result, which is 3.

Python Code for Partial Derivatives

To compute partial derivatives in Python, you can use libraries like SymPy or scipy. Here’s an example using SymPy:

import sympy as sp

x, y = sp.symbols('x y')
f = x**2 + 3*x*y - 2*y**2

# Partial derivative with respect to x
df_dx = sp.diff(f, x)

# Partial derivative with respect to y
df_dy = sp.diff(f, y)

print("Partial derivative with respect to x:", df_dx)
print("Partial derivative with respect to y:", df_dy)

Footnote

In deep learning, partial derivatives are prominently used in optimization algorithms, such as gradient descent and back-propagation. These algorithms rely on computing gradients, which are essentially vectors of partial derivatives. Gradients guide the model’s parameter updates, making partial derivatives a critical concept for training neural networks efficiently.

By understanding partial derivatives and their role in optimization, you can gain insights into how deep learning models learn and improve their performance during training.