A formal Definition

Definition of Differentiation

Differentiation is a fundamental concept in calculus, particularly in the context of deep learning. It allows us to calculate the rate of change of a function with respect to its input variables. Mathematically, the derivative of a function (f(x)) at a specific point (x) is defined as:

$$
f'(x) = \lim_{h \to 0} \frac{f(x+h) – f(x)}{h}
$$

Here’s a breakdown of the notation:

• $f'(x)$ represents the derivative of $f(x)$ with respect to (x).
• $lim_{h \to 0}$ denotes the limit as $h$ approaches 0, indicating an infinitesimally small change in (x).
• $frac{f(x+h) – f(x)}{h}$ represents the difference quotient, which measures the rate of change of $f(x)$ as $x$ changes by a small amount $h$.

5.2 Example

Let’s illustrate differentiation with a simple example. Consider the function $f(x) = 2x^2 + 3x + 1$. We want to find its derivative with respect to $x$, $f'(x))$.

Using the definition of differentiation:

$$
f'(x) = \lim_{h \to 0} \frac{f(x+h) – f(x)}{h}
$$

Substitute (f(x) = 2x^2 + 3x + 1) into the equation:

$$
f'(x) = \lim_{h \to 0} \frac{(2(x+h)^2 + 3(x+h) + 1) – (2x^2 + 3x + 1)}{h}
$$

Now, let’s simplify the expression:

$$
f'(x) = \lim_{h \to 0} \frac{2x^2 + 4xh + 2h^2 + 3x + 3h + 1 – 2x^2 – 3x – 1}{h}
$$

Simplify further:

$$
f'(x) = \lim_{h \to 0} \frac{4xh + 2h^2 + 3h}{h}
$$

Now, factor out (h) from the numerator:

$$
f'(x) = \lim_{h \to 0} \frac{h(4x + 2h + 3)}{h}
$$

Finally, cancel out (h) from the numerator and denominator:

$$
f'(x) = \lim_{h \to 0} (4x + 2h + 3)
$$

Taking the limit as (h) approaches 0:

$$
f'(x) = 4x + 3
$$

So, the derivative of (f(x) = 2x^2 + 3x + 1) is (f'(x) = 4x + 3).

5.3 Python Code

Now, let’s implement the derivative of a function in Python using the sympy library:

import sympy as sp

# Define the variable and function
x = sp.symbols('x')
f = 2*x**2 + 3*x + 1

# Calculate the derivative
f_prime = sp.diff(f, x)
print("Derivative of f(x):", f_prime)

The code above uses sympy to symbolically compute the derivative of (f(x)) with respect to (x) and prints the result.

5.4 Application in Deep Learning

In deep learning, differentiation plays a crucial role in training neural networks. Specifically, gradient descent algorithms use derivatives to update the network’s parameters in the direction that minimizes the loss function. This process enables neural networks to learn from data and make predictions, making differentiation a fundamental concept in the field of deep learning.

In conclusion, understanding the formal definition of differentiation is essential for anyone venturing into deep learning, as it underpins many optimization techniques and algorithms used in the training of neural networks.