Minima and Maxima of a function

In this chapter, we will explore the concept of finding minima and maxima of a function, a crucial topic in differential calculus. We will discuss the basics of local and global extrema, how to determine whether a point is a minimum or maximum, and provide practical examples along with Python code for optimization. Additionally, we will touch upon the application of this concept in deep learning.

Understanding Extrema

Extrema refer to the minimum and maximum values of a function within a specific domain. There are two types of extrema:

• Local Minimum/Maximum: These occur at a point within a small neighborhood of the function, where the function is either lower (local minimum) or higher (local maximum) than the surrounding values.
• Global Minimum/Maximum: These are the absolute lowest and highest values of a function over its entire domain.

Determining Local Extrema

To find local extrema, we look for points where the derivative of the function is zero or undefined. These points are known as critical points. We can use the first and second derivative tests to determine whether a critical point is a local minimum, maximum, or neither.

Example 1: Finding Local Extrema

Consider the function $f(x) = x^3 – 6x^2 + 9x + 1$. Let’s find its local extrema.

1. Calculate the first derivative: $f'(x) = 3x^2 – 12x + 9$.
2. Find critical points by setting $f'(x) = 0$:
   $$3x^2 – 12x + 9 = 0$$
   Solve this quadratic equation to get critical points.
3. Use the second derivative test to classify the critical points as minima, maxima, or neither.

Python Code:

import sympy as sp

x = sp.symbols('x')
f = x**3 - 6*x**2 + 9*x + 1
f_prime = sp.diff(f, x)
critical_points = sp.solve(f_prime, x)

for point in critical_points:
    f_double_prime = sp.diff(f_prime, x).subs(x, point)
    if f_double_prime > 0:
        print(f"Local Minimum at x = {point}")
    elif f_double_prime < 0:
        print(f"Local Maximum at x = {point}")
    else:
        print(f"Neither minimum nor maximum at x = {point}")

Application in Deep Learning

In deep learning, finding minima and maxima is essential in training neural networks. Optimization algorithms like gradient descent aim to minimize a loss function by iteratively adjusting model parameters. The minima of the loss function represent optimal model weights, leading to improved model performance.

Understanding extrema helps deep learning practitioners fine-tune models effectively, enabling the creation of more accurate and efficient neural networks.