Vector Norms (L-Norms and P-Norm)

Chapter 5: Vector Norms (L-Norms and P-Norms)

L-norms are a family of vector norms that are defined using the p-th power of the absolute values of vector components, raised to the 1/p power. The most commonly used L-norms include the L1-norm (Manhattan norm) and the L2-norm (Euclidean norm).

L1-Norm

The L1-norm of a vector $\mathbf{x} = [x_1, x_2, \ldots, x_n]$ is defined as:

$$|\mathbf{x}|_1 = |x_1| + |x_2| + \ldots + |x_n|$$

Let’s calculate the L1-norm of a vector using NumPy:

import numpy as np

x = np.array([2, -3, 5, -1, 0])
l1_norm = np.linalg.norm(x, ord=1)
print("L1-Norm of x:", l1_norm)

The output will be:

L1-Norm of x: 11.0

L2-Norm

The L2-norm of a vector $\mathbf{x}$ is defined as:

$$|\mathbf{x}|_2 = \sqrt{x_1^2 + x_2^2 + \ldots + x_n^2}$$

Let’s calculate the L2-norm of a vector using NumPy:

import numpy as np

x = np.array([2, -3, 5, -1, 0])
l2_norm = np.linalg.norm(x, ord=2)
print("L2-Norm of x:", l2_norm)

The output will be:

L2-Norm of x: 6.782329983125268

P-Norms

P-norms are a generalization of L-norms. The P-norm of a vector $\mathbf{x}$ is defined as:

$$|\mathbf{x}|_p = \left(|x_1|^p + |x_2|^p + \ldots + |x_n|^p\right)^{1/p}$$

You can choose any positive value for $p$ to compute the P-norm.

Application in Deep Learning

Vector norms, especially the L2-norm, are commonly used in deep learning. They are used in regularization techniques like L2 regularization (also known as weight decay) to prevent overfitting by adding a penalty term to the loss function. The L2-norm of weight vectors is often added to the loss function to encourage smaller weight values, leading to a simpler model with better generalization.

In summary, vector norms are fundamental in deep learning for regularization and understanding the magnitude of vectors in various mathematical operations. They provide valuable tools for managing the complexity of neural networks and improving their performance.