Identities

Vector-by-Vector:

• Identity: $\frac{\partial \mathbf{y}}{\partial \mathbf{x}} = \begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \cdots & \frac{\partial y_1}{\partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial y_m}{\partial x_1} & \cdots & \frac{\partial y_m}{\partial x_n} \end{bmatrix}$
• Example: Jacobian matrix in transformations.
• Footnote: Used in backpropagation for updating weights in neural networks.

Vector-by-Scalar:

• Identity: $\frac{\partial \mathbf{y}}{\partial x} = \begin{bmatrix} \frac{\partial y_1}{\partial x} \ \vdots \ \frac{\partial y_m}{\partial x} \end{bmatrix}$
• Example: Gradient of a vector with respect to a scalar.
• Footnote: Applied in gradient descent for optimizing single parameters.

Scalar-by-Vector:

• Identity: $\frac{\partial y}{\partial \mathbf{x}} = \begin{bmatrix} \frac{\partial y}{\partial x_1} & \cdots & \frac{\partial y}{\partial x_n} \end{bmatrix}$
• Example: Gradient vector of a scalar function.
• Footnote: Fundamental in computing the gradients for loss functions.

Scalar-by-Scalar:

• Identity: Simply the regular derivative $\frac{\partial y}{\partial x}$
• Example: Standard calculus derivative.
• Footnote: Basic building block in gradient calculations.

Scalar-by-Matrix:

• Identity: $\frac{\partial y}{\partial \mathbf{X}} = \begin{bmatrix} \frac{\partial y}{\partial X_{11}} & \cdots & \frac{\partial y}{\partial X_{1n}} \ \vdots & \ddots & \vdots \ \frac{\partial y}{\partial X_{m1}} & \cdots & \frac{\partial y}{\partial X_{mn}} \end{bmatrix}$
• Example: Derivative of a scalar with respect to each element of a matrix.
• Footnote: Used in convolutional neural networks for filter updates.

Matrix-by-Scalar:

• Identity: $\frac{\partial \mathbf{Y}}{\partial x} = \begin{bmatrix} \frac{\partial Y_{11}}{\partial x} & \cdots & \frac{\partial Y_{1n}}{\partial x} \ \vdots & \ddots & \vdots \ \frac{\partial Y_{m1}}{\partial x} & \cdots & \frac{\partial Y_{mn}}{\partial x} \end{bmatrix}$
• Example: Scaling of a matrix by a scalar derivative.
• Footnote: Relevant in scaling layers of neural networks.

Python Code for an Example Identity:

import numpy as np

# Example: Scalar-by-Vector derivative
def scalar_by_vector_derivative(f, x):
    """
    Compute the gradient of a scalar function f with respect to a vector x.
    """
    grad = np.zeros_like(x)
    for i in range(len(x)):
        grad[i] = (f(x + np.eye(len(x))[i] * 1e-5) - f(x)) / 1e-5
    return grad

# Example function and vector
f = lambda x: np.sum(x**2)
x = np.array([1.0, 2.0, 3.0])

# Compute derivative
print(scalar_by_vector_derivative(f, x))

This script calculates the gradient of a scalar function with respect to a vector using a numerical approximation method. Such techniques are foundational in gradient-based optimization algorithms in deep learning.