Tensors and Transformations

In the fields of Artificial Intelligence and Robotics, the term tensor is used in two distinct ways:

  1. Data Structures: In most AI frameworks (like PyTorch or TensorFlow), a tensor is simply a synonym for a multidimensional array. It is a generic container for numbers arranged in a grid with any number of axes.
  2. The Mathematical Perspective (Geometric Objects): In physics and classical engineering, a tensor is more than just a grid of numbers; it is a mathematical object that obeys specific transformation laws. A "true" tensor maintains a consistent relationship between components even when the underlying coordinate system is rotated or scaled (a linear transformation).
Context Definition Key Characteristic
Deep Learning Multidimensional Array Focuses on efficient storage and computation.
Physics / Robotics Geometric Object Focuses on invariance under coordinate transformation.

Tensors: The Data Structure

In modern AI, a Tensor is a generalized container for data. We categorize tensors by their rank, which describes the number of dimensions (or indices) required to access an element.

  • Scalar (Rank 0): A single number representing magnitude only (e.g., temperature, mass).
    • Notation: $s \in \mathbb{R}$
  • Vector (Rank 1): An ordered list of numbers representing magnitude and direction (e.g., a robot’s velocity in 3D space).
    • Notation: $\mathbf{v} \in \mathbb{R}^n$
  • Matrix (Rank 2): A rectangular array of numbers (e.g. covariance-matrix).
    • Notation: $\mathbf{A} \in \mathbb{R}^{m \times n}$
  • Rank 3-Tensor: For example, a color image is a Rank 3 tensor with dimensions $(\text{Height} \times \text{Width} \times \text{Channels})$.
    • Notation: $\boldsymbol{\mathcal{I}} \in \mathbb{R}^{h \times w \times c}$
  • $\dots$

The Transformation Perspective of Tensors

Imagine a robot arm physically rotating in a room. If the robot rotates its arm, the velocity vector of the gripper physically changes its direction relative to the floor. An active transformation describes this physical movement within a fixed coordinate system.

A "true" tensor is not just a collection of numbers, but a physical object that "knows" how to move. Its components transform according to specific linear rules so that the underlying physical relationship remains consistent after the movement.

Vectors (Rank 1 Tensors)

In an active rotation, we physically rotate the vector within a fixed coordinate system. If $\mathbf{R}$ is our rotation matrix, the new vector $\mathbf{x}'$ is given by

$$\mathbf{x}' = \mathbf{R}\mathbf{x}$$

In this active view, the vector "moves" within the space. The numbers change because the arrow is now pointing in a new direction.

Example: The 2D Rotation Matrix

A rotation matrix $\mathbf{R}$ in 2D takes a tensor and rotates it about the origin by an angle $\theta$, e.g. it rotates a vector without changing its length.

In 2D, the rotation matrix is defined as:

$$\mathbf{R}_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$$

If you multiply a vector (rank 1 tensor) $\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$ by this matrix, the new coordinates $\mathbf{x}'$ will be:

$$\mathbf{x}' = \mathbf{R}_\theta \mathbf{x} = \begin{bmatrix} x_1\cos\theta - x_2\sin\theta \\ x_1\sin\theta + x_2\cos\theta \end{bmatrix}$$

In [1]:
import numpy as np
import matplotlib.pyplot as plt

def plot_vectors(vectors, colors, title):
    plt.figure(figsize=(5, 5))
    plt.axhline(0, color='black', lw=1)
    plt.axvline(0, color='black', lw=1)
    for i, v in enumerate(vectors):
        plt.quiver(0, 0, v[0], v[1], angles='xy', scale_units='xy', scale=1, color=colors[i], label=f'v{i}')
    plt.xlim(-1.5, 1.5)
    plt.ylim(-1.5, 1.5)
    plt.grid(True, alpha=0.3)
    plt.title(title)
    plt.legend()
    plt.show()

# --- 1. Rotation Example ---
theta = np.radians(45) # 45 degrees
R = np.array([[np.cos(theta), -np.sin(theta)],
              [np.sin(theta),  np.cos(theta)]])

v_original = np.array([1, 0])
v_rotated = R @ v_original

print(f"Original: {v_original} -> Rotated: {v_rotated.round(2)}")
plot_vectors([v_original, v_rotated], ['blue', 'red'], "Rotation by 45°")

Original: [1 0] -> Rotated: [0.71 0.71]
No description has been provided for this image

Matrices (Rank 2 Tensors) as Linear Maps

A Rank 2 tensor (like the Inertia Tensor or a Stress Tensor) represents a relationship between two vectors.

Matrices as Linear Maps (Rank 2 Tensors): A Rank 2 tensor is an operator that maps an input vector $\mathbf{x}$ to an output vector $\mathbf{y}$. Think of it as a physical rule: "If the input force is $\mathbf{x}$, the resulting displacement is $\mathbf{y}$." Mathematically:

$$\mathbf{y} = \mathbf{A}\mathbf{x}$$

Active Rotation of the System: Now, imagine we physically rotate the entire setup (the input, the output, and the mechanism itself) by a rotation matrix $\mathbf{R}$.The rule for an active transformation of Rank 1 tensors (Vectors) is that they physically move to new positions:

$$\mathbf{x}' = \mathbf{R}\mathbf{x} \quad \text{and} \quad \mathbf{y}' = \mathbf{R}\mathbf{y}$$

We need to find the transformed matrix $\mathbf{A}'$ that describes how the rotated mechanism works. In this rotated state, the physical relationship must still hold: the rotated input $\mathbf{x}'$ must produce the rotated output $\mathbf{y}'$.

$$\mathbf{y}' = \mathbf{A}'\mathbf{x}'$$

To derive $\mathbf{A}'$, we substitute our active transformation rules into the original physical law:

  1. Start with the original rule: $\mathbf{y} = \mathbf{A}\mathbf{x}$
  2. Rotate the output vector: Multiply both sides by $\mathbf{R}$ to see where the new output points: $\mathbf{R}\mathbf{y} = \mathbf{R}(\mathbf{A}\mathbf{x})$
  3. Account for the rotated input: We know $\mathbf{x}' = \mathbf{R}\mathbf{x}$, which implies the original input was $\mathbf{x} = \mathbf{R}^{-1}\mathbf{x}'$. We substitute this in:$$\mathbf{R}\mathbf{y} = \mathbf{R}\mathbf{A}(\mathbf{R}^{-1}\mathbf{x}')$$
  4. Group the terms: $$\underbrace{\mathbf{R}\mathbf{y}}_{\mathbf{y}'} = (\mathbf{R}\mathbf{A}\mathbf{R}^{-1}) \underbrace{\mathbf{x}'}_{\mathbf{x}'}$$

By comparing this to our goal $\mathbf{y}' = \mathbf{A}'\mathbf{x}'$, the active transformation law for a Rank 2 tensor is revealed:

$$\mathbf{A}' = \mathbf{R}\mathbf{A}\mathbf{R}^{-1}$$

Note: For rotation matrices, the inverse is the transpose ($\mathbf{R}^{-1} = \mathbf{R}^T$), so we usually write:

$$\mathbf{A}' = \mathbf{R}\mathbf{A}\mathbf{R}^T$$

This "sandwich" product ensures that the matrix is rotated "on both sides"—once to handle the rotated input and once to produce the rotated output.

Example for Rank 2 Tensor: Covariance matrix

To explain why the covariance matrix is a Rank 2 tensor, we can look at it through the lens of variance. We first measure the spread of data along a chosen direction and see which expression naturally appears, and only then give that expression its name.

Variance Along a Direction

1. Projecting the Data Points

Imagine you have centered data points $\mathbf{x}_i$ (2D vectors)—centered meaning the mean has already been subtracted, so the cloud of points sits at the origin. You want to know their position along a specific direction defined by a unit vector $\mathbf{v}$. The scalar projection of a point $\mathbf{x}_i$ onto $\mathbf{v}$ is the dot product:

$$\text{projection} = \mathbf{v} \cdot \mathbf{x}_i = \mathbf{v}^T \mathbf{x}_i$$

2. Calculating the Variance

Variance is the average of the squared distances from the mean. Since our data is centered (mean is zero), the variance along direction $\mathbf{v}$ is the average of the squared projections:

$$\text{Variance}_{\mathbf{v}} = \frac{1}{n} \sum_i (\mathbf{v}^T \mathbf{x}_i)^2$$

Using the property $(ab)^2 = (ab)(ab)$, and knowing that a scalar is equal to its own transpose $(\mathbf{v}^T \mathbf{x}_i) = (\mathbf{x}_i^T \mathbf{v})$, we can rewrite the squared term:

$$(\mathbf{v}^T \mathbf{x}_i)^2 = (\mathbf{v}^T \mathbf{x}_i)(\mathbf{x}_i^T \mathbf{v})$$

3. Emerging the Covariance Matrix

Now, substitute this back into our summation:

$$\text{Variance}_{\mathbf{v}} = \frac{1}{n} \sum_i \mathbf{v}^T (\mathbf{x}_i \mathbf{x}_i^T) \mathbf{v}$$

Since $\mathbf{v}$ is a constant direction and doesn't depend on the individual data points $i$, we can pull it outside the summation:

$$\text{Variance}_{\mathbf{v}} = \mathbf{v}^T \left( \frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^T \right) \mathbf{v}$$

The term inside the parentheses is the definition of the Covariance Matrix $\mathbf{C}$. Thus:

$$\text{Variance}_{\mathbf{v}} = \mathbf{v}^T \mathbf{C} \mathbf{v}$$

Matrices as Quadratic Forms (Rank 2 Tensors)

The expression $\mathbf{v}^T \mathbf{C} \mathbf{v}$ we just derived is called a quadratic form.

What is a quadratic form?

In one dimension, a quadratic expression is something like $f(v) = c\,v^2$: a single number $v$ is squared and scaled by a constant $c$. A quadratic form generalizes this idea to vectors. Instead of a single variable we have a vector $\mathbf{v} \in \mathbb{R}^n$, and instead of a single coefficient $c$ we have a (symmetric) matrix $\mathbf{C} \in \mathbb{R}^{n \times n}$:

$$f(\mathbf{v}) = \mathbf{v}^T \mathbf{C} \mathbf{v} = \sum_i \sum_j C_{ij}\, v_i v_j$$

The result is always a single scalar. It is called quadratic because every term in the sum is a product of two components of $\mathbf{v}$ (total degree 2). For $n = 1$ it collapses back to $f(v) = C\,v^2$.

Why this explains the "Rank 2" Nature

A Rank 1 tensor (a vector $\mathbf{v}$) only carries a single direction. The quadratic form shows that the covariance matrix is instead a machine with two "slots" for vectors:

  1. The Right Slot ($\mathbf{C}\mathbf{v}$): This operation transforms the direction $\mathbf{v}$ based on the data's spread, creating a new vector that represents the "weighted" direction.
  2. The Left Slot ($\mathbf{v}^T [\dots]$): This dots that weighted direction back onto our original direction to get a single scalar number (the variance).

The rank of a tensor is exactly the number of vectors it must consume to return a scalar. A vector (Rank 1) needs just one partner: the dot product $\mathbf{v}^T \mathbf{u}$ uses a single vector to produce a number. The covariance matrix needs two vectors—one fed in on the right ($\mathbf{C}\mathbf{v}$) and one on the left ($\mathbf{v}^T[\dots]$)—before it collapses to the scalar $\mathbf{v}^T \mathbf{C} \mathbf{v}$. The same count shows up in the indices: a single entry $C_{ij}$ is addressed by two subscripts. Both ways of counting give two, and that is precisely what "Rank 2" means.

Rotation of the Covariance Matrix

When we apply an active rotation to our data points using a matrix $\mathbf{R}$, the "cloud" of data physically rotates in space. To describe this rotated cloud correctly, we must update the covariance matrix using the sandwich product:$$\mathbf{C}' = \mathbf{R} \mathbf{C} \mathbf{R}^T$$

This is not a special rule for covariance matrices: it is the general transformation law for a Rank 2 tensor that we derived earlier, $\mathbf{A}' = \mathbf{R}\mathbf{A}\mathbf{R}^T$. Every Rank 2 tensor rotates by this same "sandwich"—once on the right and once on the left—because it must transform consistently with the vectors fed into both of its slots. For the covariance matrix specifically, this exact rule is derived in detail at the end of the multivariate Gaussian distribution notebook.

This transformation ensures that the physical properties of the data—such as the Trace (the total variance) and the relative spread—remain unchanged, even though the individual numbers within the matrix have shifted to reflect the new orientation.

In [2]:
# 1. Define a simple covariance matrix (Higher variance in X than Y)
C = np.array([[2.0, 0.0], 
              [0.0, 0.5]])

# 2. Define a 45-degree Rotation Matrix (R)
# In 2D: [[cos(t), -sin(t)], [sin(t), cos(t)]]
theta = np.radians(45)
c, s = np.cos(theta), np.sin(theta)
R = np.array([[c, -s], 
              [s,  c]])

# 3. Apply the Rank 2 Transformation (The Sandwich)
C_prime = R @ C @ R.T

# 4. Results
print("Original Matrix (Variance aligned with X and Y):")
print(C)
print("\nRotated Matrix (Variance now spread across both axes):")
print(np.round(C_prime, 2))

# 5. The Invariant: Total Variance
print(f"\nOriginal Total Variance (Trace): {np.trace(C)}")
print(f"Rotated Total Variance (Trace):  {np.trace(C_prime)}")

Original Matrix (Variance aligned with X and Y):
[[2.  0. ]
 [0.  0.5]]

Rotated Matrix (Variance now spread across both axes):
[[1.25 0.75]
 [0.75 1.25]]

Original Total Variance (Trace): 2.5
Rotated Total Variance (Trace):  2.5

Why does this happen?

  • Original $\mathbf{C}$: The data is stretched only along the X and Y axes. The off-diagonal is $0$, meaning $x$ and $y$ are uncorrelated.
  • Rotated $\mathbf{C}'$: After a $45^\circ$ rotation, the "stretch" is now diagonal. Because the stretch is no longer aligned with the grid, the matrix now shows a covariance (non-zero off-diagonal elements).
  • The Invariant: Notice that the Trace (the sum of the diagonal) is exactly $2.5$ in both cases. The total amount of "shaking" in the data hasn't changed; we just changed the angle from which we are watching it.

Why the Trace is the "Total Variance"

The trace of a matrix is the sum of its diagonal entries, $\operatorname{tr}(\mathbf{C}) = \sum_i C_{ii}$. For a covariance matrix this sum has a concrete meaning.

The diagonal entries are per-axis variances. Each diagonal entry is the variance of one coordinate,

$$C_{ii} = \operatorname{Var}(x_i) = \mathbf{e}_i^T \mathbf{C}\, \mathbf{e}_i,$$

that is, the variance along the $i$-th coordinate axis $\mathbf{e}_i$ (our quadratic form $\mathbf{v}^T\mathbf{C}\mathbf{v}$ from before, evaluated in the axis direction $\mathbf{v} = \mathbf{e}_i$). Summing them over all axes gives

$$\operatorname{tr}(\mathbf{C}) = \sum_i \operatorname{Var}(x_i) = \sum_i \mathbb{E}\!\left[(x_i - \mu_i)^2\right] = \mathbb{E}\!\left[\sum_i (x_i - \mu_i)^2\right] = \mathbb{E}\!\left[\lVert \mathbf{x} - \boldsymbol{\mu}\rVert^2\right].$$

So the trace is the expected squared distance of the data from its mean—a single number that captures the total spread of the cloud over all directions at once. That is why we call it the total variance. (In our example, $\operatorname{tr}(\mathbf{C}) = 2.0 + 0.5 = 2.5$.)

Why it does not change under rotation. Rotating the cloud transforms the covariance matrix by the sandwich $\mathbf{C}' = \mathbf{R}\mathbf{C}\mathbf{R}^T$. Using the cyclic property of the trace, $\operatorname{tr}(\mathbf{A}\mathbf{B}) = \operatorname{tr}(\mathbf{B}\mathbf{A})$, together with $\mathbf{R}^T\mathbf{R} = \mathbf{I}$ for a rotation:

$$\operatorname{tr}(\mathbf{C}') = \operatorname{tr}(\mathbf{R}\mathbf{C}\mathbf{R}^T) = \operatorname{tr}(\mathbf{C}\mathbf{R}^T\mathbf{R}) = \operatorname{tr}(\mathbf{C}\,\mathbf{I}) = \operatorname{tr}(\mathbf{C}).$$

Geometrically this is obvious: a rotation moves no point closer to or farther from the center, so the total squared distance—and hence the total variance—must stay the same. The individual diagonal entries get reshuffled between the axes (in the example, from $2.0$ and $0.5$ to $1.25$ and $1.25$), but their sum is conserved.

The total variance can also be read as the sum of the eigenvalues of $\mathbf{C}$ (the variances along its principal axes); this is shown in the next chapter, the eigen-decomposition notebook.

In [3]:
import matplotlib.pyplot as plt
from matplotlib.patches import Ellipse

def draw_ellipse(mu, sigma, ax, n_std=1.0, edgecolor='black', label=None):
    # Eigen-decomposition to find the orientation and scale for the ellipse
    vals, vecs = np.linalg.eigh(sigma)
    order = vals.argsort()[::-1]
    vals, vecs = vals[order], vecs[:, order]
    theta = np.degrees(np.arctan2(*vecs[:, 0][::-1]))
    
    # The radius of the ellipse represents the standard deviation (sqrt of variance)
    width, height = 2 * n_std * np.sqrt(vals)
    ell = Ellipse(xy=mu, width=width, height=height, angle=theta,
                  edgecolor=edgecolor, facecolor='none', label=label, lw=2)
    return ax.add_patch(ell)


mu = np.array([0, 0])

# --- Vectors representing the variance axes ---
# Original axes lengths set to the std values, i.e. sqrt of (2.0 and 0.5)
v1_blue, v2_blue = np.array([np.sqrt(2.0), 0.0]), np.array([0.0, np.sqrt(0.5)])

# Rotated axes: transformed as Rank 1 tensors (Vectors)
v1_red, v2_red = R @ v1_blue, R @ v2_blue

# --- Plotting ---
fig, ax = plt.subplots(figsize=(7, 7))

# 1. Draw the Probability Distributions (Ellipses)
draw_ellipse(mu, C, ax, edgecolor='blue', label=r'Original Covariance $\mathbf{C}$')
draw_ellipse(mu, C_prime, ax, edgecolor='red', label=r'Rotated Covariance $\mathbf{C}\prime$')

# 2. Draw the Basis Vectors (Scaled to Variance)
# Original Basis (Blue)
ax.quiver(0, 0, v1_blue[0], v1_blue[1], color='blue', scale=1, 
          scale_units='xy', angles='xy', alpha=0.5, label='Orig. Std. Axes')
ax.quiver(0, 0, v2_blue[0], v2_blue[1], color='blue', scale=1, 
          scale_units='xy', angles='xy', alpha=0.5)

# Rotated Basis (Red)
ax.quiver(0, 0, v1_red[0], v1_red[1], color='red', scale=1, 
          scale_units='xy', angles='xy', alpha=0.8, label='Rotated Std. Axes')
ax.quiver(0, 0, v2_red[0], v2_red[1], color='red', scale=1, 
          scale_units='xy', angles='xy', alpha=0.8)

# Formatting
ax.set_xlim(-2.5, 2.5)
ax.set_ylim(-2.5, 2.5)
ax.set_aspect('equal')
ax.axhline(0, color='black', lw=1, alpha=0.2)
ax.axvline(0, color='black', lw=1, alpha=0.2)
ax.set_title(r"Rank 2 Tensor: $\mathbf{C}' = \mathbf{R}\mathbf{C}\mathbf{R}^T$ (45° Rotation)")
ax.legend()
ax.grid(True, linestyle=':', alpha=0.5)
plt.show()
No description has been provided for this image

Invariance: The Core of Tensor Theory

The most important takeaway from the transformation perspective is Invariance.

While the individual components (the numbers in the array) change depending on the basis you choose, the underlying physical properties do not. For a Rank 2 tensor, properties like the Determinant and the Trace (the sum of diagonal elements) remain constant regardless of the transformation.

Key Insight: In AI, we often ignore this and treat tensors just as "heaps of data." In Robotics, we must respect these laws because our data (force, torque, inertia) exists in a physical 3D world where the choice of coordinate system is arbitrary.

Geometric Intuition

  • Rank 0: A point. No matter how you turn, the point is the same.
  • Rank 1: An arrow. If you turn the world, the arrow's coordinates change to keep pointing the same way.
  • Rank 2: A transformation (like a squish or a stretch). If you turn the world, you have to turn the "squish direction" and the "stretch direction" simultaneously.

Passive Rotations: Changing the Observer

So far we have used active rotations, where we physically moved the data cloud (or the robot arm) within a fixed room. The complementary view is the passive rotation: the physical object stays exactly where it is, but the coordinate system (the observer) rotates.

The passive rotation of vectors (Rank 1)—including why the coordinates of a fixed vector transform with the inverse rotation, $\mathbf{x}' = \mathbf{R}^{-1}\mathbf{x} = \mathbf{R}^T\mathbf{x}$ (a contravariant transformation), and the active-vs-passive (alibi-vs-alias) distinction—is covered in detail in the 2D rotations notebook.

The same idea also holds for Rank 2 tensors. A passive rotation still uses the "sandwich", but with the roles of $\mathbf{R}$ and $\mathbf{R}^T$ swapped relative to the active case. If $\mathbf{A}$ is a matrix in the old coordinate system, its description in the new system is

$$\mathbf{A}' = \mathbf{R}^T \mathbf{A} \mathbf{R}$$

which is exactly the reverse of the active law $\mathbf{A}' = \mathbf{R}\mathbf{A}\mathbf{R}^T$.