Appearance
Eigenvalues and Eigenvectors — The Axes That Don't Move
The previous chapter ended with a teaser: you find eigenvalues by setting a certain determinant to zero. That connection is not a coincidence — it is the whole story. This chapter unpacks it, and along the way reveals one of the most useful ideas in all of applied mathematics.
By the end of this chapter you will be able to:
- Explain what an eigenvector and eigenvalue are, geometrically and algebraically.
- Find eigenvalues by solving the characteristic equation .
- Find eigenvectors by solving the corresponding homogeneous system.
- Understand diagonalization and why is powerful.
- Connect eigenvalues and eigenvectors to PCA, rigid body physics, and Markov chains.
The Transformation That Leaves Some Vectors Alone
Imagine pressing your palm flat on a rubber sheet and dragging it diagonally. Most points on the sheet move in complicated ways — they slide sideways, they get stretched, they land somewhere you would not have predicted. But there are a few special directions where the rubber simply stretches straight out along the same line. Points along those directions move farther from the origin or closer to it, but they never swing to a new angle.
Those special directions are eigenvectors.
Now picture applying a transformation to every vector in the plane. Almost every vector gets knocked off its original line through the origin:
Before: After applying A:
^ ^
| v | Av (rotated AND scaled)
| | /
+------> +------>But suppose there is one particular vector where the output lands exactly on the same line as — just stretched by some factor :
Before: After applying A:
^ ^
| v | Av = λv (same direction, just longer)
| |
+------> +------>That vector is an eigenvector of . The stretch factor is its eigenvalue.
The word "eigen"
Eigen is German for "own" or "characteristic." An eigenvector is, loosely, the matrix's own vector — the one that the matrix acts on in the simplest possible way.
The Formal Definition
An eigenvector of an matrix is a nonzero vector satisfying:[^1]
for some scalar , which is called the corresponding eigenvalue.[^1]
The zero vector is explicitly excluded — if , that is trivially true for any and tells you nothing useful.
What eigenvalues tell you
The eigenvalue encodes how the matrix stretches (or shrinks, or flips) along that special direction:
| value | What the transformation does along |
|---|---|
| Stretches away from the origin | |
| Shrinks toward the origin | |
| Leaves completely unchanged | |
| Flips to point the other way, same length | |
| Collapses to the zero vector |
Notice the last row: if is an eigenvalue, then for some nonzero . That means crushes at least one direction to zero — which means is not invertible. Once again, the determinant connection surfaces: exactly when is an eigenvalue.[^1]
Finding Eigenvalues: The Characteristic Equation
The equation can be rearranged:
This is a homogeneous linear system. From Chapter 6 you know it has a nontrivial solution (one other than ) if and only if the matrix is not invertible — which, from Chapter 7, happens exactly when its determinant is zero.[^2]
So eigenvalues are the values of satisfying:
This is the characteristic equation. The left side, when expanded, is a polynomial in called the characteristic polynomial. Its roots are the eigenvalues.[^2]
For an matrix, the characteristic polynomial has degree , so there are at most eigenvalues (counting complex roots and repeated roots).
A worked 2×2 example
Let
Step 1: Form .
Step 2: Set the determinant to zero.
The eigenvalues are and .
Quick formula for 2×2 matrices
For a matrix, the characteristic polynomial is always , where is the trace — the sum of the diagonal entries.[^2]
For the example above: and , giving . Same result, faster.
Notice also: the matrix above is upper triangular (all zeros below the diagonal). For any triangular matrix, the eigenvalues are simply the diagonal entries — no computation needed.[^2] Our answer matches the diagonal . Convenient.
Finding Eigenvectors: Solving the Null Space
Once you have an eigenvalue , plug it back in and solve . The solution set — all eigenvectors for that eigenvalue plus the zero vector — is called the -eigenspace.[^1]
Continuing the example above:
For :
Row reduce and solve: the system is , so . Choosing :
For :
Row reduce: the system is , and is free. Choosing :
Verification. Always worth checking:
Eigenvectors are never unique
Any nonzero scalar multiple of an eigenvector is also an eigenvector for the same eigenvalue. If , then $A(c\mathbf{v}) = c(A\mathbf{v}) = c\lambda\mathbf{v} = \lambda(c\mathbf{v})$. So when someone says "find the eigenvector," they mean "find the direction" — the specific length is arbitrary. Conventionally, eigenvectors are often normalized to unit length.
A Second Example: The Symmetric Case
Symmetric matrices — where — come up everywhere in applications (covariance matrices in statistics, inertia tensors in physics). They have a beautiful property: their eigenvectors are always perpendicular to each other.[^3]
Let
Characteristic polynomial:
Eigenvalues: and .
The eigenvalue immediately tells us is not invertible — its columns are linearly dependent (and indeed, column 1 is twice column 2).
For :
Row reduce (second row is times first): , so . Eigenvector: .
For :
Row reduce: , so . Eigenvector: .
Check orthogonality: . Perpendicular, as promised.
Diagonalization — The Skeleton of a Transformation
Here is the payoff. If an matrix has linearly independent eigenvectors with corresponding eigenvalues $\lambda_1, \dots, \lambda_n$, you can write:[^3]
where:
- is the matrix whose columns are the eigenvectors
- is the diagonal matrix with eigenvalues on the diagonal
- is the inverse of (which exists because the eigenvectors are independent)
This decomposition is called diagonalization. Geometrically, it says: change coordinates so that the eigenvectors become the axes (), apply pure stretching (), then change back (). In the eigenvector coordinate system, the transformation is just scaling — no mixing of components at all.
Original coordinates: Eigenvector coordinates:
^ ^
| v mixes components | Dw — just scale each axis
| / | |
+-----> +--|-->
v (independent stretches)Why diagonalization matters: fast powers
In game physics and simulation, you often need to apply the same transformation hundreds or thousands of times. Computing by multiplying by itself one hundred times is expensive. Diagonalization gives you a shortcut:[^3]
And is trivially cheap — just raise each diagonal entry to the th power:
No repeated matrix multiplication. This trick shows up in physics integrators, Markov chain analysis, and anywhere you need to simulate a discrete system evolving over many steps.
When is a matrix diagonalizable?
An matrix is diagonalizable if and only if it has linearly independent eigenvectors. A sufficient (but not necessary) condition: if all eigenvalues are distinct, the corresponding eigenvectors are automatically linearly independent, so the matrix is diagonalizable.[^3]
Matrices with repeated eigenvalues may or may not be diagonalizable — it depends on whether the eigenspace for the repeated eigenvalue has enough dimensions. When it does not, a more general decomposition (the Jordan normal form) is needed, but that is beyond this course.
Real Applications
PCA: Finding the Directions of Maximum Variance
Imagine you have a dataset of, say, one thousand game characters, each described by twenty numerical features (height, speed, armor, range, ...). You want to visualize them in two dimensions. Which two dimensions capture the most information?
This is the job of principal component analysis (PCA).[^4]
The trick is to compute the covariance matrix of the data — a symmetric matrix that encodes how each pair of features varies together. Then find its eigenvectors.
The eigenvectors of the covariance matrix point in the directions of maximum variance in the data. The corresponding eigenvalues tell you how much variance each direction captures. Sort by eigenvalue (largest first) and take the top two: those are your two best axes for visualization.[^4]
High-variance direction (1st eigenvector):
* * *
* *
* *
* *
* *
* * *
----> PC1 (largest eigenvalue)
Low-variance direction (2nd eigenvector):
^
| PC2 (smaller eigenvalue)The eigenvector with the largest eigenvalue is called the first principal component. It is the axis along which the data spreads the most. The second principal component is perpendicular to the first (because covariance matrices are symmetric, so their eigenvectors are orthogonal[^3]) and captures the next largest spread.
In practice, you can reduce a hundred-feature dataset to five dimensions by keeping only the top five eigenvectors — losing relatively little information while making learning algorithms dramatically faster. This is why eigenvectors sit at the core of data science.
Rigid Body Physics: Principal Axes of Rotation
When a game object spins, the physics engine needs to know how easy or hard it is to rotate that object around each possible axis. A long, thin rod is easy to spin around its length axis and much harder to spin around a perpendicular axis. A cube is symmetric — all rotation axes feel the same.
This information is captured by the inertia tensor, a symmetric matrix that maps angular velocity to angular momentum.[^5] For a generic object, the inertia tensor mixes axes together — angular velocity in the -direction produces angular momentum with and components too.
But the eigenvectors of are special: they are the principal axes of rotation, the three natural rotation axes of the body. When angular velocity points along a principal axis, angular momentum is parallel to it — no mixing, no precession.[^5] The corresponding eigenvalue is the principal moment of inertia: the scalar resistance to rotation around that axis.
Diagonalizing the inertia tensor is exactly what physics engines do when they initialize a rigid body. Once the principal axes are found, rotational dynamics become three independent scalar equations instead of a full matrix equation. This is why the eigenvector computation happens once at load time and then the simulation runs efficiently every frame.
Markov Chains: Long-Run Behavior
A Markov chain models a system that hops between states randomly, where the probability of each next state depends only on the current state. The transition probabilities form a square matrix (called a stochastic matrix).
What happens after many steps? You might ask: "If I run this random process for a long time, what fraction of the time does the system spend in each state?" The answer is the stationary distribution — a probability vector that does not change when you apply :
That equation says is an eigenvector of with eigenvalue .[^6]
Under mild conditions, every well-behaved stochastic matrix has exactly one such eigenvector (up to scaling), and the system always converges to it regardless of where it starts. Computing the long-run behavior of a Markov chain is literally an eigenvector problem.
(This is also the idea behind Google's original PageRank algorithm: the web's link structure forms a transition matrix, and the importance of each page is its entry in the dominant eigenvector.)
Chapter Recap
Four ideas to carry into the final chapter:
Eigenvectors are the axes that don't rotate. For a matrix , an eigenvector satisfies — the transformation only stretches or flips it, never rotates it off its line.
Eigenvalues come from the characteristic equation. Set and solve for . This connects eigenvalues directly to determinants and to the invertibility tests from Chapters 6 and 7.
Diagonalization reveals the transformation's skeleton. If has independent eigenvectors, then , where is diagonal. Powers become cheap: .
Applications are everywhere. PCA uses eigenvectors of a covariance matrix to find the directions of maximum variance. Rigid body physics uses eigenvectors of the inertia tensor to find natural rotation axes. Markov chains converge to the eigenvector of eigenvalue .
The next chapter ties everything together — vectors, matrices, determinants, and eigenvectors — through two extended worked examples: a full 3D game physics pipeline and a forward pass through a neural network.
References
[^1]: "Eigenvalues and Eigenvectors." Interactive Linear Algebra, Georgia Tech. https://textbooks.math.gatech.edu/ila/eigenvectors.html
[^2]: "The Characteristic Polynomial." Interactive Linear Algebra, Georgia Tech. https://textbooks.math.gatech.edu/ila/characteristic-polynomial.html
[^3]: "Diagonalization." Math 110 Course Notes, University of Victoria. https://web.uvic.ca/~eaglec/Math110/sec-Diagonalization.html
[^4]: "Principal Component Analysis." IBM Think. https://www.ibm.com/think/topics/principal-component-analysis
[^5]: "Rotation and Inertia Tensors." Gaffer On Games, Glenn Fiedler. https://gafferongames.com/post/rotation_and_inertia_tensors/
[^6]: "Lecture 8: Markov Eigenvalues and Eigenvectors." MIT OpenCourseWare, 6.262 Discrete Stochastic Processes. Spring 2011. https://ocw.mit.edu/courses/6-262-discrete-stochastic-processes-spring-2011/resources/lecture-8-markov-eigenvalues-and-eigenvectors/