Author(s): Sayan Chaudhary
Originally published on Towards AI.
most people listen “linear algebra” And expect a wall of sources. But in machine learning, you only need a small set of ideas, and each of them has a real-world interpretation. This article breaks everything down using simple examples and everyday analogies, so you walk away with a clear understanding, not confusion.
Why does linear algebra sit at the core of ML?
Machine learning does three things:
- represents data
- transforms data
- optimizes model parameters
vectors And matrix Handle representation and change.
Eigenvalues, gradients, and Hessian control the optimization. That’s what it really is.
1. Vector: How ML represents anything
A vector It’s just a list of numbers.
In ML, this becomes the basic way of storing information.
real life example
A house described as:
- size 1200 square feet
- 2 bedrooms
- age 8 years
becomes vector,
x = (1200, 2, 8)
this is one three-dimensional vector Because it contains three pieces of information.
vectors represent:
- a single data sample
- a row in the dataset
- model weight
- Guidelines for Customization
Dot Product (Why It Matters)
This tells you how similar the two are vectors Are.
Used in:
2. Matrices: The way ML stores and transforms data
A matrix is a rectangular table of numbers.
understand it this way a collection of vectors put together,
Example: a dataset
If you have 10,000 rows and 30 attributes, you literally have:
- 10,000 vectors
- Each lives in 30-dimensional space
So your dataset becomes 10,000 x 30 matrix,
Why do metrics matter?
They represent:
- dataset
- load of neural network
- changes applied to data
Each layer in a neural network is basically:
Output = Input * Load
3. Dimension: The size of your data
If a data point has 100 features, it exists 100-dimensional space,
real life intuition
Imagine describing such a person:
- age
- Height
- weight
- Income
- City
- favorite style
- browsing frequency
- credit score
- etc…
With 100 such attributes, you are essentially placing each person in a 100-dimensional world.
High-dimensional spaces often cause this (not good for models):
- rarity
- distance distortions
- Difficulty in training models
This is why dimensionality reduction exists.
4. Rank and linear independence: how much information is present
Rank tells you how many unique directions your data actually has.
If some columns in your dataset are duplicates or combinations of others, the rank drops
Example
Let’s say you have:
- Feature 3 = 2 × Feature 1
- Feature 5 = Feature 2 + Feature 4
Your dataset pretends to be 100-dimensional, but perhaps only 96 directions are actually unique.
Low rank means:
- unnecessary features
- unstable regression solution
- matrices that cannot be inverted
Understanding ranks helps diagnose data problems quickly.
5. Span and column space: what your model can represent
is the extension of a set of vectors everything you can make by mixing them,
everyday example
If you only know how to walk north and east, you can never reach the southwest.
Your possible movement space is limited. Similarly, if your model uses features that do not cover all directions, predictions fail.
In regression, predictions are always implicit column space of your feature matrix.
6. Inference: The Geometry Behind Regression and PCA
Projection means releasing a vector into another direction or subspace.
real life example
Shining a torch on an object creates a shadow. Shadow is the projection of the object on the ground.
In ML:
- Regression projects your target values ​​onto feature space
- PCA projects high-dimensional data onto principal axes
Projection Answer:
“What is the best possible prediction using the instructions you have?”
7. Criteria and Distances: Measuring Size and Error
Most important:
- L2 criteria: straight line distance
- L1 criteria: sum of absolute values
Used in:
- Regularization (Lasso, Ridge)
- clustering algorithm
- measuring slope size
Example
If the shopping patterns of two customers are represented as 20-dimensional vectors, the criterion tells how far their behavior is from each other.
8. Graduation: How models learn
The gradient is just a vector that tells you how the loss changes as a parameter increases.
It points in the direction where losses grow fastest.
gradient descent moves in the opposite direction.
everyday analogy
Imagine standing on a hill.
The slope tells you the direction of the steepest upward slope.
Go in the opposite direction to go down the hill faster.
Every deep learning model uses gradients to update weights.
9. Hessians: Understanding Curvature
The Hessian is a matrix of second derivatives. This tells you the size of the loss surface.
why it matters
- If the Hessian is positive definite → you are at the minimum
- If its eigenvalues ​​are negative → saddle point
- If eigenvalues ​​are too large → gradients explode or disappear
This is important in optimization theory and understanding why deep nets sometimes fail to converge.
10. Eigenvalues ​​and Eigenvectors: Understanding How Matrices Behave
show eigenvectors The directions on which the matrix naturally acts,
show eigenvalues how much stretch or contraction occurs,
where they appear
- PCA finds directions with maximum variation
- Optimization stability depends on Hessian eigenvalues
- Spectral clustering uses eigenvectors of graph Laplacians
real life scene
Imagine a rubber sheet drawn with lines.
Pull it in any direction.
Some lines stretch a lot, some remain almost the same.
Those particular directions are eigenvectors.
11. SVD: Foundation of PCA and many algorithms
SVD decomposes any matrix into:
- left directions
- scaling value
- right directions
it is used for:
- PCA
- low-order approximation
- noise reduction
- stable least squares solution
If eigenvalues ​​explain how matrices function, SVD gives a complete picture.
12. Positive Definite Matrix
why it matters:
- The covariance matrix is ​​PSD
- Kernel matrices must be PSD
- Minima have Hessian PD
They guarantee stability during optimization.
13. Orthogonality and orthonormal basis
Orthogonal vectors do not interfere with each other.
They act like clean, independent directions.
Used in:
- PCA
- Static initialization in neural networks
- qr decomposition
- Simplify Estimates
(Don’t be annoyed by these names. I’ll explain them soon)
Orthonormal bases make calculations efficient and reduce numerical errors.
14. Matrix Inverse and Pseudo Inverse
Not every matrix has an inverse.
When it doesn’t, we use Moore–Penrose pseudoinverse,
Used in:
- Solving Linear Regression with General Equations
- underdetermined or overdetermined systems
- dimensionality reduction
This allows ML models to work even when the data is messy.
15. Trace and Determinant
Not used every day but shown in:
- Gaussian log-likelihoods
- entropy formula
- covariance matrix identity
Good to know, rarely used manually.
16. Basic Tensor Operations
Deep learning frameworks rely heavily on tensor,
You only need:
- change shape
- to move
- broadcast
- element-wise operations
no advanced tensor General DL work requires calculus.
Linear Algebra doesn’t have to be intimidating. When seen through real-life examples and ML applications, this becomes intuitive.
View the previous version of Artificial Intelligence and Machine Learning chapter:
Tensors in Machine Learning (ML Chapter-1),
Tensors in Machine Learning: The Clearest Explanation You’ll Ever Read (ML Chapter-1)
If you’ve ever opened a machine learning textbook or played with deep-learning frameworks, you’ve seen the term tensor…
pub.towardssai.net
Data Preprocessing in Machine Learning (ML Chapter-2: Module-1):
Data Preprocessing in Machine Learning: The Complete Guide (ML Chapter-2: Module-1)
Machine learning models may be powerful, but they are only as good as the data we feed them. Even before training…
ai.plainenglish.io
Data Modeling in Machine Learning: (ML Chapter-2, Module-2)
Data Imputation in Machine Learning: A Practical, Nonsense Guide (ML Chapter-2, Module-2)
Missing data shows up everywhere: surveys, logs, sensors, medical records, finance datasets, you name it. and if you…
pub.towardssai.net
After reading this you will finally understand Machine Learning Models (ML Special),
After reading this you will finally understand Machine Learning Models (ML Special)
sayanwrites.medium.com
Published via Towards AI
