Ridge Regression (L2 Regularisation) behaves differently—retaining the variable while shrinking its impact to near-zero. While Lasso (L1) acts as a ruthless executioner that zeroes out features that do not apply to the majority, Ridge acts as a conformist smoother. It keeps your newly added features active in the equation, but it mathematically castrates their impact. It suppresses their influence so heavily that they become functional ghosts inside the model.
Ridge Regression (L2)
Instead of adding the absolute values of the weights, Ridge Regression adds the squares of all feature weights (β) to the Ordinary Least Squares (OLS) cost function. Because it squares the weights, the penalty grows rapidly for large weights. This shifts the algorithm’s entire geometric priority.
The fundamental difference between Lasso and Ridge comes down to the geometry of their constraint boundaries. While Lasso’s absolute-value penalty creates a sharp diamond with corners on the axes, Ridge’s squared penalty forms a smooth circle centred at the origin.
When the expanding contours of the error function meet this circular boundary, they almost always touch it at a tangent rather than on an axis. Mathematically, this means the weights (β) are squeezed smaller and smaller, but they never quite reach exactly zero.
If you have highly correlated features (multicollinearity) because your new localised variables overlap with existing mainstream indices, Ridge handles this by sharing the penalty. It takes the heavy mathematical importance and distributes it across all the related features, shrinking their weights simultaneously.
Lasso (L1) is honest about its erasure: it sets the weight to 0.0 and effectively removes your localised column. Ridge (L2) provides an illusion of inclusion. Your column remains in the model, and the code runs without deleting any variables.
Conclusion
When Ridge runs its optimisation loop over a lopsided dataset, it treats features based on how much global variance they explain. Ultimately, Ridge Regression (L2 Regularisation) behaves differently by shrinking the weights of localised variables close to zero, but it never completely deletes them.
Maybe engineers have to build an Inverse-Density Weighted Loss Function that forces Lasso to value minority coordinates as highly as majority clouds.
Day 21 / 30 of the #AIRewardmaxxing Series.
Tomorrow in Part 22, we're diving into Euclidean distance and how it groups data points based on dense, western-centric clusters
What's your take? Have you seen Ridge regression mask underlying data issues by keeping low-weight features alive? Let's discuss below!