Regularisation explicitly penalises and can delete newly added variables if they don’t contribute sufficiently to the model. If you try to fix a regression model by adding more features to capture different alternatives, Lasso (L1) regularisation can mathematically isolate those features and completely remove them from the model. It does not do this out of ideological bias; it does it because its objective function is designed to maximise structural simplicity by eliminating variables that contribute little to the overall prediction.
Lasso Regularisation
Standard linear regression tries to minimise the sum of squared errors through Ordinary Least Squares (OLS). Lasso Regression adds a strict mathematical constraint to this objective function called the L1 penalty. It adds the absolute values of all feature weights (β), multiplied by a tuning parameter. Lasso (L1) Regularisation acts as the ultimate automated executioner of localised nuances.
To understand why Lasso can completely delete variables rather than simply shrinking them, you have to look at its geometric constraint. In a two-dimensional feature space, Lasso’s absolute-value constraint forms a diamond shape centred at the origin. The model optimises by expanding its error contours until they hit this diamond boundary. Because a diamond has sharp corners on the axes, the optimisation line will often hit the constraint at a corner, where one of the weights (β) is exactly zero.
When the Lasso engine expands its error contours to find the best compromise between prediction accuracy and the L1 penalty, it evaluates the contribution of every feature. The optimisation engine also considers the cost-benefit ratio. To maximise its score under the strict budget imposed by the regularisation strength, the algorithm chooses the most efficient reward-hacking path: it shrinks the coefficient of the localised variable down to exactly 0.
Conclusion
Lasso Regression is designed for feature selection and sparsity. It treats any feature that does not explain sufficient global variance as a potentially unnecessary addition. In a lopsided data landscape, Lasso can act as an algorithmic filter that enforces conformity—proving that the more you tune a standard model to be “stable” and “generalised,” the more aggressively it can eliminate the specific variables that protect marginalised or unique perspectives.
Day 20 / 30 of the #AlRewardmaxxing Series.
Tomorrow in Part 21, we're moving deeper into model architectures to look at how L2 regularisation behaves differently.
What's your take? Have you seen models penalize or ignore niche edge-cases in your domain in the name of "feature selection"? Let's discuss below!