Adding more variables to a regression model seems like the most logical engineering fix. But in data science, this approach can trigger a severe computational and structural trap known as the Illusion of Dimensionality. Merely adding columns to a spreadsheet does not fix an inherently lopsided data architecture. Instead, it can introduce three deep mathematical problems that cause the regression model to become even more inaccurate and unstable.
Why Adding More Variables Fails
In regression modelling, variables do not necessarily have to be completely independent of one another, but highly correlated predictors can create a problem known as Multicollinearity. When you add new columns to a dataset that is already overwhelmingly Eurocentric, the new variables may become highly correlated with existing dominant features.
When variables are highly correlated, the matrix calculations used to estimate the regression coefficients can become unstable. The model can struggle to distinguish the individual contribution of each variable. Instead of becoming more accurate, the weight (β) of your new, localised variable may be heavily reduced or become unstable, rendering it less useful. For a regression model to calculate a stable weight (β) for a new variable, that variable needs sufficient, high-quality data across the dataset.
Also, by adding more columns to a regression model, you can mathematically decrease its training bias while massively increasing its variance. Every time you add a variable to a regression equation, you give the model another dimension through which it can fit the training data.
Because the data representing alternative realities may be rare, adding highly specific variables can cause the model to overfit to those particular data points. The model creates a highly erratic, volatile curve. The moment you expose this updated model to new, real-world data, its predictions can collapse. The system may treat the new localised variables as chaotic “noise” that reduces its overall generalisation performance.
During hyperparameter tuning or cross-validation, the automated pipeline may therefore favour simpler models or penalise features that do not improve performance on unseen data.
Conclusion
You cannot fix a lopsided ecosystem by simply giving a blind machine more metrics to track. If the underlying data pool is fundamentally skewed, adding more variables is like building a bigger, more complex house on a crooked foundation.
The regression model may simply use the new dimensions to find a more complex, mathematically optimised way to arrive at the exact same biased conclusion. To make the model truly accurate, engineers cannot just add features; they may need to alter the Loss Function itself—forcing the mathematics to prioritise and heavily weight the accuracy of rare, localised profiles over the massive, repetitive volume of the majority.
Day 19 / 30 of the #AIRewardmaxxing Series.
Tomorrow in Part 20, we're diving into L1 regularisation and how modifying the Loss Function can force algorithms to prioritize rare, localized profiles over massive majority data volume.
What's your take? Have you seen feature engineering backfire when trying to fix bias in your own datasets? Let's discuss below!