Even when you move beyond basic algorithms into the advanced stages of the machine learning pipeline—Hyperparameter Tuning, Feature Engineering, Dimensionality Reduction, and Ensembles—the exact same systemic erasure can occur.
In fact, these techniques can become powerful optimisation engines for bias. Because they are designed to maximise overall statistical performance across a dataset, they can systematically tune out, drop, or smooth over localised realities in pursuit of a strong global score.
Mechanical Breakdown
Hyperparameter Tuning (using Grid Search, Random Search, or Bayesian Optimisation) searches for the optimal internal settings—such as the maximum depth of a decision tree or the regularisation strength (λ) in a regression model—to minimise overall loss.
To prevent overfitting, tuning algorithms may favour stronger regularisation parameters, such as those used in Ridge or Lasso regression. Regularisation mathematically penalises models for fitting too closely to individual or extreme observations. Because the tuning process is guided by an averaged cross-validation score, it can select hyperparameters that reduce the influence of unique or underrepresented observations.
Feature Engineering is the process of selecting, transforming, and combining raw data fields into meaningful indicators for the machine.
In the mainstream, features are often engineered around dominant assumptions and established infrastructure. So, if the pipeline fails to engineer features that capture localised realities, the mathematics remains completely blind to them.
The algorithm cannot find a truth it has not been given coordinates for.
Techniques like Principal Component Analysis (PCA) compress thousands of data variables into a smaller number of “Principal Components” by projecting the data onto axes that capture the highest mathematical variance.
PCA prioritises directions of high variance because they explain a large proportion of the variation within the dataset. In a global data ecosystem, variables representing alternative or highly localised patterns may have relatively low variance because they rarely appear.
When PCA compresses the data, these low-variance dimensions may receive little representation in the resulting components.
Ensemble Methods (like Random Forests, Gradient Boosting Machines, or Voting Classifiers) combine the predictions of multiple sub-models to produce a final decision.
In a voting ensemble, the final output is determined by a majority vote or a weighted average of the individual models.
Conclusion
Tuning must optimise for Pareto-front efficiency, balancing overall global accuracy with a localised “Fairness/Accuracy Metric” for underrepresented subgroups.
Engineers must manually craft features that ground the data in regional realities. And when using specialised ensemble techniques, individual sub-models should be deliberately exposed to underrepresented patterns through appropriate sampling, weighting, or other methods.
Because sometimes, the unique truth does not need to win the majority vote.
It simply needs to be given a fair chance to be heard.
Day 25 / 30 of the #AIRewardmaxxing Series.
Tomorrow in Part 26, we're breaking down K-Means and Hierarchical Clustering.
What's your take? Have you seen global ML performance metrics hide poor performance for underrepresented subgroups in your models?
Let's discuss below!