When algorithms encounter a dataset heavily weighted toward a Eurocentric baseline, their internal mathematical optimisation engines—probability ratios, hyperplane margins, and information-gain metrics—can treat unique, localised truths as errors or noise that must be filtered out. And, just like the rest of the machine-learning pipeline, Naïve Bayes, Support Vector Machines (SVMs), and Decision Trees are bound by the same underlying patterns in the data.
Naïve Bayes, SVMs, and Decision Trees
Naïve Bayes relies on Bayes’ Theorem to predict classifications based on conditional probabilities. It is “naïve” because it assumes that all features are conditionally independent of one another. Naïve Bayes relies heavily on the Prior Probability (P(C)) of a class occurring in the historical training data.
An SVM searches for the optimal hyperplane in an N-dimensional space that separates data points into distinct classes with the maximum possible margin. In real-world data, classes are rarely perfectly separable by a clean line. SVMs use a regularisation parameter, C, which controls the trade-off between maximising the margin and minimising training classification errors. A standard SVM optimised for general performance uses a “soft margin” that deliberately allows the model to misclassify or tolerate some extreme data points if doing so creates a smoother, wider boundary for the rest of the data.
Decision Trees split datasets into smaller subsets by selecting the features that yield the highest Information Gain or the lowest Gini Impurity at each node. The algorithm searches for the variable that separates or purifies the data most effectively at each stage. Because Information Gain evaluates how well a feature splits the remaining dataset, features that apply to a large proportion of data points can be selected early in the tree.
Localised, highly critical nuances may apply to only a tiny fraction of the global dataset. Because such a feature may contribute little to reducing the overall entropy of a dataset dominated by Western data, the Decision Tree may push it deeper into the lower branches or ignore it entirely. The tree may therefore never reach that level of nuance, particularly if its growth is constrained by pruning or other stopping criteria.
Conclusion
Naïve Bayes uses lopsided historical frequencies to influence the probabilities assigned to alternative regional patterns. SVMs draw decision boundaries that can treat localised systemic exceptions as sacrificial soft-margin errors. Decision Trees use global information-gain or impurity metrics that can favour variables representing the dominant majority, leaving localised nuances buried or completely unmapped.
To mitigate these weaknesses, engineers can manually inject structural safeguards: applying Laplace smoothing in Naïve Bayes to prevent rare categories from receiving zero probability; utilising instance-weighted SVMs that increase the penalty for misclassifying underrepresented regional data; and using cost-sensitive Decision Trees that penalise the misclassification of unique, localised cases more heavily than standard models.
Day 23 / 30 of the #AlRewardmaxxing Series.
Tomorrow in Part 24, we're breaking down Cross-Validation-how K-Fold splits, K-packages, and evaluation pipelines can either expose these algorithmic biases or hide them completely.
What's your take? Have you seen standard ML algorithms write off rare edge cases as error noise in your data pipelines? Let's discuss below! 