Hume took the empiricist system of thought begun by Locke to its logical empiricist conclusion and posed a question that still hangs over all knowledge — from the most trivial to the most advanced — like the sword of Damocles: "How can we justify applying a generalization begun from what we've seen to what we haven't?" All of machine learning can be described as an attempt to answer this question.
- The Master Algorithm, p115
Looking back at Hume's question — there is no guarantee that a generalization obtained in one place can be applied elsewhere — I find myself thinking that overfitting in machine learning may correspond, in reality, to what we kindly call "culture, common sense, values" or, more sharply, "social convention, preconception, and prejudice."
Perhaps confirmation bias and hasty generalizations are everywhere. It is only their attributes, ranges, situations, and perspectives that differ.
Perhaps, in the process of developing technologies that mimic human intelligence, the emergence of overfitting — a phenomenon resembling human tendencies — is natural. Can an answer to overfitting be found in the space between human-like and non-human-like attributes?
The practical conclusion of "There's no free lunch" is "There is no learning without knowledge." Data alone is not enough. Starting from nothing, you only arrive at nothing. Machine learning is a knowledge pump that can draw a lot of knowledge out through data, but you must first pour priming water into the pump.
Machine learning is what mathematicians call an "ill-posed problem" — a problem that can have multiple solutions.
- The Master Algorithm, p122
Tom Mitchell, a leader of the symbolist school, calls this "the futility of bias-free learning." In everyday life, "bias" is a derogatory word. Prejudice is bad. But in machine learning, prejudice is indispensable. Without prejudice, learning is impossible. In fact, prejudice is indispensable for human cognition too — human brains are already wired, and we take prejudice as a given. This is a different kind of bias from the bias worth questioning.
Aristotle claimed there is nothing in the intellectual world that is not first sensed. Leibniz added, "except the intellect itself." The human brain is not a blank slate because it is not the same as a slate.
The reason prejudice is also important for the human (brain), and the reason it is a different bias from the bias worth suspecting, is that we need a scale we can perceive before making judgments. You could say it plays a role like the cm or inch on a ruler. A kind of cognitive frame. But one can call it a different kind of bias because, although not easy, the "length spec" of that cognitive frame holds considerably variable properties.
I wonder if the core is securing a perception range that matches your own attributes. In fact, this isn't just an issue for humans and AI. Light, lidar, touch, waves (fields) — such "specs" manifest (are measured) on the premise of each range. Beings that perceive and adapt to these attributes can survive in their own dimension, moving step by step according to their own capacities (sensing and response).
But we can't perceive and judge every moment. Even a human's biggest? energy use is from brain activity, so we define a range of perception (unit, frame) matching our physical and mental condition, and within that range learn patterns (observation, experience, knowledge). Then we compose a routine (inertia) presumed to deliver maximum effect with minimum perception and judgment, via our own validated decision patterns within that limited range. And if the stage of that range is norms, it's culture; if it's habits of living, it's taste. Each is named by a different signifier and used again as a new unit — that's the thought I have.
