Is It Necessary to Search the Whole Haystack to Find the Needle?
Recently, manynews commentators , policy wonks, government officials and private citizens have been insisting that “to find a needle in a haystack, you have to search the whole haystack”, in relation to the complex question of massive, warrantless personal data collection—whether by governments, marketers, Internet and phone service providers or anybody else. Others, equally thoughtful and concerned, have equally vigorously disagreed.
What about looking for the job-candidate “needle”? Which haystack, how big a haystack, what kind of haystack—if, indeed, any haystack at all—and with what methods, goals and limits should recruiters be randomly and meticulously searching?
Does the haystack search principle hold and stack up in recruiting or in any other important search, assuming that it has some serious associated costs in terms of consumed resources, efficiency, opportunity costs with respect to existing or conceivable alternatives and perhaps even in terms of basic relevance?
Needles and Magnets
One effective—indeed, standard—alternative in recruiting is to attract the needle, magnetically, so to speak, not search for it. In one kind of targeted recruiting, one application of this attraction-based approach is called “sting operations”; another is in attracting minds that can come up with better approaches and better metaphors that will be better in reflecting the full range of advanced technology options available and in virtue of making everybody—except the bad guys and the unqualified guys—happy.
In more conventional recruiting, a principle form of it is called “job posting”. Critics of the haystack approach would, as part of their critique, ask, “In looking through haystacks, have we not lost our compass and magnetic, attraction-based bearings?” Such is the power and persuasiveness of metaphors.
Statistically Profiled Haystacks and Needles
A third approach, applicable to both intelligence gathering and employment recruiting is what might be called “statistical needle-and-haystack profiling”, i.e., precisely and statistically targeted haystacks as well as comparably targeted needles.
This too is a standard employment recruiting approach, to the extent that the only resumes that are sought or reviewed are those that meet very specific and germane profile criteria, e.g., exclude the 10-year-old kid with a history of unlicensed lemonade stands.
An imaginative approach of this sort would be to target something other than all people, for example, specific red-flag purchases, e.g., of explosives components or career skill sets, zeroing in on individuals only when the red flags flutter on the radar. Whole haystack, but different straw.
The Hazards of Same Terms, Different Meanings
Now, another one of the great lessons to be learned from logic and critical thinking is that when intelligent and intelligence people disagree, despite the “best of intentions” on all sides, it may be because they are not talking about the same thing, despite using the same vocabulary—in this instance, metaphorical vocabulary that includes “needle” and “haystack”. Same terms, but different interpretations. It happens all the time.
For example, in one of its crudest and earliest forms, the evolution vs. special creation debate raged over the question of whether mutations and natural selection could possibly account for the highly integrated, specialized functions and structures of the eye—any eye.
To make their point, some creationist pamphleteers interpreted “mutation” metaphorically, likening it to damage done to a car that has smashed into a telephone pole, because like the latter, it is a random change in structure and functioning. Their point: How could an “accident” improve anything? [I remember the pamphlet cartoon that went with that suggestion very vividly—mostly because of how wrongheaded it seemed.]
Clearly, if you are looking for a useful mutation in a stack of wrecked non-self-replicating auto parts, you will not find it, even if you have all the parts to search through. That’s because a stack of wrecked automobile parts is the wrong “database” and metaphor for the search and the material of the evolutionary process.
To the evolutionists, this was a horrifying, or at least comical distortion of the meaning of “mutation” and the theory of evolution, making any shared understanding and insight, much less shared sense of progress, impossible.
Lessons from a Car Wreck
Isn’t it possible that this needle-in-the-haystack-debate is raging because, as the evolution debate demonstrated,
1. The key terms, viz., “needle” and “haystack” are being interpreted differently by both sides?
2. Like “stack of automobile-wreck parts”, it’s, as it stands, the wrong metaphor?
Metaphorical Garbage in, Garbage Out
The advisability—indeed, indispensability—of using extreme caution when debating, analyzing, proposing or implementing anything very important, including public policies and scientific or moral principles is a second well-established and rightly emphasized lesson from critical thinking and logic.
Here, with metaphor or analogy-based justifications, as with even the rawest of data, it’s a case of “garbage in-garbage out”—i.e., your justification and methods are only as sound as your metaphor or analogy is [“garbage-in-garbage-out” presumably being a time-tested, acceptable one].
Is It That Easy to Trip Up a Smart Farmer?
Critics of the haystack argument for massive, random searching in connection with surveillance will argue that surely the 4th amendment cannot be compromised or, worse, nullified by so folksy and simple or simple-minded an analogy as that of the needle-in-the-haystack.
Is it conceivable that the sharpest minds of colonial America that applied themselves to the task of defining and delimiting proper search methods would have been unprepared for such a critique—especially given that so many of them, as gentlemen farmers, owned and managed haystacks and pitchforks?
What Recruiters Would Say about the Needle-in-the-Haystack Approach
What would the sharpest recruiters say about the haystack principle, if they were tasked with framing a comparable search principle, e.g., “To find a candidate needle, you have to search the whole haystack”?
What they would immediately respond with would be
“Are you kidding? Who’s got the time to go through every municipal phone book or people-finder 411-style phone listings and local newspaper “job-wanted ads”, much less collect, prioritize and review everyone’s posted resumes, e.g., from the haystack of online resumes, Twitter, Facebook and LinkedIn self-promotions and job seeker ads? It’s not possible! Or if possible, it’s neither necessary nor efficient.
Even if it’s all done with super-computers, how much manpower will be required to prioritize and review the not-so-short “short list”, which, if too long, will short-circuit and fry the whole process, or at least seriously drain or distract our energies and siphon off our time?
On top of that, what do we do for follow-up? Phone everybody? Ask their neighbors to inform us of any signs of new employment, degree of seriousness of job search, or try to gather information about job-search-related credit-card expenditures by rummaging through everyone’s rubbish—and ignoring anything unrelated to those?
If you want me to search through haystacks, as opposed to luring in whomever I’m looking for, at least make it a manageably-sized and maximally relevant haystack!
Besides, a lot of those people are going to get seriously ticked off by all that snooping and unsolicited interest.”
Finding Common Ground
Proponents of the needle-in-the-haystack metaphor can effectively make their case—if, and it’s a big if, they can reach common ground with their critics regarding the qualitative and quantitative features of the “haystack”.
This means being very specific about not only the size, e.g., everybody on the planet?, but also on the criteria for inclusion as a potential needle—most reasonably based on statistical profiling that amounts to “probable cause” or some variant of that, with the proviso that the narrower grouping criteria are based on some unlawful or immoral agenda.
The Dangers of Obsolete Technological Metaphors
Then again, it may be that talking about haystacks and needles is altogether wrongheaded, irrespective of which side of the debate one takes or in what ways the haystacks and needles are delimited. One reason it may be wrongheaded is that when technology is used as a metaphor it quickly becomes obsolete and muddling.
For example, consider the metaphorical, analogical history of the human mind and brain: Early on, at the height of ancient Greek speculation, specifically, for Plato, the mind was a “bird cage”—which of course could only have so many birds inserted into it before at least one had to be removed. This completely obscured the self-repairing, threshold firing, incalculable capacity, creative and associative powers of the brain.
Much later, the brain/mind was a telephone switchboard—that likewise suggested inputs and outputs, but which lacked the holographic coding capabilities more recent models of the brain suggest, which in turn will be superseded by whatever comes next, technologically speaking.
Like the birds-in-the-cage analogy, the needle-and-haystack model may be grossly obsolete and inadequate, as the technological model that it is— at least to the extent of embodying a technology of needles lacking a complementary technology of magnets.
What the critics and supporters alike—present and future—of the needle-in-the-haystack model may discover, from this technological perspective, is that if we re-examine the premises, implications, limitations, costs and relevance of the “needle-in-the-whole haystack” metaphor and approach, we may yet find our compass, true course and magnetic bearings, or at least start looking for a different kind of straw and at a different haystack, without further inflaming the debate…
…unless, of course, the search for that too is like looking for a needle in a haystack.