Connecting the Dots: Finding Patterns in Large Piles of Numbers
http://m.theatlantic.com/technology/archive/2011/12/connecting-the-dots-finding-patterns-in-large-piles-of-numbers/250126/Connecting the Dots: Finding Patterns in Large Piles of Numbers
Rebecca J. Rosen | Dec 16, 2011
A new program can find and compare
relationships in complicated data without
having to be asked specific queries
Are there subtle patterns lurking in data that
can foretell of a coming financial-system
crash? What can explain the variations in
sports-star salaries? How about the complex
relationship between genes and certain
diseases? Scientists in various fields have
been searching for better ways to analyze
large piles of data for such patterns, but the
difficulty has always been that they need to
know what they're looking for in order to
find. A new software program, described in
the latest issue of Science , is designed to find
the patterns in data that scientists don' t know
to look for.
David Reshef, one of the scientists behind
MINE, as the program is called, explains ,
"Standard methods will see one pattern as
signal and others as noise. There can
potentially be a variety of different types of
relationships in a given data set. What's
exciting about our method is that it looks for
any type of clear structure within the data,
attempting to find all of them. ... This ability
to search for patterns in an equitable way
offers tremendous exploratory potential in
terms of searching for patterns without
having to know ahead of time what to search
for." MINE compares different possible
relationships (including linear, exponential,
and periodic) and returns those that are
strongest.
On MINE's website , the program is available
for download.
= new reply since forum marked as read
What's the meaning of pi? 
Much like a schizophrenic, that seens patterns where there are none, some algos might overinterpret the data handed to them, and nonsense conclusions might arise.
This is an interesting topic, given the tone that many threads on DU have. You often have 2 options when dealing with a polemical topic: The official version, and the conspiracy theory version. The first option is suspicious in many ways, given that it must have a tailored interpretation that suits the interests of whomever influences it; distrust of government is often due to the acknowledgement of this simple fact. The second option requires a highly disciplined and educated mind, so as to sort out the various inanities that are often encountered within it.
Disinformation campaigns operate in full recognition of the first option, and utilize the insecurities associated with the second option. For example, I remember reading in the 1990s, in the San Jose Mercury News, that the CIA had been involved in peddling crack cocaine to black communities throughout the country. Was this true? Was it disinformation? Who planted this story, if untrue? And why?
There are so many examples of this. Often times conspiracy theories turn out to be correct, such as the experience that many left wing organizations have had with COINTELPRO and MKULTRA; or the experiences that right wing organizations have had with government encroachment on civil liberties. Or the suspicions within the hacker communities regarding collaborations between the NSA and Microsoft (this is, in part, a large reason for the movement towards GNU/Linux).
My personal opinion is that the official stance is usually, if not always, incorrect and quite mendacious. But, the trickery is in being able to see through it, and form a rational picture in one's mind without:
a. Devolving into some sort of pathological paranoia,
b. Arriving at the wrong conclusion (very easy to do!),
c. Being excluded from "polite company" 
Anyways, our brains are hard at work, devising failproof algorithms for sorting out the data that we are presented with. It is not surprising, in the least, that simple numerics would run into this issue as well.
