Why not? The statistical significance is indisputable (it represented a difference of six standard deviations). The tests were clean. The scientists have rechecked their work multiple times.
The reason they aren't celebrating is they know the data may not represent what it seems to. Compare this to business, where we routinely celebrate numbers that are much fuzzier than these.
Healthy SkepticismOne useful trait many web analysts share is a deep skepticism of the numbers they promote. It is useful, because there are many times that we simply do not know what the data represents.
One reason for this uncertainty is that the measurement methodologies themselves can be wrong without us realizing it. To quote the article: "If you are measuring something incorrectly, it doesn't matter that you measure it very carefully."
Despite how carefully we measure things in business, there are so many steps in the measurement process that a slight systemic error can lead to numbers that do not really represent what we think they do. This does not mean that the numbers are wrong, just that they represent something different than what we thought.
When measuring people, we can't ever take our numbers for granted. We have to be familiar with all the common measurement errors and try to account for them. It is not enough just to recognize that there are measurement errors and then choose to ignore them because we can't do anything about them. We have to find ways (sometimes new ways) to fill in the gaps and get the data we thought we were actually going to get.
What is the Business Cycle?what motivates people to buy them or when or how long it takes them or what all of the influencing factors are in that decision. We can't create repeatable tests, because there are innumerable variables at play, many of which we don't or can't measure. Some of them we don't even know exist, so we never ask about them to begin with. Visitors to a website can be distracted by bad site design or by their neighbor's dog barking.
In business we assume that if a finding is statistically significant it is accurate. Unfortunately, it is often difficult to go much deeper than aggregated reports. Without being able to see the entire activity of a single person (which is becoming even less reliable with time), it is hard to understand why they did what they did. Without understanding that, we cannot intelligently optimize our efforts to help them.
The physicists observed the aggregated measurements of many neutrinos hitting a sensor. Physicist Joseph Lykken reviewed the study and said, "You have a proton beam ... that makes the neutrinos, but you don't know which proton made which neutrino. This makes it tough to claim nanosecond timing of the neutrinos."
In business we might look at a problem like this and decide to use statistical modeling to find the truth. To the physicists who said the same thing Lykken responded, "Maybe so, but normally in experiments you use something well understood to measure something messy, not the other way around."
This means that in business we have to use multiple sources (as opposed to a single laboratory experiment). We should triangulate our findings with surveys and focus groups. We should take advantage of social media to get a feel for how people interact with our brand and what the real triggers are in the purchase cycle. We should use multiple tools to confirm that there is real causation and not just correlation.
Measuring Messy with Messy
In many ways the physicists have it easier. People behave in a less reliable way than neutrinos. It's not impossible to get actionable data when analyzing people's behaviors. But we do have to remember that we're dealing with people, and the numbers don't always do a good job of representing what they are thinking or how we can best help them. The bigger the finding, the more cautious we have to be in confirming it.