Dark Data

As we all know, the only research that we ever see is research that is successful. I start with a null hypothesis (i.e., I believe X is driven by Y). I then collect and analyze data to look at facts to see if I can prove the null hypothesis. If I don’t prove it, I move on to another project.

Occasionally, I find out something completely surprising which makes a career or a performance year. Think about all the analysts that find correlation between different variables and the stock market. For example, the stock market does X after a democrat is elected. The stock market does Y after a long, cold winter.

In healthcare, there is an amazing amount of clinical data out there being collected and analyzed. People are looking for new cures and new drugs all the time. The question is what happens with all the “dark data” that gets put in the closet. Should it be shared? How? Would it help other people?

I don’t know the answer, but I am a big believer that more data is better. If I can predict something off just pharmacy data, I should be much more accurate with medical claims and lab values (for example).

In another Wired Magazine article from October 2007 called “Mind the Gaps” by Thomas Goetz, he talks about this topic and several efforts here:

No comments yet... Be the first to leave a reply!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: