Why The Next Big Thing Is, In Fact, A Really Big Thing

Every second, the Large Hadron Collider (LHC) near Geneva, Switzerland produces 40 terabytes of data.  That’s more data than can be currently stored and analyzed.  The scientists working on the project are forced to collect just a slim set of the data and they hesitantly ignore the rest.  While this example is at the top end of the data deluge that our increasingly digitized world is creating, we can all relate to other, closer to home examples:  Every minute, 20 hours of video is uploaded to the popular video-sharing site, YouTube.  On Facebook, the biggest and most popular social networking site, 2 billion photos are uploaded each month.  Whether it is our text messages (a volume greater than the population of the earth are sent each day), or credit card transactions that reveal the intimate details of our purchasing behaviors, or recorded search entries in search engines that tell us so much about the human experience, or the cameras that photograph and video us as we go about out daily activities, the quantity of data that humanity is collecting and storing is staggering.  And it’s increasing exponentially.  We are moving from data scarcity to data abundance and disrupting our conventional view of economics. We call it big data.

You might ask why is big data happening and what are the implications?

This data deluge is being driven by a number of factors.  First, we are increasingly migrating from an analog to digital world.  We don’t use the gelatin silver process for photos anymore; we store them as bits in an ethereal cloud.  The same applies to such things as movies, music, newspapers, books, our bank records, our bills, and our airline tickets.  It’s a very long list.  Second, it’s become cheap to store digital data.  At the time of writing, the estimated cost of a gigabyte is 10 cents.  The hard drive I am writing this blog on has 100 GB: a total cost of $10 that includes every digital photo I have taken and every song I have purchased and there is still plenty of room left over.  Finally, the tools for us to produce content, store it, and share it have become highly accessible and quite often: free!  Today, any one of us can be a publisher or credit-card collecting retailer.  This creates enormous amounts of data.

Collecting and sharing all this data clearly presents a number of challenges and in my view, incredible innovation opportunities.  This is not last year’s data-mining.  This is data-mining on steroids!  While I won’t dwell on the challenges, they are clear: increasing transparency in our lives potentially results in less privacy.  More information and equal access to it means that the value of some intellectual property is trending downward.  Easy access to content and communication tools means even the bad guys have a voice on the global stage.  And while the risks and challenges are many, I want to turn our attention to the innovation opportunities.

In order to track the spread of the H1N1 Flu virus, the Center for Disease Control (CDC) was able to leverage the search entries of a popular search engine.  Since people search for things like symptoms when they are sick and since the search engine knows where the search is located, we can easily see how large volumes of searches produce a beautiful visual of how a disease is spreading both over time and in what direction.  It’s the invisible knowledge that the data deluge unveils that makes me excited about this area.

I’ve identified five net new possibilities that big data presents:

  1. Answer formerly unanswerable questions. With so much data being collected in so many different ways, it is now possible to ask and get answers to questions we just couldn’t before.  For example, if we analyze the social connections between employees that their digital footprints create, might we be able to identify specific knowledge domain experts? Sometimes the questions will be deliberate, but I’ll bet we’ll accidentally be able to answer insightful questions we didn’t intend to.
  2. The formulation of new questions. Now that we know certain data is being collected, what new questions could we ask? Imagine how empowering that could be to C-suite executives!
  3. More informed, evidence-based decision making. How many times have you wished you had more detail on a subject in order to make a better decision? Better data can mean better and timelier decisions.  Big data opens up a whole new opportunity for competitive advantage.
  4. Democratization of data. We no longer need to build systems that silo critical data and create an enterprise digital divide.  Aggregated data in volume can be easily made available so everyone can benefit.  It can be re-purposed resulting in new value that goes way beyond its initial intent.  For example, governments all over the world are embracing open-data policies.  This gives the electorate unprecedented access to local, state, and federal insight.  For example, local community groups are using the data to combat crime and to discover societal inequalities (is this neighborhood getting an unfair access to a resource?).  Give data to your employees and they may tell you something about your organization that reduces costs or builds your next billion dollar product or service.
  5. Visualization of invisible knowledge. Big data creates amazing and valuable visualizations.  And these visualizations unveil secrets that were previously hidden.  For example, tag clouds (groups of words that become larger the more each word is used) tell us what people are talking about on social networks.  A map of the world superimposed with real-time stock trading data flows tells us a lot about global commerce.

Of course I’m only skimming the surface here.  What I’m trying to convey in this introductory blog on the subject is that big data is a big deal.  Moreover, rather than viewing it as a threat–which many folks will–this is an opportunity for incredible innovation.  Big data will bring entirely new value propositions and it will force the reinvention of entire industries and business models.

In my view, big data is the next big thing.


This entry was posted in Disruptive, Innovation, Trends. Bookmark the permalink.

5 Responses to Why The Next Big Thing Is, In Fact, A Really Big Thing

  1. Definitely agree that large data does bring about a lot of interesting opportunities. Google’s approach to translation or the personalization efforts from Netflix and Amazon are great examples of how effective large amounts of data can be.

    At the same time, big data provides a lot of interesting challenges as well. Data in itself does not help much. Being able to ask the right questions and understanding how the data can answer these questions is critical. The benefits of large data will also depend on how fast technology and algorithms can catch up to dealing with large amounts of data.

  2. Brittany Roberts says:

    I agree that is certainly is the next big thing and am very interested in seeing how “Big Data” will impact careers and future careers. For example, within the utility industry the development of a smart grid with smart meters and fiber technology will allow utilities to receive data every few seconds from every customer instead of once a month while reading their meters. It will be very interesting to see how all this data impacts not only energy usage- but also the customer service people, the data analyst people, the meter readers themselves and so on and so forth. Surely, we can predict some of these implications today- however it seems as if the majority of what we will be able to do with all this data is still unknown! People will have to make sure they keep their skills sharp and cutting edge in order to keep up with the data because as Krishna pointed out, the “data itself does not help” … but instead people must be ready to understand what to do with it!

  3. Pingback: Data hype! « Follow the Data

  4. Steve Maingot says:

    As a Health Informatician, I am particularly interested about the analysis tools to adequately support and allow you to gain value from big data repositories.

  5. Jason says:

    Jonathan, I completely agree that the amount of data available to us provides endless opportunities. New businesses/solutions are popping up regularly trying to take advantage of these opportunities.

    What I am seeing and experiencing is that there is a convergence of skills coming together to help the information explosion. Historically you have had people in an organization who have, for example, data “munging” skills (extracting, importing, parsing, filtering data) and don’t have the analysis expertise (modeling, mining) or communication expertise (data visualization, communicating results of analysis).

    The information worker of the future will be able to cover all the areas above, and then some. The challenge is finding these individuals and training those who don’t have these skillsets.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s