Technology moves through a period of inflated expectations and disillusionment before reaching productivity. Where is Big Data?
Years ago, the research firm Gartner coined the term “Hype Cycle” to explain how new technologies tend to go through distinct phases of market acceptance, from inflated expectations to disillusionment to productivity.
Where is Big Data on this cycle?
First, a quick primer on Big Data: This much-used jargon refers to a set of technologies designed for working with large volumes of data beyond those of traditional database management systems. These inventions stem from the needs, in the early part of the last decade, of Internet companies like Google, Yahoo and Facebook to manage user data and systems at a scale the industry hadn’t previously designed for.
“Not a substitute for what came before”
Software like Hadoop was released as open source, and spawned new startups to supply and maintain these complex systems. Like most new technologies, Big Data is not a direct substitute for what came before. To achieve the desired capacity, design trade-offs have been made. For instance, a Hadoop implementation consists of many commodity servers working in parallel. This configuration works extremely well for tasks that can be computed in parallel (think virus scanning millions of files) but awfully for work that can’t be done in parallel (in fact, a single high- performance computer is often faster for such tasks).
As we’ve used these technologies across our business units. we’ve learned that they must be used judiciously to maximize value. For the right problem, like consuming every price tick from multiple stock exchanges, we’ve found Big Data to be a cost-effective solution. Similarly, we’ve had success with business intelligence applications that would have taken longer and cost more with “traditional” solutions.
Big Data is not a panacea
We’ve had success with the Cassandra clustered database, building on it and ElasticSearch to store our Knowledge Graph using our own CM-Well technology.
But the Big Data suite is not a panacea. We’ve found sharing resources across projects (multi-tenancy) to be more fragile than it should be. Engineering staff need new training and time to understand what is a quite different paradigm. Mistakes create under-performing applications that cast doubt on the strategy. Systems like HBase and Cassandra can be used for mastering data, but a lot more care is required when solving using more mature technology.
Beyond that, the basics don’t change. Good project management, strong information architecture, clear vision and strategy are as vital as they ever were.
This brings us back to where Big Data is on the Hype Cycle. I think we’re well beyond disillusionment and are moving into productivity – but that comes with the caveat that this is just another tool in the box. For the right problem, a hammer works great, but sometimes you need a chisel as well.
Visit Innovation @ ThomsonReuters for more on how we bring together smart data and human expertise to find trusted answers.