Saturday, October 20, 2012

Big Data - Connecting Dots...

Advancement of information communication technology enabled us to generate almost 2.5 quintillion bytes of data everyday and more than 80% of available data so far were generated only in last two years. This data is being generated through various sources- social media posts, climate information sensors, GPS signals, surveillance, digital pictures, videos, commercial transaction records, and medical records to name a few. Isn’t it data big?

Yes! This is big data.

Big data is next wave to address the scalability of super human like intelligence to process data of any kind and any size. Big data promises & enables enterprises to pursue top opportunity and address top challenges. Big data isn’t restricted to enormous volume of data but also takes care in processing of higher velocity, greater variety and real veracity of data to drive crystallize intelligence – deriving valuable, actionable (insights) information. Big data is confluence of three fundamental technology trends to open tremendous opportunity - Big transaction data, big interaction data, and big processing data to drive better cost structure and new business avenues.

Structured data analytics is a traditional approach to manage data and derive business value out of them. Information out of structured data isn’t deriving business value in real sense as needed, because 80% of data in due course, isn’t structured. I've had the opportunity to meet some amazing people in my life those, who are capable of processing humongous amount of data (both structured and unstructured) with ultimate efficiency to derive information on runtime as needed. However such intelligence is individual centric and not scalable. Configuration of human brain - capability of storage and processing of data (historical and current) varies by and large globally.

Wave of cloud and mobile is changing the economics & role of computing and will define literal information sources available across the globe. Enterprises would like to quickly adapt the changing environment in advancement of globalization milieu. Business enterprises expect replication of super human intelligence within an automated system - so can add value of higher degree in business growth.

Social media has become main stream technology for business today, nevertheless social media is much more than facebook, tweeter, and google of the world. Social interaction driving higher volume and even greater variety of data. The value preposition of business computing is shifting from operational efficiency to organizational effectiveness. Relating social media and correlating device sensor data, big data expedites the processing of data in any form to make better decision with quality of information.

We have been dealing with relational data for last few decades, as they all were structured data and stored into massive databases, warehouses, data mart to process indexed information. But big data promises beyond just glass house to get market advantage. Big data will not operate in silos, so integration with traditional data warehouses be integral part of the game.

Let the data be stored in the size of Gigabyte, Terrabyte, Petabyte or even Yottabyte but information should not exceed even few kilobyte.

I sought to convey a simple message to the audience that integrity of information does not only depends on state of data available but rely on logical parsing of data to plot functions.

Business will get breath of fresh air through truthful information of consumer sentiments analysis and business relevance.