Big data analytics refers to the process of collecting, organizing and analyzing large sets of data (“big data“) to discover patterns and other useful information. Not only will big data analytics help you to understand the information contained within the data, but it will also help identify the data that is most important to the business and future business decisions. Big data analysts basically want theknowledge that comes from analyzing the data.
The Benefits of Big Data Analytics
Enterprises are increasingly looking to find actionable insights into their data. Many big data projects originate from the need to answer specific business questions. With the right big data analytics platforms in place, an enterprise can boost sales, increase efficiency, and improve operations, customer service and risk management.
Webopedia parent company, QuinStreet, surveyed 540 enterprise decision-makers involved in big data purchases to learn which business areas companies plan to use Big Data analytics to improve operations. About half of all respondents said they were applying big data analytics to improve customer retention, help with product development and gain a competitive advantage.
The Challenges of Big Data Analytics
For most organizations, big data analysis is a challenge. Consider the sheer volume of data and the many different formats of the data (both structured and unstructured data) collected across the entire organization and the many different ways different types of data can be combined, contrasted and analyzed to find patterns and other useful information.
The first challenge is in breaking down data silos to access all data an organization stores in different places and often in different systems. A second big data challenge is in creating platforms that can pull in unstructured data as easily as structured data. This massive volume of data is typically so large that it’s difficult to process using traditional database and software methods.
Big Data Requires High-Performance Analytics
To analyze such a large volume of data, big data analytics is typically performed using specialized software tools and applications for predictive analytics, data mining, text mining, forecasting and data optimization. Collectively these processes are separate but highly integrated functions of high-performance analytics. Using big data tools and software enables an organization to process extremely large volumes of data that a business has collected to determine which data is relevant and can be analyzed to drive better business decisions in the future.
Examples of How Big Data Analytics is Used Today
As technology to break down data silos and analyze data improves, business can be transformed in all sorts of ways. According to Datamation, today’s advances in analyzing Big Data allow researchers to decode human DNA in minutes, predict where terrorists plan to attack, determine which gene is mostly likely to be responsible for certain diseases and, of course, which ads you are most likely to respond to on Facebook. The business cases for leveraging Big Data are compelling. For instance, Netflix mined its subscriber data to put the essential ingredients together for its recent hit House of Cards, and subscriber data also prompted the company to bring Arrested Development back from the dead.
Another example comes from one of the biggest mobile carriers in the world. France’s Orange launched its Data for Development project by releasing subscriber data for customers in the Ivory Coast. The 2.5 billion records, which were made anonymous, included details on calls and text messages exchanged between 5 million users. Researchers accessed the data and sent Orange proposals for how the data could serve as the foundation for development projects to improve public health and safety. Proposed projects included one that showed how to improve public safety by tracking cell phone data to map where people went after emergencies; another showed how to use cellular data for disease containment.
Key Challenges to Big Data Analytics
The challenge of Big Data is a daunting one. We all know that data is exploding, but just how much data is out there? No one is certain, but former Google CEO Eric Schmidt has argued that we now create an entire human history’s worth of data every two days. “There was 5 exabytes of information created between the dawn of civilization through 2003,” Schmidt said a couple of years ago, “but that much information is now created every two days, and the pace is increasing.”
Those numbers may be exaggerated. RJMetrics CEO Robert J. Moore said in a TEDx talk recently that “23 exabytes of information was recorded and replicated in 2002. We now record and transfer that much information every seven days.”
Gartner believes that enterprise data will grow 650 percent in the next five years, while IDC argues thatthe world’s information now doubles about every year and a half. IDC says that in 2011 we created 1.8 zettabytes (or 1.8 trillion GBs) of information, which is enough data to fill 57.5 billion 32GB Apple iPads, enough iPads to build a Great iPad Wall of China twice as tall as the original.
The pace of data creation will surely increase, especially as machine-to-machine communications gets cheaper and more common. Think about how much data all of those sensor networks, burglar alarms and vehicle telematics systems will create.
According to IBM, every single day we create 2.5 quintillion bytes of data. IBM argues that the exponential growth of data means that 90 percent of the data that exists in the world today has been created in the last two years. “This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, e-commerce transaction records, and cell phone GPS coordinates, to name a few.”
Of course, it’s important to remember that in early human history, anything as ephemeral as a tweet just would not have been recorded, so these comparisons can only be taken so far.
To put the data explosion in context, consider this. Every minute of every day we create
- More than 204 million email messages
- Over 2 million Google search queries
- 48 hours of new YouTube videos
- 684,000 bits of content shared on Facebook
- More than 100,000 tweets
- $272,000 spent on e-commerce
- 3,600 new photos shared on Instagram
- Nearly 350 new WordPress blog posts
Another challenge facing Big Data analysts is the fact that data is stored all over the place, in different systems. Breaking down data siloes is a major challenge. Another is creating Big Data platforms that can pull in unstructured data as easily as structured data.
When you get into the Big Data weeds, though, more arcane challenges emerge. For instance, traditional databases were not designed to take advantage of multicore processors. Thus, they are much slower at processing data than they could be, which has led to the concept of “Fast Data,” with startups such as ParStream attempting to overcome various legacy issues associated with databases.
From Accumulation to Analysis
Whatever the exact number, we have a lot of data to contend with. Accumulating data is one thing. Doing something with it is another. You wouldn’t refer to a hoarder who accumulates old newspapers, empty tuna fish cans and live kittens as a “discerning collector,” after all. You wouldn’t visit a hoarder’s house to learn about history, the way you conceivably could from, say, an antiques collector. The signal-to-noise ratio is just too low.
With data, though, the world is full of hoarders. Digital storage is so cheap that people store everything—or, more accurately, don’t bother to delete anything. The same is true online, where online storagevendors now routinely give away GBs of data storage before charging a thin dime.
Today, businesses are struggling to contend with this out-of-control data sprawl—because if they don’t, they won’t stay competitive.
According to IBM, exponential data growth is leaving most organizations with serious blind spots. IBM found that one in three business leaders admit to frequently making decisions with no data to back them up. Their decisions are either based on information they don’t have or don’t really trust. Even more surprising, one in two business leaders admit that they don’t have working access to the information they need to effectively do their jobs.
Most business leaders and knowledge workers know that relevant data is out there, but they don’t know where. Even if they have a rough idea, they’re not sure how to extract it in any meaningful way. Finally, once they manage to find relevant data, they often aren’t sure how current or accurate it is.
This is where Big Data analytics comes in. What we’re after isn’t just raw data. We want the knowledge that comes from analyzing that data.
What You Can Learn from Big Data Analytics
As technology to break down data siloes and analyze data improves, business can be transformed in all sorts of ways. The advances in analyzing Big Data allow researchers to decode human DNA in minutes, which makes businesses like 23andme feasible.
Researchers are able to predict where terrorist plan to attack, which gene is mostly likely to be responsible for certain diseases and, of course, which ads you are most likely to respond to on Facebook.
In fact, a recent study published in PNAS found that the things you “like” on Facebook reveal all sorts of probable traits about you, such as your intelligence, your gender, your sexual preference, your political leanings and more.