Big Data: The Mega-Trend That Will Impact All Our Lives
There are some things that are so big that they have implications for everyone’s life, whether we want it or not. And Big Data is one of those mega trends that will impact everyone in one way or another. The name (which by the way I don’t like) might sound a bit techie or boring but believe me, it is not. With this post I want to explain what’s behind this mega buzzword and outline why it will impact everyone.
The basic idea behind the phrase ‘Big Data’ is that everything we do in our lives is (or will soon) leave a digital trace (or data), which we (and others) can use and analyze. The advances in capturing and analyzing big data allow us to decode human DNA in minutes, find cures for cancer, accurately predict human behavior, foil terrorist attacks, pinpoint marketing efforts, prevent diseases and so much more. And like most things, it can be used for good or evil, but more on that later.
Basically, big data refers to our ability to collect and analyze the vast amounts of data we are now generating in the world. The ability to harness the ever-expanding amounts of data is completely transforming our ability to understand the world and everything within it. You might ask: So what is new here? Haven’t companies and organizations captured and analyzed data for a long time? Yes, but there are two things that are changing at the moment and are making the phenomenon of ‘big data’ real:
- The rate at which we are generating new data is frightening – I call this the ‘datafication’ of our world.
- Our ability to analyze large and complex forms of data has been transformed in recent years.
The Complete Datafication of Our World
All activities (human or otherwise) will soon leave a digital trace (which can be a scary thought):
- We increasingly leave digital records of our conversations: Emails are stored in corporate systems, our social media up-dates are filed and phone conversations are digitalized and stored.
- More and more of our activities are digitally recorded: Most things we do in our digitalized world leave a data trail. For example, our bowser logs what we are searching for and what websites we visit, websites log how we click through them, as well as what and when we buy, share or like something. When we read digital books or listen to digital music the devices will collect (and share) data on what we are reading and listening to and how often we do so. And when we make payments using credit or payment cards the transactions are logged.
- Most photos and videos are now digitally captured and stored. Just think of the millions of hours of CCTV footage captured every day. In addition, we take more videos on our smart phones and digital cameras leading to around 100 hours of videos being up-loaded to YouTube every minute and something like 200,000 photos added to Facebook every 60 seconds.
- We generate data using the ever-growing amounts of smart devices and sensors: Our smart phones track the location of where we are and how fast we are moving, there are sensors in our oceans to track temperatures and currents, there are sensors in our cars that monitor our driving, there are sensors on packaging and pallets that track goods as they are shipped along supply chains. Smart watches, Google Glass and pedometers collect data. For example I wear an Up band that tells me how many steps I have taken, the calories I have burnt each day as well as how well I have slept each night, etc. Many devices are now internet-enabled so that they self-generate and share data. Smart TVs and set-top-boxes, for example, are able to track what you are watching, for how long and even detect how many people sit in front of the TV.
I am sure you are getting the point. The volume of data is growing at a freighting rate. Google’s executive chairman Eric Schmidt brings it to a point: “From the dawn of civilization until 2003, humankind generated five exabytes of data. Now we produce five exabytes every two days…and the pace is accelerating.”
So yes, we are generating unimaginable amounts of data. The other thing that has changed is that are now able to analyse more complex types of data such as digital phone records of conversations, video and photo images and conversation. In the world of ‘Big Data’ we talk about the 4 Vs that characterize big data:
- Volume – the vast amounts of data generated every second
- Velocity – the speed at which new data is generated and moves around (credit card fraud detection is a good example where millions of transactions are checked for unusual patterns in almost real time)
- Variety – the increasingly different types of data (from financial data to social media feeds, from photos to sensor data, from video capture to voice recordings)
- Veracity – the messiness of the data (just think of Twitter posts with hash tags, abbreviations, typos and colloquial speech)
So, we have a lot more data than ever before, in more complex formats, that are often fast moving and of varying quality – why would that change the world? The difference is that we now have tools that allow us to analyze vast amounts of data by breaking the task of processing very large data sets down into smaller tasks that are run in parallel using a large cluster of computers. Here are some real-life examples of how big data is used today:
- The FBI is combining data from social media, CCTV cameras, phone calls and texts to track down criminals and predict the next terrorist attack.
- Supermarkets are combining their loyalty card data with social media information to detect and leverage changing buying patterns. For example, it is easy for retailers to predict that a woman is pregnant simply based on the changing buying patterns. This allows them to target pregnant women with promotions for baby related goods.
- Facebook is using face recognition tools to compare the photos you have up-loaded with those of others to find potential friends of yours (see my post on how Facebook is exploiting your private information using big data tools).
- Politicians are using social media analytics to determine where they have to campaign the hardest to win the next election.
- Video analytics and sensor data of Baseball or Football games is used to improve performance of players and teams. For example, you can now buy a baseball with over 200 sensors in it that will give you detailed feedback on how to improve your game.
- Artists like Lady Gaga are using data of our listening preferences and sequences to determine the most popular playlist for her live gigs.
- Google’s self-driving car is analyzing a gigantic amount of data from sensor and cameras in real time to stay on the road safely.
- The GPS information on where our phone is and how fast it is moving is now used to provide live traffic up-dates.
- Companies are using sentiment analysis of Facebook and Twitter posts to determine and predict sales volume and brand equity.
- A hospital unit that looks after premature and sick babies is generating a live steam of every heartbeat. It then analyses the data to identify patterns. Based on the analysis the system can now detect infections 24hrs before the baby would show any visible symptoms, which allows early intervention and treatment.
Finally, no discussion about Big Data could be complete without mentioning the increasing concerns about privacy. Many concerns have been expressed about how retailers, credit card companies, search engine providers and mail or social media companies use our private information. However, the privacy concerns around big data started to explode with the revelations by Edward Snowden on how the U.S. National Security Agency (NSA) collects and analyses big data including the phone records and social media activities of millions of Americans. But because this is another massive issue in its own right I will address this in a future post.
As always, please let me know your thoughts on the topic. Do you find it frightening or exciting? Do you see business opportunities or ‘Big Brother’?
Bernard Marr is a best-selling business author and enterprise performance expert. He helps companies understand big data and works with executive teams to develop big data strategies. Make sure you click ‘Follow’ if you would like to hear more from Bernard Marr in the future and feel free to also connect via Twitter, Facebook and The Advanced Performance Institute
Other recent posts by Bernard Marr:
- What The Heck is a… KPI?
- How Facebook Exploits Your Personal Information
- Google: Forget Privacy When Using Gmail
- What The Heck is… Analytics?
- The 2 Words That Will Change Your Life Forever…
- What The Heck is Wrong With Mission and Vision Statements?
- How Influential Are You? Measure It
Bernard Marr’s book ‘The Intelligent Company‘ outlines how companies can better leverage the data available to them today to improve decision-making and generate a competitive advantage.
You can read a free sample chapter here.
- Featured on:
Anita DalrympleQuality Engineer and Industrial Statistician
“1984” was a book by George Orwell 35 years before it was an Apple advertisement. Isaac Asimov wrote “I, Robot” 18 years before Will Smith was born. Asimov also wrote the amazing “Foundation” series. All of these stories include the theme of how governments will control us with the trace we make. This may sound a little “conspiracy theory” to the very young or uninformed, but the world has changed, and our privacy is a thing of the past. We readily give up the remnants for tokens and convenience (think free apps). In history, we have seen this privacy trade with the Nazis, the Communists, and with other extremists. For the most part, these groups only had “Little Data.” Now we know just about every government on earth has access to “Big Data.” And in the words of George Santayana, “Those who cannot remember the past are condemned to repeat it.” Maybe we should think about the value of what we get for what we have to give.4 Replies
Sam VellaFreelance Business Intelligence Architect and Developer
My Big Data processors indicate we are only 6 months away from “The Trough of Disillusionment”1 Reply
Susha Y. PerminovaSoftware Sales Specialist at Halo Software LLC
This is very frightening, but what oi find even more frightening is that we are degradating as human beings by constantly being attached to our phones,TV’s and Computer monitors!1 Reply
James HuntAcademic Advisor at University of South Alabama
Yes, I agree with you. But we have to remember that ultimately we are in control. Facebook’s market value is estimated at $100 billion. But that isn’t stable. It’s not real. If the younger generation decided not to use Facebook, or if the public decided to stop using it, the value would be zero. I work with college students and believe me when I say that many of them are not using it. They may have an account for occasional use, but most prefer Twitter. I always hear comments like “Facebook is for my parents”. I think as more consumers are educated about privacy issues, the tide will turn. What do you think?
2 months ago
Paul H McleanDirector of Success at A Brand New Future
…I think there will be a market for ‘little data’ which actively minimizes your digital footprint to help some of your world yours alone…knowing when a woman is pregnant and targetting her with product placements is really extreme – you really do get a sense of a concerted drive for society to be force-ably pushed onto the digital nipple whether they like it or not in an effort to eliminate some of the ‘known unknowns’ of our personal lives for the specific benefit of the big business and governmental worlds…big data may become the point that future generations may come reflect upon as the moment when the world of the individual became nothing but an update on someones screen somewhere in the world…
Sumit BatraProject Manager at Accenture
Big data is the biggest challenge that customers are struggling with. What futuristic technologies can help create marketable customer intelligence from the big data?7 Replies
Princess Cox, MBA, PMPDigital Marketing & CRM Professional
@ Dave Dave, when it comes to Mexx, Zara, & what is/isn’t BigData please blame my memory and lack of detail, not the case. Because one of these two deserves their proper credit for working with what at that time was BigData (datasets so large & queries so complex the standard systems of that period could not process them) ahead of their time. They also deserve credit for using data related to customer behavior and real-time employee insights to drive everything from real-time stock control (as you mentioned) to promotions and pricing. And when we’re talking about “stock control”…we’re not referring to 30 yellow sweaters. We are talking about nearly a decade ago, being able to adjust an entire promotional strategy and supporting distribution nearly overnight because you could analyze the insights and opinions of 3,000 retail associates (if memory serves) against all purchasing trends and related customer data. I understand that nowadays you may just call that good stock control but in that time and context it was impressive in the world of customer insight. Your comment seems to stem from a clear opinion on what BigData is/isn’t. I’m curious to hear your perspective.
Princess Cox, MBA, PMPDigital Marketing & CRM Professional
@ Sumit – Thank you for the clarification. In that context, I had to smile and say touche when I read the words “fictional sci-fi” in your reply as I also posted in this forum on the implications for Big Data of futuristic technologies built around FACS microexpression recognition and science fiction at the level of the fictional show “Caprica” coming to life within 20years. Nonetheless, I would still love to see more discussion on the opportunities of today…how practitioners can create the most value for clients that dream in decades, strategize in years, but report in quarters. Because the statistics say we’re not doing that. They say less than 50% of clients would rate Big Data projects a success and technology projects in general, much worse. As you stated in your reply, there are already organisations (some of them my clients) churning Big Data into marketable customer intelligence. In my experience, these organisations have a strong strategic objective, a detailed roadmap, and a set of very specific insights they need to mine in order to drive the customer relationship. However, the majority of clients I meet have the needed tools but are still saying “where do I start? I know that no one wins points in a consultancy by telling the client “you don’t need an upgrade or to integrate the latest accelerators – you need to start asking the right questions and train your people to think differently about the answers”…but that often is the solution that delivers the most value for our clients and creates the best long-term relationship. So yes, I respect your forward-thinking and the topic – and I’d love to know your thoughts on something like micro-expression recognition & Big Data. I would just also love to see more brainstorming in this forum on increasing our clients’ success in gaining marketable customer intelligence from the Big Data possibilities that we already have…because that’s my clients’ pain today. Thanks for taking the time to share your perspective and feedback.
- Show More
Evan Quinn 2ndResearch Director, Information Management at EMA
August 27, 2013 and Bernard discovers Big Data and the 3Vs + 1. Uh, a little late, doncha think? I noticed Rob Enderle discovered big data the other week too. Thought leadership in the rearview mirror. Hey, don’t forget about visualization and value – 6Vs actually.
Catherine Kargas 2ndBusiness Strategist, Vice President at MARCON, Chair of Electric Mobility Canada, EMC Autonomous Vehicle Chair
Data has become an incredibly valuable commodity1 Reply
Princess Cox, MBA, PMPDigital Marketing & CRM Professional
Nice. A simple way of telling an intricate story. And yet…this isn’t the end of the story. The data we’re looking at right now is all 2D. What happens when we’ve effectively taught computers the 5 senses and commercialized the applications? Or a closer development, FACS (facial action coding systems – recognition of the emotions that can be identified clearly through involuntary micro-expressions / muscle movements on your face) on your SmartPhone browser telling marketers what you think of their ads in real-time? The dimensions will be indescribably complex…like the human brain. With the right analytics, behavioral scientists, and enough processing power you could even create a model of an entire human life / personality. It’s like “Caprica” walking off the TV screen and into the real-world.3 Replies
Princess Cox, MBA, PMPDigital Marketing & CRM Professional
Dave, I hear you. Sounds a bit out there…until you get into a discussion with theoretical physicists and engineers of how science fiction has influenced scientific research. In fact, here’s a little blurb from noted theoretical physicist Dr. Michio Kazu on that exact topic: http://curiosity.discovery.com/question/science-fiction-influence-scientific-research
- Show More
Dave WilliamsSenior Systems Engineer at Control Systems Laboratories, LLC
In asking what “I” see can probably be summed up as “insufficient data”. What do I mean by insufficient data? Two things, the old school reference to robotic sci-fi’s that referenced responses to natural language queries posited to a machine. I don’t remember whether it was Spook or not. We can already see that the NSA, having more than sufficient data (no one can accuse them of “under-sampling”) could not isolate the Boston bombing. What is troubling is the the rationale used to collect and amass this “Big Data” does not comport with the performance. Besides the socio-political and psychological effects of mass surveillance (either state or corporate based) has the unknown affect on free peoples. Behaviors tend to conform to the data, not the other way around. To me, this produce “insufficient results” and we rarely understand the downside of technologies because of a tendency to have blind faith in technology. There was a great thesis out of Oxford University that covered this topic (masters/PhD level treatment). Another piece of work, it was a science and technology policy research project that looked at the difference between formalism in science, the utility of that science in policy, and the disconnect between what the science says and what policy produces,. Over 800 pages, read it cover to cover in one day–it was extremely compelling. My own data, memory, is failing me–cannot remember the exact title.1 Reply
Rajesh PillaiProgram Manager at Confident Governance
BIG DATA : I think it is an exciting opportunity to learn more about the physiological behaviour of the human being. All the data that has been collected refers everything about a human behaviour, think about the predications that we can make by analyzing the “BIG DATA”. I am shocked / surprised to see the quantity of data that has been generated and monitored every second. It does bother me about the privacy but more then that I am thinking of the benefits that we can reap in future with this data. My eyes brows touching each other think about “do we really have right tools to analyse such enormous data?”
Mauricio Monteiro BragaIT Architect Specialist at Lojas Renner
From the point of view of each one: be carefull about the data you are producing. Regarding the companies, this premise is even more stronger. For some companies, complexity of each technological stack versus the pursuit of optimization of the OPEX, might transform cloud computing in a ‘good’ place for IT infrastructure, and also to big data in a ‘sustainable step’. And, here most companies are faced with: the everlasting cycle of trade-offs (information security, technical architecture, enterprise architecture, etc.). The fact is that the ‘datafication’ is happing all the time and there are several ‘actors’ that, talking politically, they might be able to use a big data platform to take advantages. So, the companies should be so stuck in the trade-offs and paradigms?
Yann Le MeurInternational HR student at INSEEC Paris
I don’t know how one can say here it’s frightening: If you’re scared, log off!1 Reply
Scott WagersHelping researchers get more funding and find more time by optimizing their gain from collaboration.
Nice post on a lightening rod topic. Like any valuable technology Big Data is a double edged sword. Every year lots of houses, buildings, and forests burn down does this mean we should stop using fire? You could almost say the more useful something is, the more potentially dangerous it is. Our growing ability to analyze and use Big Data is driving the degree to which companies and academia collaborate. Big Data in translational research is often not all that big in terms of volume, rather it is big in terms of complexity. The Pharmaceutical industry is facing a big challenge in that many drugs fail in the last phases of development. There is now a lot of interest in collaborating to combine and analyze data sets to facilitate earlier prediction of which drugs will be successful. See http://www.etriks.org for examples of what I mean.
Thomas EverchildIllustrator, Writer, Director
Did you know that the vast majority of Scientists no longer believe the standard model is applicable? One can also float controversial opinions in posts and analyse the sentiments in the comments. The notion of Big Data accurately predicting behaviour does not mean that the prediction is accurate. Governments and corporations will act on what they are persuaded to believe is accurate by the advisers who are ‘on trend’. Much of the data collected is ‘follower data’, votes for favourite films, books, songs are skewed because voters go along with the crowd. A standard method of making the population believe something is true is simply to use the media to tell them that many others already believe it. Fake the results of big data to get what you want. Pattern recognition has some value, but it won’t be long before there are softwares that claim to hide or confuse your datawake ™. Like firewalls in reverse. Facial recognition on Facebook has a way to go. I was recently presented with a photograph of a ‘friend’ to identify as part of a security measure, it was a picture of a Christmas tree that someone had tagged with the person’s name. The first sentence in this comment is a lie. Or is it? Please look out for my new book ‘Datawake’ ™, published soon.
- The facts behind the hype: What the Gartner big data 2013 report means for the industry (itproportal.com)
- Big Data Could Bring Big Profit (econmatters.com)
- Big Data – Is Big Data Relevant to me? – Big Data Questionnaires – Guest Post by Vinod Kumar (sqlauthority.com)
- Big Data to Digital Marketers (joannchiou.wordpress.com)
- Sogeti trend lab VINT launches its 4th Big Data Report (virtual-strategy.com)
- How Big Is Big Data? (hispanicbusiness.com)
- Big Data To Become Powerful Driver Of IT Spending – Gartner (misco.co.uk)
- How behavioural design can overcome the dark side of big data (theguardian.com)
- 3 keys to rule Big Data economy : tech, people + culture | #BigDataNYC (siliconangle.com)