Personal Analytics Cloud & Big Data: SJ Tech Museum Lecture Series

In this post, I provide notes and commentary on the SJ Tech Museum of Innovation lecture series event on Sunday, April 7, 2013, Big Data for Big Decision Makers, presented by Harry Blount (Founder & CEO, DISCERN), in conversation with Angie Coiro at the New Venture Hall.

Tech Museum writers noted, "The 30-year old spreadsheet is no match for the tsunami of information available today; yet it remains the primary tool for decision-makers.” Harry Blount examined how the Personal Analytics Cloud (PAC) is able to “leverage the aggregation capabilities and the processing power of the cloud to deliver personalized signals for today’s executive decision makers.” Toward these efforts, Blount provided a number of big data case studies and showcased the key points of big data theories. His call-to-action: We need to move from data noise to data signals. 



From noise to signal to understanding to action to iteration, made possible through constant research and inquiry. Sounds fun.

Tim Gasper, writing for TechCrunch, notes, "Companies will have spent $4.3 billion on Big Data technologies by the end of 2012. But here’s where it gets interesting. Those initial investments will in turn trigger a domino effect of upgrades and new initiatives that are valued at $34 billion for 2013, per Gartner. Over a 5 year period, spend is estimated at $232 billion."

Go here for a savvy article on big data and filtering through the hype. John de Goes writes, "The meteoric rise of data science in the past few years is a testament to the fact that data is the currency of the 21st century. If you don’t do anything with your data, you’re at a severe competitive disadvantage."

What is Big Data?

"Big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions" (Big Data, 2013). 

According to IBM:

"Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few. This data is big data."

The World is Interconnected at Every Level.
Blount spoke about complexity theory and the ripple effects that occur across global systems. He drew from the example of the Thailand floods that highly impacted technology manufacturing equipment, the effects of which seriously impacted technology global supply chains (and most importantly, the people of Thailand!).

“Complexity theory has been used in the fields of strategic management and organizational studies. Application areas include understanding how organizations or firms adapt to their environments and how they cope with conditions of uncertainty. The theory treats organizations and firms as collections of strategies and structures. The structure is complex; in that they are dynamic networks of interactions, and their relationships are not aggregations of the individual static entities. They are adaptive; in that the individual and collective behavior mutate and self-organize corresponding to the change-initiating micro-event or collection of events.” (Wiki, 2013).

Taking Measure
Blount spoke about the Global Volatility Index in reference to the significant changes occurring in global markets. Big data analysis might help with forecasting trends and proactive determination of market movement.

"VIX is a trademarked ticker symbol for the Chicago Board Options Exchange Market Volatility Index, a popular measure of the implied volatility of S&P 500 index options. Often referred to as the fear index or the fear gauge, it represents one measure of the market's expectation of stock market volatility over the next 30 day period. The VIX is quoted in percentage points and translates, roughly, to the expected movement in the S&P 500 index over the upcoming 30-day period, which is then annualized” (Wiki, 2013).

The Great Turbulence
Blount suggested we are entering The Great Turbulence, a time of economic uncertainty and market challenges, with fewer "shock-absorbers" in global markets than ever before to manage the negative ripple effects. Our global interconnection and interdependence is powerfully visaged in global markets.

He argued that between 1974 and 2000, there were scant reasons for top-level business decision makers to make significant changes in business process. Times are changing. Major shifts in population and economics (Japan, US, & China for example) demand greater flexibility in market thinking and planning.

There's that data set I misplaced.

Blount imagined data as a waterfall in the 1980s with archaic collection/analysis tools: spreadsheets, calculators, basic processing, pencil, ink, print, and paper. In the 1980s, information begins to flow with traditional libraries as central points to collect research. He then imagined data in the 21st century as Niagra Falls - torrents of data. New and expanding tools are needed. The ability to find key pieces of data needed in an avalanche of data cannot be achieved with tools invented before the Internet.

He argued that although the existing processes we use to make sense of big data have changed modestly, these processes are often reactive in nature. He suggested that through big data we can engage in persistent forecasting of disruptive technologies. The argument extended that people, out of habit and bias, often look for data in the wrong places.

So how much data are we talking about? Some 1.8 zettabytes in 2011. Thus, the challenge is to be proactively engaged in identifying tangible data that yields high value insights amidst this mass. It might be that using big data we can gather information to make more efficient transportation systems, enhance communal quality of life, track health outcomes and disease/illness rates, engage in better ecological conservation, or gauge the effectiveness of public policies to increase government efficiencies. (Side note - certainly, there are social justice issues and surveillance issues when it comes to big data collection and manipulation by state governments).

Three Problems 
Problem: Lack of Persistence. We tend to search only when we are looking for answers. How do we ask the right questions at the right times?

Solution: Constant Scan. Persistent asking and answering of questions on big data platforms.

Problem: Seeing the Expected. Most existing solutions surface only the most expected answers. Data-market decision makers should be less concerned with the known, the bell, and more concerned the outliers and the unexpected.

Solution: Seek & Find. Tools that do weak signal processing. Search out data from multiple platforms.

Problem: Bias. We fall into the centrist trap (personal, communal, cultural, epistemological, phenomenological).

Solution: Many Minds. Incorporate many views to understand personal views relative to the crowd, which might include expert views and community views.

Data Timing 
Blount suggested there is a time value of information - the more predictive the data, the more valuable. Many companies harness information to harvest facts not the necessary signals – these signals are the different things of the future.

He spoke briefly of MediaX, which has started a series of lectures and groups on augmented decision making – the leveraging of technology to enhance human decision making. MediaX is an industry affiliate program to Stanford’s H-STAR Institute that, with its members, explores how “the thoughtful use of technology can impact a range of fields, from entertainment to learning to commerce…researching innovative ways for people to collaborate, communicate, and interact with the information, products, and industries of tomorrow.”

The Horizon 
Personalized Analytics Cloud (PAC) will take the place of traditional pre-internet tools. Here is the leveraging of big data platforms to gather data persistently, in one place, with automated processes that free-up time for deeper analysis and value activities. The goal is to find signals and outliers, develop better questions, and detection methods to find signal markers for these data. 

Aggregation - curation – personalization. Data optimization for better decision making. 

(I imagine this last line as a line in a hip-hop track). 

One case study of note involved, SkyBox, who uses big data collected through GPS satellite images to improve business practices. According to SkyBox, the company provides “global customers easy access to reliable and frequent high resolution images of the Earth that empower more informed, data-driven decisions, by designing and building imaging satellites and cloud services.” Big data tracking and analysis here results in better tracking of shipping containers, cargo transfer, and ship movement across the supply chain.

The Equation Sum 
Some things to consider: According to Blount, by 2018, the US alone will face a shortage of 140,000 to 190,000 people with deep analytic skills, as well as 1.5 million managers and analysts with the know-how to use big data to make effective decisions. Data will grow larger in flow and more complex in scope.

For more on big data, check out the McKinsey Global Institute, which has recently released numerous studies.

Next up in the series will be, "The Power and Promise of Stem Cells," on Sunday, May 19, 2:00pm. Check back here after the lecture date for notes and analysis. More info can be found at www.thetech.org

Watch these moving picture boxes for more info on big data. 

GoogleI/O 2012: Crunching Big Data with Big Query




(Video Source: GoogleDevelopers)

Paul Zikopolous. IMB. What is Big Data. Part 1.

(Video Source: IBM Big Data)

Image Source: Niagra Falls. == Summary == Taken by Daniel Mayer in late February 2006. == Licensing == {{self|cc-by-sa-2.5}} {{Category:Niagara Falls}}

Image Source: By Subhashish Panigrahi (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Image Source: By Svebert (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Image Source: By ivanovick solano (Own work) [Public domain], via Wikimedia Commons