How to start an Analytics Revolution
Updated: Nov 13, 2019
We all know about the "shot heard around the world" in 1775 that began the American Revolution. What followed was a battle of tactics and innovation between a group of renegades and the most powerful, infamous imperial force in the world at the time—and, spoiler alert, the renegades won.
Every revolution springs from a place and time where something just isn’t working. Whether political, social, or economic, revolutionary action happens when people take a stand to make a change. The rise of big data and, with it, the all-out battle between companies for the competitive edge has created pandemonium and stagnation: an environment ripe for revolution. Not, in fact, the much-hyped AI revolution--but a revolution to change the approach used in conventional machine learning (ML) and its application in business.
First, we must all admit that analytics and ML as we know it today, whether conventional approaches or even new-aged machine learning platforms, are inherently flawed. Most analytic projects fail and are never operationalized within a business, and it is becoming a frustrating waste of time generating figures that aren’t actually applicable to predictive capabilities for businesses. Why? In fact, Dr. Nikolai Liachenko, co-founder of Compellon, was tasked with answering that question in his time as a consultant. This is what he found:
"The most intriguing part of my work was the investigation of projects which all pursued reasonable objectives, had sufficient funding, were well equipped, and were conducted by a team of highly qualified specialists…and in spite of all this—failed. I saw such examples across many fields: manufacturing, environmental monitoring, medical diagnostics, acoustical testing of materials, and others. Figuratively speaking, my job in such projects was “autopsy” and “resurrection.” Overall, the main culprit in the overwhelming majority of cases was the use of inadequate assumptions based on well-established beliefs." (Read more here)
Bias and assumptions play a well-known role in the way people think—but these terms also apply to the selection and prioritization of data in machine learning. How is the process as we know it now filled with assumptions? Here are just a few examples (some of these get a bit nerdy, but hang in there):
1: What data is important to the business problem?
Choosing the data and which variables you use to answer your business problem is the first point where assumptions are injected. Are you ignoring some data altogether? What even is feature engineering? Data and variables are chosen based on assumptions of some behavior to the outcome.
2. Which model, or algorithm, is best to use?
When selecting a model to represent your data, be careful of the square peg and the round hole. Then you select a model—a round hole—you assume that its structure supports the relationships within your data, and that your data must also be ‘round’: but this may not be true. One analyst could choose a linear model, and another random forest but each analyst then fits the data to their model. Even if they each produce a model of good quality, it does not mean it truly represents the behaviors in the data. When you try to influence that model, or optimize, you may not get the true business behaviors: you get something “squoval.”
3: What data needs to be transformed?
Transforming data to fit a model can lead to assumptions about the relationships between the data or remove the value of a variable because of 1-hot encoding, all of these begin to distort the data and relationships in disadvantages ways. Think to back when you played telephone as a kid, every time the message passed along it was distorted just enough that, when revealed at the end, everyone had a good laugh. Data transformation, unfortunately, is a necessary step where assumptions begin distorting your data.
..... you get the picture.
So why does it matter?
Bias in AI and machine learning is the favorable or unfavorable influence on data which diminishes the value of the analysis and erodes trust. Because bias can influence an outcome, using a “bias philosophy” in your process to analyze data creates outputs which are not truly representative of the environment you are trying to analyze, hence the problem. These results cannot be applied back to the business as they do not capture the true information needed to solve the problem.
Without a change, we are just submitting ourselves to a slow death by numbers disguised as answers. Stop wasting time, money, and your hopes on analytics that aren't cutting it and consider something different. Are you ready for a revolution?
Step 1: Think Differently
We need an approach the eliminates the assumptions that create bias in our analysis, the answer is in another branch of mathematics, information theory and entropy, and leveraging AI to do the hard work - the work we are incapable of as humans to accomplish without assumptions and bias. So, throw out everything you know about conventional analytics and start thinking differently.
Step 2: Intelligence
Work with intelligent AI to supplement where humans lack the capacity (not to automate what we can do ourselves), in this case, to analyze all data for information and relationships, in an assumption-free way and without bias, learn and understand, to reflect back to us the knowledge it gains through a fully describable custom model that captures the relationships between variables rather than forcing them to fit to a predefined structure.
Step 3: Trust
Build on something you can trust—beyond your own human biases. Work with an evidence-based analytical process with results you can always trust, founded on concrete evidentiary support and without manipulation or assumptions. Get past the instincts that have been holding you back and take action with confidence.
Step 4: Impact
Get to real actionability. If you are building solutions based on assumptions, you cannot affect your business with those answers. Find answers that will actually change your business based on the behaviors discovered within your data. Answers that help you gain a clear understanding of what data matters, where to focus, and how best to impact your business. We make it easy to get data-driven decisions to the front-lines of your business.
Compellon is empowering people like you to lead an analytics revolution in your companies.
Learn more about Compellon and how we changed the approach to bring the intelligence, trust, and impact you need, at compellon.com