Towards Data-Driven Decision Making

This week I worked with two very different companies on what started out as very different topics, but that ended up at the same place: bad decision making due to the reliance on experience, beliefs, customer insights sometimes decades old, etc. So, the discussion turned towards decision making and how to develop processes where data is an integral part of the decision making process.

The interesting thing is that decision making is perhaps the key activity that people get paid for and that justifies their presence in the organization. From the CEO setting strategy for the company to the developer deciding the prefered design for part of the system, we take decisions all the time. The question of course is what these decisions are based upon. For all that we like to think of ourselves as rational and conscientious people, the fact is that most decisions are habitual and are taken subconsciously.

According to Charles Duhigg’s book titled “The Power of Habit”, we spend the majority of our days in purely habitual behaviours and take very few conscious decisions. As humans, we are of course able to change our habits, but as many who have tried to do can attest to, this change not without a significant amount of effort. Organizations, consisting of several if not many humans, are even more habit driven and change is even harder as it requires everyone involved to agree to breaking a habit. Nevertheless, the goal should be to apply rational, conscious decision making to as many decisions as possible and the question remains: what do we base decisions on?

In my experience, there are broadly four sources for decision making. First, there are shadow beliefs. These are beliefs that everyone in the organization holds and these beliefs make certain decisions extremely easy and defendable as everyone in the organization “knows” that the beliefs underlying the decision are true. The problem, of course, is that shadow beliefs held by organizations may have been true at some point in the past, but most certainly are no longer true today. So, basing decisions on shadow beliefs is a recipe for failure. Decisions get made based on a fundamentally incorrect set of beliefs.

The second basis for decision making is your opinion. Your opinion is formed based on your background, education, interaction with customers, experience and the set of beliefs that you have accumulated while walking the face of the earth. Basing a decision on your opinion is superior to using shadow beliefs as you own the decision and spend at least some time consciously considering the decision.

The third basis for decision making is qualitative data. Qualitative data tends to be characterised by small sample sizes where each case is studied in significant detail. This is the domain of ethnographic researchers and design thinking experts, but everyone who spends time with customers builds up, at least informally, a semantically rich set of qualitative data. Qualitative data is more objective than opinions and shadow beliefs, but is still based on significant amounts of interpretation. However, especially in cases where you’re making future oriented decisions, extrapolating from deep, qualitative insights might be your best option.

The final foundation for decision making is quantitative data. Although this type of data comes in many forms, the basic model is that we can derive information from the relationship between two variables. The data provides the basis for deriving validating insights from your customer base and often provides the ability to predict or at least validate the effect of decisions. Of course, in the online world, one of the best known applications of the quantitative data is A/B/n testing where two or more alternatives are evaluated by exposing each to a subset of customers or systems in the field  and measuring the behaviour of customers or deployed systems in order to determine the superior alternative.

It is clear that as a basis for decision making, quantitative data is superior to qualitative data. And both types are superior to opinions and shadow beliefs. However, in practice many decisions that get made in the companies that I work with are based on shadow beliefs and opinions. Based on my experience, there are at least three reasons for that:

  • Lack of data: Even though it has never been easier to collect data, for a variety of reasons, many companies are still reticent to do so. The excuses differ widely including concerns about EU regulation on personal data, overly zealous data security officers who prefer not to collect anything, the perceived lack of a business case, etc. However, if data is not collected, it can not be used for decision making.
  • Conflict with shadow beliefs: Whenever there is a mismatch between data and the beliefs held by the organization, there will be attempts to rationalize why the data is wrong. For a human, it is always easier to reinterpret the data rather than change your belief system.
  • Long feedback loops: Even if the organization has data, a typical situation is that the deployment cycle is so slow that whatever data is available, it is based on systems that were deployed perhaps years ago and consequently the available data is no longer relevant. Similarly, in cases where companies have not adopted continuous deployment, the organization has no mechanisms to experiment with or test decisions before making them final. In these cases, there is no opportunity to make data-driven decisions.

The conclusion and take-away from this article is twofold. First, decisions based on data, preferably quantitative, are superior to those made on opinions and shadow beliefs. And preferably, the decisions to be made should be small, reversible and collectively lead to an impactful outcome. Second, many decisions that historically had to be made based on opinions can now be made based on data, but organizations tend to get stuck in their old way of doing things. So, as a call to action: review the decision processes that you are involved in and evaluate where you could increase the use of data in each decision process.

One thought on “Towards Data-Driven Decision Making

Comments are closed.