Information Evaluation — Not another “Know-How”

 We all remember vaguely this time of the year when the celebrations of next year’s work-plan begin. Management is all tied up, and you can cut the tension with a knife. Who gets what? Has my budget increased? and other issues. For the top management, things were clear, and planning the next year sounded reasonable. Even in a risky market, they knew the risk and embedded it in the calculations. When looking at 2021, nothing is as it was, and planning for a whole year ahead does not sound reasonable at all.

Since we cannot dismiss the planning process, nor the continuity of the company’s operations, we must plan, so we base our plans on the Logic Model. In short, the Logic Model starts with the expected outputs of our company’s performance, and from there, in a backward technique, define the desired inputs (this is an issue for another article).

Whether the logic model or any other strategic planning model we use, the road from inputs to outputs goes through making decisions. Good decision-making is based on relevant and accurate information, mainly gathered by CI or MI projects.

Intelligence projects are part of the inputs towards the outputs, and it has a cost. So, why should the CEO consider the Intelligence budget when deciding on the allocation of resources for the next year??!!

This question is a major one in the ongoing debate about evaluating the information. The concept of Intelligence Return on Investment (I-ROI) is the key to understanding the perceived value of information in the company (I referred to that in my previous article https://www.linkedin.com/pulse/new-model-intelligence-roi-i-roi-amir-el).

There is a difference between a theoretical model (I-ROI) and the practical implementations of it. In this article, I would like to present three methods of evaluating intelligence products for the company.

AB testing of information effectiveness

Let us assume that the CEO must decide on the future of a new product line set up lately. A wrong decision would cause losses and embarrassment to the company and upper management. The Corporate Intelligence Team comes to the CEO, saying that they have enough information to support decision X about the product-line. The CEO considers decision X or decision Y. He decides to put the CI team’s info to an evaluation test. What should he do?

At this point comes the method of AB testing for assistance. The CI team suggests that 60% of the product-line would operate under decision X and 40% under decision Y. All this for a short term in which the company can tolerate, in case of a loss. Remember that the information gathered before supports decision X.

Since AB testing is a method of identifying the preferences of the customers, engaged with the same product but with a changed attribute, we can implement this concept in the CEO’s dilemma mentioned above. If there would be a significant variance, in gross profit, between the performances of the product-line under decision X and the performance under Y, in favor of decision X, then it would mean that the information used to make decision X has a positive value.

Now we can measure the value of the information, and we can rest assured that the CEO would think highly of the company’s CI team and will invest in intelligence projects in the future.

Stability Indicators according to the Logic Model

I spoke earlier about the strategic planning method called the Logic Model. Here, if we want to measure the value of intelligence information for the company in another way, we can start with the end-front of our planning for next year by asking: what are the expected outcomes?! But this is not enough because any strategic planning method we use must consider threats and opportunities in the ecosystem around us.

Let us assume that at the end of the planning process, the CEO announced these seven outputs for next year:

• Not to change the strategy we hold
• Preserve our market-share
• Preserve our customers
• Preserve our financial resilience
• Stability in our risk rate within the market
• preserve our positioning in the market
• preserve extraordinarily talented HR

At that point, when the CEO declares these goals, The CI team gives the information to support the CEO’s decisions on how to reach these goals. We say we identify the threats and opportunities, and we gathered information to mitigate these threats.

We expect the CEO to adopt all our suggestions when deciding. We have to say what might happen if they will not embed our info in the decision process.

By the end of next year, the company’s operational results would act as an ultimate evaluation-tool of today’s info.

For example, if the outcomes are all in a positive manner and we preserved all was needed to preserve, we must review all the decisions made along the way and figure out how many of them considered our intel info. This ratio is a good indicator of the value of the preliminary information. The same goes for the evaluation of the negative manner outcomes. If the CEO and upper management realize that their info-based decisions supported the goals’ fulfillment and reduced risk, they will tend to use intelligence projects in the future too.

Corporate Decisions — Post Mortem

I argued earlier that the main use of information gathered in an intelligence project is to support the decision-making process. We make decisions every day all day. Managers make decisions in their course of work to facilitate the company’s resources towards the fulfillment of their personal goals, their department’s goals, and the company’s goals. Some decisions result in positive outcomes, and some not. Either way, the manager needs to be aware of as much information and data available to make the right decision.

It is custom to analyze big projects with the Post-Mortem technique. We can use this technique to analyze decisions with positive or negative outcomes. The analysis of a decision inspects each part of the process, especially the way the manager used or did not use the intel he had on the issue of the decision.

The evaluation of the information begins with analyzing good and bad decisions and extracting the data regarding the use of information during the making process.

The first step is to calculate the ratio of intel-use during the making process by counting how many decisions used the information out of all decisions analyzed.

The second step is to match the decisions with its outcomes while marking the decisions with no use of intel.

We assume that by using the intel available to the manager now for decision making, the results would be the best can be. Now we have three columns in our datasheet:

• decision
• use / did not use intel
• positive/negative outcome

By simple statistic calculation, we can find the correlation between the use of intel and the outcomes. If we find a significant, high correlation between them, we will argue that the value of intel, at that moment, was positive.

To sum up this article, I would say that there are many ways to evaluate information. Each company can develop a calculation method based on parameters that fit the company. If the process is coherent and steady in time, there is no reason not to rely on these measures.

My humble donation for this matter is three practical methods to approach the concept of evaluating and measuring the value of intelligence information:

• AB testing
• The Logic Model
• Postmortem Analysis

(Always at you service https://www.webintelligency.co.il/)