Experimenting with a Strategy: How to Confirm Your Theory Using KPIs

An example of how to formulate strategic hypothesis, make the controlled experiments, and track the results with a set of leading and lagging KPIs.

Experimenting with a Strategy: discuss business goals, prepare experiments, track the results.We have a community website where people can ask for advice about KPIs. It uses a question & answer model, so the idea is that the content is partially generated by users who need help and partially by the professionals that answer their questions. We have some control over the answers, but the questions always depend on the visitors and, as it appeared, they were the biggest challenge in the way of the project’s growth.

However, the purpose of this article is not to share the best practices for the webmasters, but to:

  • Show the ways to organize a discussion around the strategy,
  • Prepare controlled experiments, and
  • Track the results with KPIs.

We started with a mission statement

Probably defining a mission was not that important for such a small project, but we decided to follow the standard approach. The purpose of the mission statement is to inspire and motivate the team. What was motivating our team? I sent emails to everyone involved in the project and asked them to share their thoughts.

My two questions were:

  1. Have you ever mentioned this project to your friend or colleague?
  2. If yes, how did you describe the project to them?

I was glad to know that most people mentioned that they were involved in the project and described their involvement as “helping others with KPIs” or “brainstorming the best ways to measure the performance.”

We asked two questions in order to find out what inspired our team and formulated mission statement.That’s what was important for our experts. That is what they were proud of. As a result, we formulated the mission statement as:

“Help business professionals to find the best KPIs for their business.”

Good, short, and motivating (at least for our experts).

Formulating business objectives

In the next step we needed to think about the strategy taking into account the current market conditions, the problems that we had faced so far (the prototype of the project had already been running for several months), and the business model that we were using.

Strategy map for the project with two main objectives on it.We found two success factors that were the keystones for the whole project:

  • Visitors. As long as the content is partially generated by the visitors, we needed to make sure that we have a steady traffic to the website and that this traffic is converting into the questions. For that purpose we had an umbrella goal – “Website that is easy to use” on the strategy map.
  • Engaging content. We needed to create engaging content. Our experts were happy to help when questions contained enough details and the person who asked the question did his “homework” before asking. That was reflected under the “Provide Engaging Content” goal on the strategy map.

Tracking failure and/or success

Before going ahead with the experiments we had to find the metrics that would tell us if we were doing good or not. We chose two for the content quality:

  • Leading indicator: the number of controlled experiments (nothing special, but at least we would know that we tried hard enough).
  • Lagging indicator: % of qualified questions. A “qualified question” was a subjective idea.

Leading and lagging indicators for the "provide engaging content" goal.For example:

  • “We need a KPI for logistics” was not a good question, while
  • “We need a KPI for …, our business goal is …, we already tried …, but the problem was …” sounded like someone invested time in describing their problem and that is what was working for the creation of good content.

There were two indicators for the “Website that is easy to use” goal:

  • Leading indicator: Website Traffic, # of visitors.
  • Lagging indicator: Traffic/questions ratio, %

"Easyto use website" goal aligned with two indicators.

Probably we could go with just a “Traffic/questions” ratio, but we had to be sure that we were getting some statistically significant traffic in order to make some conclusions based on the data.

Controlled business experiments

We were ready to start controlled experiments. The first one showed that there was no problem with traffic. We tried AdWords, we gave visitors of other websites an opportunity to ask their questions and there were people coming to the new project and posting their comments.

  • The “traffic/questions ratio” was comparable with the benchmarks that some of our colleagues shared with us.

The goal of creating engaging content was more challenging. We had no data about what would work the best, so we planned to try at least 3 models.

I’d like to make one important note:

In the beginning we didn’t know what exactly we were going to try. That’s the nature of making an educated hypothesis (rather than planning). We did the first experiment, got the data, and only then understood what we should try next.

Here are the hypothesis that we tested with our experiments.

“Template” hypothesis

Our first hypothesis was to use a “Template” model. The idea was to suggest for users to fill in a template with some predefined fields (like “the purpose of KPI,” “the business context,” etc.) According to our idea visitors would give us more details about their problem if they had a template that they could use. We also added the requirement for a minimum number of words that one needed to use.

When testing "Template" hypothesis, the “% of qualified questions” indicator was on the 20-30% level.

The project was just starting and we did not have many visitors, so we tried this hypothesis for 5 months to get some representative results. The “% of qualified questions” was on the 20-30% level (red zone in our case) and never hit the target of 45%.

Introducing benchmark data

In June we had enough traffic to get some meaningful data from the experiments. Moreover, we found some benchmark data from other Q&A websites. According to our estimations their figures for “% of qualified questions” were about 60-70%, so we:

  • Updated our target for “% of qualified questions” indicator to 70%, and
  • Decided to experiment more with the question form.

“Make it easy” hypothesis

The next hypothesis to be tested was “Make it easy.” Give visitors a way to post their questions without hassle (without registration), they will spend less time on the registration and more time on the question details, and their questions will be more interesting as a result.

From July to November we tried this hypothesis:

  • For July “% of qualified questions” dropped to 15%.

We thought that the summer period might be causing this.

  • We waiting until November, but “% of qualified questions” was still around 20%.

When testing "Make it easy" hypothesis "% of qualified questions" dropped to 20%

People were asking short questions that did not make any sense either for our experts or for visitors (we could tell this by tracking the “time on page” metric). We analyzed the traffic once more, hoping to see that some source generated those “bad” visitors, but the problem was not with the traffic. The value of the “Website traffic” indicator was growing, and traffic structure (the distribution of the traffic sources) remained more or less stable.

Re-analysis of the situation

Before I was writing about XY motivation theory by Herzberg. What we were doing so far looked like following the “X” theory. We used “carrot and stick” motivation first (the “templates” hypothesis), and then just started encouraging short anonymous questions (“make it easy” hypothesis). We needed to make a shift from the “X” theory to the “Y” theory, where people are “motivated by self-esteem, creativity and personal development.”

“Y” theory hypothesis

We wanted people to behave according to the “Y” theory, and be more responsible for what they asked and actually do some basic research before asking. The idea was not straight: instead of making the process easier we decided to complicate it a little bit.

The registration was now obligatory (but still not too complicated), and people had to enter their full names (not just logins) + some other changes were also introduced.

The results were impressive. In November “% of qualified questions” jumped up to the 75-80% level.

"Y" theory hypothesis implemented - "% of qualified questions" jumped up to the 75-80% level.

But this could also mean that the total number of questions asked will decrease. We checked the indicator “Traffic/questions ratio” and the dashboard showed that it stayed on the level of 1.5 – 1.7% which was similar to the level that we had when it was possible to post the questions anonymously.

The indicator "Traffic/questions ratio" stayed on the level of 1.5 - 1.7%

Main take-aways

I hope this is not yet the end of the story and we’ll come up with some new ideas regarding this project. Here are some conclusions that we can already make in the context of the strategy execution and Balanced Scorecard:

  • Strategy is not a plan, but a number of controlled experiments. In the beginning we had no idea how to achieve a good “qualified questions” ratio (no strategic planning was possible). The best practice was found as a result of controlled experiments.
  • Benchmarking data is important. Without this data it is hard to say if you are doing good or bad. Use benchmarking to motivate your team, especially if your figures are lower than ones from the competitors.
  • Experiments are controlled by the indicators. We were clear that we needed to look at “% of qualified questions,” and “visitor/question ratio” to be sure that we were moving into the right direction.
  • Visual picture is must. We visualized business goals with KPIs and initiatives on the strategy map. We visualized the KPI data on the dashboard. This is helping not only to tell the story in this article, but to focus the discussion around what matters.

I’m sure there were similar experiments in your business. Feel free to share the best practices that worked for you and ask your questions in the comment box below.

Download the project of BSC Designer that was discussed in this article. To open the project you will need to download BSC Designer PRO.
Download .BSC Project

Related Articles

Strategy and KPIs Scorecard Expert | Speaker | CEO. Aleksey Savkin (LinkedIn, @bscdesigner) is helping companies to better formulate their strategies and make the process of strategy execution more tangible with KPIs. His areas of expertise are Balanced Scorecard, Key Performance Indicators, business performance management. Aleksey is a frequent speaker at conferences; the author of a number of articles and books on Balanced Scorecard. New book by Aleksey: 10 Step KPI System

Posted in BSC Implementation

What is BSC Designer?

BSC Designer is a balanced scorecard software available as a cloud-based service and Windows desktop application.

Getting Started with BSC Designer

Try BSC Designer for free

Download BSC Designer

Who uses BSC Designer

According to our users, BSC Designer is easy to install, configure and it helps a lot with Balanced Scorecard, strategy maps, and KPIs.

BSC Designer Customers

Training for the software

To make it easier for the users to get started, we have a lot of free training materials for the software.

Video manuals for BSC Designer

Customer support

Should you have any questions, we are at your disposal.

Pricing and buying

How much does BSC Designer cost? It depends on the configuration that you prefer. Use this price calculator to find out the total cost, get an official quote, and buy the software.

What Balanced Scorecard is all about?

Fast Track explains complex ideas in a simple way Check out Strategy Scorecard Fast Track to find easy explanation of the most important complex ideas.
  • What is a Balanced/Strategy Scorecard?
  • How to build a good strategy map?
  • How to find winning KPIs?
  • How to cascade strategy?
  • How to implement scorecard
Learn all the nuances quickly and ask follow-up questions to BSC expert. Learn more...

What are your current challenges?

We help our users to get started with Balanced Scorecard, strategy maps, and KPIs. Here is a list of frequently asked questions and respective answers to them.

Q: What is the Balanced Scorecard?

Big picture about Balanced Scorecard A: Check out: Get the big picture about Balanced Scorecard article.

Q: I'm looking for a KPIs... how do I find good ones for our business?

How to find a KPI for... A: Check out our recommendations.

Q: Do you have some examples of the Balanced Scorecard?

Examples of Balanced ScorecardA: Sure, we have some here.

Q: Balanced Scorecard is about strategy execution, right? But what is a strategy?

Define strategy, create strategy mapA: Check these articles about strategy definition and creating strategy map.

Q: How do we cascade our top level scorecard throughout the company?

How to cascade a strategy mapA: Do it by business goals, some examples will also be useful.

More questions and answers...

BSC Lessons

Download BSC Designer and we will follow up with you with lessons about the Balanced Scorecard:

BSC Designer experts

Aleksey Savkin Performance Management and Balanced Scorecard

Kazim Ladimeji HR KPIs and Performance Expert

Levi Newman Effective KPIs and social metrics

Oana Boteanu Performance Management

Oleg Tumarkin Business Measurements

Upcoming events and talks

We organize, sponsor, and participate in various events related to the strategy execution.

  • "Building Balanced Scorecard Step by Step." 15th July, 2017, online training. Read more...

BSC Designer Roadshow in Athens, Greece, June 9-15, 2017. - Firsthand experience with BSC Designer software Drop us a note if you want to meet up at these events.

Online training: Build BSC step by step

The best way to get started with the Balanced Scorecard is to join our online training.

Balanced Scorecard Training Online Strategy map, KPIs, cascading - it might be hard to put it all together. In the online training "Building BSC Step by Step" we explain all the nuances in simple words and help companies to build a prototype of their balanced scorecards.

Find us on Google Plus

Thank you for sharing!

Whether you are looking for a professional Balanced Scorecard software, or just researching information about Balanced Scorecard and business strategies, we recommend you to download and try our BSC Designer software (no credit card is required).

We will follow up with you with lessons about the Balanced Scorecard and will keep you informed about the trending articles on bscdesigner.com

Follow us in Social Media

Send this to friend

More in BSC Implementation
5 Steps for Effortless Balanced Scorecard Implementation

Who else wants an easy to follow Balanced Scorecard implementation guide? Here is a simple 5 step formula that will...

Balanced Scorecard Cascading: Make Your Strategy Resonate with Company

Good strategy implies the coherence between objectives and the way people interpret them on each level of the organization. In the business...