An example of how to formulate strategic hypothesis, make the controlled experiments, and track the results with a set of leading and lagging KPIs.
We have a community website where people can ask for advice about KPIs. It uses a question & answer model, so the idea is that the content is partially generated by users who need help and partially by the professionals that answer their questions. We have some control over the answers, but the questions always depend on the visitors and, as it appeared, they were the biggest challenge in the way of the project’s growth.
However, the purpose of this article is not to share the best practices for the webmasters, but to:
- Show the ways to organize a discussion around the strategy,
- Prepare controlled experiments, and
- Track the results with KPIs.
Probably defining a mission was not that important for such a small project, but we decided to follow the standard approach. The purpose of the mission statement is to inspire and motivate the team. What was motivating our team? I sent emails to everyone involved in the project and asked them to share their thoughts.
My two questions were:
- Have you ever mentioned this project to your friend or colleague?
- If yes, how did you describe the project to them?
I was glad to know that most people mentioned that they were involved in the project and described their involvement as “helping others with KPIs” or “brainstorming the best ways to measure the performance.”
“Help business professionals to find the best KPIs for their business.”
Good, short, and motivating (at least for our experts).
Formulating business objectives
In the next step we needed to think about the strategy taking into account the current market conditions, the problems that we had faced so far (the prototype of the project had already been running for several months), and the business model that we were using.
- Visitors. As long as the content is partially generated by the visitors, we needed to make sure that we have a steady traffic to the website and that this traffic is converting into the questions. For that purpose we had an umbrella goal – “Website that is easy to use” on the strategy map.
- Engaging content. We needed to create engaging content. Our experts were happy to help when questions contained enough details and the person who asked the question did his “homework” before asking. That was reflected under the “Provide Engaging Content” goal on the strategy map.
Tracking failure and/or success
Before going ahead with the experiments we had to find the metrics that would tell us if we were doing good or not. We chose two for the content quality:
- Leading indicator: the number of controlled experiments (nothing special, but at least we would know that we tried hard enough).
- Lagging indicator: % of qualified questions. A “qualified question” was a subjective idea.
- “We need a KPI for logistics” was not a good question, while
- “We need a KPI for …, our business goal is …, we already tried …, but the problem was …” sounded like someone invested time in describing their problem and that is what was working for the creation of good content.
There were two indicators for the “Website that is easy to use” goal:
- Leading indicator: Website Traffic, # of visitors.
- Lagging indicator: Traffic/questions ratio, %
Probably we could go with just a “Traffic/questions” ratio, but we had to be sure that we were getting some statistically significant traffic in order to make some conclusions based on the data.
Controlled business experiments
We were ready to start controlled experiments. The first one showed that there was no problem with traffic. We tried AdWords, we gave visitors of other websites an opportunity to ask their questions and there were people coming to the new project and posting their comments.
- The “traffic/questions ratio” was comparable with the benchmarks that some of our colleagues shared with us.
The goal of creating engaging content was more challenging. We had no data about what would work the best, so we planned to try at least 3 models.
I’d like to make one important note:
In the beginning we didn’t know what exactly we were going to try. That’s the nature of making an educated hypothesis (rather than planning). We did the first experiment, got the data, and only then understood what we should try next.
Here are the hypothesis that we tested with our experiments.
Our first hypothesis was to use a “Template” model. The idea was to suggest for users to fill in a template with some predefined fields (like “the purpose of KPI,” “the business context,” etc.) According to our idea visitors would give us more details about their problem if they had a template that they could use. We also added the requirement for a minimum number of words that one needed to use.
The project was just starting and we did not have many visitors, so we tried this hypothesis for 5 months to get some representative results. The “% of qualified questions” was on the 20-30% level (red zone in our case) and never hit the target of 45%.
Introducing benchmark data
In June we had enough traffic to get some meaningful data from the experiments. Moreover, we found some benchmark data from other Q&A websites. According to our estimations their figures for “% of qualified questions” were about 60-70%, so we:
- Updated our target for “% of qualified questions” indicator to 70%, and
- Decided to experiment more with the question form.
“Make it easy” hypothesis
The next hypothesis to be tested was “Make it easy.” Give visitors a way to post their questions without hassle (without registration), they will spend less time on the registration and more time on the question details, and their questions will be more interesting as a result.
From July to November we tried this hypothesis:
- For July “% of qualified questions” dropped to 15%.
We thought that the summer period might be causing this.
- We waiting until November, but “% of qualified questions” was still around 20%.
People were asking short questions that did not make any sense either for our experts or for visitors (we could tell this by tracking the “time on page” metric). We analyzed the traffic once more, hoping to see that some source generated those “bad” visitors, but the problem was not with the traffic. The value of the “Website traffic” indicator was growing, and traffic structure (the distribution of the traffic sources) remained more or less stable.
Re-analysis of the situation
Before I was writing about XY motivation theory by Herzberg. What we were doing so far looked like following the “X” theory. We used “carrot and stick” motivation first (the “templates” hypothesis), and then just started encouraging short anonymous questions (“make it easy” hypothesis). We needed to make a shift from the “X” theory to the “Y” theory, where people are “motivated by self-esteem, creativity and personal development.”
“Y” theory hypothesis
We wanted people to behave according to the “Y” theory, and be more responsible for what they asked and actually do some basic research before asking. The idea was not straight: instead of making the process easier we decided to complicate it a little bit.
The registration was now obligatory (but still not too complicated), and people had to enter their full names (not just logins) + some other changes were also introduced.
The results were impressive. In November “% of qualified questions” jumped up to the 75-80% level.
But this could also mean that the total number of questions asked will decrease. We checked the indicator “Traffic/questions ratio” and the dashboard showed that it stayed on the level of 1.5 – 1.7% which was similar to the level that we had when it was possible to post the questions anonymously.
I hope this is not yet the end of the story and we’ll come up with some new ideas regarding this project. Here are some conclusions that we can already make in the context of the strategy execution and Balanced Scorecard:
- Strategy is not a plan, but a number of controlled experiments. In the beginning we had no idea how to achieve a good “qualified questions” ratio (no strategic planning was possible). The best practice was found as a result of controlled experiments.
- Benchmarking data is important. Without this data it is hard to say if you are doing good or bad. Use benchmarking to motivate your team, especially if your figures are lower than ones from the competitors.
- Experiments are controlled by the indicators. We were clear that we needed to look at “% of qualified questions,” and “visitor/question ratio” to be sure that we were moving into the right direction.
- Visual picture is must. We visualized business goals with KPIs and initiatives on the strategy map. We visualized the KPI data on the dashboard. This is helping not only to tell the story in this article, but to focus the discussion around what matters.
I’m sure there were similar experiments in your business. Feel free to share the best practices that worked for you and ask your questions in the comment box below.