I have written previously about how the uncareful optimization for growth metrics can be harmful, but primarily examining potential harms to the user.  Here I consider a hypothetical case study of how the uncareful choice of a growth metric as a key performance indicator (KPI) can incentivize bad behaviour in a business.  I present a hypothetical example for analysis, but this is a serious problem that has been written about extensively.  See the references below for more examples including an example of poor choice of metrics incentivizing customer-hostile behaviour [1] or poor prioritization [2].

A KPI is a metric that a company designates as especially important and business activities will be oriented around working to increase it.  Employee bonuses, especially at higher levels, are often tied to a targeted improvement in the metric.  A good KPI metric could be something that is connected to business success (if it goes up, it means the business is doing well), something that is sensitive to the business’s activities (there is something that can be done to make it go up), and should not be “gameable” (there should not be ways of making it go up while not driving business success).  This alignment is discussed in much more detail in [1].

The last condition is especially important.  Many metrics may be correlated with business success, but when a metric is set as a KPI, it unleashes a mass of energy and creativity in efforts to increase it – this can break the metric’s correlation with business success (see “Goodhart’s Law” [3]).  When people are presented with a clear metric for measuring their success, it can create a shortsightedness in that people are disinclined to carefully consider if their activities are really doing good, as long as they are increasing the metric.  In more extreme cases, especially when bonuses are tied to the metric, it can even incentivize intentional efforts to move the metric in ways that don’t necessarily generate business value and unfortunately, the ways of increasing the metric are often easier than those that actually help the business.

In a hypothetical case study, let’s consider a business that builds a messaging app.  As is typical of these apps, the business hopes that the users will use the product frequently.  Further, let’s consider that some action should be taken by the user in order to receive value from the app which might include adding contacts or sending or reading messages.  This setup is typical in quite a range of digital products.

When our hypothetical company (we’ll call it “MessApp”) wants a KPI metric, they should use something similar to the typical approach of counting “daily active users” (DAU) of their product.  But they find that the DAU varies significantly due to day of week and seasonal fluctuations.  Using “monthly active users¹” (MAU) seems much smoother.  And, while they can detect when a user opens “MessApp”, they haven’t yet set up the instrumentation to be able to determine if a user does anything in it, so they decide to just count a user as active if they open the app.

What’s wrong with this KPI metric?  First, for a product that we expect users to use fairly frequently, using MAU is going to give more weight to low-frequency or single-use users than it should.  Consider two cases:

1. During a month, 100 people try the app once and never use it again.
2. During the same month 100 people try the app and become regular users.

MAU will be equal in both cases, whereas DAU would differ dramatically².

Secondly, as the app requires the user to take some action to receive value, simply opening the app is not a good measure of activity.  Why?  Let’s look at an example that comes out of the interaction of both of these problems.

MessApp decides they want to try sending push notifications to users that have previously installed the app, but have not used it in a long time.  The hope is that these notifications will convince the users to come back and become regular users.  However, it turns out that these messages are completely ineffective.  Actually, they may persuade a significant fraction of the users to uninstall the app, because push notifications are irritating.  However, some users also will tap on the notification (this is inevitable – if you put a button in front of a lot of people, some will press it), which then takes them into the app.  The user then immediately exits the app, and perhaps uninstalls it.

This campaign is clearly harmful to the business but can actually increase the KPI metric for a 28-day period after the messages are sent.  Since it brings some users into the app, they will be counted as MAU, even if they do nothing.  And because MAU is used, those users will continue to be counted for 27 more days.  An unscrupulous team might decide to launch such a campaign approximately 28 days before the end of the year, when the KPI metric is compared to its target to decide on what bonuses will be paid.  Ultimately, the only long-term effect of this campaign is to drive some dormant users to uninstall the app, also possibly harming the company’s reputation as it sends irritating push notifications.

The poor choice of a KPI metric has incentivized a course of action that is harmful to the business.  Choosing good metrics is a difficult but critical task.  Alignment between what one is trying to measure and actually measuring is one general challenge, while alignment between the metric and business strategy is the special challenge for KPI metrics.

References

1. Michael Harris and Bill Tayler. Don’t Let Metrics Undermine Your Business. Harvard Business Review. September–October 2019. https://hbr.org/2019/09/dont-let-metrics-undermine-your-business
2. Michael J. Mauboussin. The True Measures of Success. Harvard Business Review. October 2012. https://hbr.org/2012/10/the-true-measures-of-success
3. Goodhart’s law. Wikipedia. https://en.wikipedia.org/wiki/Goodhart%27s_law

Footnotes

1. Count of unique users in the past 28 days. 28 is chosen as it is about the length of a month and is a multiple of 7, which minimizes day of week effects in the metric.
2. One common solution is to consider an “engagement ratio” metric, i.e. DAU divided by MAU – this could be an additional KPI to “balance” the incentives created by the use of MAU.