3.3 Metrics 🎯
This section includes an Activity 🎯
As a product manager, you will be responsible for some or all of the metrics used to measure your product's performance. These metrics are usually tied to your company's business model and industry, which is why you're learning about it now.
Many products share the same metrics, but a few are different in special ways that you'll need to be aware of. You'll be expected to be fluent in these terms, so the sooner you master them, the sooner you can spend your time focused on the rest of your product work.
Conversely, if you're not aware of your company's top metrics, you could waste your time trying to optimize the wrong part of your business. It's your job to know how your business works so that you can build the best product possible.
By the end of this checkpoint, you should be able to do the following:
- Describe business metrics common across all products
- Describe metrics specific to certain industries or business models
- Identify the KPIs a given company or business model is likely to use
Why metrics?
Companies typically prefer to quantitatively measure progress and performance. If you're working in a public company like Oracle, your company's top goal is to show increasing revenue over time. Your revenue probably ties to other metrics, like the number of companies that subscribe to your products and the price of your service. The number of subscribers relates to the features of your products, and those features are built by teams led by product managers.
Your performance as a product manager will be measured in part by how successful you are at improving your product's metrics. A great PM always knows their key metrics—including their current value and the overall trend—without having to look them up. You should also know which metrics cause other metrics to move—like how a change in price could increase your revenue but might cause some customers to find a cheaper alternative leading to lower sales.
If your metrics are moving in the wrong direction, you should take all the necessary actions to turn them around.
KPIs
Key performance indicators (KPIs) are one common tool for analysis. KPIs are a small number of top metrics that are used to summarize the overall health of a product. For instance, say you're running the support forum for Google products. Your goal is to ensure that people get their issues solved and questions answered successfully. Some of your KPIs might include these factors:
- Number of issues and messages created each month
- Percentage of issues that are successfully resolved
- Average time for an issue to be successfully resolved
If you're doing your job well, the issues have a swift, successful resolution. That means the percentage of issues with a successful resolution increases, and the average time to arrive at a resolution decreases. This is a good example of how some KPIs are moving in the right direction when they increase, and others are improving when they decrease.
Not all metrics are KPIs. For example, on Google support forums, the number of users per month is a good metric, but might not be a KPI. That number of users could depend on factors outside your control, like a new product launch from a different team. On the other hand, that metric could be a KPI for another team. For example, if there's a support forum for Android and there's a huge spike in issues in February, that could be an indication to the Android team of a bigger problem. In that case, it's a KPI that they'll want to monitor.
Mike Clayton comes up with a brilliant video explaining KPI's
Moving metrics and KPIs
In your day-to-day work, you'll be managing your product so that your KPIs move in the right direction. You'll be in direct control of some of these metrics. For instance, if you manage Facebook's notification service and one of your KPIs is the number of notifications sent per user per day, you could always create or suppress notifications to manipulate that KPI. On the other hand, if your KPI is the number of notifications that get clicked, you don't have as much control over that because the click rate is affected by other factors. These could include the notification's content or who it's from.
Similarly, if you're running Amazon's checkout flow, you may have KPIs around the completion rate of the checkout process. A user could bail out of that flow (in other words, leave the site) if they realize they can't afford all the items in their cart. You may have some ability to change that outcome, like letting users remove an item from their cart at the last moment before buying or letting users pay with multiple payment methods.
As a PM, your job is to figure out the most effective way to improve your product's metrics.
This video below provides further insight into what you need to do if your metrics are moving in the wrong direction.
If you're building your own company at some point, you will come across or hear about Y Combinator. Not delving deeper into what they do but here's an amazing video where Suhail Doshi explains on how to measure your Product
If you want to improve your product's metrics, you'll also need to understand those metrics. Below you will learn more about some key metrics that you'll run into across all products, then you'll cover some of the metrics specific to certain types of products.
Common product metrics
Some metrics are common regardless of the type of product or service you're working on. Here are some examples of those.
Profit
Most companies care about profit—which is computed by subtracting costs from revenue. Costs include everything that goes into the creation and distribution of goods and services. If you sell physical goods, costs can include buying, producing, storing, and shipping those goods. If you sell software or services, then your costs include the labor of creating those, overhead (like office space, laptops, and other business staples), and infrastructure (such as servers, networking, and storage). Even nonprofit organizations want to manage the costs of running their product to make sure it's within the organization's budget, even though they're not out to make a profit.
Margin
A margin is the ratio between profits and loss. Say you pay someone $1,000 to write a short book. You then print the book for $10 per copy.
If you want to make a 10% margin on every book, here's how much you'd need to sell the book for a given number of copies.
| Books printed | Total cost | Cost per book w/ 10% margin |
|---|---|---|
| 1 | $1,010 | $1,111 |
| 10 | $1,100 | $121 |
| 100 | $2,000 | $22 |
| 1,000 | $11,000 | $12.1 |
| 10,000 | $101,000 | $11.11 |
When you sell a small number of books, most of your sale goes toward paying for the labor that produced it. But if you sell 10,000 books, most of the cost is the printing of the book. Remember that it costs $10 per copy, and you can make a 10% margin if you sell it for $11. At that price, 90% of what the customer paid goes toward the $10 you spent to print the physical copy of the book.
Hardware products work similarly. If you produce a line of Roku digital players, it could cost $10 million to build 1 million devices ($10 apiece). But you also need to account for the costs of creating the software, marketing, and other fixed costs, similar to how you have to pay for writing the book before you sell it.
Pure software products differ from physical goods in crucial ways. If you only sell one copy of a piece of software and need to break even, the buyer has to pay the full price of producing it. If you sell two copies, you could give away the second copy for free if the first buyer paid full price. Each additional copy you make of that software is essentially free. By comparison, think of how much work it takes to make a copy of a file—it takes practically no effort. The same thing applies to software.
So taking that same chart as above, assume you have a piece of software that you paid someone $1,000 to write. As you sell more, the price per copy will always keep decreasing because there's no additional cost for new copies of the software.
| Copies of software | Total cost | Price per copy for 10% margin |
|---|---|---|
| 1 | $1,000 | $1,100 |
| 10 | $1,000 | $110 |
| 100 | $1,000 | $11 |
| 1,000 | $1,000 | $1.10 |
| 10,000 | $1,000 | $0.11 |
This is an example of economies of scale. In other words, this is how the price of a good continues to decrease as you produce more of it. Unlike a book, software has zero copy costs, and you need to take this into account when you price it. You'll learn more about this in a future checkpoint.
Users
The most naive metric you can share about your product is the number of people who use it. For example, Facebook has over 2 billion users. This can be really misleading though—you might have signed up a year ago but stopped using it after six months. Should you report on all the people who have ever signed up? Just the ones who really tried using it? People who tried it, left, and came back again later?
Product managers often prefer slicing up user numbers in a few ways to help explain those trends better.
Total users generally refers to all the people who have ever used your product. In particular, people will want to know the total amount of unique users who use your site. If a person visits your site twice, you would count that as one unique user. It's also worth tracking the ratio of new users to returning users so you can understand how much you're growing as opposed to the number of people you're retaining.
If you want to count the number of visits to your site, those are usually reported as sessions. One session is when someone spends a continuous amount of time on your site and ends after they leave your site or if they go idle for a long time. A good rule of thumb is that a session starts when someone first visits your site and ends 30 minutes after that user's last activity on that site. For example, a user who visits your site, is inactive for an hour, then starts browsing again would be counted as one unique user and two sessions.
Besides total users, product managers often report on active users—the total number of unique users who used your product within a timeframe. For example, Wired may have 300 million total users that have used their site overall. In a particular month, they might have 15 million active users (unique people who visited the site or read an article). In a week, they could have six million active users.
One other note about active users is that you need to ensure you define active appropriately. For a media site, a person who is only active for five seconds then leaves isn't really active. If they are engaged with your site's content for 30 seconds, however, they are clearly active.
Finally, many sites measure their churn. This is the business metric that represents attrition, or the rate at which the business is losing recurring users. For example, if Wired has six million weekly active users and churns half of them every week, here's what that would look like in terms of totals.
| Week of month | New users | Returning users | Churned from last week | Total unique visitors this month | Weekly active users |
|---|---|---|---|---|---|
| 1 | 3,000,000 | 3,000,000 | 6,000,000 | 6,000,000 | |
| 2 | 3,000,000 | 3,000,000 | 3,000,000 | 9,000,000 | 6,000,000 |
| 3 | 3,000,000 | 3,000,000 | 3,000,000 | 12,000,000 | 6,000,000 |
| 4 | 3,000,000 | 3,000,000 | 3,000,000 | 15,000,000 | 6,000,000 |
If Wired was able to reduce their churn rate by 10% (that is, 10% of 50%, so the new churn rate is 45%), they could increase their weekly active users by nearly half a million readers even with the same number of total visitors. That's an 8% increase in weekly active users after one month compared to the 50% churn rate. Small churn changes add up quickly over time if you can continue to compound the results.
| Week of month | New users | Returning users | Churned from last week | Total unique visitors this month | Weekly active users |
|---|---|---|---|---|---|
| 1 | 3,000,000 | 3,000,000 | 6,000,000 | 6,000,000 | |
| 2 | 3,000,000 | 3,300,000 | 2,700,000 | 9,000,000 | 6,300,000 |
| 3 | 3,000,000 | 3,465,000 | 2,835,000 | 12,000,000 | 6,465,000 |
| 4 | 3,000,000 | 3,555,750 | 2,909,250 | 15,000,000 | 6,555,750 |
Why does churn matter? The lower your churn, the less effort and resources you have to spend on acquiring new people to make up for the ones who left. Similarly, high churn could also be a sign that your product isn't working for your users.
Conversions
When a user completes an important action, such as completing a purchase on eBay or signing up for a free Spotify account, that's a conversion. Product managers care about a few different conversion metrics. Total conversions is just the raw number of conversion events that happen, like the number of clicks from a Google search results list to a destination page. Sometimes you want to measure the unique conversions, such as the number of people who bought something on Amazon (versus the total number of purchases from all sources).
The conversion rate is the ratio of people who had an opportunity to convert to those who actually did convert. For example, you might search for something on Google but not see any good results, so you choose not to click anything. That is a failed conversion, and the conversion rate will go down as a consequence. So if 100,000 people search for something on Google in a minute and 50,000 of those searches get a click to a final page, the conversion rate is 50%.
You might also think about conversion rates per session or user. Taking the Google Search example again, maybe 50% of individual searches converted. Now, look at the same data by session—say there were 50,000 sessions and 40,000 of them converted. That would be an 80% conversion rate per session. If you compare the conversion rates, you can see that people are trying multiple times to find what they need. This can point you to opportunities to improve your product.
Finally, you may also want to track the distribution of time to convert. In e-commerce, for example, sometimes it takes a while for someone to decide they want to buy a product. They may visit a page multiple times over a week before finally purchasing. Looking at all users, you might find that the average time to convert is two weeks, so this person was below average. Poor time to convert metrics could be an indication that your conversion process is too complicated or that your buyers have a lot to consider before making a purchase.
CAC
Your customer acquisition cost (CAC) is the amount of money that you spend to get customers to convert. You could have a product where each sale you make is profitable, but you spend so much on advertising and marketing that overall you're still losing money.
Companies are willing to spend huge amounts to acquire customers. For example, Google paid $9 billion to be the default search engine for Apple's products. Imagine how much money Google can make from Apple's searchers if they're willing to spend that much to be the default search engine.
As a product manager, you probably won't be managing the CAC directly. That is usually done by your marketing team. However, you should pay close attention to it because it's a leading sign of your product's future health.
NPS
The Net Promoter Score (NPS) is one of the most important metrics that you'll need to understand as a product manager. In short, it's a way to track your users' happiness with your product. It generally looks like this: users are prompted to answer the question "How likely are you to recommend this product to a friend?" on a 1 to 10 point scale, where 1 is not likely, and 10 is very likely.
NPS is calculated like this:
(% of people who answered 9 or 10) − (% of people who answered 6 or less) = NPS
So if 50% of people answered the "How likely are you to recommend this product to a friend?" with a 9 or 10, 20% answered with a 7 or 8, and 30% with a 6 or less, your NPS score is: 50% − 30% = 20. (NPS scores are communicated as bare numbers, not percentages.)
The larger and more positive your NPS score, the more likely your brand has a positive perception in the market—and that it really works for your users. Top brands can have NPS scores of 40 or more. If yours is zero or lower, you should be very concerned about your brand's perception in the marketplace. Here are some tech company NPS scores taken from a survey of popular brands:
| Brand | NPS score |
|---|---|
| Apple | 47 |
| -21 | |
| Microsoft | 45 |
| Amazon | 7 |
| 11 |
You can check more NPS scores for top companies here.
NPS scores can also be misleading. Recall that answers of 7 or 8 don't count towards your NPS at all. So for instance, if 5% of people answered 9 or 10, 90% answered 7 or 8, and 5% answered 6 or less, your NPS score will be 0. However, the implications for your product are much different compared to a situation where your NPS is 0 because 30% answered 9 or 10, 40% answered 7 or 8, and 30% answered 2 or less. While in both situations the NPS is 0, the latter tells you that 30% of your users are really, really happy with your product while another 30% are really, really unsatisfied with it. To gain the best insight into how your product is performing, always make sure to look at both the total NPS score and the distribution of the scores.
As a substitute for the NPS question, you could ask customer satisfaction questions like, "How satisfied are you with this product? Rate it from 1-5." But be careful because NPS is the standard satisfaction metric. Many stakeholders will not accept substitutes.
Regardless of how you ask, you should always keep track of your users' happiness with your product and how it changes over time. Any changes could indicate problems, such as issues with your product or a new competitor.
Slicing your metrics
Most metrics are dynamic; they change over time and across segments of your users. For example, if you work for Walmart, you'll notice a huge change in your unique visitors and conversion rates in November and December as shoppers buy Christmas gifts.
To better understand these dynamic changes, you'll want to slice your numbers in ways that make them more meaningful. Here are a few common ways to slice and scale your metrics:
- Time. Most metrics can be tracked over periods of time like "per month" or "per week." Smaller slices could make sense depending on your product, but in most cases "per day" or "per hour" or smaller segments are too narrow. The exception is special cases, like Walmart's Black Friday sales or Amazon's Prime Day.
- Per user. For example, Walmart probably wants to know the average order size per user because they want to encourage people to buy more items.
- Per transaction. For example, Uber would love to know their margins per ride (what Uber earns versus what they pay to the driver), and Amazon needs to know their margin per sale. If your company is profitable on most single transactions, then you're probably on the right track for success.
- By demographics. Slicing your data by demographic factors like location, age, or gender can help you analyze your product. Some sites try to appeal to specific groups of people like teens or women, while others might discover that a product meant for everyone is actually most successful for one specific group.
Product-specific metrics
Certain kinds of products track metrics that are unique to their industry. You should be aware of what they are, especially if you're interested in working in any of these industries.
E-commerce
E-comm sites are always obsessing over the amount and value of transactions they're generating. Without sales, your e-commerce site is just useless text.
Average order value (AOV) is the average amount that people spend per order. Related to AOV, unit margin looks at just the variable costs in one transaction, not including the fixed costs (like writing software). For example, the AOV of a ride on Uber might be $20, but Uber's unit margin might be $1 after you subtract the costs paid to the driver, transaction fees, and other costs.
Even if your unit margins are profitable, your company could still lose money as a whole when you factor in the costs of producing, selling, and marketing your product. Also, unit margins can be negative for some orders and positive for others. Digging into this kind of data can give you insight into how to manage the overall profitability of your site (we will go deeper into data analysis later).
Abandonment is another metric often used in E-comm. When someone starts the checkout/purchase process and then gives up, they "abandon" their cart. A high abandonment rate could mean that your checkout process needs improvement or indicate that your buyers get "sticker shock" when they see how much they'll have to pay at the end.
Return customer rate is the opposite of churn. Most e-commerce sites depend on people coming back to buy more. Depending on the kind of purchase, they could be tracking a daily return rate, weekly return rate, or more.
Subscriptions
If you work on a product with a subscription-based business model, you'll track a few other specialized metrics too.
Lifetime value (LTV) is the average amount you expect a new customer to spend over their lifetime as a customer. The longer they keep subscribing, the better your LTV is. E-commerce sites can track their LTV too over the lifetime of all orders from an individual.
Monthly recurring revenue (MRR) is the total amount of revenue the company earns from subscriptions. If you run a subscription service, your MRR should continue to climb as long as your subscribers stick with your service and you acquire new subscribers.
Subscription churn is the ratio of subscribers that you lose over a period of time, typically monthly (or whatever period your subscription covers). You want your churn to be as low as possible because acquiring replacement subscribers is much more expensive than keeping the ones you already have.
Ads and Media
For products that rely on advertising revenue and for media sites, success means eyes looking at content. Both these types of products track a few similar metrics.
An impression is an opportunity for someone to see something. In ads, an impression is when someone sees an ad appear on their screen. Related to this are views, as in when someone watches a video. Page views are counted when the individual page content matters, such as in a news article. Different companies have different definitions of views or impressions, so you should always be aware of the specifics and ask: are they being counted immediately? After 30 seconds of watching? Something else?
The click rate for an ad is the ratio of people who click on an ad and go to the landing page on the advertiser's site. Advertisers want to know the click rates of their ads so they can understand which ones are performing best. Publishers (the owners of the sites where you see the ads displayed) want to know click rates so they can tell if they've placed their ads in a good spot. Click rates also help them confirm that they are getting the appropriate revenue for those clicks. Advertisers also like to track the cost per click (CPC) as a measure of their advertising effectiveness. Videos and music products track their play rate, which is equivalent to an ad's click rate.
And once again, conversion rate is the rate of people who saw an ad to those who took a meaningful action at the other end. For example, if you're advertising a demo for your music streaming service, you would want to track the conversion rate of ad viewers to registrants and ad clickers to registrants. This helps you calculate the cost per conversion—the amount you spent on the advertising divided by the number of people who actually converted. If your cost per conversion is more than your unit margin, that means you're losing money overall, and you should look deeper into how you can either reduce your cost per conversion or increase your unit margin. (Cost per conversion can also be abbreviated as CPC, so be sure not to confuse it with the above-mentioned cost per click).
Lying with your metrics
Not all metrics are created equal. Some of them look impressive but don't really mean anything. Others are defined in ways that make your product look much better than it really is. Below, you'll dig into some ways you can use metrics to tell lies—and why you shouldn't do that.
Vanity metrics
There's an infinite number of metrics you could track for your product, but many of them are unnecessary or not indicative of anything interesting. These flashy, but less meaningful, metrics are called vanity metrics. They are numbers that are neat to show off but are actually unrelated to any success or failure.
For example, page views could be a good metric or a vanity metric depending on the context. If you run a news website where your ad revenue correlates to the number of pages your viewers see, then page views is a good metric and possibly even a KPI. If you run a marketplace where the only thing that really matters is whether someone is buying your product, then page views is probably a vanity metric.
Another way to think about it is this: does your metric help you make decisions about what you should do next for your product? For example, the fact that WeWork has 6.5 million square feet of office space looks good, but it's not indicative of the overall health of the business. What percent of that space is leased? What's the margin on each office? Those are much more relevant questions that can lead to better decisions than total number of square feet.
Poorly defined metrics
Another danger to avoid is defining your metric poorly. Imagine you run a website that has 10,000 unique visitors every month. You'd like some way to know whether or not these visitors are actively using your site, or if they run away the moment they arrive.
You decide to call a user active only if they stay on the site for at least 30 seconds and browse two pages. You do the analysis and see that your active user count for the month is 1,000. After you dig deeper, you see that you can bump it to 2,000 if you count a user active after only one page view for at least 10 seconds. Which metric should you report?
This is a trick that people sometimes use to make a product's performance seem better than it really is. Google famously tried this trick with its Google+ social network. In 2013, Google reported that the network had over 300 million users, but that also included people who engaged with the Google notification bell in any Google product (like Gmail or Google Search), not people who engaged directly with the Google+ news feed or social features. This is a great example of a poorly defined metric.
As a product manager, your job is to pick metrics that accurately reflect your product's health. In general, you should always discuss your KPIs and other metrics with your manager and teammates so you're being as accurate as possible. Choosing the right KPIs will make your job easier and help you focus on the right opportunities for your product's success.
Activity 🎯
Spend some time thinking about the metrics the PM for each of these products needs to track:
- Twitter newsfeed
- Instagram messages
- Etsy marketplace
- Washington Post mobile app
Make a list of five KPIs for each of these products. Make sure they are meaningful (no vanity metrics please!).
Add a link to your answers in your notebook/notion page . If you want to further hone your skills, analyze the appropriate metrics for a few more products, focusing especially on the type of products or industry where you plan to work. This is a core PM skill, so make sure you know your metrics before you continue!