Measuring deep impact

We have made it to the the final guide of the LEARN series. This guide complements the last two guides, as it is all about measuring the impact we are making as we discover what to build and turn on our growth engines. Measuring the right data for your product and business is an essential part of your startup journey. Otherwise you are flying blind, and you will likely never reach your mission. In this guide we will take a pragmatic approach to defining and implementing analytics for your product and business. The optics you gain from such will help you iterate your product, inform your strategic business decisions and show investors that you are onto something worth investing in.

KPIs and Vanity Metrics

Vanity metrics may be attractive, but tracking the wrong metrics can be a pitfall. For an example, take the number of users who have visited your website or the number of sign ups for your product. While a trend up in either of these metrics are likely a step in the right direction, they only represent a piece of the larger data puzzle. You could easily become misled into making wrong decisions for improving your product and business. Thus, it is important to define, design and analyze metrics on a deeper level. We need to better understand the internal dynamics of what is happening when users interact with our product and how this leads to them becoming actively engaged and coming back for more. Ultimately, we want our users so elated that they are sharing our product/business with others and paying us.

Case in Point

As mentioned in the last case, we were able to get hundreds of thousands of users to visit our website while working on the Pastimes app. We were even able to get tens of thousands of users to download our app and sign up. However, I must admit that I became infatuated with these metrics to a flaw. Rather than building a proper analytics strategy around what was happening within the product usage, I was anchored to the growth at the top of the funnel in our userbase. Furthermore, I was assuming our users were actively engaging in the app based on the number of connections being made. However, here is what I was missing.

Many of the users who were downloading the app were actually spam bot accounts. Many of these accounts were inflating both the overall user numbers and the “Activation” metrics due to their automated messages. While we ultimately discovered this, implemented better tooling to prevent these accounts from spamming our users, we made many uninformed decisions based on the lack of usage data we were not tracking.

Outside of the bots and spam, there were many unanswered messages. This was because many of our users were using Pastimes as a dating app. They were blasting many other users of the opposite sex dating messages. Because Pastimes was not a formal dating app, this behavior and use case led to less response from users.

Thus, while it looked like we were growing and our users were engaging, it was actually not as much as our numbers indicated. I learned that it is important to have a holistic view of your data. In order to do such, you need to build a data blueprint using the following framework. Doing so ultimately helped us gain better insights into how to build a better product and business.

Defining Analytics

Dave McClure’s (500) Pirate metrics have become a standard framework for designing and measuring startup analytics. These metrics AARRRe a solid framework for measuring the holistic picture end to end. Awareness, Activation, Retention, Referral, and Revenue frame a holistic blueprint for how to best define your analytics strategy. You can add another A in the mix to measure how many users go from Awareness to creating an Account. This way we do not include the account creation as an Active usage metric and can track how many users go from creating an account to becoming active. Thus, McClure’s pirate metrics become AAARRR. Let’s double click into each of these metrics.

Awareness

These are all the metrics you can define, implement and track for how people discover your product from each awareness channel. Awareness metrics are sometimes referred to as Acquisition metrics. They include both the traditional paid marketing channels as well as the growth channels we explored in the previous Boostrapping Growth guide. This is where vanity metrics for how many users visited your website for your product become a bit more insightful. This is because now you can track how effective and expensive each awareness channel is. Examples for awareness channels include:

  • Traditional Ad Campaigns

  • SEO/SEM

  • Social Media

  • Ambassadors

  • Partnerships

  • Bootstrapped growth channels discussed here

Account

As previously mentioned, this is the additional category to McClure’s pirate metrics framework. The metrics help you understand how many users are signing up for an account to further explore your product beyond just the marketing on your website. This is an important step in the metrics funnel, as the Account is typically the first step where users provide you with their personally identifiable data. Not only, will you have a data point to measure drop off from awareness to account to activation in the funnel, but you can use this information to re-market to users who dropped off. Additionally, you will also be able to measure the drop off of users who were not convinced to take the next step and sign up for an account. Tracking this churn overtime can be very effective as you make improvements to your product marketing and value proposition.

Activation

In order to measure activation, we must define how a user becomes active when interacting with our product. This will help you understand whether people are interacting with your product in the way you designed it to provide value. While the “active” definition will be subjective, it should objectively connect to unlocking the intended value you are creating in your product. Once you have defined what an active user is, you can then start to measure how many users who created an account, have become active. This will easily be one of the most important aspects of informing your product’s roadmap. If you can’t activate the users you are building awareness to create accounts, they will likely not come back after their initial experience, refer others to check your product or pay you. Thus, this is a vital drop off to take note of in your product analytics.

Additionally, once you have defined an active user, you will be able to measure and track the true Active Usage. You may hear investors ask you what your DAU/WAU/MAU is. While some tools will report a proxy for daily/weekly/monthly active usage based on their own definition of activity, it is more important to track DAU/WAU/MAU normalized to your definition of active usage. Otherwise, this could be another example where a vanity metric misleads the reporting of how much true activation your product is generating.

Retention

Once you have defined active behavior, you will want to see how it translates to active users returning to engage with your product. Thus, the next drop off to track and improve is how many active users do not return. As you improve your product’s value proposition, you should start to see improved retention metrics. Retention is a critical aspect of Product Market Fit (PMF). We discuss PMF at length in the next guide in the context of tracking monetization from your business model.

Referral

As we have previously discussed in the previous Boostrapping Growth guide, referral is all about the viral coefficient your product is generating. For every user who comes into experience your product, how many users are they telling others about your product. As your value proposition improves, you should see improved referral metrics due to organic word of mouth that is taking place. Often times a Net Promoter Scores (NPS) is used as a proxy for referral. You have likely seen the question asked, “How likely are you to recommend this product/business/experience to a friend?”. The NPS response is a numbered scale from 0-10. A net promoter is any user who responds with a 9 or 10. A detractor is any user who responds with a 0-6. A user who responds with a 7-8 is seen as a passive user. Thus, when tracking your NPS, you aggregate the net of promoters with detractors and passive users to give you an objective score. If your NPS score is trending upwards over time, you must be improving the outcome your experience is having on your end users or customers.

Revenue

If your product’s value proposition is strong enough, it should be generating revenue. Thus, you will want to track how activation and retention are translating into revenue. This can then help you determine the health of your business model. Doing such can help you also determine how much to invest in your cost of acquisition. We explore monetization, business models and PMF in the next guide.

In summary, your product analytics blueprint should be a list of all the behaviors you want to track within your product that map to each of the AAARRR categories. There are many product analytics tools that act as a system of record for your metrics. You can also just use a spreadsheet. It doesn’t matter what you use, it is just important that you define, implement and track the metrics that answer the following:

  1. Acquisition - How are people discovering your product from the awareness you generate?

  2. Account - Are these users sold on the value proposition you marketed by signing up for an account?

  3. Activation - Are these users receiving the intended value from your product?

  4. Referral - Are these active users coming back over and over again?

  5. Referral - Are these active users telling other users about your product?

  6. Revenue - Are these active users willing to pay you for the value of your product?

Implementing Analytics

Once you have defined all the behaviors you want to measure, you can now implement events that track how your product innovations and solutions have an impact on each step of the analytics funnel. In order to do such, you will need to build requirements for your engineering team to code events for each feature that gets released. Typically this requires a small amount of coding for engineers. The heavier lift is on the product analytics and metrics design. Again, you will regret not having events coded for tracking usage in your product. While you can always add tracking events after the fact, you will not have any historical data to benchmark what is happening once the usage data starts to come in.

Tracking and Reporting

Now that the usage of each feature being released in your product can be measured, you will need to use an analytics tool to aggregate trends among cohorts of users of your product. While there are many tools for doing such, Firebase and Google Analytics are often used to build funnels and reporting product analytics. Advanced reporting beyond out of the box funnel reports may require some data warehousing and business intelligence tools and skills. Having someone on your team that has a data analytics or data science background could be very helpful once the operations of your product start to take off and your data really starts to scale. However, in the meantime, you should be able to keep your data and product analytics simple and work with the out of the box tools and funnel reports available.

If you have put the work into planning, defining and implementing your product analytics, you should now begin to understand what is and is not working within your product. Data science is both an art and science. While it is not actual rocket science, it does require a gambit of technical and analytical skills. As a product manager, your goal will be to run as many experiments as possible to understand how they impact your product analytics.

Rocket Science Checklist

This guide has hopefully helped you understand the difference between vanity metrics and product metrics that can track the impact of your startup’s mission. In the next guide we expand outside of product analytics to explore monetization and product market fit. This guide summarizes the LEARN series, and in the next guide we shift gears towards how to LAND successful business and exit.

Rocket Science Checklist:

  1. Did you define your product analytics for each AAARRR category?

  2. Did create a product analytics blueprint that maps to your roadmap of product features?

  3. Did you implement events in the code for all of the new features you are releasing in your product?

  4. Did you use a analytics tool to build reports that track your product analytics?

  5. Did you run experiments as a scientist and product manager to improve your the AAARRR of your product analytics?

  • Add a short summary or a list of helpful resources here.