Case Interview: Measure Success

1. Analytics Framework

For metrics selection, the funnel anaysis model is the most frequently applied type of frameworks. In this case, we commonly applied the AARRR framework to allocate proper metrics. For details on models and frameworks, refer to ongoing

The according metrics of each objectives under various business models are listed below:

Business Model	Acquisition	Activation	retention	Revenue	referral
E-commerce
SaaS
Free Mobile APP
Media Site
UGC
TS Marketplace
For all	DAU/MAU, DNU by platform

2. Criterion for Metrcis Selection

3. Procedure of Measure Success Problem

This type of cases usually come as "find a metric to measure the extent.....(how good a new feature/strategy/product is)". IT evaluates one's ability to design a measure according to certain business objectives

1. Clarify the objective:

   Ask questions about:

   • How this feature works?
   • who is it for

2. Confirm the business objective

   according to the explanation from the interviewer, conclude the objective of the feature to the following aspects:

   • Acquisition: bring in more user, help them discover the product
   • Activation: make user taking actions we want them to, keep users engaged
   • conversion and retention: keep user continuing to use the product, convert them from free-user to paid user
   • Money: make revenue from users

3. Select metrics according to objective

   Objective
   Acquisition	DAU/MAU, DNU by platform,
   Activation	ACU/PCU, PV/UV, time spending, interaction behaviour
   Conversion and Retention	CTR,CO, pay rate, 7 day retention rate/churn rate
   Money	ROI,GMV, AOV, ACV, ARPU/ARPPU

   • Select 1 main success metric, this metric should be directly relevant to the business objective

   • Select 1 second success metrics, this metrics should be a supplement. It improve the completeness of the logic relationship of the key metrics and business goal. To fing the second goal, consider the following questions:

     • Can the change of key metrics be caused by factors other than the feature?
     • Is the key metric equivalent to the business goal? Is there any intermediate process
     • If the key metrics changes on a group level, should we have same changes on single user level?

     Alternatively, we can do a funnel analysis to find the secondary success metrics

   • Select 1 guardrail metric, this metric is critical to the main goal of the whole product. The feature should not have significant negative effect on it

4. structure the answer

   if the guardrail metric shift towards a negative direction, the feature is probably not success.

   Otherwise, if the key success metric and the second both shift towards a positive direction, the feature is successful.