Case Interview: Metrics Diagnosis

1. Procedure of Metrics Shift Diagnosis

This type of cases usually come as a important metrics shift(usually towards a negative direction), and the objective the problem is to find the root cause of the shift and try to fix it

Framework:

1. Clarify the definition of the metric:

   how the metrics is defined. This help you decompose the metrics correctly. This is useful for metric decomposition

2. Check Data:

   Check the log of the data source and data pipeline to make sure the shift is not caused by any bugs, logical errors or outliers

3. Check external factors:

   Figure out whether the shift is caused by any external factors, including:

   • Seasonality: Is there a seasonal factor in the lifecycle of the product? Like there would be a drastic decline in certain weeks or months of every year. Conduct year-on-year comparison to see such decline always appear in a particular stage in a year
   • Industry/market trend: If there exist a descending trend in the industry? Look to the data of the metrics for the previous few weeks or month. Also see this trend happens on other products or competitors or industry. Figure out the scope of the trend
   • Competitors: If the competitors are doing a sales campaign? Does the loss of the metrics goes to the competitors?
   • Special events: Is there a change of policy? Is there any social events or phenomenon happening that it might have a potential impact on the metrics?

4. Check Internal facotrs:

   1. Change or treatment on product:

      is there any change or treatment implemented on the product around the time of the metric's decline? if so, we can do a causal inference to evaluate the effect

   2. Decompose the metrics:

      decompose the metrics in a additive or multiplicative way, for example: order amont = show pv * CTR * C_O, ETA = waiting time for a drive to take the order + time the driver arrive the pick up point + time the rider find the driver

      For each sub-metrics, calculate the changing rate and find out which submetric is the most influential part of the original metrics, focus on these sub- metrics

   3. Segment the metric by user demographic channel, or user behavior

   Segment the metric by dimensions like region, new/old user, platform, content category. For each subgroup, calculate its contributions ot the decline of the metrics. For example, if you find some facts like 90% of the decline are caused by new user, than you probably would suspect there's a association, like the gui is not friendly to new user

2. Fermi Estimation

Fermi estimation is a method of making rough estimates of unknown quantities using approximate values and simple reasoning. The basic idea is to decompose the metric we want to estimate into some submetrics in a additive or multiplicative way. For example, to answer to question: how many red vehicles are there in New York?

We can make such decomposition:

Red vichle in New York = Population in New York * Vichle ownership rate * City factor(large city usually has higher ownership rate) * the ratio of red car in all cars

Then, for each metirc, we given them a upper boundary and a lower boundary according to our common sense. We use the median of the range as an estimation and use them to calculate the estimation for the target metric.