Evaluating Recommender Systems

Total time needed: ~3 hours
Learning Objectives
This list will cover error-based methods, rank-based methods, and other metrics for evaluating recommender systems.
Potential Use Cases
Assess performance of product recommendation, friend recommendation, movie/video recommendation, etc.
Target Audience
Go through the following annotated items in order:
ARTICLE 1. List of Methods to evaluate RecSys
  • What is Mean Absolute Error?
  • What are MSE and RMSE?
  • What is Precision and Recall?
  • What is ROC curve?
  • What is nDCG?
  • What are Coverage, Popularity, and Novelty?
30 minutes
ARTICLE 2. List of Evaluation Methods
  • What is MAE?
  • What is HIT rate?
  • What is Coverage, Diversity, and Novelty?
15 minutes
ARTICLE 3. Root Means Squared Error and Mean Absolute Eerror
  • What is RMSE?
  • What is MAE?
  • Which method (RMSE or MAE) is better to use in what context?
5 minutes
ARTICLE 4. Mean Average Precision
  • What is Precision and Recall at Cutoff-K?
  • What is Average Precision?
  • What are some variants of Average Precision formula?
20 minutes
ARTICLE 5. Normalized Discounted Cumulative Gain (nDCG)
  • What is cumulative gain?
  • What is normalized discounted CG?
  • What are some limitations of nDCG?
10 minutes
ARTICLE 6. Comparison of Rank-based Evaluation Methods
  • What is MRR?
  • What is MAP?
  • What is nDCG?
  • How do these different methods compare?
30 minutes
PAPER 7. Beyond Accuracy: Coverage and Serendipity in RecSys
  • What are different ways to calculate coverage (e.g. precision coverage)?
  • What is serendipity in recommender systems?
30 minutes

Concepts Covered