L2 norm weight penalty regularization

Total time needed: ~2 hours
Learning Objectives
This list will help you get an intuitive idea of L2 regularization for weights in deep learning.
Potential Use Cases
Mathematical foundations behind Deep Learning
Target Audience
INTERMEDIATEPeople interested in knowing how Deep Learning model training works.
Go through the following annotated items in order:
ARTICLE 1. Conceptual understanding of applying L2 penalty on weights.
  • Gives a visual understanding of what L2 penalty does for a simple 2-d case.
20 minutes
ARTICLE 2. How is L2 regularization implemented practically?
  • How is L2 regularization different from its conceptual understanding?
15 minutes
BOOK_CHAPTER 3. What does L2 penalty actually does to the weights in comparison to unregularized cost function?
  • How to mathematically associate the weights learnt using regularized cost function with weights learnt for unregularized cost function?
30 minutes
ARTICLE 4. [Example] A case of L2 regularization for Linear regression.
  • How L2 penalty applied to Linear regression problem help to handle strong multicollinearity?
30 minutes
BOOK_CHAPTER 5. What does it mean to penalize weights using norm ?
  • How does penalizing weights lead to regularization?
  • Why penalty is applied only to weight matrices and not biases?
10 minutes
BOOK_CHAPTER 6. What is regularization?
  • What is regularization?
  • What is the need of preventing model from overfitting?
20 minutes

Concepts Covered