ARTICLEDifference between restrictions imposed by L1 and L2 penalties on weights

Covers: theory of L1 norm weight penalty
Estimated time needed: 15 minutes
Questions this item adddesses:
  • Difference between restrictions imposed by L1 and L2 penalties on weights
How to use this item?

Read the entire article

URL: https://explained.ai/regularization/L1vsL2.html#sec:3.1
Author(s) / creator(s) / reference(s)
Terence Parr
Shortlist
publicShare

L1 norm weight penalty

Hardik SahiTotal time needed: ~2 hours
Learning Objectives
Mathematical foundations behind Deep Learning
Potential Use Cases
Mathematical foundations behind Deep Learning
Target Audience
INTERMEDIATEPeople interested in knowing how Deep Learning model training works.
Go through the following annotated items in order:
ARTICLE 1. Conceptual understanding of L1 norm weight penalty
  • Gives a visual understanding of what L1 penalty does for a simple 2-d case.
20 minutes
ARTICLE 2. Practical implementation of L1 penalty
  • How is L1 regularization implementation different from its conceptual understanding?
20 minutes
BOOK_CHAPTER 3. Mathematical understanding of what L1 penalty does to weights learnt as compared to weights of unregularized cost function.
  • Mathematical understanding of what L1 penalty does to weights
30 minutes
ARTICLE 4. Difference between restrictions imposed by L1 and L2 penalties on weights
  • Difference between restrictions imposed by L1 and L2 penalties on weights
15 minutes
BOOK_CHAPTER 5. What is regularization for neural networks?
  • What is regularization?
15 minutes
BOOK_CHAPTER 6. How does penalizing parameters lead to regularization?
  • Why is penalty applied only to weights and not biases
10 minutes

Concepts Convered