VIDEORasa's series for Explanation of the Transformer Architecture and its components

Covers: theory of Transformer-based LMs
Estimated time needed: 50 minutes
Questions this item adddesses:
  • What is self-attention?
  • What is the Transformer?
How to use this item?

Watch all 4 parts of the Transformer mechanism

Fail to play? Open the link directly: https://www.youtube.com/watch?v=yGTUuEx3GkA&ab_channel=Rasa
Author(s) / creator(s) / reference(s)
Rasa Youtube content
Shortlist
publicShare

Language Modelling from N-grams to Transformer

Zach NguyenTotal time needed: ~4 hours
Learning Objectives
With this list, you will learn about Language Modelling and the three families of methodology to model language and use it for text understanding or text generation.
Potential Use Cases
Chatbot, Speech recognition, NLU and NLG for Downstream tasks
Target Audience
INTERMEDIATEPython Developers who have basic understanding of NLP (experienced with text processing, n-grams, word-vectors)
Go through the following annotated items in order:
ARTICLE 1. Introduction to Language Modelling and N-gram LM
  • What is a language model?
  • How does an N-gram language model work?
10 minutes
ARTICLE 2. In-depth take into N-gram modelling
  • How was language modelling traditionally done?
  • What was the drawbacks of traditional LM techniques?
  • What methods were proposed to mitigate the drawbacks?
40 minutes
ARTICLE 3. An Intuitive understanding of the use of NN in LM
  • How does a NN achieve LM?
  • What are the typical NN architecture that can do LM?
15 minutes
VIDEO 4. A mathematical understanding of how RNNs model language
  • How exactly does a Neural Network model sequences?
12 minutes
PAPER 5. A Survey on Neural Network Language Models
  • What are NN architecture used for Language Modelling?
  • How did they contribute to improve SOTA performance on Language Modelling?
15 minutes
VIDEO 6. Rasa's series for Explanation of the Transformer Architecture and its components
  • What is self-attention?
  • What is the Transformer?
50 minutes
VIDEO 7. The Narrated Transformer Language Model
  • How does the Transformer achieve Language Modelling?
  • What are the reasons why this modern architecture performs well?
30 minutes
USE_CASE 8. Implementation of Language Modelling with three common algorithms
  • How do we implement language models?
20 minutes
USE_CASE 9. 8 Leading Language Models For NLP In 2020
  • What are the pre-trained model you can use in the market?
  • What are the details about each pre-trained model (architecture, achievement, use cases)?
10 minutes

Concepts Convered