Recipe
publicShareStar

Activation Functions for Neural Networks in Natural Language Processing

Collaborators
Reviewers
Total time needed: ~2 hours
Learning Objectives
You will learn about activation functions in general, and about how they are used in neural networks for NLP.
Potential Use Cases
Creating an NLP neural network, or experimenting with different activation functions for an NLP neural network.
Target Audience
BEGINNERThis list would be useful for someone new to NLP or neural networks.
Go through the following annotated items in order:
ARTICLE 1. Fundamentals of Deep Learning – Activation Functions and When to Use Them?
  • What are the different types of activation functions?
15 minutes
BOOK_CHAPTER 2. Neural Network Methods for Natural Language Processing
  • What activation functions are used in NLP neural networks?
10 minutes
PAPER 3. A Primer on Neural Network Models for Natural Language Processing
  • What are the activation functions used within a neural network for NLP, and for the output layer?
10 minutes
ARTICLE 4. 7 types of Artificial Neural Networks for Natural Language Processing
  • What types of neural networks are used for NLP?
15 minutes
PAPER 5. Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks
  • When should you use alternative types of activation functions?
20 minutes

Concepts Covered