Covers: theory of Activation Functions
Estimated time needed to finish: 20 minutes
Questions this item addresses:
  • When should you use alternative types of activation functions?
How to use this item?

Read sections 1, 2, 4, 5.

Author(s) / creator(s) / reference(s)
Steffen Eger, Paul Youssef, Iryna Gurevych
0 comment

Activation Functions for Neural Networks in Natural Language Processing

Total time needed: ~2 hours
You will learn about activation functions in general, and about how they are used in neural networks for NLP.
Potential Use Cases
Creating an NLP neural network, or experimenting with different activation functions for an NLP neural network.
Who is This For ?
BEGINNERThis list would be useful for someone new to NLP or neural networks.
Click on each of the following annotated items to see details.
ARTICLE 1. Fundamentals of Deep Learning – Activation Functions and When to Use Them?
  • What are the different types of activation functions?
15 minutes
BOOK_CHAPTER 2. Neural Network Methods for Natural Language Processing
  • What activation functions are used in NLP neural networks?
10 minutes
PAPER 3. A Primer on Neural Network Models for Natural Language Processing
  • What are the activation functions used within a neural network for NLP, and for the output layer?
10 minutes
ARTICLE 4. 7 types of Artificial Neural Networks for Natural Language Processing
  • What types of neural networks are used for NLP?
15 minutes
PAPER 5. Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks
  • When should you use alternative types of activation functions?
20 minutes

Concepts Covered

0 comment