PAPERUniversal Approximation with Deep Narrow Networks

Covers: theory of Universal Approximation with Deep Narrow Networks
Estimated time needed: 30 minutes
Questions this item adddesses:
  • Does there exist a universal approximation theorem for neural networks on bounded width and arbitrary depth that acts on Euclidean spaces?
How to use this item?

Read the background and main discussion

URL: https://arxiv.org/pdf/1905.08539.pdf
Author(s) / creator(s) / reference(s)
Patrick Kidger, Terry Lyons
Shortlist
publicShare

Non-Euclidean Universal Approximation

Nour FahmyTotal time needed: ~2 hours
Learning Objectives
Learn about the developments of Non-Euclidean Universal Approximation, and how it allows estimation of approximation bounds given a density of your neural network.
Potential Use Cases
Test how estimation changes with different NN densities.
Target Audience
ADVANCEDAdvanced audience looking to mathematically deduce estimation capabilities of NN given specified densities
Go through the following annotated items in order:
VIDEO 1. Intro to Universal Approximation Theorem
  • What even is the universal approximation theorem and why should I care about it?
10 minutes
PAPER 2. Approximation by Superpositions of a Sigmoidal Function
  • Can we estimate the approximation capabilities of a feed forward network with sigmoidal activation functions and one single hidden internal layer?
30 minutes
PAPER 3. Universal Approximation with Deep Narrow Networks
  • Does there exist a universal approximation theorem for neural networks on bounded width and arbitrary depth that acts on Euclidean spaces?
30 minutes
PAPER 4. Quantitative Rates and Fundamental Obstructions to Non-Euclidean Universal Approximation with Deep Narrow Feed-Forward Networks
  • Can we specify estimates of a deep and narrow network with general activation functions, irregardless if the spaces it acts on are Euclidean?
30 minutes

Concepts Convered