Neural Networks and Deep Learning¶

(Draft status)

Practical Implementations with Python, Theano and Tensorflow.

Prerequisite:¶

• Basics of machine learning
• Short review gradient descent

Content¶

• General Introduction to Deep Learning
• Introduction/Applications (slides)
• Programming models for Deep Learning with examples in Theano and TensorFlow
• Introduction into Theano
  • Univariate Linear Regression with Theano
• Feedforward Neural Networks
  • Feedforward Neural Networks in a Nutshell (incl. a numpy implementation)
  • Using TensorFlow to build feedforward neural networks:
    • One Neuron - Logistic Regression - in German
    • Simple Feed Forward Neural Network - in German
  • Using Theano to build feedforward neural networks:
    • Simple Feed Forward Neural Network with Theano
    • Slides
    • For an implementation of neural networks also work through the first two chapters of the deep learning tutorial (external link)
• Neuron Types
• Autoencoder and Manifold Hypotheses
  • Slides: Unsupervised Pretraining, Autoencoder and Manifolds
• Weight initialization
• Optimization:
  • Momentum and Nesterov Momentum, Adagrad and Adadelta, 2nd order optimization and Hessian free optimization
• Dropout and Dropconnect

Convolutional Neural Networks (CNNs)¶

• Tensoflow Implementation of [Die16] Dieleman, Sander, Jeffrey De Fauw, and Koray Kavukcuoglu; "Exploiting cyclic symmetry in convolutional neural networks"

Recurrent Neural Networks (RNNs)¶

• Recurrent neural networks
• LSTM: Long Short Term Memory
• Deep transition recurrent neural networks
• Stacked recurrent neural networks
• Applications of RNNs

Energy based models¶

• Intro RBM: Introduction to Restricted Boltzmann Machines
• Gibbs Sampling on RBM
• Contrastive Divergence
• ...
• Renormalization Group and RBMs