Seminarium "Modeling of Complex Systems"

The lecture presents practical insights into training of Deep Neural Networks. We will start by introducing basic components of modern Artificial Neural Networks and explain why the training of such networks is not a trivial task. Next, we will discuss the most important approaches to training neural networks, including Stochastic Gradient Descent and Adam, show practical variants of the methods improving their performance and convergence time and discuss applicability of the methods. Finally, we will take a look into the most recent advances in the field of optimizing neural networks, including Circular Learning Rates, warm-up and Lookahead optimizer.