Lipschitz and approximation bounds for neural networks

Kurzfassung/Abstract

The present cumulative thesis is concerned with theoretical properties of (deep) fully connected neural networks. It is based on five publications and can be divided into two parts.
In the first part, we establish bounds on the ℓᵖ-Lipschitz constants of deep ReLU neural networks at initialization, that is, networks with random weights and biases. More precisely, we assume that the weights and biases are drawn according to a generalization of the popular He initialization. In the zero-bias case, we prove upper and lower high-probability bounds for wide networks that differ only by a logarithmic factor in the network width and a polynomial factor in the network depth. We then extend the analysis to symmetric bias distributions, for which similar bounds are established.
The second part of the thesis investigates the approximation capabilities of complex-valued neural networks (CVNNs), that is, networks with complex-valued weights and biases and with activation functions mapping from ℂ to ℂ. We prove sharp quantitative bounds for the worst-case approximation error when approximating Cʳ-functions by shallow CVNNs, both under continuous weight selection and without any assumptions on the weight selection. As part of this analysis, we generalize well-known results on the approximation properties of univariate ridge functions to the multivariate setting. Moreover, we study the universality of deep, narrow CVNNs, that is, classes of CVNNs with restricted width but arbitrary depth.