Universal approximation with complex-valued deep narrow neural networks

Kurzfassung/Abstract

We study the universality of complex-valued neural networks with bounded widths
and arbitrary depths. Under mild assumptions, we give a full description of those
activation functions � : C → C that have the property that their associated networks
are universal, i.e., are capable of approximating continuous functions to arbitrary
accuracy on compact domains. Precisely, we show that deep narrow complex-valued
networks are universal if and only if their activation function is neither holomorphic,
nor antiholomorphic, nor R-affine. This is a much larger class of functions than in the
dual setting of arbitrary width and fixed depth. Unlike in the real case, the sufficient
width differs significantly depending on the considered activation function. We show
that a width of 2n + 2m + 5 is always sufficient and that in general a width of
max {2n, 2m} is necessary. We prove, however, that a width of n +m +3 suffices for
a rich subclass of the admissible activation functions. Here, n and m denote the input
and output dimensions of the considered networks. Moreover, for the case of smooth
and non-polyharmonic activation functions, we provide a quantitative approximation
bound in terms of the depth of the considered networks.