Neural network identifiability for a family of sigmoidal nonlinearities
AuthorsVerner Vlačić and Helmut Bölcskei
ReferenceConstructive Approximation, May 2021, (invited paper).
AbstractThis paper addresses the following question of neural network identifiability: Does the input-output map realized by a feed-forward neural network with respect to a given nonlinearity uniquely specify the network architecture, weights, and biases? Existing literature on the subject , ,  suggests that the answer should be yes, up to certain symmetries induced by the nonlinearity, and provided the networks under consideration satisfy certain “genericity conditions”. The results in  and  apply to networks with a single hidden layer and in  the networks need to be fully connected. In an effort to answer the identifiability question in greater generality, we derive necessary genericity conditions for the identifiability of neural networks of arbitrary depth and connectivity with an arbitrary nonlinearity. Moreover, we construct a family of nonlinearities for which these genericity conditions are minimal, i.e., both necessary and sufficient. This family is large enough to approximate many commonly encountered nonlinearities to within arbitrary precision in the uniform norm.
KeywordsDeep neural networks, identifiability, sigmoidal nonlinearities
Download this document:
Copyright Notice: © 2021 V. Vlačić and H. Bölcskei.
This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.