Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping
Using an extended and formalized version of the Q/C map analysis of Poole et al.(2016),
along with Neural Tangent Kernel theory, we identify the main pathologies present in deep …
along with Neural Tangent Kernel theory, we identify the main pathologies present in deep …
Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations
Deep neural networks have become the state-of-the-art models in numerous machine
learning tasks. However, general guidance to network architecture design is still missing. In …
learning tasks. However, general guidance to network architecture design is still missing. In …
Flexpoint: An adaptive numerical format for efficient training of deep neural networks
Deep neural networks are commonly developed and trained in 32-bit floating point format.
Significant gains in performance and energy efficiency could be realized by training and …
Significant gains in performance and energy efficiency could be realized by training and …
dynoNet: A neural network architecture for learning dynamical systems
M Forgione, D Piga - … Journal of Adaptive Control and Signal …, 2021 - Wiley Online Library
This article introduces a network architecture, called dynoNet, utilizing linear dynamical
operators as elementary building blocks. Owing to the dynamical nature of these blocks …
operators as elementary building blocks. Owing to the dynamical nature of these blocks …
Training behavior of deep neural network in frequency domain
Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is
a mystery [24]. To find a potential mechanism, we focus on the study of implicit biases …
a mystery [24]. To find a potential mechanism, we focus on the study of implicit biases …
Time-delay neural networks: Representation and induction of finite-state machines
In this work, we characterize and contrast the capabilities of the general class of time-delay
neural networks (TDNNs) with input delay neural networks (IDNNs), the subclass of TDNNs …
neural networks (TDNNs) with input delay neural networks (IDNNs), the subclass of TDNNs …
Deep learning with dynamic spiking neurons and fixed feedback weights
A Samadi, TP Lillicrap, DB Tweed - Neural computation, 2017 - ieeexplore.ieee.org
Recent work in computer science has shown the power of deep learning driven by the
backpropagation algorithm in networks of artificial neurons. But real neurons in the brain are …
backpropagation algorithm in networks of artificial neurons. But real neurons in the brain are …
Online spatio-temporal learning in deep neural networks
Biological neural networks are equipped with an inherent capability to continuously adapt
through online learning. This aspect remains in stark contrast to learning with error …
through online learning. This aspect remains in stark contrast to learning with error …
GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
Although deep neural networks (DNNs) are being a revolutionary power to open up the AI
era, the notoriously huge hardware overhead has challenged their applications. Recently …
era, the notoriously huge hardware overhead has challenged their applications. Recently …
Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies
Circuits of biological neurons, such as in the functional parts of the brain can be modeled as
networks of coupled oscillators. Inspired by the ability of these systems to express a rich set …
networks of coupled oscillators. Inspired by the ability of these systems to express a rich set …