The core components of many modern neural network architectures, such as transformers,
convolutional, or graph neural networks, can be expressed as linear layers with* weight …

Adaptive gradient optimizers like Adam (W) are the default training algorithms for many deep
learning architectures, such as transformers. Their diagonal preconditioner is based on the …

Optimizing deep neural networks is one of the main tasks in successful deep learning.
Current state-of-the-art optimizers are adaptive gradient-based optimization methods such …

Why do deep neural networks (DNNs) benefit from very high dimensional parameter
spaces? Their huge parameter complexities vs. stunning performances in practice is all the …

Symplectic numerical integrators for Hamiltonian systems form the paramount class of
geometric numerical integrators, and have been very well investigated in the past forty …

Adaptive optimization algorithms such as Adam see widespread use in Deep Learning.
However, these methods rely on diagonal approximations of the preconditioner, losing much …