Empirical studies of the loss landscape of deep networks have revealed that many local
minima are connected through low-loss valleys. Yet, little is known about the theoretical …

Existing gradient-based optimization methods update parameters locally, in a direction that
minimizes the loss function. We study a different approach, symmetry teleportation, that …

In many neural networks, different values of the parameters may result in the same loss
value. Parameter space symmetries are loss-invariant transformations that change the …

Abstract Deep Neural Networks (DNN) have made significant advances in various fields
including speech recognition and image processing. Typically, modern DNNs are both …

Empirical studies have revealed that many minima in the loss landscape of deep learning
are connected and reside on a low-loss valley. We present a general framework for finding …

Parameter space symmetries, or loss-invariant transformations, are important for
understanding neural networks' loss landscape, training dynamics, and generalization …

Most activation functions operate component-wise, which restricts the equivariance of neural
networks to permutations. We introduce Conic Linear Units (CoLU) and generalize the …