Despite extensive studies, the underlying reason as to why overparameterizedneural
networks can generalize remains elusive. Existing theory shows that common stochastic …

Sharpness-Aware Minimization (SAM) is a highly effective regularization technique for
improving the generalization of deep neural networks for various settings. However, the …

We study gradient descent under linearly correlated noise. Our work is motivated by recent
practical methods for optimization with differential privacy (DP), such as DP-FTRL, which …

Sharpness-Aware Minimization (SAM) is a highly effective regularization technique for
improving the generalization of deep neural networks for various settings. However, the …

This paper proposes a flexible probabilistic activation function that enhances the training
and operation of artificial neural networks by intentionally injecting noise to gain additional …

It is well known that the finite step-size ($ h $) in Gradient Descent (GD) implicitly regularizes
solutions to flatter minima. A natural question to ask is" Does the momentum parameter …

Studying the properties of stochastic noise to optimize complex non-convex functions has
been an active area of research in the field of machine learning. Prior work~\citep …

Fine-tuning pretrained language models (PLMs) for downstream tasks is a large-scale
optimization problem, in which the choice of the training algorithm critically determines how …

Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized
the usage of Large Language Models (LLMs). Recent studies have shown that a small …

It is often observed that stochastic gradient descent (SGD) and its variants implicitly select a
solution with good generalization performance; such implicit bias is often characterized in …