Abstract Stochastic Gradient Descent (SGD) is arguably the most important single algorithm
in modern machine learning. Although SGD with unbiased gradient estimators has been …

We propose a novel randomized first order optimization method---SEGA (SkEtched GrAdient
method)---which progressively throughout its iterations builds a variance-reduced estimate …

Prompt-tuning has emerged as a promising method for adapting pre-trained models to
downstream tasks or aligning with human preferences. Prompt learning is widely used in …

We introduce a general framework for large-scale model-based derivative-free optimization
based on iterative minimization within random subspaces. We present a probabilistic worst …

Derivative-free algorithms seek the minimum value of a given objective function without
using any derivative information. The performance of these methods often worsens as the …

Deep neural networks are vulnerable to small input perturbations known as adversarial
attacks. Inspired by the fact that these adversaries are constructed by iteratively minimizing …

In this paper, we consider a stochastic distributed nonconvex optimization problem with the
cost function being distributed over n agents having access only to zeroth-order (ZO) …

We consider the problem of minimizing a high-dimensional objective function, which may
include a regularization term, using only (possibly noisy) evaluations of the function. Such …

Abstract The key to Black-Box Optimization is to efficiently search through input regions with
potentially widely-varying numerical properties, to achieve low-regret descent and fast …

Neuroevolutionary machine learning is an emerging topic in the evolutionary computation
field and enables practical modeling solutions for data-driven engineering applications …