Robust mixture-of-expert training for convolutional neural networks
Abstract Sparsely-gated Mixture of Expert (MoE), an emerging deep model architecture, has
demonstrated a great promise to enable high-accuracy and ultra-efficient model inference …
demonstrated a great promise to enable high-accuracy and ultra-efficient model inference …
Selectivity drives productivity: efficient dataset pruning for enhanced transfer learning
Massive data is often considered essential for deep learning applications, but it also incurs
significant computational and infrastructural costs. Therefore, dataset pruning (DP) has …
significant computational and infrastructural costs. Therefore, dataset pruning (DP) has …
Constrained bi-level optimization: Proximal lagrangian value function approach and hessian-free algorithm
This paper presents a new approach and algorithm for solving a class of constrained Bi-
Level Optimization (BLO) problems in which the lower-level problem involves constraints …
Level Optimization (BLO) problems in which the lower-level problem involves constraints …
Soul: Unlocking the power of second-order optimization for llm unlearning
Large Language Models (LLMs) have highlighted the necessity of effective unlearning
mechanisms to comply with data regulations and ethical AI practices. LLM unlearning aims …
mechanisms to comply with data regulations and ethical AI practices. LLM unlearning aims …
Principled penalty-based methods for bilevel reinforcement learning and rlhf
Bilevel optimization has been recently applied to many machine learning tasks. However,
their applications have been restricted to the supervised learning setting, where static …
their applications have been restricted to the supervised learning setting, where static …
Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategy
This work focuses on addressing two major challenges in the context of large-scale
nonconvex Bi-Level Optimization (BLO) problems, which are increasingly applied in …
nonconvex Bi-Level Optimization (BLO) problems, which are increasingly applied in …
Challenging forgets: Unveiling the worst-case forget sets in machine unlearning
The trustworthy machine learning (ML) community is increasingly recognizing the crucial
need for models capable of selectively'unlearning'data points after training. This leads to the …
need for models capable of selectively'unlearning'data points after training. This leads to the …
Federated Learning Can Find Friends That Are Beneficial
In Federated Learning (FL), the distributed nature and heterogeneity of client data present
both opportunities and challenges. While collaboration among clients can significantly …
both opportunities and challenges. While collaboration among clients can significantly …
Decentralized Bilevel Optimization over Graphs: Loopless Algorithmic Update and Transient Iteration Complexity
Stochastic bilevel optimization (SBO) is becoming increasingly essential in machine
learning due to its versatility in handling nested structures. To address large-scale SBO …
learning due to its versatility in handling nested structures. To address large-scale SBO …
A multiscale Consensus-Based algorithm for multi-level optimization
A novel multiscale consensus-based optimization (CBO) algorithm for solving bi-and tri-level
optimization problems is introduced. Existing CBO techniques are generalized by the …
optimization problems is introduced. Existing CBO techniques are generalized by the …