Despite its flexibility to learn diverse inductive biases in machine learning programs, meta
learning (ie,\learning to learn) has long been recognized to suffer from poor scalability due …

Offline model-based optimization aims to maximize a black-box objective function with a
static dataset of designs and their scores. In this paper, we focus on biological sequence …

Deep Equilibrium (DEQ) Models, an emerging class of implicit models that maps inputs to
fixed points of neural networks, are of growing interest in the deep learning community …

The essential difficulty of gradient-based bilevel optimization using implicit differentiation is
to estimate the inverse Hessian vector product with respect to neural network parameters …

Differentiable optimization algorithms often involve expensive computations of various meta-
gradients. To address this, we design and implement TorchOpt, a new PyTorch-based …

Despite its practical importance across a wide range of modalities, recent advances in self-
supervised learning (SSL) have been primarily focused on a few well-curated domains, eg …

Meta-learning has emerged as a powerful approach for leveraging knowledge from previous
tasks to solve new tasks. The mainstream methods focus on training a well-generalized …

The design of highly accurate deep learning architectures is related to different parameters
through different optimizations at different levels of the design. While the architectural design …

Large-scale pretraining followed by task-specific finetuning has achieved great success in
various NLP tasks. Since finetuning all parameters of large pretrained models poses …

Low-rank adaptation (LoRA) is a popular method for fine-tuning large-scale pre-trained
models in downstream tasks by learning low-rank incremental matrices. Though LoRA and …