Learning in audio-visual context: A review, analysis, and new perspective
Sight and hearing are two senses that play a vital role in human communication and scene
understanding. To mimic human perception ability, audio-visual learning, aimed at …
understanding. To mimic human perception ability, audio-visual learning, aimed at …
A review on multimodal zero‐shot learning
Multimodal learning provides a path to fully utilize all types of information related to the
modeling target to provide the model with a global vision. Zero‐shot learning (ZSL) is a …
modeling target to provide the model with a global vision. Zero‐shot learning (ZSL) is a …
Gan inversion: A survey
GAN inversion aims to invert a given image back into the latent space of a pretrained GAN
model so that the image can be faithfully reconstructed from the inverted code by the …
model so that the image can be faithfully reconstructed from the inverted code by the …
Binding touch to everything: Learning unified multimodal tactile representations
The ability to associate touch with other modalities has huge implications for humans and
computational systems. However multimodal learning with touch remains challenging due to …
computational systems. However multimodal learning with touch remains challenging due to …
Sound to visual scene generation by audio-to-visual latent alignment
How does audio describe the world around us? In this paper, we propose a method for
generating an image of a scene from sound. Our method addresses the challenges of …
generating an image of a scene from sound. Our method addresses the challenges of …
Conditional generation of audio from video via foley analogies
The sound effects that designers add to videos are designed to convey a particular artistic
effect and, thus, may be quite different from a scene's true sound. Inspired by the challenges …
effect and, thus, may be quite different from a scene's true sound. Inspired by the challenges …
Gluegen: Plug and play multi-modal encoders for x-to-image generation
Abstract Text-to-image (T2I) models based on diffusion processes have achieved
remarkable success in controllable image generation using user-provided captions …
remarkable success in controllable image generation using user-provided captions …
Chat with the environment: Interactive multimodal perception using large language models
Programming robot behavior in a complex world faces challenges on multiple levels, from
dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large …
dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large …
Touch and go: Learning from human-collected vision and touch
The ability to associate touch with sight is essential for tasks that require physically
interacting with objects in the world. We propose a dataset with paired visual and tactile data …
interacting with objects in the world. We propose a dataset with paired visual and tactile data …
Gan-based facial attribute manipulation
Facial Attribute Manipulation (FAM) aims to aesthetically modify a given face image to
render desired attributes, which has received significant attention due to its broad practical …
render desired attributes, which has received significant attention due to its broad practical …