[HTML][HTML] To compress or not to compress—self-supervised learning and information theory: A review

R Shwartz Ziv, Y LeCun - Entropy, 2024 - mdpi.com
Deep neural networks excel in supervised learning tasks but are constrained by the need for
extensive labeled data. Self-supervised learning emerges as a promising alternative …

Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery

Y Cong, S Khanna, C Meng, P Liu… - Advances in …, 2022 - proceedings.neurips.cc
Unsupervised pre-training methods for large vision models have shown to enhance
performance on downstream supervised tasks. Developing similar techniques for satellite …

An introduction to neural data compression

Y Yang, S Mandt, L Theis - Foundations and Trends® in …, 2023 - nowpublishers.com
Neural compression is the application of neural networks and other machine learning
methods to data compression. Recent advances in statistical machine learning have opened …

Improving self-supervised learning by characterizing idealized representations

Y Dubois, S Ermon, TB Hashimoto… - Advances in Neural …, 2022 - proceedings.neurips.cc
Despite the empirical successes of self-supervised learning (SSL) methods, it is unclear
what characteristics of their representations lead to high downstream accuracies. In this …

Task-oriented communication for multidevice cooperative edge inference

J Shao, Y Mao, J Zhang - IEEE Transactions on Wireless …, 2022 - ieeexplore.ieee.org
This paper investigates task-oriented communication for multi-device cooperative edge
inference, where a group of distributed low-end edge devices transmit the extracted features …

Compute-efficient deep learning: Algorithmic trends and opportunities

BR Bartoldson, B Kailkhura, D Blalock - Journal of Machine Learning …, 2023 - jmlr.org
Although deep learning has made great progress in recent years, the exploding economic
and environmental costs of training neural networks are becoming unsustainable. To …

Supervised compression for resource-constrained edge computing systems

Y Matsubara, R Yang, M Levorato… - Proceedings of the …, 2022 - openaccess.thecvf.com
There has been much interest in deploying deep learning algorithms on low-powered
devices, including smartphones, drones, and medical sensors. However, full-scale deep …

Optimal representations for covariate shift

Y Ruan, Y Dubois, CJ Maddison - arXiv preprint arXiv:2201.00057, 2021 - arxiv.org
Machine learning systems often experience a distribution shift between training and testing.
In this paper, we introduce a simple variational objective whose optima are exactly the set of …

Is a caption worth a thousand images? a controlled study for representation learning

S Santurkar, Y Dubois, R Taori, P Liang… - arXiv preprint arXiv …, 2022 - arxiv.org
The development of CLIP [Radford et al., 2021] has sparked a debate on whether language
supervision can result in vision models with more transferable representations than …

Compressive visual representations

KH Lee, A Arnab, S Guadarrama… - Advances in Neural …, 2021 - proceedings.neurips.cc
Learning effective visual representations that generalize well without human supervision is a
fundamental problem in order to apply Machine Learning to a wide variety of tasks …