Dipam Goswami

I am a PhD Student at the Computer Vision Center, Universitat Autonoma de Barcelona working with Dr. Joost van de Weijer and Dr. Bartlomiej Twardowski at the Learning and Machine Perception group. Previously, I graduated with a B.E. in Computer Science and M.Sc. in Mathematics from Birla Institute of Technology and Science, Pilani India in 2022. I was also a research assistant at the Augmented Vision Group at DFKI Kaiserslautern.

I am mostly working on Continual Learning problems. Recently, I am also exploring Federated Learning, Document Retrieval and Large Language Models. I have published research papers on Continual Learning for Image Classification, Few-Shot Continual Learning, Incremental Object Detection and Semantic Segmentation.

I serve as reviewer for several conferences (ICLR, AISTATS, NeurIPS, CVPR, ECCV, WACV, BMVC) and journals (IEEE Transactions on Neural Networks and Learning Systems, IEEE Transactions on Image Processing). I have been recognized as one of the outstanding/top reviewers in BMVC 2024 and NeurIPS 2024.

news

Aug 30, 2024	Finished my 3-month long research stay at IDEAS NCBR, Warsaw where I worked with dense retrieval embedding models.
Jun 21, 2024	Attended CVPR in Seattle and presented two posters - one paper in the main conference and one paper in the CLVision workshop.
Jun 13, 2024	Gave an invited talk at the GMUM seminar at the Jagiellonian University in Krakow. Also presented my research at the Data Science Summit 2024 in Warsaw.
Dec 21, 2023	Gave a talk at the Deep Learning Barcelona Symposium (DLBCN) 2023. Also presented my work at the CVML reading sessions at the University of Barcelona.
Dec 10, 2023	Attended NeurIPS 2023 in New Orleans and presented the poster of our paper in the main conference.
Oct 03, 2023	Attended ICCV 2023 in Paris. Presented one poster for our paper in the main conference and gave an oral talk at the Visual Continual Learning Workshop where our team “Continual Learners” won the innovation award at the “Continual Test-time Adaptation for Semantic Segmentation and Object Detection” challenge.

selected publications

CVPRW 2024

Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers

Dipam Goswami, Bartłomiej Twardowski, and Joost Van De Weijer

In CVPR Workshops - CLVISION, 2024

Abs arXiv Code

Few-shot class-incremental learning (FSCIL) aims to adapt the model to new classes from very few data (5 samples) without forgetting the previously learned classes. Recent works in many-shot CIL (MSCIL) (using all available training data) exploited pre-trained models to reduce forgetting and achieve better plasticity. In a similar fashion, we use ViT models pre-trained on large-scale datasets for few-shot settings, which face the critical issue of low plasticity. FSCIL methods start with a many-shot first task to learn a very good feature extractor and then move to the few-shot setting from the second task onwards. While the focus of most recent studies is on how to learn the many-shot first task so that the model generalizes to all future few-shot tasks, we explore in this work how to better model the few-shot data using pre-trained models, irrespective of how the first task is trained. Inspired by recent works in MSCIL, we explore how using higher-order feature statistics can influence the classification of few-shot classes. We identify the main challenge of obtaining a good covariance matrix from few-shot data and propose to calibrate the covariance matrix for new classes based on semantic similarity to the many-shot base classes. Using the calibrated feature statistics in combination with existing methods significantly improves few-shot continual classification on several FSCIL benchmarks.
CVPR 2024

Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning

Dipam Goswami, Albin Soutif-Cormerais, Yuyang Liu, Sandesh Kamath, Bartłomiej Twardowski, and Joost Van De Weijer

In Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Abs arXiv Code

Continual learning methods are known to suffer from catastrophic forgetting, a phenomenon that is particularly hard to counter for methods that do not store exemplars of previous tasks. Therefore, to reduce potential drift in the feature extractor, existing exemplar-free methods are typically evaluated in settings where the first task is significantly larger than subsequent tasks. Their performance drops drastically in more challenging settings starting with a smaller first task. To address this problem of feature drift estimation for exemplar-free methods, we propose to adversarially perturb the current samples such that their embeddings are close to the old class prototypes in the old model embedding space. We then estimate the drift in the embedding space from the old to the new model using the perturbed images and compensate the prototypes accordingly. We exploit the fact that adversarial samples are transferable from the old to the new feature space in a continual learning setting. The generation of these images is simple and computationally cheap. We demonstrate in our experiments that the proposed approach better tracks the movement of prototypes in embedding space and outperforms existing methods on several standard continual learning benchmarks as well as on fine-grained datasets.
NeurIPS 2023

FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning

Dipam Goswami, Yuyang Liu, Bartłomiej Twardowski, and Joost Van De Weijer

In Advances in Neural Information Processing Systems (NeurIPS), 2023

Abs arXiv Code

Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks and thus suffers from catastrophic forgetting. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. In this paper, we explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes. In an analysis of the feature distributions of classes, we show that classification based on Euclidean metrics is successful for jointly trained features. However, when learning from non-stationary data, we observe that the Euclidean metric is suboptimal and that feature distributions are heterogeneous. To address this challenge, we revisit the anisotropic Mahalanobis distance for CIL. In addition, we empirically show that modeling the feature covariance relations is better than previous attempts at sampling features from normal distributions and training a linear classifier. Unlike existing methods, our approach generalizes to both many- and few-shot CIL settings, as well as to domain-incremental settings. Interestingly, without updating the backbone network, our method obtains state-of-the-art results on several standard continual learning benchmarks.