We use the field guides to create class prototypes that are used at inference time to make a prediction.Abstract
We exploit field guides to learn bird species recognition, in particular zero-shot recognition of unseen species. The illustrations contained in field guides deliberately focus on discriminative properties of a species, and can serve as side information to transfer knowledge from seen to unseen classes. We study two approaches: (1) a contrastive encoding of illustrations that can be fed into zero-shot learning schemes; and (2) a novel method that leverages the fact that illustrations are also images and as such structurally more similar to photographs than other kinds of side information. Our results show that illustrations from field guides, which are readily available for a wide range of species, are indeed a competitive source of side information. On the iNaturalist2021 subset, we obtain a harmonic mean from 749 seen and 739 unseen classes greater than 45% (@top-10) and 15% (@top-1). Which shows that field guides are a valuable option for challenging real-world scenarios with many species.
Andres C Rodriguez
Bridge Fellow (PostDoc) at Kapok.ai/ETH Zurich
My interest is in developing algorithms that leverage Remote Sensing data for environmental and sustainability purposes.