Superimposed Image Description and Retrieval for Fish Species Identification

Abstract

Fish species identification is critical to the study of fish ecology and management of fisheries. Traditionally, dichotomous keys are used for fish identification. The keys consist of questions about the observed specimen. Answers to these questions lead to more questions till the reader identifies the specimen. However, such keys are incapable of adapting or changing to meet different fish identification approaches, and often do not focus upon distinguishing characteristics favored by many field ecologists and more user-friendly field guides. This makes learning to identify fish difficult for Ichthyology students. Students usually supplement the use of the key with other methods such as making personal notes, drawings, annotated fish images, and more recently, fish information websites, such as Fishbase. Although these approaches provide useful additional content, it is dispersed across heterogeneous sources and can be tedious to access. Also, most of the existing electronic tools have limited support to manage user created content, especially that related to parts of images such as markings on drawings and images and associated notes. We present SuperIDR, a superimposed image description and retrieval tool, developed to address some of these issues. It allows users to associate parts of images with text annotations. Later, they can retrieve images, parts of images, annotations, and image descriptions through text-and content-based image retrieval. We evaluated SuperIDR in an undergraduate Ichthyology class as an aid to fish species identification and found that the use of SuperIDR yielded a higher likelihood of success in species identification than using traditional methods, including the dichotomous key, fish web sites, notes, etc.

Use of subimages in fish species identification

“…We confirmed that participants engaged in two broad activities within fish identification -learning about a species and identifying species (found in our previous evaluation [12] as well). Sometimes both activities would happen simultaneously as a participant tried to identify an unknown specimen and then learn about its details.…”

“…Experienced users would give us wider exposure to types of users in the fisheries domain as compared to the earlier classroom evaluation [12]. In that evaluation, the participants were undergraduate students, who were relatively new to fisheries and to fish identification.…”

“…Therefore, we chose qualitative design and inquiry as we wanted to explore emergent behaviors with regard to use of SuperIDR and how it supported working with subimages in fish identification. Another reason to choose a qualitative method was that a controlled experiment, such as the one that we conducted earlier [12], might not replicate situated use [16,19]. Such situated use would give us rich data on using SuperIDR in the natural work/study setting of a person doing fish identification.…”

“…These questions required a different mode of inquiry than the controlled experiment previously conducted [12]. Therefore, we chose qualitative design and inquiry as we wanted to explore emergent behaviors with regard to use of SuperIDR and how it supported working with subimages in fish identification.…”

“…In previous work [12], in which we addressed research themes R2 and R3, we evaluated the use of SuperIDR [11,12,14], a prototype personal SI-DL, in an undergraduate Ichthyology class as an aid to fish identification. SuperIDR enables a user to work with subimages (contextualized parts of images) in multiple ways, including capabilities to annotate, search using text-and content-based image retrieval and their combination, browse, and compare subimages and all associated information.…”