A fuzzy rule based multimodal framework for face sketch-to-photo retrieval

Khan, Mohd. Aamir; Jalal, Anand Singh

doi:10.1016/j.eswa.2019.05.040

Cited by 22 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[73] employs a generative representation method with extreme learning machines for cross-modal classification. [27] have fused the facial attributes of a person and the semantic color information using a fuzzy rule based layered classifier. [8] presents an Attribute-Image Hierarchical Matching model for text attribute description based person search without any query imagery.…”

Section: Related Workmentioning

confidence: 99%

“…Multi-modal information retrieval takes queries in one modality of data to retrieve relevant data from other modalities, augmenting information from a single source with information from other sources as in the cases of video-text matching [36,59], image-text matching [10], situational knowledge delivery [8,27,41], etc. The main challenge in cross-modal retrieval lies in the heterogeneity gap between different modalities [18,43,62].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Feature Centric Multi-modal Information Retrieval in Open World Environment (FemmIR)

Solaiman¹,

Bhargava²

2023

Preprint

View full text Add to dashboard Cite

<p>Multi-modal information retrieval has great implications for search engines, situational knowledge delivery and complex data management systems. Existing cross-modal learning models use separate information models for each data modality and lack the compatibility to utilize pre-existing features in an application domain. Moreover, supervised learning methods lack the capability to include user preference to define data relevancy without training samples and need modality-specific translation methods. To address these problems, we propose a novel multi-modal information retrieval framework (FemmIR) with two retrieval models based on graph similarity search (RelGSim) and relational database querying (EARS). FemmIR uses extracted features from different modalities and translates them into a common information model. For RelGSim, we propose to build a localized graph for each data object with the features and define a novel distance metric to measure the similarity between two data objects. A neural network based graph similarity approximation model is trained to map the data objects to a similarity score. Furthermore, for handling feature extraction in an open world environment, appropriate extraction models are discussed for different application domains. To tackle the problem of finer attribute analysis in text, a novel human attribute extraction model is proposed for unstructured text. Contrary to existing methods, FemmIR can integrate application domains with existing features and can include user preference for relevancy determination for situational knowledge discovery. The single information model (common schema or graph) reduces the data representation overhead. Comprehensive experimental results on a novel open world cross-media dataset show the efficacy of our models.</p>

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Feature Centric Multi-modal Information Retrieval in Open World Environment (FemmIR)

Solaiman¹,

Bhargava²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Data fusion among multiple modalities has been employed in many application domains, such as sentiment analysis, 17 image-text matching, 14 face retrieval, 8 and visual question answering for a better understanding of context. These approaches have performed well for their respective application domains, but they lack generalization capabilities.…”

Section: Cross-modal Matching and Correlation Learningmentioning

confidence: 99%

Applying Machine Learning and Data Fusion to the “Missing Person” Problem

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Data fusion among multiple modalities has been used in many application domains such as sentiment analysis [17], image-text matching [14], face retrieval [8], and visual questionanswering for a better understanding of context. These approaches have performed well for respective application domains, but they lack generalization capabilities.…”

Section: Related Workmentioning

confidence: 99%

Applying Machine Learning and Data Fusion to the "Missing Person" Problem

Solaiman¹,

Sun²,

Nesen³

et al. 2022

Preprint

View full text Add to dashboard Cite

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.<div><br></div><div>We present a system for integrating multiple sources of data for finding missing persons. This system can assist authorities in finding children during amber alerts, mentally challenged persons who have wandered off, or person-of-interests in an investigation. Authorities search for the person in question by reaching out to acquaintances, checking video feeds, or by looking into the previous histories relevant to the investigation. In the absence of any leads, authorities lean on public help from sources such as tweets or tip lines. A missing person investigation requires information from multiple modalities and heterogeneous data sources to be combined.<div>Existing cross-modal fusion models use separate information models for each data modality and lack the compatibility to utilize pre-existing object properties in an application domain. A framework for multimodal information retrieval, called Find-Them is developed. It includes extracting features from different modalities and mapping them into a standard schema for context-based data fusion. Find-Them can integrate application domains with previously derived object properties and can deliver data relevant for the mission objective based on the context and needs of the user. Measurements on a novel open-world cross-media dataset show the efficacy of our model. The objective of this work is to assist authorities in finding uses of Find-Them in missing person investigation.</div></div>

show abstract

A fuzzy rule based multimodal framework for face sketch-to-photo retrieval

Cited by 22 publications

References 11 publications

Feature Centric Multi-modal Information Retrieval in Open World Environment (FemmIR)

Feature Centric Multi-modal Information Retrieval in Open World Environment (FemmIR)

Applying Machine Learning and Data Fusion to the “Missing Person” Problem

Applying Machine Learning and Data Fusion to the "Missing Person" Problem

Contact Info

Product

Resources

About