Knowledge-Based Regularization in Generative Modeling

Takeishi, Naoya; Kawahara, Yoshinobu

doi:10.24963/ijcai.2020/331

Cited by 6 publications

(8 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several studies use prior knowledge on the input feature space for regularizing models. For example, they use the similarity between features (Krupka and Tishby 2007;Mollaysa, Strasser, and Kalousis 2017;Takeishi and Kawahara 2021) and relevant features for prediction per labeled training instance (Zaidan, Eisner, and Piatko 2007;Rieger et al 2020;Du et al 2019). These studies have shown the effectiveness of using prior knowledge on the input feature space.…”

Section: Related Workmentioning

confidence: 99%

Zero-Shot Task Adaptation with Relevant Feature Information

Kumagai,

Iwata,

Fujiwara

2024

AAAI

View full text Add to dashboard Cite

We propose a method to learn prediction models such as classifiers for unseen target tasks where labeled and unlabeled data are absent but a few relevant input features for solving the tasks are given. Although machine learning requires data for training, data are often difficult to collect in practice. On the other hand, for many applications, a few relevant features would be more easily obtained. Although zero-shot learning or zero-shot domain adaptation use external knowledge to adapt to unseen classes or tasks without data, relevant features have not been used in existing studies. The proposed method improves the generalization performance on the target tasks, where there are no data but a few relevant features are given, by meta-learning from labeled data in related tasks. In the meta-learning phase, it is essential to simulate test phases on target tasks where prediction model learning is required without data. To this end, our neural network-based prediction model is meta-learned such that it correctly responds to perturbations of the relevant features on randomly generated synthetic data. By this modeling, the prediction model can explicitly learn the discriminability of the relevant features without real target data. When unlabeled training data are available in the target tasks, the proposed method can incorporate such data to boost the performance in a unified framework. Our experiments demonstrate that the proposed method outperforms various existing methods with four real-world datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Zero-Shot Task Adaptation with Relevant Feature Information

Kumagai,

Iwata,

Fujiwara

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…They highlight the difficulty in eliciting expert knowledge from people but their technique is similar to the other works presented here in that the knowledge loss is still represented as an additive term to the standard network loss. Takeishi and Kawahara (2020) present an example of how the knowledge of relations of objects can be used to regularise a generative model. Again, the solution involves appending terms to the loss function, but they demonstrate that relational information can aid a learning algorithm.…”

Section: Related Workmentioning

confidence: 99%

MultiplexNet: Towards Fully Satisfied Logical Constraints in Neural Networks

Hoernle

Karampatsis

Belle

et al. 2022

AAAI

View full text Add to dashboard Cite

We propose a novel way to incorporate expert knowledge into the training of deep neural networks. Many approaches encode domain constraints directly into the network architecture, requiring non-trivial or domain-specific engineering. In contrast, our approach, called MultiplexNet, represents domain knowledge as a quantifier-free logical formula in disjunctive normal form (DNF) which is easy to encode and to elicit from human experts. It introduces a latent Categorical variable that learns to choose which constraint term optimizes the error function of the network and it compiles the constraints directly into the output of existing learning algorithms. We demonstrate the efficacy of this approach empirically on several classical deep learning tasks, such as density estimation and classification in both supervised and unsupervised settings where prior knowledge about the domains was expressed as logical constraints. Our results show that the MultiplexNet approach learned to approximate unknown distributions well, often requiring fewer data samples than the alternative approaches. In some cases, MultiplexNet finds better solutions than the baselines; or solutions that could not be achieved with the alternative approaches. Our contribution is in encoding domain knowledge in a way that facilitates inference. We specifically focus on quantifier-free logical formulae that are specified over the output domain of a network. We show that this approach is both efficient and general; and critically, our approach guarantees 100% constraint satisfaction in a network's output.

show abstract

“…However, these approaches do not always guarantee the correct identification of the mechanistic part, and the outcomes depend on the specific regularization term used [28]. To the best of our knowledge, the identifiability analysis of the mechanistic parameters in a HNODE model has not been investigated in the literature so far.…”

Section: Introductionmentioning

confidence: 99%

“…Firstly, while model calibration in mechanistic models usually relies on global optimization techniques to explore the parameter search space [11], training HNODE models necessitates the use of local and gradient-based methods [25]. Secondly, incorporating a universal approximator, such as a neural network, into a dynamical model may compromise the identifiability of the HNODE mechanistic components [28, 29]. In this sense, the existing literature has focused on trying to enforce the identifiability of the mechanistic parameters within a HNODE by integrating a regularization term into the cost function [30].…”

Section: Introductionmentioning

confidence: 99%

Robust parameter estimation and identifiability analysis with Hybrid Neural Ordinary Differential Equations in Computational Biology

Giampiccolo,

Reali,

Fochesato

et al. 2024

Preprint

View full text Add to dashboard Cite

Parameter estimation is one of the central problems in computational modeling of biological systems. Typically, scientists must fully specify the mathematical structure of the model, often expressed as a system of ordinary differential equations, to estimate the parameters. This process poses significant challenges due to the necessity for a detailed understanding of the underlying biological mechanisms. In this paper, we present an approach for estimating model parameters and assessing their identifiability in situations where only partial knowledge of the system structure is available. The partially known model is extended into a system of Hybrid Neural Ordinary Differential Equations, which captures the unknown portions of the system using neural networks.Integrating neural networks into the model structure introduces two primary challenges for parameter estimation: the need to globally explore the search space while employing gradient-based optimization, and the assessment of parameter identifiability, which may be hindered by the expressive nature of neural networks. To overcome the first issue, we treat biological parameters as hyperparameters in the extended model, exploring the parameter search space during hyperparameter tuning. The second issue is then addressed by ana posteriorianalysis of parameter identifiability, computed by introducing a variant of a well-established approach for mechanistic models. These two components are integrated into an end-to-end pipeline that is thoroughly described in the paper. We assess the effectiveness of the proposed workflow on test cases derived from three different benchmark models. These test cases have been designed to mimic real-world conditions, including the presence of noise in the training data and various levels of data availability for the system variables.Author summaryParameter estimation is a central challenge in modeling biological systems. Typically, scientists calibrate the parameters by aligning model predictions with measured data once the model structure is defined. Our paper introduces a workflow that leverages the integration between mechanistic modeling and machine learning to estimate model parameters when the model structure is not fully known. We focus mainly on analyzing the identifiability of the model parameters, which measures how confident we can be in the parameter estimates given the available experimental data and partial mechanistic understanding of the system. We assessed the effectiveness of our approach in variousin silicoscenarios. Our workflow represents a first step to adapting traditional methods used in fully mechanistic models to the scenario of hybrid modeling.

show abstract

Knowledge-Based Regularization in Generative Modeling

Cited by 6 publications

References 5 publications

Zero-Shot Task Adaptation with Relevant Feature Information

Zero-Shot Task Adaptation with Relevant Feature Information

MultiplexNet: Towards Fully Satisfied Logical Constraints in Neural Networks

Robust parameter estimation and identifiability analysis with Hybrid Neural Ordinary Differential Equations in Computational Biology

Contact Info

Product

Resources

About