Transformers for modeling physical systems

Geneva, Nicholas; Zabaras, Nicholas

doi:10.1016/j.neunet.2021.11.022

Cited by 81 publications

(41 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…17 Transformer for physical system After the groundbreaking success in natural language processing, 29,56 attention has also been demonstrated as a promising tool for various other machine learning tasks. [57][58][59] Following the success in these fields, several previous works 17,18,[60][61][62] have explored using attention to model and simulate physical system, which can generally be divided into two orthogonal lines. In the first group, attention layers are used to capture the structures and patterns lie in the PDEs' spatial domain.…”

Section: Related Workmentioning

confidence: 99%

“…17,18,62 On the contrary, in the second group, the attention is used to model the temporal evolution of the system while the spatial encoding is done by other mechanisms like CNNs or GNNs. 60,61 Our model lies in the former group where attention-based layers are used for encoding the spatial information of the input and query points, while the time marching is performed in the latent space using recurrent MLPs.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Transformer for Partial Differential Equations' Operator Learning

Li¹,

Meidani²,

Farimani³

2022

Preprint

View full text Add to dashboard Cite

Data-driven learning of partial differential equations' solution operators has recently emerged as a promising paradigm for approximating the underlying solutions. The solution operators are usually parameterized by deep learning models that are built upon problem-specific inductive biases. An example is a convolutional or a graph neural network that exploits the local grid structure where functions' values are sampled. The attention mechanism, on the other hand, provides a flexible way to implicitly exploit the patterns within inputs, and furthermore, relationship between arbitrary query locations and inputs. In this work, we present an attention-based framework for data-driven operator learning, which we term Operator Transformer (OFormer). Our framework is built upon self-attention, cross-attention, and a set of point-wise multilayer perceptrons (MLPs), and thus it makes few assumptions on the sampling pattern of the input function or query locations. We show that the proposed framework is competitive on standard benchmark problems and can flexibly be adapted to randomly sampled input.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Transformer for Partial Differential Equations' Operator Learning

Li¹,

Meidani²,

Farimani³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The population dynamics is thus learnt with the resulting embedding via a temporal transformer for the purpose of inference. There are many complex ways to create this embedding: [16] and [17] extracted embeddings from multivariate physical systems with Koopman operators before feeding the resulting representation into a temporal transformer; [18] used a graph neural network before transformers to embed interconnected-structures to perform skeleton-based action recognition; [19] used convolutional architecture to extract image embeddings before feeding them into transformer.…”

Section: Background and Related Workmentioning

confidence: 99%

Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers

Liu

Azabou

Dabagia

et al. 2022

Preprint

View full text Add to dashboard Cite

Complex time-varying systems are often studied by abstracting away from the dynamics of individual components to build a model of the population-level dynamics from the start. However, when building a population-level description, it can be easy to lose sight of each individual and how each contributes to the larger picture. In this paper, we present a novel transformer architecture for learning from time-varying data that build descriptions of both the individual as well as the collective population dynamics. Rather than combining all of our data into our model at the onset, we develop a separable architecture that operates on individual time-series first before passing them forward; this induces a permutation-invariance property and can be used to transfer across systems of different size and order. After demonstrating that our model can be applied to successfully recover complex interactions and dynamics in many-body systems, we apply our approach to populations of neurons in the nervous system. On neural activity datasets, we show that our multi-scale transformer not only yields robust decoding performance, but also provides impressive performance in transfer. Our results show that it is possible to learn from neurons in one animal's brain and transfer the model on neurons in a different animal's brain, with interpretable neuron correspondence across sets and animals. This finding opens up a new path to decode from and represent large collections of neurons.

show abstract

“…Multiple architectures have been proposed for using RNN for accomplishing the task future state prediction of a dynamical system. Recently Geneva & Zabara [44] and Eivazi et al [45] used LSTM and transformers as time integrator to predict flow evolution state. Hosseinyalamdary [46] used simple RNN for IMU modeling in deep Kalman filter.…”

Section: Recurrent Neural Networkmentioning

confidence: 99%

State estimation with limited sensors -- A deep learning based approach

Kumar,

Bahl,

Chakraborty

2021

Preprint

View full text Add to dashboard Cite

The importance of state estimation in fluid mechanics is well-established; it is required for accomplishing several tasks including design/optimization, active control, and future state prediction. A common tactic in this regards is to rely on reduced order models. Such approaches, in general, use measurement data of one-time instance. However, oftentimes data available from sensors is sequential and ignoring it results in information loss. In this paper, we propose a novel deep learning based state estimation framework that learns from sequential data. The proposed model structure consists of the recurrent cell to pass information from different time steps enabling utilization of this information to recover the full state. We illustrate that utilizing sequential data allows for state recovery from only one or two sensors. For efficient recovery of the state, the proposed approached is coupled with an auto-encoder based reduced order model. We illustrate the performance of the proposed approach using two examples and it is found to outperform other alternatives existing in the literature.

show abstract

Transformers for modeling physical systems

Cited by 81 publications

References 33 publications

Transformer for Partial Differential Equations' Operator Learning

Transformer for Partial Differential Equations' Operator Learning

Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers

State estimation with limited sensors -- A deep learning based approach

Contact Info

Product

Resources

About