Neural algorithmic reasoning

Veličković, Petar; Blundell, Charles

doi:10.1016/j.patter.2021.100273

Cited by 24 publications

(17 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, our key contribution is to develop an SDP analog of our original LP formulation, and show how to lift LP-based extensions into a corresponding high-dimensional SDP-based extensions. Our general procedure for lifting low-dimensional representations into higher dimensions aligns with the neural algorithmic reasoning blueprint (Veličković & Blundell, 2021), and suggests that classical techniques such as SDPs may be effective tools for combining deep learning with algorithmic processes more generally.…”

Section: Introductionmentioning

confidence: 80%

“…In both machine learning and optimization, it has been observed that high-dimensional representations can make problems "easier". For instance, neural networks rely on high-dimensional internal representations for representational power and to allow information to flow through gradients, and performance suffers considerably when undesirable low-dimensional bottlenecks are introduced into network architectures (Belkin et al, 2019;Veličković & Blundell, 2021). In optimization, lifting to higher-dimensional spaces can make the problem more well-behaved (Goemans & Williamson, 1995;Shawe-Taylor et al, 2004;Du et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions

Karalias¹,

Robinson²,

Loukas³

et al. 2022

Preprint

View full text Add to dashboard Cite

Integrating functions on discrete domains into neural networks is key to developing their capability to reason about discrete objects. But, discrete domains are (1) not naturally amenable to gradient-based optimization, and (2) incompatible with deep learning architectures that rely on representations in high-dimensional vector spaces. In this work, we address both difficulties for set functions, which capture many important discrete problems. First, we develop a framework for extending set functions onto low-dimensional continuous domains, where many extensions are naturally defined. Our framework subsumes many well-known extensions as special cases. Second, to avoid undesirable low-dimensional neural network bottlenecks, we convert low-dimensional extensions into representations in high-dimensional spaces, taking inspiration from the success of semidefinite programs for combinatorial optimization. Empirically, we observe benefits of our extensions for unsupervised neural combinatorial optimization, in particular with high-dimensional representations. * Equal contribution.Preprint. Under review.

show abstract

Section: Introductionmentioning

confidence: 80%

Section: Introductionmentioning

confidence: 99%

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions

Karalias¹,

Robinson²,

Loukas³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To thoroughly assess the generalisation capability with respect to graph size, we generated 9 different test sets -128 graphs each -with increasing number of nodes, namely 16,32,64,96,128,160,192,224,256 nodes. In order to train the network on intermediate steps, we utilise the CLRS benchmark (Veličković et al, 2021) to generate training data.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Learning heuristics for A*

Numeroso¹,

Bacciu²,

Veličković³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Path finding in graphs is one of the most studied classes of problems in computer science. In this context, search algorithms are often extended with heuristics for a more efficient search of target nodes. In this work we combine recent advancements in Neural Algorithmic Reasoning to learn efficient heuristic functions for path finding problems on graphs. At training time, we exploit multi-task learning to learn jointly the Dijkstra's algorithm and a consistent heuristic function for the A* search algorithm. At inference time, we plug our learnt heuristics into the A* algorithm. Results show that running A* over the learnt heuristics value can greatly speed up target node searching compared to Dijkstra, while still finding minimal-cost paths.

show abstract

“…This has motivated research on networks that can perform algorithmic reasoning (Graves et al, 2014;Zaremba & Sutskever, 2014;Bieber et al, 2020). Neural networks that accurately represent the semantics of programs could enable a variety of downstream tasks, including program synthesis (Devlin et al, 2017), program analysis (Allamanis et al, 2018), and other algorithmic reasoning tasks (Velickovic & Blundell, 2021).…”

Section: Introductionmentioning

confidence: 99%

Show Your Work: Scratchpads for Intermediate Computation with Language Models

Nye¹,

Andreassen²,

Gur-Ari³

et al. 2021

Preprint

View full text Add to dashboard Cite

Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text (Brown et al., 2020) or synthesizing computer programs Austin et al., 2021). However, they struggle with tasks that require unbounded multi-step computation, such as adding integers (Brown et al., 2020) or executing programs (Austin et al., 2021). Surprisingly, we find that these same models are able to perform complex multistep computations-even in the few-shot regime-when asked to perform the operation "step by step", showing the results of intermediate computations. In particular, we train Transformers to perform multi-step computations by asking them to emit intermediate computation steps into a "scratchpad". On a series of increasingly complex tasks ranging from long addition to the execution of arbitrary programs, we show that scratchpads dramatically improve the ability of language models to perform multi-step computations.

show abstract

Neural algorithmic reasoning

Abstract: We present neural algorithmic reasoning-the art of building neural networks that are able to execute algorithmic computation-and provide our opinion on its transformative potential for running classical algorithms on inputs previously considered inaccessible to them.

Cited by 24 publications

References 58 publications

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions

Learning heuristics for A*

Show Your Work: Scratchpads for Intermediate Computation with Language Models

Contact Info

Product

Resources

About