The Hardware Lottery

Hooker, Sara

doi:10.48550/arxiv.2009.06489

Cited by 26 publications

(30 citation statements)

References 73 publications

(79 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An important objection that might be raised against the position taken in this paper is the fact that black box approaches display remarkable performance, partly due to the fact that such approaches are eminently suited for current hardware, e.g., GPUs [26], while it is far from clear whether systems based on the principles proposed here would be able to reach similar performance, given current computers. However, as argued in [6], there is no fundamental reason to believe that one would have to sacrifice performance to achieve better interpretability; at the very least, we argue that this issue should be thoroughly investigated.…”

Section: Discussionmentioning

confidence: 94%

The five Is: Key principles for interpretable and safe conversational AI

Wahde¹,

Virgolin²

2021

Preprint

View full text Add to dashboard Cite

In this position paper, we present five key principles, namely interpretability, inherent capability to explain, independent data, interactive learning, and inquisitiveness, for the development of conversational AI that, unlike the currently popular black box approaches, is transparent and accountable. At present, there is a growing concern with the use of black box statistical language models: While displaying impressive average performance, such systems are also prone to occasional spectacular failures, for which there is no clear remedy. In an effort to initiate a discussion on possible alternatives, we outline and exemplify how our five principles enable the development of conversational AI systems that are transparent and thus safer for use. We also present some of the challenges inherent in the implementation of those principles.

show abstract

Section: Discussionmentioning

confidence: 94%

The five Is: Key principles for interpretable and safe conversational AI

Wahde¹,

Virgolin²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Lottery tickets [Frankle and Carbin, 2018] are a set of small sub-networks derived from a larger dense network, which outperforms their parent networks. Many insightful studies [Morcos et al, 2019, Orseau et al, 2020, Frankle et al, 2019, 2020, Malach et al, 2020, Pensia et al, 2020] are carried out to analyze these tickets, but it remains difficult to generalize to large models due to training cost. In an attempt, follow-up works , Tanaka et al, 2020 show that one can find tickets without training labels.…”

Section: Related Workmentioning

confidence: 99%

“…State-of-the-art sparse training methods require up to 5× more training epochs compared to dense models[Evci et al, 2020] 2 An unstructured sparse model with 1% nonzero weights can be as slow as a dense model[Hooker, 2020] …”

mentioning

confidence: 99%

Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models

Chen¹,

Dao²,

Liang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Overparameterized neural networks generalize well but are expensive to train. Ideally, one would like to reduce their computational cost while retaining their generalization benefits. Sparse model training is a simple and promising approach to achieve this, but there remain challenges as existing methods struggle with accuracy loss, slow training runtime, or difficulty in sparsifying all model components. The core problem is that searching for a sparsity mask over a discrete set of sparse matrices is difficult and expensive. To address this, our main insight is to optimize over a continuous superset of sparse matrices with a fixed structure known as products of butterfly matrices. As butterfly matrices are not hardware efficient, we propose simple variants of butterfly (block and flat) to take advantage of modern hardware. Our method (Pixelated Butterfly) uses a simple fixed sparsity pattern based on flat block butterfly and low-rank matrices to sparsify most network layers (e.g., attention, MLP). We empirically validate that Pixelated Butterfly is 3× faster than butterfly and speeds up training to achieve favorable accuracy-efficiency tradeoffs. On the ImageNet classification and WikiText-103 language modeling tasks, our sparse models train up to 2.5× faster than the dense MLP-Mixer, Vision Transformer, and GPT-2 medium with no drop in accuracy. * Equal contribution. Order determined by coin flip.

show abstract

“…Moreover, deep NNs necessitate more computational resources, have higher energy consumption, and consequently lead to substantially higher CO 2 emissions [12]. The essay of the hardware lottery [13] highlighted the impact of available hardware system in determining which research ideas succeed (and fail). It is therein emphasized how the hardware lottery can delay research progress by casting successful ideas as failures.…”

Section: Introductionmentioning

confidence: 99%

A shallow neural model for relation prediction

Demir

Moussallem

Ngomo

2021

2021 IEEE 15th International Conference on Semantic Computing (ICSC)

View full text Add to dashboard Cite

Knowledge graph completion refers to predicting missing triples. Most approaches achieve this goal by predicting entities, given an entity and a relation. We predict missing triples via the relation prediction. To this end, we frame the relation prediction problem as a multi-label classification problem and propose a shallow neural model (SHALLOM) that accurately infers missing relations from entities. SHALLOM is analogous to C-BOW as both approaches predict a central token (p) given surrounding tokens ((s, o)). Our experiments indicate that SHAL-LOM outperforms state-of-the-art approaches on the FB15K-237 and WN18RR with margins of up to 3% and 8% (absolute), respectively, while requiring a maximum training time of 8 minutes on these datasets. We ensure the reproducibility of our results by providing an open-source implementation including training and evaluation scripts at https://github.com/dice-group/Shallom.

show abstract

The Hardware Lottery

Cited by 26 publications

References 73 publications

The five Is: Key principles for interpretable and safe conversational AI

The five Is: Key principles for interpretable and safe conversational AI

Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models

A shallow neural model for relation prediction

Contact Info

Product

Resources

About