arXiv Notify me• Oct 2, 2019paper

Accelerating Deep Learning by Focusing on the Biggest Losers

Jeff Dean(guest)

Get the full transcript next time arXiv releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Jeff Dean.

Training Efficiency in Deep Learning Selective-Backprop Mechanism

0 quotes

Concerns Raised

The performance is only benchmarked on specific image classification datasets (CIFAR, SVHN), and generalizability to other domains like NLP is not mentioned.
Potential for introducing bias by systematically skipping low-loss examples, which could affect model robustness on edge cases.

Opportunities Identified

Significant reduction in DNN training time and associated computational costs for image-based models.
The principle of using stale results for selection could be applied to further optimize other importance sampling techniques.

Key Themes

Research Findings6

On CIFAR10, CIFAR100, and SVHN datasets, Selective-Backprop converges to target error rates up to 3.5 times faster than standard Stochastic Gradient Descent (SGD).

On CIFAR10, CIFAR100, and SVHN datasets, Selective-Backprop converges to target error rates between 1.02 and 1.8 times faster than a state-of-the-art importance sampling approach.

Selective-Backprop is a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration.

The Selective-Backprop technique uses the output of a training example's forward pass to decide whether to compute gradients and update parameters for that example or to skip it.

Selective-Backprop accelerates training by reducing the number of computationally-expensive backpropagation steps performed.

Using stale forward pass results for selection in Selective-Backprop can achieve an additional 26% acceleration in training by skipping forward passes of low-priority examples.

Topics

Deep Learning Neural Networks Model Training Training Acceleration Optimization Algorithms Selective-Backprop Backpropagation Forward Pass Stochastic Gradient Descent (SGD)Importance Sampling Computational Efficiency Machine Learning Image Classification CIFAR10 SVHN

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

Accelerating Deep Learning by Focusing on the Biggest Losers

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

Training Efficiency in Deep Learning

Selective-Backprop Mechanism

Importance Sampling and Data Prioritization

Empirical Performance and Benchmarking

Research Findings6

Topics