Tutorials
Large Scale Distributed Systems for Training Neural Networks
- intro: By Jeff Dean & Oriol Vinyals, Google. NIPS 2015.
- slides: https://media.nips.cc/Conferences/2015/tutorialslides/Jeff-Oriol-NIPS-Tutorial-2015.pdf
- video: http://research.microsoft.com/apps/video/default.aspx?id=259564&l=i
- mirror: http://pan.baidu.com/s/1mgXV0hU
Papers
Large Scale Distributed Deep Networks
- intro: distributed CPU training, data parallelism, model parallelism
- paper: http://www.cs.toronto.edu/~ranzato/publications/DistBeliefNIPS2012_withAppendix.pdf
- slides: http://admis.fudan.edu.cn/~yfhuang/files/LSDDN_slide.pdf
Implementation of a Practical Distributed Calculation System with Browsers and JavaScript, and Application to Distributed Deep Learning
- project page: http://mil-tokyo.github.io/
- arxiv: https://arxiv.org/abs/1503.05743
SparkNet
SparkNet: Training Deep Networks in Spark
- arxiv: http://arxiv.org/abs/1511.06051
- github: https://github.com/amplab/SparkNet
- blog: http://www.kdnuggets.com/2015/12/spark-deep-learning-training-with-sparknet.html
A Scalable Implementation of Deep Learning on Spark
- intro: Alexander Ulanov
- slides: http://www.slideshare.net/AlexanderUlanov1/a-scalable-implementation-of-deep-learning-on-spark-alexander-ulanov
- mirror: http://pan.baidu.com/s/1jHiNW5C
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- arxiv: http://arxiv.org/abs/1603.04467
- gitxiv: http://gitxiv.com/posts/57kjddp3AWt4y5K4h/tensorflow-large-scale-machine-learning-on-heterogeneous
Distributed Supervised Learning using Neural Networks
- intro: Ph.D. thesis
- arxiv: http://arxiv.org/abs/1607.06364
Distributed Training of Deep Neuronal Networks: Theoretical and Practical Limits of Parallel Scalability
How to scale distributed deep learning?
- intro: Extended version of paper accepted at ML Sys 2016 (at NIPS 2016)
- arxiv: https://arxiv.org/abs/1611.04581
Projects
Theano-MPI: a Theano-based Distributed Training Framework
CaffeOnSpark: Open Sourced for Distributed Deep Learning on Big Data Clusters
- intro: Yahoo Big ML Team
- blog: http://yahoohadoop.tumblr.com/post/139916563586/caffeonspark-open-sourced-for-distributed-deep
- github: https://github.com/yahoo/CaffeOnSpark
- youtube: https://www.youtube.com/watch?v=bqj7nML-aHk
Tunnel: Data Driven Framework for Distributed Computing in Torch 7
Distributed deep learning with Keras and Apache Spark
- project page: http://joerihermans.com/work/distributed-keras/
- github: https://github.com/JoeriHermans/dist-keras
BigDL: Distributed Deep learning Library for Apache Spark
Videos
A Scalable Implementation of Deep Learning on Spark
Distributed TensorFlow on Spark: Scaling Google’s Deep Learning Library (Spark Summit)
Deep Recurrent Neural Networks for Sequence Learning in Spark (Spark Summit)
Distributed deep learning on Spark
- author: Alexander Ulanov July 12, 2016
- intro: Alexander Ulanov offers an overview of tools and frameworks that have been proposed for performing deep learning on Spark.
- video: https://www.oreilly.com/learning/distributed-deep-learning-on-spark
Blogs
Hadoop, Spark, Deep Learning Mesh on Single GPU Cluster
http://www.nextplatform.com/2016/02/24/hadoop-spark-deep-learning-mesh-on-single-gpu-cluster/
The Unreasonable Effectiveness of Deep Learning on Spark
https://databricks.com/blog/2016/04/01/unreasonable-effectiveness-of-deep-learning-on-spark.html
Distributed Deep Learning with Caffe Using a MapR Cluster
https://www.mapr.com/blog/distributed-deep-learning-caffe-using-mapr-cluster
Deep Learning with Apache Spark and TensorFlow
https://databricks.com/blog/2016/01/25/deep-learning-with-apache-spark-and-tensorflow.html
Deeplearning4j on Spark
http://deeplearning4j.org/spark
Distributed Deep Learning, Part 1: An Introduction to Distributed Training of Neural Networks
GPU Acceleration in Databricks: Speeding Up Deep Learning on Apache Spark
https://databricks.com/blog/2016/10/27/gpu-acceleration-in-databricks.html
Distributed Deep Learning with Apache Spark and Keras