Skip to content

Elastic Deep Learning using PaddlePaddle and Kubernetes

License

Notifications You must be signed in to change notification settings

guru4elephant/edl

This branch is 140 commits behind elasticdeeplearning/edl:develop.

Folders and files

NameName
Last commit message
Last commit date
Feb 9, 2018
Mar 13, 2018
Aug 27, 2018
Mar 20, 2019
Oct 18, 2019
Aug 27, 2018
Aug 8, 2018
Aug 28, 2018
Mar 15, 2018
Mar 8, 2018
Mar 17, 2018
Mar 16, 2018
Oct 15, 2019
Mar 13, 2018
Mar 8, 2018
Oct 18, 2019
Mar 15, 2018
Mar 15, 2018

Repository files navigation

PaddlePaddle EDL: Elastic Deep Learning

While many hardware and software manufacturers are working on improving the running time of deep learning jobs, EDL optimizes

  1. the global utilization of the cluster, and
  2. the waiting time of job submitters.

For more about the project EDL, please refer to this invited blog post on the Kubernetes official blog.

EDL includes two parts:

  1. a Kubernetes controller for the elastic scheduling of distributed deep learning jobs, and

  2. making PaddlePaddle a fault-tolerable deep learning framework. This directory contains the Kubernetes controller. For more information about fault-tolerance, please refer to the design.

We deployed EDL on a real Kubernetes cluster, dlnel.com, opened for graduate students of Tsinghua University. The performance test report of EDL on this cluster is here.

Tutorials

Design Docs

Future

  • Resource Adjustments by EDL
  • Support Full-Tolerant Distributed Training in PadldePaddle Fluid.

FAQ

TBD

License

PaddlePaddle EDL is provided under the Apache-2.0 license.

About

Elastic Deep Learning using PaddlePaddle and Kubernetes

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 65.3%
  • Python 28.2%
  • Shell 5.8%
  • Dockerfile 0.7%