Data Parallelism: How to Train Deep Learning Models on Multiple GPUs (DPHTDLM)

 

Résumé du cours

This workshop teaches you techniques for data-parallel deep learning training on multiple GPUs to shorten the training time required for data-intensive applications. Working with deep learning tools, frameworks, and workflows to perform neural network training, you’ll learn how to decrease model training time by distributing data to multiple GPUs, while retaining the accuracy of training on a single GPU.

Please note that once a booking has been confirmed, it is non-refundable. This means that after you have confirmed your seat for an event, it cannot be cancelled and no refund will be issued, regardless of attendance.

Moyens Pédagogiques :
  • Quiz pré-formation de vérification des connaissances (si applicable)
  • Réalisation de la formation par un formateur agréé par l’éditeur
  • Formation réalisable en présentiel ou en distanciel
  • Mise à disposition de labs distants/plateforme de lab pour chacun des participants (si applicable à la formation)
  • Distribution de supports de cours officiels en langue anglaise pour chacun des participants
    • Il est nécessaire d'avoir une connaissance de l'anglais technique écrit pour la compréhension des supports de cours
Moyens d'évaluation :
  • Quiz pré-formation de vérification des connaissances (si applicable)
  • Évaluations formatives pendant la formation, à travers les travaux pratiques réalisés sur les labs à l’issue de chaque module, QCM, mises en situation…
  • Complétion par chaque participant d’un questionnaire et/ou questionnaire de positionnement en amont et à l’issue de la formation pour validation de l’acquisition des compétences

Pré-requis

Experience with deep learning training using Python

Objectifs

By participating in this workshop, you’ll:

  • Understand how data parallel deep learning training is performed using multiple GPUs
  • Achieve maximum throughput when training, for the best use of multiple GPUs
  • Distribute training to multiple GPUs using Pytorch Distributed Data Parallel
  • Understand and utilize algorithmic considerations specific to multi-GPU training performance and accuracy

Suite de parcours

Contenu

Introduction

  • Meet the instructor.
  • Create an account at courses.nvidia.com/join

Stochastic Gradient Descent and the Effects of Batch Size

  • Learn the significance of stochastic gradient descent when training on multiple GPUs
  • Understand the issues with sequential single-thread data processing and the theory behind speeding up applications with parallel processing.
  • Understand loss function, gradient descent, and stochastic gradient descent (SGD).
  • Understand the effect of batch size on accuracy and training time with an eye towards its use on multi-GPU systems.

Training on Multiple GPUs with PyTorch Distributed Data Parallel (DDP)

  • Learn to convert single GPU training to multiple GPUs using PyTorch Distributed Data Parallel
  • Understand how DDP coordinates training among multiple GPUs.
  • Refactor single-GPU training programs to run on multiple GPUs with DDP.

Maintaining Model Accuracy when Scaling to Multiple GPUs

  • Understand and apply key algorithmic considerations to retain accuracy when training on multiple GPUs
  • Understand what might cause accuracy to decrease when parallelizing training on multiple GPUs.
  • Learn and understand techniques for maintaining accuracy when scaling training to multiple GPUs.

Workshop Assessment

  • Use what you have learned during the workshop: complete the workshop assessment to earn a certificate of competency

Final Review

  • Review key learnings and wrap up questions.
  • Take the workshop survey.

Prix & Delivery methods

Formation en ligne

Durée
1 jour

Prix
  • US$ 500,–
Dates et Inscription
Demande de date

Actuellement aucune session planifiée