Sitemap

Member-only story

The Sequence Scope: Distributed ML Training is Going to be Everyone’s Problem

Weekly newsletter with over 100,000 subscribers that discusses impactful ML research papers, cool tech releases, the money in AI, and real-life implementations.

4 min readAug 22, 2021

📝 Editorial: Distributed ML Training is Going to be Everyone’s Problem

Large-scale, distributed training is one of those machine learning (ML) problems that is easy to ignore. After all, only large AI labs like Google, Facebook, and Microsoft work with these massively large models that require many GPUs to be trained. I definitely thought that way until the transformers came into the picture. If there is one takeaway from the emergence of transformer models, it is that bigger models are better, at least for the time being. Training a basic BERT-based transformer model requires quite a bit of infrastructure and distributed processes. As a result, distributed training is slowly becoming a mainstream problem for the…

--

--

Jesus Rodriguez
Jesus Rodriguez

Written by Jesus Rodriguez

Co-Founder and CTO of Sentora( fka IntoTheBlock), President of LayerLens, Faktory and NeuralFabric. Founder of The Sequence , Lecturer at Columbia, Wharton

No responses yet