采用直接反馈对齐的大规模现代深度学习架构的简化光学训练
Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment.
摘要(中文译文)
现代深度学习几乎完全依赖于专用的电子硬件加速器。光子方法具有低功耗和高运算速度，越来越多地被考虑用于推理，但迄今为止，它们大多局限于相对基本的任务。同时，训练深度和复杂的神经网络的问题，绝大多数是通过反向传播进行的，仍然是当前架构的大小和性能的一个重大限制，也是一个主要的计算和能源瓶颈。在这里，我们在混合电子-光子平台上实验性地实现了一种通用且可扩展的训练算法，称为直接反馈对准。光学处理单元执行大规模随机矩阵乘法，这是该算法的核心操作。我们使用超过1B的参数对现代深度学习架构（包括Transformers）进行光学训练，并在语言、视觉和基于扩散的生成任务上取得了良好的性能。我们研究了训练时间的缩放，并展示了我们的混合光电方法在超深和超宽神经网络方面的潜在优势，从而开辟了一条有前景的路线，以维持现代人工智能在传统冯·诺伊曼方法之外的指数级增长。
Abstract
Modern deep learning relies nearly exclusively on dedicated electronic hardware accelerators. Photonic approaches, with low consumption and high operation speed, are increasingly considered for inference but, to date, remain mostly limited to relatively basic tasks. Simultaneously, the problem of training deep and complex neural networks, overwhelmingly performed through backpropagation, remains a significant limitation to the size and, consequently, the performance of current architectures and a major compute and energy bottleneck. Here, we experimentally implement a versatile and scalable training algorithm, called direct feedback alignment, on a hybrid electronic-photonic platform. An optical processing unit performs large-scale random matrix multiplications, which is the central operation of this algorithm. We perform optical training of modern deep learning architectures, including Transformers, with more than 1B parameters, and obtain good performances on language, vision, and diffusion-based generative tasks. We study the scaling of the training time and demonstrate a potential advantage of our hybrid opto-electronic approach for ultra-deep and wide neural networks, thus opening a promising route to sustain the exponential growth of modern artificial intelligence beyond traditional von Neumann approaches.
如何引用
AMA
Ziao Wang, Kilian Müller, Matthew Filipovich, Julien Launay, Ruben Ohana, Gustave Pariente, et al. Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment.. Proceedings of the National Academy of Sciences of the United States of America. 2026; doi:10.1073/pnas.2532022123.
APA
Ziao Wang, Kilian Müller, Matthew Filipovich, Julien Launay, Ruben Ohana, Gustave Pariente, et al (2026). Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment.. Proceedings of the National Academy of Sciences of the United States of America. https://doi.org/10.1073/pnas.2532022123
GB/T 7714
Ziao Wang, Kilian Müller, Matthew Filipovich, Julien Launay, Ruben Ohana, Gustave Pariente, et al. Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment.[J]. Proceedings of the National Academy of Sciences of the United States of America, 2026 doi:10.1073/pnas.2532022123.
在 PubMed增强版 阅读全文
翻译全文 · AI 文献问答 · PDF 原文下载 · 引用导出